Flight control method and device based on endogenous electroencephalogram signals, equipment and medium

Abstract

The application discloses a flight control method and device based on endogenous electroencephalogram signals, equipment and medium. The method comprises the following steps: acquiring endogenous electroencephalogram signals, performing feature extraction on the signals, analyzing a plurality of obtained endogenous electroencephalogram features, and obtaining output results of the endogenous electroencephalogram features. The output results of at least one endogenous electroencephalogram feature comprise intensity information of the endogenous electroencephalogram feature. Fusion processing is performed based on the output results of the endogenous electroencephalogram features to generate a target comprehensive brain wave state vector. A flight control instruction for an aircraft is generated, the instruction comprising a target flight control vector mapped from the target comprehensive brain wave state vector. The vector changes with the change of the intensity information of the at least one endogenous electroencephalogram feature. The aircraft is controlled through the endogenous electroencephalogram signals, so that the naturalness of brain-computer interaction is improved. The target comprehensive brain wave state vector is mapped into the target flight control vector, which is beneficial to improving the delicacy and fluency of control.

Description

Technical Field

[0001] This application relates to the field of aircraft technology, and in particular to a flight control method, device, equipment and medium based on endogenous electroencephalogram (EEG) signals. Background Technology

[0002] Brain-computer interface (BCI) technology, as a direct communication channel connecting the human brain to external devices, has made significant progress in the fields of neurorehabilitation, assistive control, and human-computer interaction in recent years. Among these applications, unmanned aerial vehicles (UAVs) have become a highly representative and important carrier for brain-controlled technology due to their high flexibility and maneuverability. Brain-controlled UAV systems decode the operator's electroencephalogram (EEG) signals and convert them into flight commands, not only freeing the operator's hands but also demonstrating unique application value in specific complex environments or special tasks.

[0003] Currently, mainstream brain-controlled drone technologies are primarily based on two visual evoked paradigms: Steady-State Visual Evoked Potential (SSVEP) and P300 event-related potentials (ERPs). SSVEP-based systems typically require the user to focus on a visual stimulus block flashing at a specific frequency (such as arrows or icons on a screen). The system identifies the user's intention by detecting spectral components in the EEG signal that match the stimulus frequency. This method is highly dependent on visual stimulation; prolonged viewing of a high-frequency flashing screen can easily lead to severe visual fatigue. Furthermore, the SSVEP paradigm is extremely sensitive to lighting conditions; in bright outdoor environments, ambient light interference can significantly reduce recognition accuracy. P300 ERP-based systems utilize the "strange sphere" paradigm, controlling the user by focusing on a target stimulus (such as a specific row or column on a matrix keyboard) and generating a P300 potential upon its appearance. However, this paradigm also heavily relies on visual feedback. Therefore, there is an urgent need for a brain-controlled drone technology solution that does not rely on visual stimulation. Summary of the Invention

[0004] This application discloses a flight control method, device, equipment, and medium based on endogenous brainwave signals. By controlling an aircraft using endogenous brainwave signals, it is possible to eliminate the dependence on visual stimulation in traditional brain control technologies, and the interaction method is more in line with human intuition, thus improving the naturalness of brain-computer interaction. In addition, by mapping the target integrated brainwave state vector to the target flight control vector, it is beneficial to improve the precision and smoothness of control.

[0005] In a first aspect, embodiments of this application provide a flight control method based on endogenous electroencephalogram (EEG) signals. This method can be executed by an electronic device or by a device matched with the electronic device (such as a chip or processor placed inside the electronic device). The electronic device can be a server, an aircraft, or a control terminal of the aircraft. The method includes: acquiring the user's endogenous EEG signal, which is a spontaneous EEG signal independent of external stimuli; performing multi-modal feature extraction on the endogenous EEG signal to obtain multiple endogenous EEG features; decoding and analyzing the multiple endogenous EEG features separately to obtain the output results of each endogenous EEG feature; the output result of at least one endogenous EEG feature includes the intensity information of the endogenous EEG feature; performing fusion processing based on the output results of various endogenous EEG features to generate a target comprehensive EEG state vector, which includes the intensity information of at least one endogenous EEG feature; generating flight control commands for the aircraft, which include a target flight control vector, which is obtained by mapping the target comprehensive EEG state vector based on the mapping relationship between the comprehensive EEG state vector and the flight control vector; wherein the numerical value of the target flight control vector changes accordingly with the change of the intensity information of at least one endogenous EEG feature.

[0006] In this technical solution, firstly, controlling the aircraft using endogenous EEG signals eliminates the reliance on visual stimulation inherent in traditional brain-control technologies, and the interaction method is more intuitive, thus improving the naturalness of brain-computer interaction. Secondly, mapping the target's comprehensive brainwave state vector to the target's flight control vector enhances the precision and fluency of control. Furthermore, employing multi-modal feature extraction and fusion integrates multi-dimensional endogenous EEG features, avoiding the susceptibility to interference and unstable recognition rates associated with single EEG features, thereby enhancing the anti-interference capability and robustness of the brain-control system.

[0007] In some possible implementations, multiple endogenous EEG features include at least two of the following: motor imagery features, attentional state features, and cognitive state features; wherein, the output of the motor imagery features includes the user's motor imagery intention category, the output of the attentional state features includes the user's attention intensity index, and the output of the cognitive state features includes the user's cognitive state category.

[0008] In some possible implementations, the intensity information of at least one endogenous EEG feature includes the intensity information of motor imagery features and the intensity information of attentional state features; the target flight control vector includes at least two of roll control, pitch control, yaw control, and throttle control. The method further includes at least two of the following: linearly mapping the intensity information of attentional state features to throttle control; mapping the intensity information of single-handed motor imagery features to a roll control to the left or right when the motor imagery features include single-handed motor imagery features; mapping the intensity information of double-handed motor imagery features to a pitch control to move forward or backward when the motor imagery features include double-handed motor imagery features; and mapping the intensity information of foot motor imagery features to a yaw control to the left or right when the motor imagery features include foot motor imagery features.

[0009] In this technical solution, the motion imagination characteristics of different limbs are mapped to the corresponding attitude control quantities of the aircraft, which can realize multi-directional flight control of the aircraft.

[0010] In some possible implementations, the magnitude of the roll control variable is positively correlated with the intensity of the event-related desynchronization / synchronization ERD / ERS of the single-handed motor imagery feature; the magnitude of the pitch control variable is positively correlated with the intensity of the ERD / ERS of the two-handed motor imagery feature.

[0011] This technical solution breaks away from traditional discrete switch control, allowing for smoother transitions in flight maneuvers and significantly improving control precision and naturalness.

[0012] In some possible implementations, the intensity information of the imagined foot motion features is mapped to a yaw control quantity to the left or right, including: mapping the intensity information of the imagined foot motion features to a first yaw control quantity to the left or right. The method also includes: acquiring the user's head rotation angle detected by the inertial measurement unit; and modifying the first yaw control quantity to a second yaw control quantity based on the head rotation angle.

[0013] In this technical solution, by integrating EEG signals and head posture data, the problem of insufficient precision in single EEG control can be compensated for, thereby improving the accuracy of yaw control.

[0014] In some possible implementations, the output of the cognitive state features includes the user's cognitive state category; the method also includes: acquiring the user's peripheral physiological signals; determining whether the peripheral physiological signals match the user's cognitive state category; and if they do not match, modifying the user's cognitive state category so that the modified cognitive state category matches the peripheral physiological signals.

[0015] In this technical solution, by introducing peripheral physiological signals, multimodal verification with EEG features can be performed to correct misjudgments in single EEG recognition, thereby improving the accuracy and reliability of cognitive state categories.

[0016] In some possible implementations, the method further includes: generating hovering instructions and first indication information for the aircraft when the user's cognitive state category is fatigue category and / or the duration of the intensity information of the attention intensity feature being lower than a preset intensity threshold is greater than a preset duration; the first indication information is used to instruct the aircraft to execute the hovering instructions and stop executing flight control instructions.

[0017] This technical solution can take over the flight status in a timely manner when the user's control ability declines, effectively avoiding the risk of control errors and improving the safety of brain-controlled flight.

[0018] In some possible implementations, motor imagery features include ERD / ERS power spectrum features of the Mu and Beta bands extracted from the sensorimotor cortex of the brain, as well as cosmotor pattern features; attentional state features include the power ratio of the Alpha and Beta bands extracted from the frontal lobe of the brain, as well as the cognitive load index of the Theta band; and cognitive state features include the Alpha band asymmetry index and Gamma band energy features extracted from the prefrontal lobe of the brain.

[0019] Secondly, embodiments of this application provide a flight control device based on endogenous electroencephalogram (EEG) signals, the device including units for implementing the method described in the first aspect.

[0020] Thirdly, embodiments of this application provide an electronic device, including a processor and a communication interface; wherein the communication interface is used to connect the electronic device to other devices, and the processor is used to execute the method described in the first aspect. The electronic device may be a server, an aircraft, or an aircraft control terminal.

[0021] In some possible implementations, the electronic device may also include a memory for storing computer programs or instructions; and a processor specifically for retrieving the computer programs or instructions from the memory to perform the method described in the first aspect.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or computer instructions, which, when executed, cause an electronic device to perform the method described in the first aspect.

[0023] Fifthly, embodiments of this application provide a computer program product including a computer program or instructions, which, when executed on an electronic device, causes the electronic device to perform the method described in the first aspect. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating an application scenario applicable to an embodiment of this application; Figure 2 This is a flowchart illustrating a flight control method based on endogenous electroencephalogram (EEG) signals provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a flight control device based on endogenous electroencephalogram (EEG) signals provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] It should be understood that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish different objects, not to describe a specific order. "At least one" in the embodiments of this application refers to one or more, and "multiple" refers to two or more. "And / or" in the embodiments of this application describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. The character " / " can indicate that the preceding and following related objects are in an "or" relationship. Additionally, the symbol " / " can also represent a division sign, i.e., performing a division operation.

[0026] In the embodiments of this application, "at least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent the following seven cases: a, b, c, a and b, a and c, b and c, a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0027] This invention provides a flight control method based on endogenous electroencephalogram (EEG) signals, primarily applicable to scenarios where brain-computer interfaces (BCIs) and aircraft operate in collaboration. A BCI is a normal output channel that does not rely on peripheral nerves and muscle tissue; it directly establishes communication and control connections between the brain and external devices.

[0028] Please see Figure 1 , Figure 1This is a schematic diagram of an application scenario applicable to an embodiment of this application. The application scenario may include, but is not limited to, a server, an aircraft, a control terminal, and a user wearing an electroencephalogram (EEG) head-mounted device. Figure 1 The number and form of the devices shown are for illustrative purposes and do not constitute a limitation on the embodiments of this application. Figure 1 The application scenario shown takes an example including an aircraft (10), an EEG headset (10), a control terminal (12), and a server (13). Figure 1 As shown, any two devices are connected via a wireless communication link to exchange data.

[0029] like Figure 1 As shown, the user wears an EEG headset integrating multi-lead dry electrodes. The EEG headset collects the user's endogenous electroencephalogram (EEG) signals in real time and transmits them to a control terminal via wireless networks (such as Wi-Fi, 5G, etc.). The EEG headset records macroscopic electrophysiological signals of brain neuronal population activity through electrodes placed on the scalp, reflecting the brain's electrical activity state.

[0030] A control terminal is a hardware and software system used to control aircraft, plan mission paths, and analyze flight data. This application does not limit the form of the control terminal; for example, the form of the control terminal may include, but is not limited to, fixed control stations, portable handheld terminals, and integrated control platforms in mobile vehicles. Portable handheld terminals may include, but are not limited to, remote controls, mobile phones, tablets, desktop computers, laptops, all-in-one computers, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, and wearable devices.

[0031] The control terminal needs to establish a pairing relationship (or matching relationship) with the aircraft in advance. Only after the pairing relationship is successfully established can the control terminal gain control of the aircraft.

[0032] The server may have one or more of the following functions: managing the pairing, authentication and binding of devices between the control terminal and the aircraft; storing other configuration data; recording flight logs, operation records, etc.

[0033] The aircraft in this application embodiment may include a flight control module, a mission payload module, and a multi-channel communication module. The mission payload refers to the onboard functional equipment / working devices carried by the aircraft, excluding the aircraft body, for completing operational tasks; the mission payload is not part of the flight control structure. The multi-channel communication module may include at least a 5G / 4G cellular network communication module and a point-to-point image transmission communication module (such as a Wi-Fi or self-organizing network module).

[0034] The aircraft may include, but is not limited to, unmanned aerial vehicles (UAVs) and manned aircraft. This application does not limit the form of the aircraft; for example, the form of the aircraft may include, but is not limited to, fixed-wing aircraft, rotorcraft, model aircraft, airships, hot air balloons, and robots. Among them, rotorcraft may include, but is not limited to, helicopters and multi-rotor aircraft.

[0035] In this embodiment, after the control terminal acquires the user's endogenous EEG signals collected by the EEG head-mounted device, it can further extract features from the endogenous EEG signals. Alternatively, the control terminal can forward the endogenous EEG signals to other devices (such as servers or aircraft), which can then extract features from the endogenous EEG signals.

[0036] In some possible implementations, the EEG headset may have a built-in Inertial Measurement Unit (IMU) for real-time acquisition of the user's head posture data.

[0037] It is understood that the application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will know that with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0038] The following describes the flight control method based on endogenous EEG signals proposed in the embodiments of this application. This flight control method based on endogenous EEG signals can be executed by an electronic device, or by a device matched with the electronic device (such as a chip or processor, etc., placed inside the electronic device). The embodiments of this application illustrate the example of the flight control method based on endogenous EEG signals being executed by an electronic device. The electronic device can be... Figure 1 The aircraft, server, or control terminal shown.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating a flight control method based on endogenous electroencephalogram (EEG) signals, provided as an embodiment of this application. Figure 2As shown, the flight control method based on endogenous EEG signals may include, but is not limited to, the following steps.

[0040] S201: Acquire the user's endogenous EEG signals, which are spontaneous EEG signals that do not depend on external stimuli.

[0041] The user's endogenous electroencephalogram (EEG) signals are collected by an EEG headset worn on the user's head and then transmitted to an electronic device. It should be noted that data transmission can occur directly or indirectly between the EEG headset and the electronic device. Taking the electronic device as a control terminal (such as a mobile phone) as an example, direct data transmission is possible between the EEG headset and the mobile phone. Taking the electronic device as a server as an example, direct data transmission is possible between the EEG headset and the server. Alternatively, indirect data transmission is possible, where the EEG headset first sends the collected endogenous EEG signals to the mobile phone, which then forwards the signals to the server.

[0042] Endogenous EEG signals refer to neural electrical activity spontaneously generated by the brain, independent of external stimuli (such as flashing light sources or buzzing sounds), reflecting internal cognition, motor intentions, or spontaneous rhythm regulation. Unlike exogenous EEG signals induced by external stimuli in the SSVEP or P300 paradigm, endogenous EEG signals belong to an actively regulated EEG pattern. Controlling aircraft based on endogenous EEG signals can eliminate visual fatigue caused by visual stimuli and is unaffected by strong outdoor light environments, effectively improving environmental adaptability and continuous usage time.

[0043] S202: Multi-modal feature extraction of endogenous EEG signals yields a variety of endogenous EEG features.

[0044] In some possible implementations, the various endogenous EEG features obtained by multimodal feature extraction of endogenous EEG signals may include, but are not limited to, at least two of the following: motor imagery features, attentional state features, and cognitive state features.

[0045] In some possible implementations, motor imagery features can include event-related desynchronization / synchronization (ERD / ERS) power spectrum features extracted from the Mu and Beta bands of the brain's sensorimotor cortex, as well as common spatial pattern (CSP) features. The Mu band has a frequency of 8–12 Hz, and the Beta band has a frequency of 13–30 Hz. The Mu and Beta bands are mainly distributed in the brain's sensorimotor cortex and are typical EEG bands characterizing limb movement and motor imagery. When a person generates a motor intention, the signal in these bands will change regularly. After a specific event (such as motor imagery) triggers the EEG signal, the power of the corresponding band decreases to ERD (desynchronization) and increases to ERS (synchronization). The ERD / ERS power spectrum features are characteristic quantities obtained by quantifying and statistically analyzing these power changes, which can intuitively reflect the strength of motor imagery. CSP is a spatial filtering algorithm for EEG that filters multi-channel EEG signals from the sensorimotor area, amplifies the signal differences between different motor imagery states, extracts highly discriminative spatial features, and improves the accuracy of motor intention recognition.

[0046] In some possible implementations, attentional state characteristics can include the power ratio of the alpha to beta frequency bands extracted from the frontal lobe of the brain, as well as a cognitive load index for the theta frequency band. The alpha frequency band in the frontal lobe ranges from 8 to 12 Hz, and the signal strength in this band is related to the brain's relaxation and focus states. The beta frequency band in the frontal lobe ranges from 13 to 30 Hz, reflecting the level of brain activity and focus. A lower alpha / beta power ratio indicates a more concentrated user attention. The theta frequency band in the frontal lobe ranges from 4 to 7 Hz, and this band is positively correlated with cognitive load and mental exertion. Constructing a cognitive load index using this band can quantify the level of mental effort invested by the user during operation.

[0047] In some possible implementations, cognitive state characteristics may include an alpha band asymmetry index and gamma band energy characteristics extracted from the prefrontal cortex. The alpha band asymmetry index is calculated by the power difference of the alpha signals on the left and right sides of the prefrontal cortex. The gamma band in the prefrontal cortex has a frequency of 30 Hz or higher and can reflect the brain's emotional activity, tension level, and fatigue state.

[0048] Among them, motor imagery features can reflect the user's intention to move their limbs, attentional state features can reflect the intensity of the user's focus, and cognitive state features can reflect the user's real-time cognitive state. Cognitive states can be categorized into positive, neutral, negative, fatigued, and tense states. By employing a multi-modal feature extraction strategy, features reflecting different intentions and states can be extracted from different brain regions, which helps improve the accuracy and reliability of identifying motor intentions, attentional states, and cognitive states.

[0049] S203: Decode and analyze the various endogenous EEG features separately to obtain the output results of each endogenous EEG feature; the output result of at least one endogenous EEG feature includes the intensity information of the endogenous EEG feature.

[0050] After obtaining various endogenous EEG features, electronic devices can send them to their respective trained decoding models (such as SVM classifiers, regression models, etc.) for decoding and analysis, thereby converting each endogenous EEG feature into an output result with a clear physical meaning.

[0051] In some possible implementations, the output of motor imagery features includes the user's motor imagery intention category, the output of attention state features includes the user's attention intensity index, and the output of cognitive state features includes the user's cognitive state category. The motor imagery intention category can include, but is not limited to, the user's left / right limb or front / back limb motor imagery intention categories (e.g., left hand, right hand, both hands pushing forward, both feet, etc.) and the corresponding ERD / ERS feature intensity information. For example, if a user imagines moving their left hand, the output category is "left hand," and the output ERD intensity is 0.75 (higher intensity indicates stronger imagery). Taking the output of attention state features as an example, which includes continuous attention intensity values ​​in the range of 0-100, assuming the output attention intensity index is 85, it indicates that the user is currently highly focused. Cognitive states can be categorized as positive, neutral, negative, fatigued, tense, etc.

[0052] In some possible implementations, after the electronic device decodes and analyzes the various endogenous EEG features separately, it can also obtain the confidence level of the output results for each endogenous EEG feature. The confidence level is used to characterize the reliability of the decoding result of a single type of EEG feature, and its value is generally between 0 and 1 or between 0 and 100%. The higher the value, the more reliable the current feature recognition result; the lower the value, the greater the signal interference and the weaker the reference value of the recognition result. For example, the output user's cognitive state category is fatigue, and the confidence level is 0.9.

[0053] S204: Based on the output results of various endogenous EEG features, perform fusion processing to generate a target comprehensive EEG state vector, which includes the intensity information of at least one endogenous EEG feature.

[0054] Electronic devices can integrate multi-dimensional EEG information such as user's motor intention, attention, and cognitive state by fusing the output results of various endogenous EEG features. This can compensate for the shortcomings of single feature recognition, such as its limited scope and poor stability, effectively suppress interference and misjudgment of single feature signals, and generate a comprehensive brainwave state vector that can fully and accurately represent the user's true control intention. This improves the accuracy and robustness of EEG intention decoding and provides reliable feature support for subsequent high-precision, continuous flight control mapping.

[0055] In some possible implementations, S204 can be specifically implemented as follows: Based on Bayesian posterior probability fusion and a confidence-weighted voting algorithm, the output results of various endogenous EEG features are fused to generate a target comprehensive EEG state vector. Bayesian posterior probability fusion refers to combining prior laws of EEG signals to calculate the true probability of the output result of a single endogenous EEG feature under the current signal, quantifying the theoretical credibility of the output result itself, and filtering out abnormal identification conclusions that clearly violate the laws. For example, through learning from past data: when a user is "fatigued," 80% of the "turn left" commands issued are misoperations (a slight hand tremor); when a user is "highly focused," 95% of the "turn left" commands issued are genuine intentions. Bayesian posterior probability fusion can use "historical patterns" to correct "current observations," where "current observations" refer to the output results of each endogenous EEG feature. The confidence-weighted voting algorithm uses the confidence level of the decoded output as weight to weight and synthesize the output results of various endogenous EEG features after probability verification. High-confidence features dominate the final result, while the influence of low-confidence features is reduced. The combination of these two approaches verifies the rationality of the result from the perspective of signal regularity and allocates decision weights from the perspective of real-time signal quality, making the fusion result more accurate and resistant to interference.

[0056] Alternatively, S204 can be implemented as follows: When the confidence levels of the output results of various endogenous EEG features are all higher than the corresponding safety thresholds, the output results of various endogenous EEG features are fused based on a Bayesian posterior probability fusion and a confidence-weighted voting algorithm to generate a target comprehensive EEG state vector. That is, the fusion calculation to generate the target comprehensive EEG state vector is only performed when the confidence levels of the output results of various endogenous EEG features are all higher than the corresponding safety thresholds. If the confidence level of the classification result of any endogenous EEG feature is lower than the corresponding safety threshold, the fusion calculation to generate the target comprehensive EEG state vector can be refused. For example, if the confidence level of the motor imagery intention category is 0.4 (lower than the threshold of 0.6), the electronic device can refuse fusion. In this case, no new instructions are generated for the aircraft, and the aircraft can continue to execute previous instructions or enter a hovering state. This method prevents low-confidence erroneous operations from being mapped as flight instructions.

[0057] S205: Generate flight control commands for the aircraft. The flight control commands include a target flight control vector, which is obtained by mapping the target integrated brainwave state vector based on the mapping relationship between the integrated brainwave state vector and the flight control vector. The magnitude of the target flight control vector changes accordingly with the intensity information of the at least one endogenous EEG feature.

[0058] The mapping relationship between the integrated brainwave state vector and the flight control vector is pre-established. By substituting the fused target integrated brainwave state vector into the mapping relationship, the target flight control vector corresponding to the aircraft can be determined, and then the target flight control vector can be encapsulated into flight control commands. The magnitude of the target flight control vector changes synchronously with the intensity information of at least one endogenous EEG feature. This mapping mechanism breaks the traditional "on / off" control and realizes "proportional control" similar to a real joystick. That is, the greater the intensity of the EEG feature, the higher the value of the target flight control vector, and the greater the amplitude and speed of the aircraft's movements; the weaker the intensity, the lower the value of the target flight control vector. In this way, the aircraft can be continuously and smoothly controlled based on the magnitude of the target flight control vector.

[0059] In this embodiment, controlling the aircraft using endogenous brainwave signals eliminates the reliance on visual stimulation inherent in traditional brain-control technologies, and the interaction method is more intuitive, thus enhancing the naturalness of brain-computer interaction. Furthermore, mapping the target's integrated brainwave state vector to the target's flight control vector improves the precision and smoothness of control.

[0060] In this embodiment, if the electronic device is an aircraft, after S205, the electronic device can directly execute the flight control command. If the electronic device is a server or control terminal, after S205, the electronic device can send the flight control command to the aircraft so that the aircraft can execute the flight control command.

[0061] In some possible implementations, the intensity information of at least one of the aforementioned endogenous EEG features includes the intensity information of motor imagery features and the intensity information of attentional state features; the target flight control vector includes at least two of roll control, pitch control, yaw control, and throttle control. After determining the integrated EEG state vector, the electronic device may also execute at least two of the following: The intensity information of attention state features is linearly mapped to throttle control values. In some possible implementations, the intensity information of attention state features is positively correlated with the throttle control value; that is, as the attention index (i.e., the intensity information of the attention state features) increases, a larger throttle control value is mapped, and the aircraft's climb rate increases. For example, as the attention index increases from 80 to 100, the throttle is increased from 80% to 100%. In some possible implementations, when the attention index is in a preset dead zone (e.g., in the 40-60 range), the throttle control value is zero to maintain hovering.

[0062] When the motion visualization features include single-hand motion visualization features, the intensity information of the single-hand motion visualization features is mapped to a roll control amount to the left or right. For example, imagining a left-hand movement corresponds to a left roll of the aircraft; imagining a right-hand movement corresponds to a right roll.

[0063] When the motion visualization features include hand motion visualization features, the intensity information of the hand motion visualization features is mapped to pitch control variables for forward or backward movement. For example, imagine pushing your hands forward, and the aircraft pitches forward; imagine pulling your hands back, and the aircraft pitches backward.

[0064] When the motion visualization features include foot motion visualization features, the intensity information of the foot motion visualization features is mapped to a yaw control variable to the left or right. For example, visualizing the left foot moving corresponds to the aircraft yawing to the left; visualizing the right foot moving corresponds to the aircraft yawing to the right.

[0065] Based on the motor imagination characteristics of different limbs, they are mapped to the corresponding attitude control quantities of the aircraft, which can realize multi-directional flight control of the aircraft.

[0066] In some possible implementations, the magnitude of the roll control variable is positively correlated with the ERD / ERS strength of the single-handed motion imaginary feature; the magnitude of the pitch control variable is positively correlated with the ERD / ERS strength of the two-handed motion imaginary features. For example, when imagining a right-handed movement, the higher the ERD / ERS strength of the right-handed motion imaginary feature, the greater the rightward roll angle and / or speed of the aircraft; the higher the ERD / ERS strength of the two-handed forward push motion imaginary feature, the greater the pitch angle of the aircraft's forward dive. In this way, traditional discrete on / off control can be broken, resulting in smoother transitions in flight maneuvers and a significant improvement in control precision and naturalness.

[0067] In some possible implementations, mapping the intensity information of foot motion imagery features to yaw control values ​​for left or right can be implemented as follows: The intensity information of foot motion imagery features is mapped to a first yaw control value for left or right. The electronic device can also acquire the user's head rotation angle detected by the Inertial Measurement Unit (IMU) and modify the first yaw control value to a second yaw control value based on the head rotation angle. In other words, a basic first yaw control value is first generated based on the intensity of the foot motion imagery features; then, combined with the user's head rotation angle acquired by the IMU, the basic control value (i.e., the first yaw control value) is corrected to obtain the final usable second yaw control value. For example, when the user imagines foot movements, a first yaw control value for the aircraft turning left is generated; simultaneously, the IMU detects a slight leftward turn of the user's head, and the system fine-tunes the first yaw control value accordingly, outputting the corrected second yaw control value. By fusing EEG signals and head posture data, the problem of insufficient precision in single EEG control can be compensated for, improving the accuracy of yaw control.

[0068] In some possible implementations, the output of cognitive state features includes the user's cognitive state category. The electronic device can also acquire the user's peripheral physiological signals, determine whether the peripheral physiological signals match the user's cognitive state category, and modify the user's cognitive state category if they do not match, so that the modified cognitive state category matches the peripheral physiological signals. Peripheral physiological signals refer to physiological indicator signals collected from the human body surface, excluding electroencephalogram (EEG) signals, used to reflect information such as human emotions, fatigue, tension, and physical condition; they belong to peripheral physiological monitoring data. Peripheral physiological signals may include, but are not limited to: heart rate, electrocardiogram (ECG), skin resistance / skin conductance, respiratory rate, and electromyography (EMG).

[0069] The cognitive state category is first determined based on EEG characteristics, and then cross-validated using peripheral physiological signals (such as skin resistance and heart rate). If the two types of information reflect inconsistent states, the cognitive state category is corrected based on the peripheral physiological signals. For example, if EEG characteristics determine the user's cognitive state as neutral, but the collected peripheral physiological signals show a low heart rate and abnormal skin resistance, indicating that the user is actually in a fatigued or passive state, the cognitive state category can be corrected from neutral to fatigued or passive. By introducing peripheral physiological signals, multimodal verification can be performed with EEG characteristics, correcting misjudgments from single EEG recognition, thereby improving the accuracy and reliability of cognitive state classification.

[0070] Understandably, in S204, the electronic device can perform fusion processing based on the original output results of various endogenous EEG features, or it can perform fusion processing on the new output results obtained after modifying the original output results.

[0071] In some possible implementations, when the user's cognitive state is classified as fatigued, and / or the duration for which the intensity of attention intensity is below a preset threshold exceeds a preset duration, the electronic device can generate a hovering command and a first indication message for the aircraft. The first indication message instructs the aircraft to execute the hovering command and cease executing flight control commands. In other words, when it is determined that the user is fatigued and / or their attention intensity is below a preset threshold and this state persists for an extended period, a hovering command and the first indication message are issued to the aircraft, causing it to stop its original flight maneuvers and remain hovered. For example, if the user is fatigued or distracted for more than 3 seconds, the current operation command can be forcibly overridden, causing the aircraft to hover in place and avoiding a crash or collision. In this way, the flight status can be taken over in a timely manner when the user's control ability declines, effectively avoiding the risk of control errors and improving the safety of brain-controlled flight.

[0072] In some possible implementations, when an emergency is detected, the electronic device can generate an automatic return-to-home command for the aircraft, causing it to cease its current flight maneuvers and automatically return to its origin. This approach enhances flight safety and emergency response capabilities. An emergency can refer to an abnormal situation that the electronic device determines will affect flight safety and prevents the normal execution of brain-controlled commands. This can include, but is not limited to, abnormal user status and device / link malfunctions. Abnormal user status refers to a user being persistently fatigued or having attention levels consistently below a preset threshold, resulting in a significant decline in control ability. Device / link malfunctions refer to failures or signal interruptions in EEG acquisition, wireless communication, aircraft hardware, or dangerous conditions in the flight environment.

[0073] In some possible implementations, the electronic device can also calculate the total delay of the entire system, construct a kinematic model of the aircraft, and use a Kalman filter to predict the expected state of the aircraft at the compensation time based on the current aircraft state feedback. It can also calculate the error between the desired target state and the expected state corresponding to the target integrated brainwave state vector, and use this error as a lead-ahead control variable to correct the target flight control vector. The total delay of the entire system includes brainwave acquisition delay, signal processing delay, wireless communication delay, and flight control response delay. By statistically analyzing and calculating the total delay of the entire control link, the lag time from brainwave signal acquisition to command execution can be clearly identified. Based on the aircraft kinematic model, the state the aircraft will reach within the delay period is predicted using a Kalman filter. The predicted state is compared with the user's expected state, and the error between the two is calculated. This error is used as a lead-ahead control variable to correct the original flight control vector, thus offsetting the deviation caused by the delay in advance. This effectively compensates for the control lag caused by the total delay, weakens the impact of signal transmission, computation, and execution delays on control, allows the aircraft's actions to closely follow the user's control intentions, and improves flight tracking, control smoothness, and control accuracy.

[0074] This application does not limit the values ​​or setting methods of the various thresholds (such as preset intensity thresholds, preset durations, etc.) involved. Electronic devices can set the values ​​of these thresholds by default, or they can set or modify the values ​​of these thresholds based on user operations.

[0075] In this embodiment, spontaneous EEG signals from the user, requiring no external stimulation, are first collected. Then, multiple EEG features are extracted and decoded one by one to obtain an output containing feature intensity information. All decoding results are then fused to generate a comprehensive brainwave state vector carrying intensity information. Finally, based on a preset mapping relationship, this vector is converted into a target flight control vector to generate flight commands. The control vector value dynamically changes with the intensity of the EEG features, achieving continuous control. In this way, firstly, controlling the aircraft using endogenous EEG signals eliminates the dependence on visual stimulation found in traditional brain-control technologies, and the interaction method is more intuitive, improving the naturalness of brain-computer interaction. Secondly, mapping the target comprehensive brainwave state vector to the target flight control vector improves the precision and fluency of control. Thirdly, employing multi-modal feature extraction and fusion integrates multi-dimensional endogenous EEG features, avoiding the susceptibility to interference and unstable recognition rates of single EEG features, thus enhancing the anti-interference capability and robustness of the brain-control system.

[0076] Please see Figure 3 , Figure 3 This is a schematic diagram of a flight control device based on endogenous electroencephalogram (EEG) signals provided in an embodiment of this application. Figure 3 As shown, the flight control device 30 based on endogenous EEG signals includes an acquisition unit (301) and a processing unit (302). The flight control device based on endogenous EEG signals can perform the relevant steps of the electronic device (such as an aircraft, server, or control terminal) in the aforementioned method embodiments.

[0077] The acquisition unit is used to acquire the user's endogenous EEG signals, which are spontaneous EEG signals that do not depend on external stimuli. The processing unit is used to extract multi-modal features from endogenous EEG signals to obtain various endogenous EEG features. The processing unit is also used to decode and analyze various endogenous EEG features to obtain the output results of various endogenous EEG features; the output result of at least one endogenous EEG feature includes the intensity information of the endogenous EEG feature; The processing unit is also used to perform fusion processing on the output results based on various endogenous EEG features to generate a target comprehensive EEG state vector, which includes intensity information of at least one endogenous EEG feature. The processing unit is also used to generate flight control commands for the aircraft. The flight control commands include a target flight control vector, which is obtained by mapping the target integrated brainwave state vector based on the mapping relationship between the integrated brainwave state vector and the flight control vector. The magnitude of the target flight control vector changes accordingly with the intensity information of at least one endogenous EEG feature.

[0078] Specifically, in this case, the operations performed by the acquisition unit and the processing unit can be referred to the description of electronic devices (such as aircraft, servers or control terminals) in the method embodiments.

[0079] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device 40 provided in an embodiment of this application. It can be used to implement the functions of the electronic device in the above method embodiments; that is, the electronic device 40 can be an aircraft, a server, or a control terminal. The electronic device 40 may include a processor (401) and a communication interface (402). Optionally, the electronic device 40 may also include a memory (403). The processor, memory, and communication interface can be connected via a bus (404) or other means. The bus is in... Figure 4 The connections between other components are shown in bold lines only and are not intended to be limiting. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0080] The coupling in this application embodiment is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. This application embodiment does not limit the specific connection medium between the aforementioned processor, memory, and communication interface.

[0081] Communication interfaces are used to receive and / or send information, and can also be used to connect electronic devices to other devices. For example, taking an electronic device as a server, the server's communication interface can be used to connect the server to an aircraft or control terminal; the specific implementation of the communication interface can be a transceiver. Memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory.

[0082] The processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or optionally, any conventional processor.

[0083] In one example, when the electronic device adopts Figure 4 When in the form shown, Figure 4 The processor in the device can execute the method executed by the electronic device in any of the above method embodiments.

[0084] In one alternative implementation, a memory is used to store computer programs or instructions; a processor is used to invoke the computer programs or instructions stored in the memory to perform the steps executed by the electronic device in the method embodiment.

[0085] In the embodiments of this application, the methods provided in the embodiments of this application can be implemented by running a computer program (including program code) capable of performing the steps involved in the above-described methods on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a CPU, random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into an electronic device through the computer-readable recording medium, and run therein.

[0086] Based on the same inventive concept, the principle and beneficial effects of the electronic device 40 provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the electronic device (such as an aircraft, server or control terminal) in the method embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0087] This application also provides a computer-readable storage medium storing a computer program or computer instructions, which is adapted to be loaded by an electronic device and execute the method provided in the above-described method embodiments.

[0088] This application also provides a computer program product containing a computer program or instructions, which, when run on an electronic device, causes the electronic device to perform the method provided in the above-described method embodiments.

[0089] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for various devices and products applied to or integrated into a chip, all of their modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, all of their modules / units can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The implementation is achieved through a software program that runs on a processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into electronic devices, each of their modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the electronic device. Alternatively, at least some modules / units can be implemented using a software program that runs on a processor integrated within the electronic device, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0090] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0091] In the above embodiments, the descriptions of each embodiment have their own emphasis, and any multiple embodiments can be used in combination. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0093] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0094] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by program instructions and related hardware. The program instructions can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0095] The above-disclosed embodiments are merely one example of this application and only a part of the embodiments of this application. They should not be construed as limiting the scope of this application.

Claims

1. A flight control method based on endogenous electroencephalogram (EEG) signals, characterized in that, The method includes: Acquire the user's endogenous EEG signals, which are spontaneous EEG signals that do not depend on external stimuli; Multimodal feature extraction was performed on the endogenous EEG signals to obtain various endogenous EEG features; The various endogenous EEG features are decoded and analyzed to obtain the output results of each endogenous EEG feature; the output result of at least one of the endogenous EEG features includes the intensity information of the endogenous EEG feature; The output results of the various endogenous EEG features are fused together to generate a target comprehensive EEG state vector, which includes the intensity information of at least one endogenous EEG feature. A flight control command for the aircraft is generated, the flight control command including a target flight control vector, the target flight control vector being obtained by mapping the target integrated brainwave state vector based on the mapping relationship between the integrated brainwave state vector and the flight control vector; wherein, the numerical value of the target flight control vector changes accordingly with the intensity information of the at least one endogenous EEG feature.

2. The method according to claim 1, characterized in that, The aforementioned multiple endogenous EEG features include at least two of the following: motor imagery features, attentional state features, and cognitive state features; The output of the motion imagery feature includes the user's motion imagery intention category, the output of the attention state feature includes the user's attention intensity index, and the output of the cognitive state feature includes the user's cognitive state category.

3. The method according to claim 2, characterized in that, The intensity information of the at least one endogenous EEG feature includes the intensity information of the motor imagery feature and the intensity information of the attention state feature; the target flight control vector includes at least two of the following: roll control, pitch control, yaw control, and throttle control. The method further includes at least two of the following: The intensity information of the attention state features is linearly mapped to the throttle control quantity; When the motion visualization feature includes a single-hand motion visualization feature, the intensity information of the single-hand motion visualization feature is mapped to the roll control amount to the left or right. When the motion visualization features include hand motion visualization features, the intensity information of the hand motion visualization features is mapped to the pitch control amount for forward or backward movement. When the motion visualization features include foot motion visualization features, the intensity information of the foot motion visualization features is mapped to the yaw control amount to the left or right.

4. The method according to claim 3, characterized in that, The magnitude of the roll control quantity is positively correlated with the intensity of the event-related desynchronization / synchronization ERD / ERS of the single-handed motor imagery characteristics; The magnitude of the pitch control quantity is positively correlated with the ERD / ERS intensity of the hand movement imagery features.

5. The method according to claim 3, characterized in that, The process of mapping the intensity information of the foot movement imagery features to the yaw control amount to the left or right includes: The intensity information of the foot movement imagination feature is mapped to a first yaw control value to the left or right; The method further includes: The inertial measurement unit detects the user's head rotation angle. Based on the head rotation angle, the first yaw control value is modified to the second yaw control value.

6. The method according to any one of claims 2-5, characterized in that, The output of the cognitive state features includes the user's cognitive state category; the method further includes: Acquire the user's peripheral physiological signals; Determine whether the peripheral physiological signals match the user's cognitive state category; In the event of a mismatch, the user's cognitive state category is modified so that the modified cognitive state category matches the peripheral physiological signals.

7. The method according to any one of claims 2-5, characterized in that, The method further includes: If the user's cognitive state category is fatigue, and / or the duration of the intensity information of the attention intensity feature being lower than a preset intensity threshold is greater than a preset duration, a hovering command and a first indication information are generated for the aircraft. The first indication information is used to instruct the aircraft to execute the hovering command and stop executing the flight control command.

8. The method according to any one of claims 2-5, characterized in that, The motor imagery features include ERD / ERS power spectrum features of the Mu and Beta bands extracted from the sensorimotor cortex of the brain, as well as co-space mode features; The attentional state characteristics include the power ratio of the Alpha band to the Beta band extracted from the frontal lobe of the brain, and the cognitive load index of the Theta band. The cognitive state features include the Alpha band asymmetry index and Gamma band energy features extracted from the prefrontal cortex of the brain.

9. A flight control device based on endogenous electroencephalogram (EEG) signals, characterized in that, Includes a unit for performing the method as described in any one of claims 1-8.

10. An electronic device, characterized in that, The device includes a memory, a processor, and a communication interface; wherein the communication interface is used to connect the electronic device to other devices, the memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions in the memory, such that when the computer programs or instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or computer instructions that, when executed by an electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

Images