A brain-computer interface system and method for air-ground collaborative multi-unmanned system

Abstract

This invention discloses a brain-computer interface system and method for air-ground collaborative multi-unmanned systems. The system includes: a human-computer interaction module for presenting visual stimuli through augmented reality technology; an EEG decoding module for decoding brainwaves, obtaining decoding results, and outputting brain-controlled commands; a reliability analysis module for establishing a relationship between the brain-controlled commands and probability distributions using the Softmax function, and performing reliability analysis on the brain-controlled commands; and an auxiliary control module for optimizing robot performance and correcting brain-controlled commands. This invention improves the portability and practicality of brain-computer interfaces by combining augmented reality technology with the SSVEP brain-computer interface.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction science, brain-computer interface technology and artificial intelligence control technology, and in particular to a brain-computer interface system and method for air-ground collaborative multi-unmanned systems. Background Technology

[0002] Modern intelligent air-ground collaborative robot systems (composed of drones and unmanned vehicles) can leverage their respective strengths to jointly complete complex tasks such as search environment detection, target perception and recognition, and subsequent rescue. These devices can remotely enter complex environments, replacing operators in dangerous tasks and providing better solutions for search and rescue, counter-terrorism, and riot control. However, current unmanned equipment suffers from poor collaborative performance, unstable intelligence levels, and low efficiency, making it unsuitable for application in complex real-world environments in the short term. Human-machine shared control technology can compensate for the shortcomings of fully autonomous air-ground collaborative systems. However, due to the complexity of operating air-ground collaborative robot systems, if multiple operators control the drone and unmanned vehicle separately, the performance of the system will be affected by the varying levels of proficiency among the operators. To address the development needs of such air-ground collaborative robot systems, this invention proposes a brain-computer interface (BCI) system for multi-unmanned air-ground collaborative systems. A single operator can control the drone using both hands and simultaneously control the unmanned vehicle through the BCI, achieving efficient operation of the air-ground collaborative robot system.

[0003] Brain-computer interface technology realizes the dream of humans communicating with and even controlling the external environment through brain activity. It can bypass the body's own organs, allowing the brain to interact efficiently with external equipment without the usual mediums—peripheral nerves and limbs. It has great potential in giving operators additional executive capabilities.

[0004] Currently, methods for acquiring brain activity signals can be categorized into invasive, partially invasive, and non-invasive methods. Non-invasive brain-computer interface (BCI) avoids complex, expensive, and high-risk surgeries, and its brain activity recording methods include electroencephalography (EEG). EEG is widely used in BCI systems due to its low acquisition cost, sufficient temporal resolution, safety, hygiene, ease of use, and ability to significantly shorten experimental preparation time. Brain-computer interface technology based on Steady-State Visual Evoked Potentials (SSVEP) is widely used due to its advantages such as fast command generation speed and high signal-to-noise ratio.

[0005] Early brain-controlled robot systems were mostly direct-control systems. In this method, BCI (Brain Intervention Center) analyzes the subject's EEG signals to obtain the control intention and converts it into control commands that the robot can recognize, which are then directly applied to the robot. However, due to limitations in BCI performance, such as low recognition accuracy, long command transmission intervals, and the tendency to cause operator fatigue, this direct-control method often cannot meet the needs of practical applications.

[0006] Traditional SSVEP brain-computer interfaces mostly generate visual stimuli through a computer screen to induce brain signals. This stimulation method significantly reduces the portability and practicality of brain-computer interface systems. Furthermore, in traditional SSVEP-based brain-computer interface systems, operators are in a relatively quiet, ideal environment to complete a single designated task. However, in real-world applications, external environmental interference or operator factors (distraction, fatigue, etc.) can lead to a decline in brain-computer interface performance, ultimately greatly affecting the overall system control performance and even system safety. The performance of directly controlled brain-controlled mobile robot systems is heavily dependent on the performance of the brain-computer interface used, and the mobile robot may collide with obstacles during tasks. Because current brain-computer interfaces still have limitations in command frequency, accuracy, and response time, the safety of directly controlled brain-controlled mobile robot systems cannot be fully guaranteed. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a brain-computer interface system and method for air-ground collaborative multi-unmanned systems. Taking brain-computer interaction and shared control as the research object and air-ground collaborative robot system application as the design goal, the invention controls unmanned vehicles through brain-computer interface to achieve efficient operation of air-ground collaborative robot systems.

[0008] On the one hand, to achieve the above objectives, the present invention provides a brain-computer interface system for air-ground collaborative multi-unmanned systems, comprising:

[0009] Human-computer interaction module: used to present visual stimuli through augmented reality technology;

[0010] EEG decoding module: used to decode brain waves, obtain decoding results, and output brain control commands;

[0011] Reliability Analysis Module: Used to establish the relationship between the mind control commands and the probability distribution using the Softmax function, and to perform reliability analysis on the mind control commands;

[0012] Auxiliary control module: Used to optimize the robot's performance and correct brain-controlled commands.

[0013] Preferably, the auxiliary control module includes a control performance optimization unit, which optimizes the robot's control performance through an auxiliary controller, perceives surrounding environmental information in real time through lidar, avoids perceived obstacles by combining with model predictive control methods, and corrects brain-controlled commands.

[0014] On the other hand, to achieve the above objectives, the present invention also provides an operation method for a brain-computer interface system for air-ground collaborative multi-unmanned systems, comprising:

[0015] Collect the operator's EEG signals, and perform preprocessing and decoding operations on the EEG signals to obtain EEG control commands;

[0016] The reliability of the control commands is evaluated using a reliability analysis module.

[0017] Based on the credibility evaluation results, the robot is controlled through the auxiliary control module, and the brain-controlled commands are corrected.

[0018] The operator controls the ground unmanned vehicle through brain-controlled commands and directly controls the drone through a remote control device.

[0019] Preferably, the acquisition of the operator's electroencephalogram (EEG) signals includes:

[0020] The operator wears a standard EEG acquisition device to collect EEG signals while gazing at a stimulus scene displayed by an augmented reality helmet. The acquisition frequency is set, and the collected EEG signals are segmented and downsampled.

[0021] Preferably, the electroencephalogram (EEG) signal is preprocessed, including:

[0022] The acquired EEG signals were preprocessed by baseline correction, bandpass filtering, and notch filtering to remove noise.

[0023] Preferably, decoding the EEG signal includes:

[0024] The preprocessed EEG signal is filtered through a filter bank to obtain several sub-bands;

[0025] Each sub-band is analyzed using canonical correlation analysis to obtain the correlation coefficient between the sub-band component and the predefined reference signal corresponding to all stimulus frequencies.

[0026] Target recognition is performed based on the correlation coefficient to obtain the decoded EEG signal.

[0027] Preferably, target identification based on the correlation coefficient includes:

[0028] Calculate the correlation vector of the k-th reference signal:

[0029]

[0030] In the formula, ρ(X,Y) represents the correlation coefficient between X and Y, ρ k Let ρ be the correlation vector of the k-th reference signal. k N The correlation coefficient of the Nth filter corresponding to the kth reference signal. W is the transpose of the EEG signal after filtering by the Nth filter corresponding to the kth reference signal. x This represents the weight vector for x in canonical correlation analysis. The EEG signal after filtering by the Nth filter corresponding to the kth reference signal, Y fk For the k-th reference signal, Y T W is the template signal vector. y This represents the weight vector of y in canonical correlation analysis;

[0031] The correlation coefficients corresponding to all sub-bands The weighted sum of squares is used as a feature for target recognition:

[0032]

[0033] Where n is the nth sub-band obtained from the decomposition. ρ is the weighted correlation coefficient corresponding to the stimulus frequency. k n Let N be the nth filter, and N be the number of filters.

[0034] The sub-band weighting function w(n) is defined as:

[0035] w(n) = n -a +bn∈[1,N]

[0036] Where a and b are both constants;

[0037] Extract the weighted correlation coefficient corresponding to the highest stimulus frequency. The corresponding stimulation frequency, i.e., the decoded EEG signal:

[0038]

[0039] In the formula, f target Let f be the probability of the target. i Let S be the frequency corresponding to the i-th stimulus target, and S be the total number of stimulus targets.

[0040] Preferably, the reliability evaluation of the control commands includes:

[0041] The Softmax function is used to establish an item relationship between the EEG control commands and the probability distribution, with cross-entropy as the cost function, specifically:

[0042]

[0043] In the formula, P(H i |ρ) represents the posterior probability assuming the target is the i-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let N be the natural logarithm with the j-th obtained correlation coefficient as the exponent. t Let $\mathbf{H}$ be the target total, $j$ be each term in the summation process, and $Cost(H)$ be the total number of terms. i The result is assumed to be H. i The value of C(H) j H i ) is based on H j H i The value of the function C with parameters;

[0044] Introducing the concept of penalty, by comparing the cost H of the highest probability assumption. q The evaluation decision is based on the cost H0 of rejecting the hypothesis, specifically as follows:

[0045]

[0046] In the formula, C(H) j H q ) is based on H j H q The value of C is a function of the parameter, where q is the stimulus label corresponding to the maximum probability. ′ The stimulus label corresponding to the second highest probability, C(H) j H0) is based on H j H0 is the value of the function C of the parameters, and ε is the threshold parameter;

[0047] The larger the value of ε, the more inclined it is to judge the command as unreliable;

[0048] Maximum probability assumption cost H q The method for calculating the cost H0 of rejecting the hypothesis is as follows:

[0049]

[0050] In the formula, P(H j |ρ) represents the posterior probability assuming the target is the j-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let ρ be the natural logarithm with the k-th obtained correlation coefficient as the exponent. kTo calculate the k-th correlation coefficient during the summation process, It is the natural logarithm with the maximum correlation coefficient as the exponent. ρ is the natural logarithm with the second largest correlation coefficient as the exponent. max For the largest correlation coefficient value, ρ 2ndmax This is the second largest correlation coefficient value;

[0051] When the cost of rejecting the hypothesis is lower than the cost of the recognition result, the brain control command is considered unreliable and requires the intervention of an auxiliary controller.

[0052] Preferably, the robot is controlled by an auxiliary control module based on the credibility evaluation results, including:

[0053] The robot's perception unit provides obstacle information and determines whether the mobile robot will be in a safe environment within a certain period of time. If the mobile robot is determined to be in a safe environment, the auxiliary controller tracks the user's intention; otherwise, the auxiliary controller intervenes and outputs a control command that meets safety constraints to the mobile robot to achieve collaborative control. The robot avoids the perceived obstacles by combining with a model predictive control method.

[0054] Preferably, the objective function of the model predictive control is:

[0055]

[0056] Where k represents the current time, and S(k) represents the current state. This represents the control sequence, α is the penalty factor for penalizing the control action weights, and u = [v, ω] represents the command output by the auxiliary controller. (k) The command output by the auxiliary controller at the current moment. The commands output by the mind control at the current moment. H is the penalty matrix that controls the rate of change of the output. c To control the time domain, u (k+i) The command output by the auxiliary controller is i steps back from the current time, u (k+i-1) This is the command output by the auxiliary controller i-1 steps back from the current time.

[0057] Compared with the prior art, the present invention has the following advantages and technical effects:

[0058] This invention improves the portability and practicality of brain-computer interfaces by combining augmented reality technology with the SSVEP brain-computer interface. The Softmax function is used to perform reliability analysis on the EEG decoding results, and in conjunction with an auxiliary controller, the issue of low recognition accuracy of brain-computer interfaces in practical applications is mitigated, thereby improving the overall control performance of the air-ground collaborative robot system. Simultaneously, the MPC-based auxiliary controller enables autonomous obstacle avoidance, enhancing the control safety of the brain-controlled robot. Attached Figure Description

[0059] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0060] Figure 1 This is a schematic diagram of a brain-computer interface system structure for an air-ground collaborative multi-unmanned system according to an embodiment of the present invention;

[0061] Figure 2 This is a flowchart illustrating the operation method of a brain-computer interface system for air-ground collaborative multi-unmanned systems according to an embodiment of the present invention.

[0062] Figure 3 This is a schematic diagram of the stimulus presentation in a virtual reality headset according to an embodiment of the present invention.

[0063] Figure 4 This is a schematic diagram of the pseudo-online experiment results in an embodiment of the present invention. Detailed Implementation

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0065] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0066] This invention proposes a brain-computer interface system for air-ground collaborative multi-unmanned systems, such as... Figure 1 ,include:

[0067] Human-computer interaction module: used to present visual stimuli through augmented reality technology;

[0068] EEG decoding module: used to decode brain waves, obtain decoding results, and output brain control commands;

[0069] Reliability Analysis Module: Used to establish the relationship between the mind control commands and the probability distribution using the Softmax function, and to perform reliability analysis on the mind control commands;

[0070] Auxiliary control module: Used to optimize the robot's performance and correct brain-controlled commands.

[0071] Specifically, this embodiment focuses on brain-computer interface and shared control, with the design goal of application in air-ground collaborative robot systems. A single operator can control a drone using both hands while simultaneously controlling an unmanned vehicle via a brain-computer interface, achieving efficient operation of the air-ground collaborative robot system.

[0072] In human-computer interaction modules, traditional vision-based brain-computer interfaces mostly use a computer screen as the interaction interface, which has poor portability and cannot meet the needs of practical applications. In this embodiment, the brain-computer interface will be implemented using augmented reality (AR) technology, presenting visual stimuli in an augmented reality environment. This not only eliminates the constraints of fixed devices, but also allows the operator to observe the real world through the interaction interface, greatly improving the operator's freedom of interaction and the system's practicality.

[0073] The EEG decoding module employs the Filter Bank Canonical Correlation Analysis (FBCCA) algorithm, which can extract effective information embedded in harmonic frequency components through the filter bank, thereby accelerating target recognition and improving the performance of SSVEP-BCI.

[0074] During EEG decoding, the Softmax function is used to project the correlation coefficients obtained during decoding into a probability space, establishing a hypothesis testing model. The risk function is used as an evaluation of the "credibility" of the classification results. This indicator will play a decisive role in adjusting the intervention level of the auxiliary controller. The hypothesis testing model calculates the cost function to obtain reliability analysis.

[0075] Furthermore, the auxiliary control module includes a control performance optimization unit. The control performance optimization unit optimizes the robot's control performance through the auxiliary controller, perceives the surrounding environment information in real time through lidar, avoids perceived obstacles by combining with model predictive control methods, and corrects brain-controlled commands.

[0076] Specifically, this embodiment employs a model predictive control (MPC) method to establish an auxiliary controller for optimizing robot control performance. Using LiDAR, the robot can perceive environmental information in real time and avoid perceived obstacles by combining this with the MPC method. Furthermore, a weight term is introduced into the cost function to correct potentially erroneous (low-reliability) brain-controlled commands.

[0077] This embodiment also provides an operation method for a brain-computer interface system for air-ground collaborative multi-unmanned systems, such as... Figures 1-2 ,include:

[0078] Collect the operator's EEG signals, and perform preprocessing and decoding operations on the EEG signals to obtain EEG control commands;

[0079] The reliability of the control commands is evaluated using a reliability analysis module.

[0080] Based on the credibility evaluation results, the robot is controlled through the auxiliary control module, and the brain-controlled commands are corrected.

[0081] The operator controls the ground unmanned vehicle through brain-controlled commands and directly controls the drone through a remote control device.

[0082] This invention improves the portability and practicality of brain-computer interfaces by combining augmented reality technology with the SSVEP brain-computer interface. The Softmax function is used to perform reliability analysis on the EEG decoding results, and in conjunction with an auxiliary controller, the issue of low recognition accuracy of brain-computer interfaces in practical applications is mitigated, thereby improving the overall control performance of the air-ground collaborative robot system. Simultaneously, the MPC-based auxiliary controller enables autonomous obstacle avoidance, enhancing the control safety of the brain-controlled robot.

[0083] Furthermore, the operator's electroencephalogram (EEG) signals are collected, including:

[0084] The operator wears a standard EEG acquisition device to collect EEG signals while gazing at a stimulus scene displayed by an augmented reality helmet. The acquisition frequency is set, and the collected EEG signals are segmented and downsampled.

[0085] Specifically, traditional SSVEP brain-computer interfaces suffer from poor flexibility and portability because the screen position is fixed, requiring subjects to constantly shift their gaze between the screen and the controlled robot during task execution. Combining augmented reality (AR) technology with BCI technology can largely solve the problem of poor flexibility in traditional BCI systems, improving the portability and practicality of SSVEP-based brain-computer interfaces. In this embodiment, HoloLens 2 augmented reality glasses are used to provide the stimulation interface.

[0086] To better obtain the brainwave signal stimulation response from the mind-controlled user and avoid harmonic interference, the stimulation frequency of SSVEP is generally set between 8 and 16 Hz. Previous studies have found that the display method of HoloLens devices differs from common computer monitors. If white sinusoidal flashing stimulation is used, the actual result will be a superposition of four color sinusoidal stimuli (RGBW, etc.) flashing sequentially with a difference of π / 2. Because this embodiment requires high stability of flashing, using monochromatic flashing stimulation is necessary to avoid the above situation and induce the desired SSVEP signal. Furthermore, since red refreshes first among the primary colors, in this embodiment, the stimulation interface is set to a red flashing black background, such as... Figure 3 .

[0087] In Unity3D, a red, blinking 3D cube was created using the MRTK toolkit, with its built-in transparent color as the background. The R value in the cube's color (RGBA) changes sinusoidally between 0 and 255 as the display refreshes.

[0088] The operator wears a 64-channel EEG cap conforming to the "10-20 international standard leads", injects conductive gel, and collects EEG signals from eight channels: PO5, PO3, PO2, PO4, PO6, O1, O2, and O2. The acquisition frequency is 1000Hz. The acquired EEG signals are divided into 2-second windows and downsampled to 500Hz.

[0089] This embodiment is designed so that the operator can control the drone with both hands and the unmanned vehicle through a brain-computer interface, thus achieving efficient operation of the air-ground collaborative robot system.

[0090] The drone is directly controlled via a remote control handle that comes with it.

[0091] The unmanned vehicles on the ground are controlled via brainwaves.

[0092] In this embodiment, the brain-computer interface uses stimulation frequencies of 13Hz, 12Hz, 11Hz, and 9Hz, which correspond to the brain-controlled robot's commands to move forward, turn left, turn right, and decelerate, respectively.

[0093] Furthermore, the EEG signals undergo preprocessing, including:

[0094] The acquired EEG signals were preprocessed by baseline correction, bandpass filtering, and notch filtering to remove noise.

[0095] Specifically, the preprocessing mainly includes several steps such as baseline correction, bandpass filtering, and notch filtering, which are used to remove noise from EEG signals.

[0096] Furthermore, decoding operations are performed on the EEG signals, including:

[0097] The preprocessed EEG signal is filtered through a filter bank to obtain several sub-bands;

[0098] Each sub-band is analyzed using canonical correlation analysis to obtain the correlation coefficient between the sub-band component and the predefined reference signal corresponding to all stimulus frequencies.

[0099] Target recognition is performed based on the correlation coefficient to obtain the decoded EEG signal.

[0100] Specifically, canonical correlation analysis (CCA) is a statistical method that analyzes the overall correlation between indicators by utilizing the degree of correlation between comprehensive variables. Due to its efficiency, robustness, and simplicity, it is widely used in brain-computer interfaces based on steady-state visual evoked potentials. Considering two multidimensional variables X and Y and their linear combination x = X... T W x and y=Y T W y Canonical correlation analysis method finds the weight vector W x and W y And use the following formula to maximize the correlation between x and y:

[0101]

[0102] ρ relative to W x and W y The maximum value is the maximum canonical correlation, and E is the variance. This is the transpose of the weight coefficient vector corresponding to x.

[0103] When the above method is applied to SSVEP frequency detection, X represents the multi-channel SSVEP signal, and Y represents the reference signal. Typically, a sinusoidal signal is chosen as the reference signal Y. f :

[0104]

[0105] In the formula, f is the stimulation frequency, and N is the frequency of stimulation. h It is the harmonic number.

[0106] To identify the frequency of SSVEPs, CCA calculates the typical correlation frequencies between multi-channel SSVEPs and the reference signal corresponding to each stimulus. The frequency of the reference signal with the highest correlation is considered the frequency of the SSVEP.

[0107] SSVEP consists of brain responses at the same fundamental, harmonic, and subharmonic frequencies as its stimulation frequency. Besides the fundamental frequency component, the harmonic components can also provide useful information for frequency detection. Filter bank analysis can decompose SSVEP into multiple sub-band components, improving the performance of SSVEP BCI and accelerating target recognition. This embodiment uses the filter bank canonical correlation analysis method. Specifically, a bandpass filter bank is used to decompose the input EEG signal into N sub-band components, and then canonical correlation analysis is performed on these N sub-components. Afterwards, a weighted average is calculated based on the correlation coefficients obtained from the N sub-bands, corresponding to each stimulation frequency f. k This yields an overall correlation coefficient value. Finally, the highest correlation coefficient value is selected from these values, and its corresponding frequency is taken as the final identification result.

[0108] Further, target identification based on the correlation coefficient includes:

[0109] Calculate the correlation vector of the k-th reference signal:

[0110]

[0111] In the formula, ρ(X,Y) represents the correlation coefficient between X and Y. k Let ρ be the correlation vector of the k-th reference signal. k N The correlation coefficient of the Nth filter corresponding to the kth reference signal. W is the transpose of the EEG signal after filtering by the Nth filter corresponding to the kth reference signal. x This represents the weight vector for x in canonical correlation analysis. The EEG signal after filtering by the Nth filter corresponding to the kth reference signal, Y fk For the k-th reference signal, Y T W is the template signal vector. y This represents the weight vector of y in canonical correlation analysis;

[0112] The correlation coefficients corresponding to all sub-bands The weighted sum of squares is used as a feature for target recognition:

[0113]

[0114] Where n is the nth sub-band obtained from the decomposition. ρ is the weighted correlation coefficient corresponding to the stimulus frequency. k n Let N be the nth filter, and N be the number of filters.

[0115] The sub-band weighting function w(n) is defined as:

[0116] w(n) = n -a +bn∈[ 1,N ] (5)

[0117] Where a and b are both constants;

[0118] Extract the weighted correlation coefficient corresponding to the highest stimulus frequency. The corresponding stimulation frequency, i.e., the decoded EEG signal:

[0119]

[0120] In the formula, f target Let f be the probability of the target. i Let S be the frequency corresponding to the i-th stimulus target, and S be the total number of stimulus targets.

[0121] Specifically, firstly, filter bank analysis performs subband decomposition using multiple filters with different passbands. In this embodiment, the EEG signal X is decomposed into subbands using a filter bank. There are N sub-bands. The standard CCA procedure is applied to each sub-band component to obtain the sub-band component and a predefined reference signal corresponding to all stimulus frequencies. Where S is the correlation coefficient value between the number of stimuli. For the k-th reference signal, the correlation coefficient is calculated from N correlation coefficient values ​​ρ. k The relevant vectors are as follows:

[0122]

[0123] Where ρ(x,y) represents the correlation coefficient between x and y, corresponding to the correlation coefficients of all sub-bands. The weighted sum of squares is used as a feature for target recognition:

[0124]

[0125] Where n is the nth sub-band obtained from the decomposition, and the weighting function w(n) of the sub-band is defined as follows:

[0126] w(n) = n -a +bn∈[ 1,N ] (9)

[0127] Here, a and b are both constants, and their values ​​depend on the optimal state of the classifier after calibration. This yields the weighted correlation coefficients corresponding to the S stimulus frequencies. The largest of them The corresponding stimulation frequency is the identified target, i.e., the brain control command:

[0128]

[0129] Furthermore, the reliability of the control commands is evaluated, including:

[0130] The Softmax function is used to establish an item relationship between the EEG control commands and the probability distribution, with cross-entropy as the cost function, specifically:

[0131]

[0132] In the formula, P(H i |ρ) represents the posterior probability assuming the target is the i-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let N be the natural logarithm with the j-th obtained correlation coefficient as the exponent. t Let $\mathbf{H}$ be the target total, $j$ be each term in the summation process, and $Cost(H)$ be the total number of terms. i The result is assumed to be H. i The value of C(H) j H i ) is based on H j H i The value of the function C with parameters;

[0133] Introducing the concept of punishment, the evaluation decision is made by comparing the cost of the highest probability hypothesis Hq and the cost of rejecting the hypothesis H0, specifically as follows:

[0134]

[0135] In the formula, C(H) j H q ) is based on H j H q The value of C is a function of the parameter, where q is the stimulus label corresponding to the maximum probability. ′ The stimulus label corresponding to the second highest probability, C(H) j H0) is based on H j H0 is the value of the function C of the parameters, and ε is the threshold parameter;

[0136] The larger the value of ε, the more inclined it is to judge the command as unreliable;

[0137] The calculation methods for the cost of the maximum probability hypothesis Hq and the cost of rejecting the hypothesis H0 are as follows:

[0138]

[0139] In the formula, P(H j |ρ) represents the posterior probability assuming the target is the j-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let ρ be the natural logarithm with the k-th obtained correlation coefficient as the exponent. k To calculate the k-th correlation coefficient during the summation process, e is the natural logarithm with the maximum correlation coefficient as the exponent. ρ2ndmax ρ is the natural logarithm with the second largest correlation coefficient as the exponent. max For the largest correlation coefficient value, ρ 2ndmax This is the second largest correlation coefficient value;

[0140] When the cost of rejecting the hypothesis is lower than the cost of the recognition result, the brain control command is considered unreliable and requires the intervention of an auxiliary controller.

[0141] Specifically, the Softmax function is first used to establish an item relationship between the FBCCA calculation results and the probability distribution, with cross-entropy as the cost function.

[0142]

[0143] Since only the maximum probability hypothesis Hresult and the rejection hypothesis H0 need to be considered, the algorithm can make a decision simply by comparing the cost values ​​of these two hypotheses.

[0144] To improve the robustness of the algorithm, a penalty concept is introduced. Previous research has shown that the difference between the first and second largest feature values ​​(the two values ​​with the largest and second largest correlation coefficients) is also a usable classification criterion. The more significant this difference, the higher the probability of correct classification. Furthermore, the frequency detected by the second largest feature is a "neighbor" of the actual stimulus frequency, which may be close in the frequency domain or visual domain.

[0145]

[0146] The larger the value of ε, the more inclined the command is to be judged as "unreliable". The value of the threshold ε is determined based on the actual usage effect.

[0147] The cost calculation method for rejecting hypothesis H0 and the maximum probability hypothesis Hq is as follows:

[0148]

[0149]

[0150] When the cost of rejecting the hypothesis is lower than the cost of the recognition result, the brain control command is considered unreliable and the intervention of the auxiliary controller is required.

[0151] Regarding the value of ε, a pseudo-online experiment was first conducted using previously acquired EEG signals. The experimental results are shown below. Figure 4 .

[0152]

[0153] TP represents the number of commands that are correctly classified and considered reliable, FP represents the number of commands that are incorrectly classified but considered reliable, TN represents the number of commands that are correctly classified but whose analysis results are unreliable, and FN represents the number of commands that are incorrectly classified and whose analysis results are unreliable.

[0154] Furthermore, based on the credibility evaluation results, the robot is controlled via an auxiliary control module, including:

[0155] The robot's perception unit provides obstacle information and determines whether the mobile robot will be in a safe environment within a certain period of time. If the mobile robot is determined to be in a safe environment, the auxiliary controller tracks the user's intention; otherwise, the auxiliary controller intervenes and outputs a control command that meets safety constraints to the mobile robot to achieve collaborative control. The robot avoids the perceived obstacles by combining with a model predictive control method.

[0156] Specifically, the robot's perception module (two-dimensional LiDAR) provides obstacle information and determines whether the mobile robot will be in a safe state within a certain period of time. If the mobile robot is determined to be in a safe environment, the controller tries its best to track the user's intention. bci =[v bci ,ω bci ], v bci ω represents the speed of a mind-controlled mobile robot. bci This is the angular velocity of the mobile robot output by the brain control; otherwise, the auxiliary controller will intervene and output control commands that meet safety constraints to the mobile robot to achieve collaborative control.

[0157] Since predicting the future state of a robot based on a kinematic model has a smaller error in a shorter prediction time when the speed is low, this embodiment uses the above-mentioned kinematic model as the prediction model for the mobile robot.

[0158] Furthermore, the objective function of the model predictive control is:

[0159]

[0160] Where k represents the current time, and S(k) represents the current state. This represents the control sequence, α is the penalty factor for penalizing the control action weights, and u = [v, ω] represents the command output by the auxiliary controller. (k) The command output by the auxiliary controller at the current moment. The commands output by the mind control at the current moment. H is the penalty matrix that controls the rate of change of the output. c To control the time domain, u(k+i) The command output by the auxiliary controller is i steps backward from the current time, u (k+i-1) This is the command output by the auxiliary controller i-1 steps back from the current time.

[0161] The constrained optimization function for model predictive control is:

[0162] min J(S(k),U k )

[0163] st

[0164] S(k+i+1)=f(S(k+i),u(k+i)),i=0,…,H p -1 (24)

[0165] D safe -d (k+i+1) (d obs ,δ obs ,u (k) ,…,u (k+i) )≤0,i=0,…,H p -1 (25)

[0166] u min (k+i) max ,i=0,…,H c -1 (26)

[0167] Δu (k+i) =u (k+i) -u (k+i-1) ,i=0,…,H c -1 (27)

[0168] Δu min <Δu (k+i) <Δu max ,i=0,…,H c -1 (28)

[0169] Δu (k+i) =0, i=H c H c +1,…,H p -1(29)

[0170] In the formula, equation (24) is used to ensure the safety of the robot, D safe Let S(k+i+1) be the safe distance for the mobile robot, S(k+i) be the state of the mobile robot i+1 steps after k at the current time, S(k+i) be the state of the mobile robot i steps after k at the current time, u(k+i) be the command output by the auxiliary controller i steps after k at the current time, and H be the safe distance for the mobile robot. p ​​For the prediction time domain, i represents the sampling duration at each step; d obs ,δ obs These represent the distance and orientation information of the nearest obstacle to the mobile robot, obtained by the mobile robot's perception module.

[0171] Constraints (26)-(28) are physical constraints. c and H p T represents the control time domain and the prediction time domain, respectively. s For sampling duration, u min To output the minimum limits of velocity and angular velocity of the command, u (k+i) The command output by the auxiliary controller is i steps backward from the current time, u max To output the maximum limits of velocity and angular velocity of the command, Δu (k+i) H represents the change in velocity and angular velocity of the command output by the auxiliary controller i steps back from the current moment. c To control the time domain.

[0172] Set H c ≤H p And assume that when H c ≤k≤H p At the current time (between the prediction and control time domains), the control signal remains unchanged to reduce computational complexity, as shown in equation (26). In equation (27), u (k+i-1) This represents the output of the mobile robot controller in the previous control cycle.

[0173] At this point, the design of the mobile robot controller is complete. In each control cycle, the controller predicts a future time period, solves a constrained optimization function, and calculates and outputs the optimal control sequence that ensures safety. The first control command of the control sequence is then input into the mobile robot, and this process is repeated for rolling control.

[0174] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A brain-computer interface system for air-ground collaborative multi-unmanned systems, characterized in that, include: Human-computer interaction module: used to present visual stimuli through augmented reality technology; EEG decoding module: used to decode brain waves, obtain decoding results, and output brain control commands; Reliability Analysis Module: Used to establish the relationship between the mind control commands and the probability distribution using the Softmax function, and to perform reliability analysis on the mind control commands; Auxiliary control module: used to optimize the robot's performance and correct brain-controlled commands; The reliability analysis of the brain control commands includes: The Softmax function is used to establish an item relationship between the EEG control commands and the probability distribution, with cross-entropy as the cost function, specifically: In the formula, Let be the posterior probability assuming the target is the i-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let be the natural logarithm with the j-th obtained correlation coefficient as the exponent. Let j be the target total, and j be each term in the summation process. The result is assumed to be The value of the time, For The value of the function C with parameters; Introducing the concept of penalty, by comparing the costs of the highest probability assumption. and the cost of rejecting the assumption The evaluation and decision-making process is as follows: In the formula, For The value of function C as parameter. The stimulus label corresponding to the highest probability. The stimulus label corresponds to the second highest probability. For The value of function C as parameter. For threshold parameters; in, The larger the value, the more inclined it is to judge the command as unreliable; Cost of maximum probability assumption and the cost of rejecting the assumption The calculation method is as follows: In the formula, Let be the posterior probability assuming the target is the j-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let be the natural logarithm with the k-th correlation coefficient as the exponent. To calculate the k-th correlation coefficient during the summation process, It is the natural logarithm with the maximum correlation coefficient as the exponent. It is the natural logarithm with the second largest correlation coefficient as the exponent. The highest correlation coefficient value, This is the second largest correlation coefficient value; When the cost of rejecting the hypothesis is lower than the cost of the recognition result, the brain control command is considered unreliable and requires the intervention of an auxiliary controller.

2. The brain-computer interface system for air-ground collaborative multi-unmanned systems according to claim 1, characterized in that, The auxiliary control module includes a control performance optimization unit, which optimizes the robot's control performance through an auxiliary controller, perceives surrounding environmental information in real time through lidar, avoids perceived obstacles by combining with model predictive control methods, and corrects brain-controlled commands.

3. An operation method for a brain-computer interface system for air-ground collaborative multi-unmanned systems, characterized in that, include: Collect the operator's EEG signals, and perform preprocessing and decoding operations on the EEG signals to obtain EEG control commands; The reliability of the control commands is evaluated using a reliability analysis module. Based on the credibility evaluation results, the robot is controlled through the auxiliary control module, and the brain-controlled commands are corrected. Among them, the operator controls the ground unmanned vehicle through the brain control command and directly controls the drone through the remote control device; The reliability of the control commands is evaluated, including: The Softmax function is used to establish an item relationship between the EEG control commands and the probability distribution, with cross-entropy as the cost function, specifically: In the formula, Let be the posterior probability assuming the target is the i-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let be the natural logarithm with the j-th obtained correlation coefficient as the exponent. Let j be the target total, and j be each term in the summation process. The result is assumed to be The value of the time, For The value of the function C with parameters; Introducing the concept of penalty, by comparing the costs of the highest probability assumption. and the cost of rejecting the assumption The evaluation and decision-making process is as follows: In the formula, For The value of function C as parameter. The stimulus label corresponding to the highest probability. The stimulus label corresponds to the second highest probability. For The value of function C as parameter. For threshold parameters; in, The larger the value, the more inclined it is to judge the command as unreliable; Cost of maximum probability assumption and the cost of rejecting the assumption The calculation method is as follows: In the formula, Let be the posterior probability assuming the target is the j-th stimulus. Let be the natural logarithm with the i-th obtained correlation coefficient as the exponent. Let be the natural logarithm with the k-th correlation coefficient as the exponent. To calculate the k-th correlation coefficient during the summation process, It is the natural logarithm with the maximum correlation coefficient as the exponent. It is the natural logarithm with the second largest correlation coefficient as the exponent. The highest correlation coefficient value, This is the second largest correlation coefficient value; When the cost of rejecting the hypothesis is lower than the cost of the recognition result, the brain control command is considered unreliable and requires the intervention of an auxiliary controller.

4. The operating method according to claim 3, characterized in that, Collecting the operator's electroencephalogram (EEG) signals, including: The operator wears a standard EEG acquisition device to collect EEG signals while gazing at a stimulus scene displayed by an augmented reality helmet. The acquisition frequency is set, and the collected EEG signals are segmented and downsampled.

5. The operating method according to claim 4, characterized in that, Preprocessing of the EEG signals includes: The acquired EEG signals were preprocessed by baseline correction, bandpass filtering, and notch filtering to remove noise.

6. The operating method according to claim 3, characterized in that, Decoding the EEG signal includes: The preprocessed EEG signal is filtered through a filter bank to obtain several sub-bands; Each sub-band is analyzed using canonical correlation analysis to obtain the correlation coefficient between the sub-band component and the predefined reference signal corresponding to all stimulus frequencies. Target recognition is performed based on the correlation coefficient to obtain the decoded EEG signal.

7. The operating method according to claim 6, characterized in that, Target identification based on the correlation coefficient includes: Calculate the first Correlation vector of each reference signal: In the formula, This represents the correlation coefficient between X and Y. The correlation vector of the k-th reference signal. For the corresponding number The correlation coefficient of the Nth filter for a reference signal For the corresponding number The transpose of the EEG signal after filtering by the Nth filter of the reference signal. This represents the weight vector for x in canonical correlation analysis. Corresponding to the The EEG signal after filtering by the Nth filter of the reference signal. For the k-th reference signal, The template signal vector, This represents the weight vector of y in canonical correlation analysis; The correlation coefficients corresponding to all sub-bands The weighted sum of squares is used as a feature for target recognition: in, For the decomposed first Sub-band The weighted correlation coefficient corresponding to the stimulus frequency. Let N be the nth filter, and N be the number of filters. Subband weighting function Defined as: in, and All are constants; Extract the weighted correlation coefficient corresponding to the highest stimulus frequency. The corresponding stimulation frequency, i.e., the decoded EEG signal: In the formula, The desired probability is... Let S be the frequency corresponding to the i-th stimulus target, and S be the total number of stimulus targets.

8. The operating method according to claim 3, characterized in that, Based on the credibility evaluation results, the robot is controlled via an auxiliary control module, including: The robot's perception unit provides obstacle information and determines whether the mobile robot will be in a safe environment within a certain period of time. If the mobile robot is determined to be in a safe environment, the auxiliary controller tracks the user's intention; otherwise, the auxiliary controller intervenes and outputs a control command that meets safety constraints to the mobile robot to achieve collaborative control. The robot avoids the perceived obstacles by combining with a model predictive control method.

9. The operating method according to claim 8, characterized in that, The objective function of the model predictive control is: in, Representing the current moment, Indicates the current state. Indicates a control sequence. The penalty factor is used to penalize the weight of the control action. This indicates the command output by the auxiliary controller. The command output by the auxiliary controller at the current moment. The commands output by the mind control at the current moment. It is a penalty matrix that controls the rate of change of the output. To control the time domain, This is the command output by the auxiliary controller i steps back from the current time. This is the command output by the auxiliary controller i-1 steps back from the current time.

Images