Brain-computer interface system and method for air-ground cooperative multi-unmanned system

Abstract

The invention discloses a brain-computer interface system and method for an air-ground cooperative multi-unmanned system. The system comprises a man-machine interaction module used for presenting a visual stimulation interface through an augmented reality technology; the electroencephalogram decoding module is used for decoding the electroencephalogram to obtain a decoding result and outputting a brain control instruction; the reliability analysis module is used for establishing a project relationship between the brain control instruction and probability distribution by using a Softmax function and performing reliability analysis on the brain control instruction; and the auxiliary control module is used for optimizing the performance of the robot and correcting the brain control instruction. According to the invention, the augmented reality technology is combined with the SSVEP brain-computer interface, so that the portability and practicability of the brain-computer interface are improved.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction science, brain-computer interface technology and artificial intelligence control technology, and in particular to a brain-computer interface system and method for air-to-ground collaborative multi-unmanned systems. Background Art

[0002] Modern intelligent air-ground collaborative robot systems (composed of drones and unmanned vehicles) can give full play to their respective advantages to jointly complete complex tasks such as search environment detection, target perception and recognition, and subsequent rescue. The equipment can remotely enter complex environments and perform dangerous tasks instead of operators, providing better solutions for search and rescue and anti-terrorism and riot control. However, unmanned equipment currently has problems such as poor collaborative performance, unstable intelligence level, and low efficiency, and cannot be applied to complex practical environments in the short term. Human-machine shared control technology can make up for the shortcomings of fully autonomous air-ground collaborative systems. However, due to the complex operation of the air-ground collaborative robot system, if multiple operators control drones and unmanned vehicles separately, the performance of the air-ground collaborative system will be affected by the proficiency of multiple people's cooperation. In response to the development needs of this type of air-ground collaborative robot system, the present invention proposes a brain-computer interface (Brain-Computer Interface, BCI) system for air-ground collaborative multi-unmanned systems. A single operator can control the drone with both hands and control the unmanned vehicle through the brain-computer interface at the same time, so as to achieve efficient control of the air-ground collaborative robot system.

[0003] Brain-computer interface technology realizes the human dream of using brain activities to communicate with and even control the external environment. It can bypass the body's own organs and enable the brain to interact efficiently with external equipment without the usual media - the help of peripheral nerves and limbs. It has great potential in giving operators additional execution capabilities.

[0004] At present, the methods of collecting brain activity signals can be divided into invasive, partially invasive and non-invasive. Non-invasive BCI avoids complicated, expensive and high-risk surgery, and its brain activity recording methods include scalp electroencephalography (EEG) and so on. Since EEG acquisition has low cost, sufficient time resolution, safety, hygiene, ease of use, and can greatly shorten the experimental preparation time, it is widely used in BCI systems. Brain-computer interface technology based on Steady-State Visual Evoked Potentials (SSVEP) has been widely used due to its advantages such as fast command generation speed and high signal-to-noise ratio.

[0005] Most of the early brain-controlled robot systems belong to direct control brain-controlled robot systems. In this control mode, BCI obtains control intentions by analyzing the subject's EEG signals and converts them into control commands that can be recognized by the robot, and directly acts on the robot. However, due to the performance limitations of BCI, such as low recognition accuracy, long command sending intervals, and easy operator fatigue, this direct control method cannot meet the needs of practical applications in most cases.

[0006] Traditional SSVEP brain-computer interfaces mostly generate visual stimulation through computer screens to induce EEG signals. This stimulation method greatly reduces the portability and practicality of the brain-computer interface system. In addition, in traditional SSVEP-based brain-computer interface systems, operators are in a relatively quiet and ideal environment to complete a single designated task. In actual application scenarios, the performance of the brain-computer interface is reduced due to external environmental interference or operator factors (distraction, fatigue, etc.), which will ultimately greatly affect the overall control performance of the system and even affect the safety of the system. The performance of the directly controlled brain-controlled mobile robot system is heavily dependent on the performance of the brain-computer interface used. The mobile robot may collide with obstacles during the task. Since the current brain-computer interface still has limitations in command frequency, accuracy and response time, the safety of the directly controlled brain-controlled mobile robot system cannot be fully guaranteed. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a brain-computer interface system and method for air-ground collaborative multi-unmanned systems, takes brain-computer interaction and shared control as the research object, and takes the application of air-ground collaborative robot systems as the design goal. The unmanned vehicle is controlled through the brain-computer interface to achieve efficient control of the air-ground collaborative robot system.

[0008] On the one hand, to achieve the above-mentioned purpose, the present invention provides a brain-computer interface system for air-ground collaborative multi-unmanned system, comprising:

[0009] Human-computer interaction module: used to present visual stimulation interface through augmented reality technology;

[0010] EEG decoding module: used to decode brain waves, obtain decoding results, and output brain control instructions;

[0011] Reliability analysis module: used to establish a project relationship between the brain control command and the probability distribution using a Softmax function, and perform reliability analysis on the brain control command;

[0012] Auxiliary control module: used to optimize the performance of the robot and correct brain control commands.

[0013] Preferably, the auxiliary control module includes a control performance optimization unit, which optimizes the control performance of the robot through an auxiliary controller, perceives the surrounding environment information in real time through a lidar, and avoids perceived obstacles and corrects brain control instructions in combination with a model predictive control method.

[0014] On the other hand, in order to achieve the above-mentioned purpose, the present invention also provides an operation method of a brain-computer interface system for an air-ground collaborative multi-unmanned system, comprising:

[0015] Collect the operator's EEG signals, pre-process and decode the EEG signals, and obtain EEG control instructions;

[0016] Performing credibility evaluation on the control instruction through a reliability analysis module;

[0017] The robot is controlled through the auxiliary control module according to the credibility evaluation results, and the brain control instructions are corrected;

[0018] Among them, the operator controls the ground unmanned vehicle through the brain control instructions and directly controls the drone through the remote control device.

[0019] Preferably, collecting the EEG signal of the operator includes:

[0020] The operator is asked to wear a standard EEG acquisition device to collect EEG signals by looking at the stimulation scene displayed by the augmented reality helmet, set the acquisition frequency, and segment and downsample the collected EEG signals.

[0021] Preferably, the EEG signal is preprocessed, including:

[0022] The collected EEG signals are subjected to baseline correction, bandpass filtering and notch filtering to remove noise from the EEG signals and complete preprocessing.

[0023] Preferably, decoding the EEG signal comprises:

[0024] The preprocessed EEG signal is filtered through a filter bank to obtain several sub-bands;

[0025] Analyze each of the sub-frequency bands by a canonical correlation analysis method to obtain a correlation coefficient between the sub-frequency band component and a predefined reference signal corresponding to all stimulation frequencies;

[0026] Target recognition is performed based on the correlation coefficient to obtain a decoded EEG signal.

[0027] Preferably, target identification is performed based on the correlation coefficient, including:

[0028] Calculate the correlation vector of the kth reference signal:

[0029]

[0030] In the formula, ρ(X,Y) represents the correlation coefficient between X and Y, ρ k is the correlation vector of the kth reference signal, ρ k N is the correlation coefficient of the Nth filter corresponding to the kth reference signal, is the transpose of the EEG signal filtered by the Nth filter corresponding to the kth reference signal, W x is the weight vector corresponding to x in canonical correlation analysis, The EEG signal after filtering by the Nth filter corresponding to the kth reference signal, Y fk is the kth reference signal, Y T is the template signal vector, W y is the weight vector corresponding to y in the canonical correlation analysis;

[0031] The correlation coefficients corresponding to all sub-bands The weighted sum of squares is used as the feature for target recognition:

[0032]

[0033] Where n is the nth sub-band decomposed, is the weighted correlation coefficient corresponding to the stimulation frequency, ρ k n is the nth filter, N is the number of filters;

[0034] The weight function w(n) of the sub-band is defined as:

[0035] w(n)=n -a +bn∈[1,N]

[0036] Where a and b are both constants;

[0037] Extract the weighted correlation coefficient corresponding to the maximum stimulation frequency The corresponding stimulation frequency, that is, the decoded EEG signal:

[0038]

[0039] In the formula, f target is the target probability, f i is the frequency corresponding to the i-th stimulus target, and S is the total number of stimulus targets.

[0040] Preferably, the credibility evaluation of the control instruction includes:

[0041] The Softmax function is used to establish a project relationship between the EEG control command and the probability distribution, and the cross entropy is used as the cost function, specifically:

[0042]

[0043] In the formula, P(H i |ρ) is the posterior probability of assuming that the target is the i-th stimulus, is the natural logarithm with the ith correlation coefficient as the exponent, is the natural logarithm with the jth correlation coefficient as the exponent, N t is the total number of targets, j is each small item in the summation process, Cost(H i ) is the result, assuming H i The cost value, C(H j ,H i ) is H j ,H i is the value of the function C of the parameter;

[0044] Introducing the idea of ​​penalty, by comparing the maximum probability hypothesis cost H q And reject the hypothesis cost H0 to make an evaluation decision, specifically:

[0045]

[0046] In the formula, C(H j ,H q ) is H j ,H q is the value of the parameter function C, q is the stimulus number corresponding to the maximum probability, q ′ is the stimulus number corresponding to the second highest probability, C(H j ,H0) is H j ,H0 is the value of the parameter function C, ε is the threshold parameter;

[0047] The larger the value of ε is, the more likely it is that the command is judged to be unreliable.

[0048] Maximum probability hypothesis cost H q The cost of rejecting the hypothesis H0 is calculated as follows:

[0049]

[0050] In the formula, P(H j |ρ) is the posterior probability of assuming that the target is the jth stimulus, is the natural logarithm with the ith correlation coefficient as the exponent, is the natural logarithm with the kth correlation coefficient as the exponent, ρ kis the kth correlation coefficient in the summation process, is the natural logarithm with the maximum correlation coefficient as the exponent, is the natural logarithm of the second largest correlation coefficient, ρ max is the maximum correlation coefficient value, ρ 2ndmax is the second largest correlation coefficient value;

[0051] When the cost value of rejecting the hypothesis is lower than the cost value of the recognition result, it is considered that the brain control command at this time is unreliable and requires the intervention of an auxiliary controller.

[0052] Preferably, controlling the robot through an auxiliary control module according to the credibility evaluation result includes:

[0053] The robot perception unit provides obstacle information and determines whether the mobile robot will be in a safe environment within a certain period of time. If the mobile robot is determined to be in a safe environment, the auxiliary controller tracks the user's intention. Otherwise, the auxiliary controller performs control intervention and outputs control commands that meet safety constraints to the mobile robot to achieve collaborative control; wherein the robot is combined with the model predictive control method to avoid perceived obstacles.

[0054] Preferably, the objective function of the model predictive control is:

[0055]

[0056] Among them, k represents the current moment, S(k) represents the current state, represents the control sequence, α is the penalty factor for the penalty control action weight, u=[v,ω] represents the command output by the auxiliary controller, and u (k) is the command output by the auxiliary controller at the current moment, is the command output by the brain control at the current moment, is the penalty matrix that controls the output change rate, H c To control the time domain, u (k+i) is the command output by the auxiliary controller for pushing i steps backward at the current moment, u (k+i-1) This is the command output by the auxiliary controller that is pushed back i-1 steps from the current moment.

[0057] Compared with the prior art, the present invention has the following advantages and technical effects:

[0058] The present invention combines augmented reality technology with the SSVEP brain-computer interface to improve the portability and practicality of the brain-computer interface. The reliability analysis of the EEG decoding results is performed using the Softmax function, and the auxiliary controller is used to weaken the problem of low accuracy of brain-computer interface recognition in actual application, thereby improving the overall control performance of the air-ground collaborative robot system. At the same time, the auxiliary controller based on MPC can perform autonomous obstacle avoidance and improve the control safety of the brain-controlled robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1 This is a schematic diagram of the structure of a brain-computer interface system for an air-to-ground collaborative multi-unmanned system according to an embodiment of the present invention;

[0061] Figure 2 This is a flow chart of the operation method of the brain-computer interface system for air-to-ground collaborative multi-unmanned system according to an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of stimulation forms presented in a virtual reality helmet according to an embodiment of the present invention;

[0063] Figure 4 It is a schematic diagram of pseudo-online experiment results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] The present invention proposes a brain-computer interface system for air-ground collaborative multi-unmanned systems. Figure 1 ,include:

[0067] Human-computer interaction module: used to present visual stimulation interface through augmented reality technology;

[0068] EEG decoding module: used to decode brain waves, obtain decoding results, and output brain control instructions;

[0069] Reliability analysis module: used to establish a project relationship between the brain control command and the probability distribution using a Softmax function, and perform reliability analysis on the brain control command;

[0070] Auxiliary control module: used to optimize the performance of the robot and correct brain control commands.

[0071] Specifically, this embodiment takes brain-computer interaction and shared control as the research object, and takes the application of air-ground collaborative robot system as the design goal. A single operator can control the drone with both hands and control the unmanned vehicle through the brain-computer interface at the same time, realizing efficient control of the air-ground collaborative robot system.

[0072] In the human-computer interaction module, traditional vision-based brain-computer interfaces mostly use computer screens as interactive interfaces, which are less portable and cannot meet the needs of practical applications. The brain-computer interface interaction interface in this embodiment will be realized through augmented reality (AR) technology, presenting a visual stimulation interface in an augmented reality environment. Not only can it get rid of the constraints of fixed equipment, but the operator can also observe the real world through the interactive interface, greatly improving the operator's interactive freedom and the practicality of the system.

[0073] The EEG decoding module adopts the Filter Bank Canonical Corrlation Analysis (FBCCA) algorithm, which can extract effective information embedded in the harmonic frequency components through the filter bank, speed up the target recognition speed, and improve the performance of SSVEP-BCI.

[0074] In the EEG decoding process, the Softmax function is used to project the correlation coefficient obtained in the decoding process into the probability space, and a hypothesis testing model is established, with the risk function as the evaluation of the "credibility" of the classification result. This indicator will play a decisive role in adjusting the degree of intervention of the auxiliary controller. The hypothesis testing model calculates the cost function to obtain reliability analysis.

[0075] Furthermore, the auxiliary control module includes a control performance optimization unit, which optimizes the control performance of the robot through an auxiliary controller, perceives the surrounding environment information in real time through a lidar, and avoids perceived obstacles and corrects brain control commands by combining with a model predictive control method.

[0076] Specifically, this embodiment uses the model predictive control (MPC) method to establish an auxiliary controller to optimize the robot's control performance. Through the laser radar, the robot can perceive the surrounding environment information in real time, and avoid perceived obstacles by combining with the model predictive control method. And introduce a weight term in the cost function to correct possible erroneous (low reliability) brain control instructions.

[0077] This embodiment also provides an operation method of a brain-computer interface system for an air-ground collaborative multi-unmanned system. Figure 1-Figure 2 ,include:

[0078] Collect the operator's EEG signals, pre-process and decode the EEG signals, and obtain EEG control instructions;

[0079] Performing credibility evaluation on the control instruction through a reliability analysis module;

[0080] The robot is controlled through the auxiliary control module according to the credibility evaluation results, and the brain control instructions are corrected;

[0081] Among them, the operator controls the ground unmanned vehicle through the brain control instructions and directly controls the drone through the remote control device.

[0082] The present invention combines augmented reality technology with the SSVEP brain-computer interface to improve the portability and practicality of the brain-computer interface. The reliability analysis of the EEG decoding results is performed using the Softmax function, and the auxiliary controller is used to weaken the problem of low accuracy of brain-computer interface recognition in actual application, thereby improving the overall control performance of the air-ground collaborative robot system. At the same time, the auxiliary controller based on MPC can perform autonomous obstacle avoidance and improve the control safety of the brain-controlled robot.

[0083] Furthermore, the operator's EEG signals are collected, including:

[0084] The operator is asked to wear a standard EEG acquisition device to collect EEG signals by looking at the stimulation scene displayed by the augmented reality helmet, set the acquisition frequency, and segment and downsample the collected EEG signals.

[0085] Specifically, the traditional SSVEP brain-computer interface has a fixed screen position, so the subject is required to constantly shift his or her gaze between the screen and the controlled robot during the task, which has poor flexibility and portability. Combining augmented reality technology with BCI technology can largely solve the problem of poor flexibility of traditional BCI systems and improve the portability and practicality of brain-computer interfaces based on SSVEP. In this embodiment, HoloLens2 augmented reality glasses are used to provide a stimulation interface.

[0086] In order to better obtain the EEG signal stimulation response of the brain-controlled user and avoid harmonic interference, the stimulation frequency of SSVEP is generally set between 8 and 16 Hz. Previous studies have found that the display method of the HoloLens device is different from that of common computer monitors. If white sinusoidal flickering stimulation is used, what is actually produced will be the superposition of four colors of sinusoidal stimulation such as RGBW, which flash in sequence with a difference of π / 2. Because this embodiment has high requirements on the stability of the flickering, the use of unit color flickering stimulation can avoid the above situation and induce the desired SSVEP signal. Because red is the first to refresh among the primary colors, in this embodiment, the stimulation interface is set to flash red with a black background, such as Figure 3 .

[0087] In Unity3D, a red flashing cube is created using the MRTK toolkit, and its own transparent color is used as the background color. The R value in the cube's color (RGBA) changes sinusoidally between 0 and 255 as the display is refreshed.

[0088] The operator wears a 64-channel EEG cap that meets the "10-20 international standard lead", injects conductive paste and collects EEG signals from eight channels: PO5, PO3, POz, PO4, PO6, O1, Oz, and O2. The collection frequency is 1000 Hz, and the collected EEG signals are divided into 2s windows and downsampled to 500 Hz.

[0089] This embodiment is designed so that the operator can control the drone with both hands and control the unmanned vehicle through the brain-computer interface at the same time, thereby achieving efficient control of the air-ground collaborative robot system.

[0090] The aerial drone is directly controlled by the remote control handle that comes with the drone.

[0091] Ground unmanned vehicles are controlled through brain waves.

[0092] The stimulation frequencies used in the brain-computer interface part of this embodiment are 13Hz, 12Hz, 11Hz, and 9Hz, which correspond to the brain-controlled robot's forward, left turn, right turn, and deceleration commands, respectively.

[0093] Furthermore, the EEG signal is preprocessed, including:

[0094] The collected EEG signals are subjected to baseline correction, bandpass filtering and notch filtering to remove noise from the EEG signals and complete preprocessing.

[0095] Specifically, the preprocessing process mainly includes several steps such as baseline correction, bandpass filtering and notch filtering, which are used to remove noise from EEG signals.

[0096] Furthermore, the EEG signal is decoded, including:

[0097] The preprocessed EEG signal is filtered through a filter bank to obtain several sub-bands;

[0098] Analyze each of the sub-frequency bands by a canonical correlation analysis method to obtain a correlation coefficient between the sub-frequency band component and a predefined reference signal corresponding to all stimulation frequencies;

[0099] Target recognition is performed based on the correlation coefficient to obtain a decoded EEG signal.

[0100] Specifically, canonical correlation analysis (CCA) is a statistical method that uses the correlation between comprehensive variables to analyze the overall correlation between indicators. It is widely used in brain-computer interfaces based on steady-state visual evoked potentials because of its high efficiency, robustness and simple implementation. Considering two multidimensional variables X, Y and their linear combination x = X T W x and y = Y T W y , the canonical correlation analysis method finds the weight vector W x and W y , and maximize the correlation between x and y using the following formula:

[0101]

[0102] ρ relative to W x and W y The maximum value of is the maximum canonical correlation, E is the variance, is the transpose of the weight coefficient vector corresponding to x.

[0103] When the above method is applied to the frequency detection of SSVEP, X represents the multi-channel SSVEP signal and Y represents the reference signal. Usually, a sinusoidal signal is selected as the reference signal Y. f :

[0104]

[0105] Where f is the stimulation frequency, N h is the harmonic number.

[0106] To identify the frequency of SSVEP, CCA calculates the typical correlation frequency between multi-channel SSVEP and the reference signal corresponding to each stimulus. The frequency of the reference signal with the largest correlation is considered to be the frequency of SSVEP.

[0107] SSVEP is composed of brain responses at the fundamental, harmonic and subharmonic frequencies that are the same as its stimulation frequency. In addition to the fundamental frequency component, the harmonic component can also provide useful information for frequency detection. The use of filter bank analysis can decompose SSVEP into multiple sub-band components, improve the performance of SSVEP BCI, and speed up target recognition. In this embodiment, the filter bank canonical correlation analysis method is used. That is, a bandpass filter bank is used to decompose the input EEG signal into N sub-band components, and then a canonical correlation analysis algorithm is performed on these N sub-components. Then, a weighted average is taken of the correlation coefficients obtained on the N sub-bands, corresponding to each stimulation frequency f k , and get an overall correlation coefficient value. Finally, the largest correlation coefficient value is selected from these correlation coefficient values, and its corresponding frequency is used as the final recognition result.

[0108] Further, target recognition is performed based on the correlation coefficient, including:

[0109] Calculate the correlation vector of the kth reference signal:

[0110]

[0111] In the formula, ρ(X,Y) represents the correlation coefficient between X and Y, ρ k is the correlation vector of the kth reference signal, ρ k N is the correlation coefficient of the Nth filter corresponding to the kth reference signal, is the transpose of the EEG signal filtered by the Nth filter corresponding to the kth reference signal, W x is the weight vector corresponding to x in canonical correlation analysis, The EEG signal after filtering by the Nth filter corresponding to the kth reference signal, Y fk is the kth reference signal, Y T is the template signal vector, W y is the weight vector corresponding to y in the canonical correlation analysis;

[0112] The correlation coefficients corresponding to all sub-bands The weighted sum of squares is used as the feature for target recognition:

[0113]

[0114] Where n is the nth sub-band decomposed, is the weighted correlation coefficient corresponding to the stimulation frequency, ρ k n is the nth filter, N is the number of filters;

[0115] The weight function w(n) of the sub-band is defined as:

[0116] w(n)=n -a +bn∈[ 1,N ] (5)

[0117] Where a and b are both constants;

[0118] Extract the weighted correlation coefficient corresponding to the maximum stimulation frequency The corresponding stimulation frequency, that is, the decoded EEG signal:

[0119]

[0120] In the formula, f target is the target probability, f i is the frequency corresponding to the i-th stimulus target, and S is the total number of stimulus targets.

[0121] Specifically, first, filter bank analysis uses multiple filters with different passbands to perform sub-band decomposition. In this embodiment, the EEG signal X is decomposed into N sub-bands. The standard CCA process is applied to each sub-band component separately, and the sub-band components are compared with the predefined reference signal corresponding to all stimulation frequencies ( Where S is the number of stimuli). For the kth reference signal, the N correlation coefficient values ​​ρ k The associated vectors are as follows:

[0122]

[0123] Among them, ρ(x,y) represents the correlation coefficient between x and y, corresponding to the correlation coefficient of all sub-bands The weighted sum of squares is used as the feature for target recognition:

[0124]

[0125] Where n is the nth sub-band decomposed, and the weight function w(n) of the sub-band is defined as follows:

[0126] w(n)=n -a +bn∈[ 1,N ] (9)

[0127] Among them, a and b are constants, and their values ​​depend on the best case of classifier labeling. In this way, the weighted correlation coefficients corresponding to S stimulation frequencies are obtained. The largest of these The corresponding stimulation frequency is the identified target, i.e. the brain control command:

[0128]

[0129] Furthermore, the control instruction is evaluated for credibility, including:

[0130] The Softmax function is used to establish a project relationship between the EEG control command and the probability distribution, and the cross entropy is used as the cost function, specifically:

[0131]

[0132] In the formula, P(H i |ρ) is the posterior probability of assuming that the target is the i-th stimulus, is the natural logarithm with the ith correlation coefficient as the exponent, is the natural logarithm with the jth correlation coefficient as the exponent, N t is the total number of targets, j is each small item in the summation process, Cost(H i ) is the result, assuming H i The cost value, C(H j ,H i ) is H j ,H i is the value of the function C of the parameter;

[0133] The penalty idea is introduced to make an evaluation decision by comparing the maximum probability hypothesis cost Hq and the rejection hypothesis cost H0, specifically:

[0134]

[0135] In the formula, C(H j ,H q ) is H j ,H q is the value of the parameter function C, q is the stimulus number corresponding to the maximum probability, q ′ is the stimulus number corresponding to the second highest probability, C(H j ,H0) is H j ,H0 is the value of the parameter function C, ε is the threshold parameter;

[0136] The larger the value of ε is, the more likely it is that the command is judged to be unreliable.

[0137] The calculation method of the maximum probability hypothesis cost Hq and the rejection hypothesis cost H0 is as follows:

[0138]

[0139] In the formula, P(H j |ρ) is the posterior probability of assuming that the target is the jth stimulus, is the natural logarithm with the ith correlation coefficient as the exponent, is the natural logarithm with the kth correlation coefficient as the exponent, ρ k is the kth correlation coefficient in the summation process, is the natural logarithm with the maximum correlation coefficient as the exponent, e ρ2ndmax is the natural logarithm of the second largest correlation coefficient, ρ max is the maximum correlation coefficient value, ρ 2ndmax is the second largest correlation coefficient value;

[0140] When the cost value of rejecting the hypothesis is lower than the cost value of the recognition result, it is considered that the brain control command at this time is unreliable and requires the intervention of an auxiliary controller.

[0141] Specifically, the Softmax function is first used to establish an item relationship between the calculation results of FBCCA and the probability distribution, and the cross entropy is used as the cost function.

[0142]

[0143] Since only the maximum probability hypothesis Hresult and the rejected hypothesis H0 need to be considered, the algorithm only needs to make a decision by comparing the cost values ​​of these two hypotheses.

[0144] In order to improve the robustness of the algorithm, the idea of ​​penalty is introduced into the algorithm. Previous studies have shown that the difference between the first and second largest eigenvalues ​​(the two values ​​with the largest and second largest correlation coefficients) is also a usable basis for classification. The more obvious this difference is, the higher the probability of correct classification. In addition, the frequency detected by the second largest feature is a "neighbor" of the real stimulus frequency, and they may be close in the frequency domain or visual domain.

[0145]

[0146] The larger the value of ε is, the more likely it is that the command is judged to be “unreliable”. The value of the threshold ε is determined based on actual usage results.

[0147] The cost of rejecting hypothesis H0 and the maximum probability hypothesis Hq is calculated as follows:

[0148]

[0149]

[0150] When the cost value of rejecting the hypothesis is lower than the cost value of the recognition result, it is considered that the brain control command at this time is unreliable and requires the intervention of an auxiliary controller.

[0151] For the value of ε, we first try to use the previously collected EEG signals to conduct a pseudo-online experiment. The experimental results are shown in Figure 4 .

[0152]

[0153] TP is the number of commands that are correctly classified and considered reliable, FP is the number of commands that are incorrectly classified but considered reliable, TN is the number of commands that are correctly classified but the analysis results are unreliable, and FN is the number of commands that are incorrectly classified and the analysis results are unreliable.

[0154] Furthermore, the robot is controlled by an auxiliary control module according to the credibility evaluation result, including:

[0155] The robot perception unit provides obstacle information and determines whether the mobile robot will be in a safe environment within a certain period of time. If the mobile robot is determined to be in a safe environment, the auxiliary controller tracks the user's intention. Otherwise, the auxiliary controller performs control intervention and outputs control commands that meet safety constraints to the mobile robot to achieve collaborative control; wherein the robot is combined with the model predictive control method to avoid perceived obstacles.

[0156] Specifically, the robot perception module (two-dimensional laser radar) provides obstacle information and determines whether the mobile robot will be in a safe state within a certain period of time. If the mobile robot is judged to be in a safe environment, the controller will try its best to track the user's intention u bci =[v bci ,ω bci ],v bci is the speed of the mobile robot output by brain control, ω bci The angular velocity of the mobile robot is output by brain control; otherwise, the auxiliary controller will intervene and output control commands that meet the safety constraints to the mobile robot to achieve collaborative control.

[0157] Since the error of predicting the future state of the robot based on the kinematic model is small within a short prediction time under low-speed conditions, this embodiment adopts the above kinematic model as the prediction model of the mobile robot.

[0158] Furthermore, the objective function of the model predictive control is:

[0159]

[0160] Among them, k represents the current moment, S(k) represents the current state, represents the control sequence, α is the penalty factor for the penalty control action weight, u=[v,ω] represents the command output by the auxiliary controller, and u (k) is the command output by the auxiliary controller at the current moment, is the command output by the brain control at the current moment, is the penalty matrix that controls the output change rate, H c To control the time domain, u(k+i) is the command output by the auxiliary controller for pushing i steps backward at the current moment, u (k+i-1) This is the command output by the auxiliary controller that is pushed back i-1 steps from the current moment.

[0161] The constrained optimization function of model predictive control is:

[0162] min J(S(k),U k )

[0163] st

[0164] S(k+i+1)=f(S(k+i),u(k+i)),i=0,…,H p -1 (24)

[0165] D safe -d (k+i+1) (d obs ,δ obs ,u (k) ,…,u (k+i) )≤0,i=0,…,H p -1 (25)

[0166] u min (k+i) max ,i=0,…,H c -1 (26)

[0167] Δu (k+i) =u (k+i) -u (k+i-1) ,i=0,…,H c -1 (27)

[0168] Δu min <Δu (k+i) <Δu max ,i=0,…,H c -1 (28)

[0169] Δu (k+i) =0,i=H c ,H c +1,…,H p -1(29)

[0170] In the formula, formula (24) is used to ensure the safety of the robot, D safe is the safe distance of the mobile robot, S(k+i+1) is the state of the mobile robot at the current moment k after pushing i+1 steps, S(k+i) is the state of the mobile robot at the current moment k after pushing i steps, u(k+i) is the command output by the auxiliary controller at the current moment i after pushing i steps, H p ​​For the prediction time domain, i is the sampling time of each step; d obs ,δ obs They respectively represent the distance and orientation information of the nearest obstacle of the mobile robot obtained by the perception module of the mobile robot,

[0171] Constraints (26)-(28) are physical constraints. c and H p Represent the control time domain and prediction time domain respectively, T s is the sampling duration, u min is the minimum limit of the speed and angular velocity of the output command, u (k+i) is the command output by the auxiliary controller for pushing i steps backward at the current moment, u max is the maximum limit of the speed and angular velocity of the output command, Δu (k+i) is the speed and angular velocity change of the command output by the i-step backward auxiliary controller at the current moment, H c To control the time domain.

[0172] Set H c ≤H p And suppose that when H c ≤k≤H p When (the current time is between the prediction time domain and the control time domain), the control signal does not change to reduce the complexity of the calculation, as shown in equation (26). In equation (27), u (k+i-1) represents the output of the mobile robot controller in the previous control cycle.

[0173] At this point, the design of the mobile robot controller has been completed. In each control cycle, the controller will make a prediction for a period of time in the future, solve the optimization function with constraints, and calculate and output the optimal control sequence to ensure safety. Then the first control command of the control sequence is input into the mobile robot, and rolling control is repeated in this way.

[0174] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A brain-computer interface system for air-ground collaborative multi-unmanned systems, characterized in that: include: Human-computer interaction module: used to present visual stimulation interface through augmented reality technology; EEG decoding module: used to decode brain waves, obtain decoding results, and output brain control instructions; Reliability analysis module: used to establish a project relationship between the brain control command and the probability distribution using a Softmax function, and perform reliability analysis on the brain control command; Auxiliary control module: used to optimize the performance of the robot and correct brain control commands.

2. The brain-computer interface system for air-ground collaborative multi-unmanned systems according to claim 1 is characterized in that: The auxiliary control module includes a control performance optimization unit, which optimizes the control performance of the robot through an auxiliary controller, perceives the surrounding environment information in real time through a lidar, and avoids perceived obstacles and corrects brain control instructions in combination with a model predictive control method.

3. An operating method of a brain-computer interface system for air-ground collaborative multi-unmanned systems, characterized in that: include: Collect the operator's EEG signals, pre-process and decode the EEG signals, and obtain EEG control instructions; Performing credibility evaluation on the control instruction through a reliability analysis module; The robot is controlled through the auxiliary control module according to the credibility evaluation results, and the brain control instructions are corrected; Among them, the operator controls the ground unmanned vehicle through the brain control instructions and directly controls the drone through the remote control device.

4. The operating method according to claim 3, characterized in that: Collecting the electroencephalogram signal of the operator, including: The operator is asked to wear a standard EEG acquisition device to collect EEG signals by looking at the stimulation scene displayed by the augmented reality helmet, set the acquisition frequency, and segment and downsample the collected EEG signals.

5. The operating method according to claim 4, characterized in that: Preprocessing the EEG signal includes: The collected EEG signals are subjected to baseline correction, bandpass filtering and notch filtering to remove noise from the EEG signals and complete preprocessing.

6. The operating method according to claim 1, characterized in that: The electroencephalogram signal is decoded, comprising: The preprocessed EEG signal is filtered through a filter bank to obtain several sub-bands; Analyze each of the sub-frequency bands by a canonical correlation analysis method to obtain a correlation coefficient between the sub-frequency band component and a predefined reference signal corresponding to all stimulation frequencies; Target recognition is performed based on the correlation coefficient to obtain a decoded EEG signal.

7. The operating method according to claim 6, characterized in that: Performing target recognition based on the correlation coefficient includes: Calculate the correlation vector of the kth reference signal: In the formula, ρ(X,Y) represents the correlation coefficient between X and Y, ρ k is the correlation vector of the kth reference signal, ρ k N is the correlation coefficient of the Nth filter corresponding to the kth reference signal, is the transpose of the EEG signal filtered by the Nth filter corresponding to the kth reference signal, W x is the weight vector corresponding to x in canonical correlation analysis, The EEG signal filtered by the Nth filter corresponding to the kth reference signal, is the kth reference signal, Y T is the template signal vector, W y is the weight vector corresponding to y in the canonical correlation analysis; The correlation coefficients corresponding to all sub-bands The weighted sum of squares is used as the feature for target recognition: Where n is the nth sub-band decomposed, is the weighted correlation coefficient corresponding to the stimulation frequency, ρ k n is the nth filter, N is the number of filters; The weight function w(n) of the subband is defined as: w(n)=n -a +b n∈[1,N] Among them, a and b are constants; Extract the weighted correlation coefficient corresponding to the maximum stimulation frequency The corresponding stimulation frequency, that is, the decoded EEG signal: In the formula, f target is the target probability, f i is the frequency corresponding to the i-th stimulus target, and S is the total number of stimulus targets.

8. The operating method according to claim 1, characterized in that: The control instruction is evaluated for credibility, including: The Softmax function is used to establish a project relationship between the EEG control command and the probability distribution, and the cross entropy is used as the cost function, specifically: In the formula, P(H i |ρ) is the posterior probability of assuming that the target is the i-th stimulus, is the natural logarithm with the ith correlation coefficient as the exponent, is the natural logarithm with the jth correlation coefficient as the exponent, N t is the total number of targets, j is each small item in the summation process, Cost(H i ) is the result, assuming H i The cost value, C(H j ,H i ) is H j ,H i is the value of the function C of the parameter; Introducing the idea of ​​penalty, by comparing the maximum probability hypothesis cost H q And reject the hypothesis cost H0 to make an evaluation decision, specifically: In the formula, C(H j ,H q ) is H j ,H q is the value of the parameter function C, q is the stimulus number corresponding to the maximum probability, q ′ is the stimulus number corresponding to the second highest probability, C(H j ,H0) is H j ,H0 is the value of the parameter function C, ε is the threshold parameter; The larger the value of ε is, the more likely it is that the command is judged to be unreliable. Maximum probability hypothesis cost H q The cost of rejecting the hypothesis H0 is calculated as follows: In the formula, P(H j |ρ) is the posterior probability of assuming that the target is the jth stimulus, is the natural logarithm with the ith correlation coefficient as the exponent, is the natural logarithm with the kth correlation coefficient as the exponent, ρ k is the kth correlation coefficient in the summation process, is the natural logarithm with the maximum correlation coefficient as the exponent, is the natural logarithm of the second largest correlation coefficient, ρ max is the maximum correlation coefficient value, ρ 2ndmax is the second largest correlation coefficient value; When the cost value of rejecting the hypothesis is lower than the cost value of the recognition result, it is considered that the brain control command at this time is unreliable and requires the intervention of an auxiliary controller.

9. The operating method according to claim 8, characterized in that: The robot is controlled through the auxiliary control module according to the credibility evaluation results, including: The robot perception unit provides obstacle information and determines whether the mobile robot will be in a safe environment within a certain period of time. If the mobile robot is determined to be in a safe environment, the auxiliary controller tracks the user's intention. Otherwise, the auxiliary controller performs control intervention and outputs control commands that meet safety constraints to the mobile robot to achieve collaborative control; wherein the robot is combined with the model predictive control method to avoid perceived obstacles.

10. The operating method according to claim 9, characterized in that: The objective function of the model predictive control is: Among them, k represents the current moment, S(k) represents the current state, represents the control sequence, α is the penalty factor for the penalty control action weight, u=[v,ω] represents the command output by the auxiliary controller, and u (k) is the command output by the auxiliary controller at the current moment, is the command output by the brain control at the current moment, is the penalty matrix that controls the output change rate, H c To control the time domain, u (k+i) is the command output by the auxiliary controller for pushing i steps backward at the current moment, u (k+i-1) This is the command output by the auxiliary controller that is pushed back i-1 steps from the current moment.

Images