Multi-agent cooperative control system based on multi-mode brain-computer interface-visual tracking

Abstract

The invention discloses a multi-agent cooperative control system based on multi-mode brain-computer interface-visual tracking, and the system comprises the following modules: a signal collection and processing module which employs an 8-lead electroencephalogram cap to collect electroencephalogram original data, selects occipital region parts (PO7, PO8, P3, P4, Pz, O1, O2 and Oz) as an electroencephalogram data collection region, and carries out the EEG data collection; the feature extraction and classification module is used for extracting SSVEP signal features by using a space-time convolutional network, optimizing classification precision in combination with an integrated learning strategy, extracting EOG signals based on a Transform model, integrating an LMS adaptive filter and denoising the EEG signals in combination with wavelet transform, expanding the frequency tracking range of the adaptive filter, and reducing signal loss to the greatest extent; the cooperative control strategy design module is used for mapping the electroencephalogram signals into control instructions of different intelligent agents, and cooperative control over the unmanned aerial vehicle and the trolley is achieved; and the target detection and tracking module is used for optimizing an unmanned aerial vehicle visual tracking system based on a PP-YOLOE model and realizing accurate tracking and automatic adjustment of the trolley.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and specifically relates to a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking. Background Art

[0002] Brain-computer interface technology mainly collects EEG signals, processes and extracts EEG signals, analyzes and identifies the user's intentions, and finally translates EEG signals into specific control instructions to control external devices and realize the interaction between the brain and the outside world. Common EEG-BCI systems mainly include P300 potential, motor imagery and steady-state visual evoked potential. Among them, the SSVEP-BCI system has become the focus of BCI technology research due to its high precision, fast response and easy control. In 2020, Lei et al. proposed a new intelligent remote operation system based on SSVEP brain-computer interface for a mobile wall-climbing cleaning robot. The user can control the moving direction of the cleaning robot by looking at four flashing dots of different frequencies; in 2022, Guo et al. developed an SSVEP-BCI-controlled soft robot glove system for functional rehabilitation after stroke. The user can control the two soft robot gloves by looking at flashing dots of different frequencies marked left or right, thereby performing rehabilitation training, which has better effects than simple robot glove rehabilitation; Zhang et al. proposed a speed modulation method for controlling the robotic arm in an SSVEP-based BCI, so that the flashing cursor and the robotic arm move synchronously, reducing the visual fatigue of the user who constantly switches his eye perspective between cursor stimulation and the robot; Chen et al. applied the Transformer model to SSVEP classification for the first time, and proposed an extended SSVEPformer based on filter bank technology. Its average classification accuracy and ITR are better than the other five commonly used baseline methods, optimizing the classification accuracy of SSVEP.

[0003] In order to improve the accuracy and flexibility of brain-computer interface technology, BCI systems usually combine a variety of auxiliary means, among which EOG (Electrooculography) is a bioelectrophysiological technology used to measure and record the electrical activity of eye movements. EOG signals are relatively simple and immune to EEG noise, which can simplify signal processing and improve the stability of BCI systems. It is often combined with BCI technology for research.

[0004] At present, there are still some challenges in various applications of SSVEP-based BCI systems: First, since SSVEP signals are easily affected by factors such as ambient light and noise, and the strength of SSVEP signals varies from user to user, the accuracy of SSVEP signal processing and differential measurement technology needs to be improved; second, since SSVEP signals are often confused with periorbital motion signals, improving the accuracy of SSVEP signal classification is a major challenge. As for the SSVEP-BCI system combined with EOG, since EOG signals are easily affected by the environment and other muscle movements, SSVEP signals and EOG signals are very likely to interfere with each other. Accurate identification, classification and feature extraction of the two signals is a major challenge currently faced by the SSVEP-EOG-BCI system.

[0005] With the continuous deepening of research, BCI technology has also been more optimized and developed. Sadiq et al. proposed a new automation framework based on pre-trained convolutional neural network (CNN) to optimize the performance of BCI system; Abenna et al. improved the accuracy of multi-classification of EEG signals in BCI system by combining SCA algorithm and LGBM classifier. Zhou et al. proposed a shared robotic arm control system based on hybrid asynchronous brain-computer interface and computer vision, which realized the control of robotic arm to complete precise tasks in three-dimensional space; Zhang et al. of Stanford University proposed NOIR system which synergistically integrated robot learning algorithm based on small sample strategy, which can be adapted according to the needs and behavioral habits of different users, and adopted modular ideas to decode neural signals to obtain human intentions, which improved the efficiency of BRI and can efficiently complete up to 20 tasks. In the future, how to improve the robustness of BCI multimodal fusion, integrate big data and machine learning algorithms and control research will also become new challenges facing the development of BCI technology;

[0006] At present, the research on brain-controlled multi-agents still faces some urgent problems: First, the control instructions between multi-agents are too scattered, and the collaborative control based on EEG signals is not efficient enough; second, the operating environment between multi-agents is different, and different agents are difficult to adapt to the environment when running under the same control mechanism, which also leads to problems such as slow response time of the collaborative system; third, the real-time image transmission interface of brain-controlled multi-agents is single, and it is difficult to grasp information in real time. Therefore, how to achieve mutual coordination of decision-making among various machines and balance the goals and needs of different machines based on BCI technology is a major challenge in research.

[0007] Multi-target tracking is the core task of UAV visual tracking. UAVs continuously take images or videos and use tracking algorithms to estimate the position, speed and trajectory of the target. There are two main multi-target tracking strategies: one is tracking by detection (TBD), and the other is tracking based on the initial frame (Detection Free Tracking, DFT). Since DFT requires manual marking, the tracking result is unstable, so the multi-target tracking strategy is mainly TBD.

[0008] TBD's main target detection algorithms are SORT and DeepSORT. The SORT algorithm was proposed by Alex Bewley and others. Since detection quality is a key factor affecting tracking performance, changing the detector with the SORT algorithm can improve tracking quality by up to 18.9%, which is more than 20 times faster than other trackers. The DeepSORT algorithm is a multi-target tracking algorithm based on deep learning. It introduces a deep learning model, the convolutional neural network, on the basis of the SORT algorithm to extract appearance features and motion characteristics from the detected targets, which can better handle the appearance changes and occlusion of the targets. Both the SORT algorithm and the DeepSORT algorithm need to rely on target detection with good real-time performance. The YOLO detector is a single-stage target detector that puts feature extraction, regression and classification in a convolutional network, which simplifies the network. In real-time conditions, the average detection accuracy mAP reaches 63.4%, and the detection speed reaches 45frame / s. Combining the advantages of Faster RCNN and YOLO, Liu et al. proposed the SSD algorithm, which runs at 59 frames / s on Nvidia Titan X, significantly better than YOLO, and achieves an mAP of 79.8% on the VOC2007 dataset, better balancing detection accuracy and speed. In 2022, the improved SSD algorithm proposed by Sun Peng et al. exceeded the original SSD model by 3% in small object detection AP value. It effectively solves the problem of small target detection difficulty in aerial photography; Ma et al. proposed a multi-UAV real-time tracking algorithm based on improved PaddlePaddle-YOLO (PP-YOLO) and simple online real-time tracking based on deep association metric (Deep-SORT) to address the problems of poor detection performance and unstable tracking of small target UAVs by traditional target detection algorithms; Sun et al. proposed a lightweight target detector based on YOLO with the functions of structural simplification and attention enhancement; Liu et al. designed a lightweight target detection algorithm through deep separable convolution and multiple efficient network structures, which maintained high detection accuracy while reducing the model size; Xu et al. optimized PP-VOLOv2 and proposed the PP-VOLPE model, which improved 1.9AP and 1.3AP respectively compared with PP-VOLOv2 and VOLOX in COCO test development, achieving a higher performance target detection model.

[0009] Due to the continuous optimization and improvement of target detection algorithms, the accuracy of UAV visual tracking has been greatly improved under ideal conditions, but the real-time and precision of tracking still need to be improved, and there are problems such as low resolution and poor quality of image transmission. Although the fixed rectangular frame used in traditional target detection can simplify the detection task by pre-defining the target position, it lacks good generalization ability for the changes of the target at different angles and cannot adapt well to the special situation of the car's rapid speed change. Therefore, developing a target detection and tracking algorithm that adapts to various environmental changes is a major challenge in the current UAV visual tracking research. Summary of the invention

[0010] The purpose of the present invention is to provide a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, so as to solve the problem that the general process of BCI technology proposed in the above background technology is EEG signal acquisition and processing, EEG signal decoding and recognition, and EEG command conversion and execution. Brain-controlled drones have been deeply studied and applied under various EEG controls such as single-mode and multi-modal brain-computer interfaces. Regarding the BCI system combining SSVEP and EOG, there are mainly problems such as insufficient precision in SSVEP signal and EOG signal recognition and insufficient precision in signal classification; At present, the research on brain-controlled drones is still focused on the control of a single drone, and there is still little research on brain-controlled multiple drones and multiple agents. For the research on multi-agent collaborative control based on BCI technology, there are generally problems such as scattered instructions, a single control mechanism that is difficult to adapt to different operating environments, and weak generalization ability of collaborative control strategies; For the target detection algorithm applied to drone visual tracking technology, the current research has the problem that the low resolution and poor quality of the image transmission picture make it difficult to find the target, and the real-time and accuracy of tracking still need to be optimized and improved. At the same time, the fixed rectangular bounding box used to predefine the target position in traditional target detection has problems such as weak generalization ability, and it is difficult to adapt to special situations such as complex environments and rapid changes in target speed or angle.

[0011] To achieve the above object, the present invention provides the following technical solution: a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, the system comprising the following modules:

[0012] In the signal acquisition and processing module, an 8-lead EEG cap was used to collect raw EEG data. The occipital region (PO7, PO8, P3, P4, Pz, O1, O2, Oz) was selected as the EEG data collection area with a sampling rate of 1000 Hz. Before the experiment, conductive gel was applied to the electrodes to ensure that the impedance was reduced to about 5 kΩ. The experimenter observed the flickering blocks on the screen to form EEG signals of corresponding frequencies, which were collected by the device and summarized into csvEEG data sets.

[0013] A feature extraction and classification module, the feature extraction and classification module includes a feature extractor based on spatiotemporal convolution to extract features of SSVEP signals, an SSVEP detection algorithm that solves the problem of difficult target recognition in brain-computer interface system applications caused by factors such as low signal-to-noise ratio, non-stationarity, and strong individual differences of SSVEP signals, and an EOG that processes electrooculogram signals, and also includes a control instruction encoding and output module. For the SSVEP paradigm, a three-level menu is included; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted functional commands are different; in addition, in order to facilitate collaborative control operations and ensure that the subjects can simultaneously grasp the operation of the drone and the car, the present invention includes the transmission images of the drone and the car during operation in the center of the GUI, providing the subjects with visual feedback of the movement of the machine equipment, thereby enhancing their control experience;

[0014] Stimulus paradigm module, used to provide the system with a practical, stable and secure human-computer interaction system interface;

[0015] The collaborative control strategy module constitutes a command mapping mechanism in the system, which is used to map the user's EEG signals to the commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices can be achieved;

[0016] The multi-device collaborative control module constitutes a command mapping mechanism and is used to map the user's EEG signals to the commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices can be achieved;

[0017] The drone and car control algorithm module includes a PID control algorithm module and a drone detection and tracking module. The controller equipped with the PID control algorithm module is used for drone and car control, translating EEG signals into instructions, transmitting them to the control device through a mapping system, and finally realizing the action through the motor.

[0018] As a preferred technical solution in the present invention, the signal acquisition and processing module has a built-in preprocessing module, and the processing steps of the preprocessing module are as follows:

[0019] Step 1: Use a digital filter to filter the raw EEG data to remove low-frequency noise and baseline drift, usually between 0.1 Hz and 1 Hz, and remove high-frequency noise and high-frequency oscillations, usually between 30 Hz and 70 Hz;

[0020] Step 2: Remove possible noise sources, including eye movement, muscle activity, and electrode noise. Use independent component analysis (ICA) technology to analyze the weight coefficients of different signals and decompose the mixed signal into non-independent components, thereby removing noise related to eye movement.

[0021] Step 3: Through the IIR filter, the feedback generated by the response filter output forms multiple pulses, thereby removing the noise related to muscle activity and correcting or eliminating the noise introduced by poor electrode contact or drift.

[0022] As a preferred technical solution of the present invention, in the step 2, the EEG signal is denoised by combining wavelet transform with adaptive filtering (i.e. LMS adaptive filter) for the noise interference generated during the experiment, and the least mean square (LMS) adaptive filter is used to adjust the filter coefficient to adapt to the statistical characteristics of the input signal to achieve a specific signal processing goal; the multi-resolution analysis characteristics of the wavelet decompose the signal at different scales with multiple resolutions, and decompose the mixed signal composed of various different frequencies intertwined into sub-signals of different frequency bands, and process the signal by frequency band; the wavelet transform is through the following steps: first, the noisy signal is decomposed into different frequency bands at a certain scale according to the needs, and then the frequency band where the noise is located is set to zero, or the frequency band where the useful signal is located is directly extracted to perform wavelet reconstruction;

[0023] The least mean square (LMS) adaptive filtering denoising is done through the following steps:

[0024] S1. Input signal Xn;

[0025] S2. Generates output signal Yn after passing through a digital filter with adjustable parameters;

[0026] S3. Compare the signal Yn with the reference signal Dn;

[0027] S4. Form an error signal En.

[0028] As a preferred technical solution in the present invention, the operation steps of the SSVEP detection algorithm are as follows:

[0029] First, for each EEG signal segment, GCN (graph convolutional neural network) is used to explore the dynamic and intrinsic connections between different EEG channels, that is, graph convolution operations are performed in the spatial dimension, and then a feature matrix of size N*Nf is obtained, where Nf is the number of graph convolution filters; each node in the matrix corresponds to an EEG channel, and the feature at each node is the DE feature vector extracted from the Ffb frequency band of the corresponding channel;

[0030] Next, the temporal CNN is used to extract the temporal context information of the EEG samples, that is, 2D-CNN is used to merge the feature matrices of different time periods, where the size of the convolution kernel is (Kt, 1), where Kt is the size of the convolution kernel in the time dimension; finally, the output of the temporal convolution layer is flattened and input into the dense layer to obtain a high-level spatiotemporal representation (feature vector) of the EEG, and a relatively stable SSVEP signal classification output result with a high signal-to-noise ratio is obtained;

[0031] Finally, an integrated learning strategy is adopted. When constructing the CNN model, different hyperparameters are used to build multiple models, and parallel training is performed on the same data set. When the SSVEP signal classification results are finally output, a "soft voting" method will be used, and the category label with the highest probability in the result is the result of this round of reasoning. This strategy enables the model to learn different types of signal features and obtain smaller variance when facing complex data sets.

[0032] As a preferred technical solution of the present invention, the EOG is first processed by the reference electrode and the low-pass filter, and then divided into non-overlapping periods of 4 seconds, and EOG features related to blinking are extracted from them, and then the EOG feature extraction model based on Transformer is used to identify and extract the EOG features;

[0033] The model consists of six modules: input, complex spectrum representation, channel combination, encoder, multilayer perceptron, and output;

[0034] Among them, the "complex spectrum representation module" converts the input EOG data into a complex spectrum representation and passes it to the channel combination module;

[0035] The "channel combination module" uses multiple convolutional layers to perform weighted combination of multiple channels to obtain multiple weighted results and pass them into the encoder;

[0036] The "encoder module" is an improvement on the traditional Transformer encoder. It is mainly composed of two identical and consecutive sub-encoders. Each sub-encoder consists of two parts: a CNN module and a channel MLP module.

[0037] In the CNN module, convolution kernels are used to extract features in the channel dimension and fuse the features of each module; while in the MLP module, linear transformation is used to extract features on each channel;

[0038] The EOG features extracted above are passed into a multi-layer perceptron, which consists of two fully connected layers; this module first flattens and refines the input EOG feature data, and finally expresses the output as the probability distribution of each EOG category (single blink / double blink) through the softmax function, thereby realizing the output of the classification results of the EOG features.

[0039] The present invention uses MindBridge to collect data of electrooculogram (EOG) signals, and a series of operations of the EOG feature extractor based on Transformer. The designed EOG switch can freely and flexibly switch the control state of the system.

[0040] As a preferred technical solution in the present invention, the mapping process in the collaborative control strategy module is divided into the following three steps:

[0041] ①. Based on the fact that EEG signals of different frequencies reflect different aspects of brain activity, this system can capture the user's thoughts and intentions by identifying these frequency features in EEG signals, and achieve classification of EEG signals of different frequencies;

[0042] ②, based on the spatiotemporal convolution feature extractor to detect and classify SSVEP signals, and then based on the LSTM feature extractor to identify different EOG signals, the user's intention is determined according to different frequency components, and then the signals of different frequencies are mapped to the corresponding control commands;

[0043] ③. Map the EEG signal to the corresponding command ID and convert it into an executable operation;

[0044] At the same time, in terms of collaborative control strategy, the present invention also designs an SSVEP paradigm, including a three-level menu as shown in the figure below; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted functional commands are different; in addition, to facilitate collaborative control operations and ensure that the subjects can grasp the operation of the drone and the car at the same time, the present invention includes the transmitted images of the drone and the car during operation in the center of the GUI, providing the subjects with visual feedback of the movement of the machine and equipment, thereby enhancing their control experience.

[0045] As a preferred technical solution in the present invention, the PID control algorithm module includes three parts: proportion (P), integration (I), and differentiation (D): proportion controls the reaction error, integration eliminates the steady-state error, and differentiation predicts the future error change; to simplify the PID parameter adjustment, the AOA-HHO algorithm is used to optimize the PID parameters (Kp, Ki, Kd) to minimize the integral absolute error (IAE). The algorithm is divided into initialization and iteration stages, which respectively execute the exploration stage of AOA and the development stage of HHO, and finally find the optimal PID parameter configuration.

[0046] As a preferred technical solution in the present invention, the drone detection and tracking module uses the PP-YOLOE model for target detection, and improves the drone target detection performance by optimizing the Anchor-Free structure, Backbone & Neck and Task Alignment Learning (TAL); the optimized PP-YOLOE system is used to detect and track the car, and the processing speed and accuracy are improved;

[0047] To deal with the situation where the drone loses the car, the drone detection and tracking module also includes an automatic tracking system based on boundary thresholds; when the car exceeds the drone's field of view, the drone will automatically adjust its speed and direction to keep the car in the center of the picture; to optimize the system response time, the MLP model is used to dynamically adjust the threshold boundary according to the car state, shorten the state mutation response time, and capture the key motion information of the car by reasonably dividing the time window, thereby improving the real-time performance of the system;

[0048] The drone detection and tracking module also has a special-shaped boundary recognition system, which enables the drone to adopt different tracking strategies in different areas, improving the robustness and flexibility of automatic tracking; using a TOF sensor to measure the distance between the drone and the car, improving the accuracy and real-time performance of target tracking, and ensuring that the drone can accurately lock the position of the car and make timely adjustments.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] Optimize the EEG signal preprocessing and classification and recognition scheme; based on the data preprocessing operation of traditional EEG signals, the LMS adaptive filter combined with the wavelet transform is used to denoise the EEG signal, which expands the frequency tracking range of the adaptive filter and minimizes the signal loss; in the design of the multimodal brain-computer interface system, the present invention solves the problems of low signal-to-noise ratio and non-stationarity of the SSVEP signal through a spatiotemporal convolution feature extractor, and uses the "soft voting" integrated learning strategy to optimize the classification results; at the same time, for EOG signals, the present invention designs a feature extraction model of EOG signals based on the Transformer model, improves the traditional Transformer encoder, integrates the CNN module and the channel MLP module, and improves the accuracy of the extraction and classification of blink features;

[0051] Realize flexible switching of multi-agent control modes based on multi-modal brain-computer interface; Innovatively design the traditional brain-computer interface system. In order to expand the multi-device collaborative control function, on the basis of the traditional SSVEP-based brain-computer interface system, the EOG electrooculogram signal is integrated as the switch for the entire collaborative system to change the control mode, so as to achieve wider user participation and higher control accuracy. At the same time, considering that current academic research is mainly focused on the research field of single-device control, a multi-agent collaborative control system based on the SSVEP+EOG three-level menu is designed. Through the organic integration of multi-modal control signals and different control modes, multi-agents can be controlled in collaborative control mode and unmanned mode. Both the single-control mode of the drone and the single-control mode of the car can issue accurate and reasonable control instructions. When the drone and the car are in different operating environments, they can adapt to different obstacles and achieve the effect of smooth and accurate collaborative control of multiple devices. In addition, in order to optimize the collaborative control effect, the car is designed as the object to be detected and tracked. The drone provides visual tracking to identify the position of the car to realize positioning and tracking, and sets a dual feedback interface in the center of the GUI, which simultaneously presents the video streams shot by the drone and the car, making it convenient for the control end to grasp the real-time operation status of multiple devices at the same time, and improve the overall performance of the system during the collaborative control process. Compared with the control mode of a single device, it has a wider range of uses in practical applications;

[0052] Design a UAV cooperative control strategy based on PP-YOLOE; Based on the high requirements of the cooperative control system for target detection, adopt a UAV visual tracking strategy based on PP-YOLOE, integrate the high-performance and highly adaptable real-time target detection algorithm PP-YOLOE formed by optimized operation, and apply it to the UAV system to achieve fast and accurate target recognition and tracking; In the cooperative control mode, the UAV can accurately detect the target car in real time based on this scheme, and switch to the UAV single control mode to judge the position of the target car based on the boundary threshold set in the upper camera video stream, so as to achieve real-time autonomous tracking and improve the fluency of cooperative control; This not only improves the autonomy and practicality of the UAV, but also provides efficient solutions for various tasks, which is of certain significance to the research in the field of UAV visual tracking;

[0053] A high degree of coordination among multiple agents in the collaborative control process is achieved; in view of the delay problem that may affect the performance of the collaborative system in the multi-agent collaborative control system, a classification method based on the MLP model is introduced to dynamically adjust the boundary threshold in the drone video stream according to the current real-time state of the car, effectively alleviating the delay problem caused by the drone video stream return delay, target detection processing time and instruction transmission delay; this scheme significantly improves the response time of the collaborative control system, especially when the speed of the car changes sharply, by timely adjusting the threshold, the agent can quickly identify and respond to the change, thereby effectively improving the performance of the collaborative system; at the same time, in view of the problem that the common rectangular boundaries in the drone video stream have poor tracking performance when the drone and the car are facing different directions, the present invention also proposes a multi-level special-shaped recognition boundary configuration strategy, which configures different tracking strategies by judging that the tracking target is located in different areas to optimize the performance of the drone automatic tracking system, thereby improving the coordination of the drone and the car collaborative control process and adapting to the multi-angle movement of the agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall system diagram of the present invention;

[0055] Figure 2 Schematic diagram of the spatiotemporal convolution module of the present invention;

[0056] Figure 3 Schematic diagram of the distribution of channels of the present invention.

[0057] Figure 4 This is the design diagram of the system control interface of the present invention DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figures 1 to 4 The present invention provides a technical solution: a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, the system includes the following modules:

[0060] In the signal acquisition and processing module, an 8-lead EEG cap was used to collect raw EEG data. The occipital region (PO7, PO8, P3, P4, Pz, O1, O2, Oz) was selected as the EEG data collection area with a sampling rate of 1000 Hz. Before the experiment, conductive gel was applied to the electrodes to ensure that the impedance was reduced to about 5 kΩ. The experimenter observed the flickering blocks on the screen to form EEG signals of corresponding frequencies, which were collected by the device and summarized into csvEEG data sets.

[0061] A feature extraction and classification module, the feature extraction and classification module includes a feature extractor based on spatiotemporal convolution to extract features of SSVEP signals, an SSVEP detection algorithm that solves the problem of difficult target recognition in brain-computer interface system applications caused by factors such as low signal-to-noise ratio, non-stationarity, and strong individual differences of SSVEP signals, and an EOG that processes electrooculogram signals, and also includes a control instruction encoding and output module. For the SSVEP paradigm, a three-level menu is included; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted functional commands are different; in addition, in order to facilitate collaborative control operations and ensure that the subjects can simultaneously grasp the operation of the drone and the car, the present invention includes the transmission images of the drone and the car during operation in the center of the GUI, providing the subjects with visual feedback of the movement of the machine equipment, thereby enhancing their control experience;

[0062] The stimulation paradigm module is used to provide the system with a practical, stable and safe human-computer interaction system interface. The system interaction interface can be divided into three levels, namely the system startup interface, the system controllable interface and the system shutdown interface; the top layer of the system is the brain switch interface. When the user wants to use this system, he must first turn on the system switch, because the system is locked by default, and the drone and the car are in the shutdown state; the existence of the brain switch can significantly reduce the false alarm rate of the system, especially for such applications that require robust control. Reducing or even eliminating the false trigger rate of the system is of great significance to safety and stability; after turning on the system, you can enter the system control interface. The system is in the collaborative control mode of the drone and the car by default. The user can select the instructions to be executed on the device in this control state (joint forward, joint backward, joint left turn or joint right turn) on this interface, and each group of buttons is a control instruction; when the user wants to shut down the system, he needs to first trigger the button to enter the system shutdown interface on the system control interface, and then shut down the system after entering the system shutdown interface;

[0063] The system switch interface consists of a system start interface and a system shutdown interface. The label below the switch button on the start interface is "Start", and the label below the shutdown interface is "Stop"; there are six groups of buttons, corresponding to the six numbers "1", "2", "3", "4", "5" and "6", of which there are five groups of pseudo-keys and only one group is the real switch key, which is marked by labels below the switch key; each group of buttons consists of a central square and rectangular buttons around it; the buttons on the periphery of each group of buttons flash at a fixed frequency to modulate the SSVEP stimulation paradigm. This interface selects 6 frequencies, namely 7.5Hz, 9Hz, 10.5Hz, 12Hz, 13.5Hz and 15Hz; the flashing form of each group of buttons is the button background color change, shape change, etc. The shape of the image changes from a diamond to a circle; the flashing duration is 100ms, and the time interval between two adjacent flashes is 200ms; thus, by looking at the button, an SSVEP potential can be generated; the designed system control interface; six groups of buttons, corresponding to the six letters "A", "B", "C", "D", "E" and "F", the flashing and changing frequency of the six groups of buttons are consistent with the system switch interface buttons; considering that the simultaneous flashing of the six visual stimulus blocks may cause mutual interference, the six stimulus blocks are arranged around the screen when designing the induced interface; and in order to enable the subjects to better coordinate the control of the drone and the car, and adapt to the operation under different control modes, the drone and car images fed back by the camera in real time are displayed in the middle of the interface to achieve synchronous dual feedback;

[0064] The collaborative control strategy module constitutes a command mapping mechanism in the system, which is used to map the user's EEG signals to the commands for controlling devices such as cars and drones. Through the intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices are achieved. The command mapping mechanism mainly achieves the following three main goals: First, adaptive switching is achieved; this mechanism allows users to control different types of devices through EEG signals, and switch to control different devices in different situations without having to face the cumbersome device switching and reconfiguration process; the command mapping mechanism analyzes the user's EEG signals and intelligently maps them to the control commands of the corresponding devices, thereby achieving adaptive switching; second, collaborative control is achieved; this mechanism ensures that multiple devices can work together to complete more complex tasks, so that users can control the two devices at the same time to achieve more efficient collaborative actions; third, based on the experiment of collaborative control of drones and cars, this mechanism promotes support for a variety of different types of devices to meet various application needs;

[0065] The multi-device collaborative control module constitutes a command mapping mechanism and is used to map the user's EEG signals to the commands for controlling devices such as cars and drones. Through the intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices are achieved. The command mapping mechanism mainly achieves the following three main goals: First, adaptive switching is achieved; this mechanism allows users to control different types of devices through EEG signals, and switch to control different devices in different situations without having to face the cumbersome device switching and reconfiguration process; the command mapping mechanism analyzes the user's EEG signals and intelligently maps them to the control commands of the corresponding devices, thereby achieving adaptive switching; second, collaborative control is achieved; this mechanism ensures that multiple devices can work together to complete more complex tasks, allowing users to control both devices at the same time to achieve more efficient collaborative actions; third, based on the experiment of collaborative control of drones and cars, this mechanism promotes support for a variety of different types of devices to meet various application needs;

[0066] The drone and car control algorithm module includes a PID control algorithm module and a drone detection and tracking module. The controller equipped with the PID control algorithm module is used for drone and car control, translating EEG signals into instructions, transmitting them to the control device through a mapping system, and finally realizing the action through the motor.

[0067] In this embodiment, considering that the EEG signals collected in the experiment are affected by a variety of interference factors, including biological noise (such as heartbeat and eye movement), environmental noise, electrode noise and baseline drift, the preprocessing steps generally include filtering to remove noise outside the frequency range, denoising to eliminate eye movement, muscle activity and electrode noise, baseline correction to remove baseline drift, bad track processing, data resampling, data segment selection, data averaging to obtain average EEG response, and finally checking and visualizing the data to ensure the quality and consistency of the signal. The signal acquisition and processing module has a built-in preprocessing module, and the processing steps of the preprocessing module are as follows:

[0068] Step 1: Use a digital filter to filter the raw EEG data to remove low-frequency noise and baseline drift, usually between 0.1 Hz and 1 Hz, and remove high-frequency noise and high-frequency oscillations, usually between 30 Hz and 70 Hz;

[0069] Step 2: Remove possible noise sources, including eye movement, muscle activity, and electrode noise. Use independent component analysis (ICA) technology to analyze the weight coefficients of different signals and decompose the mixed signal into non-independent components, thereby removing noise related to eye movement.

[0070] Step 3: Through the IIR filter, the feedback generated by the response filter output forms multiple pulses, thereby removing the noise related to muscle activity and correcting or eliminating the noise introduced by poor electrode contact or drift.

[0071] In this embodiment, in step 2, in view of the noise interference generated during the experiment, a method combining wavelet transform and adaptive filtering (i.e., LMS adaptive filter) is used to denoise the EEG signal, and the least mean square (LMS) adaptive filter is used to adjust the filter coefficient to adapt to the statistical characteristics of the input signal to achieve a specific signal processing goal; the multi-resolution analysis characteristics of the wavelet decompose the signal at different scales with multiple resolutions, and decompose the mixed signal composed of various different frequencies intertwined into sub-signals of different frequency bands, and process the signal by frequency band; the wavelet transform is through the following steps: first, the noisy signal is decomposed into different frequency bands at a certain scale according to the needs, and then the frequency band where the noise is located is set to zero, or the frequency band where the useful signal is located is directly extracted to perform wavelet reconstruction;

[0072] When designing, it is not necessary to know the statistical characteristics of the input signal and noise in advance. It can gradually "understand" or estimate the required statistical characteristics in its own working process, and automatically adjust its parameters based on this to achieve the best filtering effect; once the statistical characteristics of the input signal change, it can track this change and automatically adjust the parameters to make the filter reach the best again;

[0073] In the experimental verification, the db5 wavelet is selected and decomposed into 5 layers; the power spectrum of the signal after denoising based on the above method becomes flatter (the flatter the power spectrum, the better the denoising effect), which proves that the method is effective and has great feasibility and flexibility.

[0074] In this embodiment, the operation steps of the SSVEP detection algorithm are as follows:

[0075] First, for each EEG signal segment, GCN (graph convolutional neural network) is used to explore the dynamic and intrinsic connections between different EEG channels, that is, graph convolution operations are performed in the spatial dimension, and then a feature matrix of size N*Nf is obtained, where Nf is the number of graph convolution filters; each node in the matrix corresponds to an EEG channel, and the feature at each node is the DE feature vector extracted from the Ffb frequency band of the corresponding channel;

[0076] Next, the temporal CNN is used to extract the temporal context information of the EEG samples, that is, 2D-CNN is used to merge the feature matrices of different time periods, where the size of the convolution kernel is (Kt, 1), where Kt is the size of the convolution kernel in the time dimension; finally, the output of the temporal convolution layer is flattened and input into the dense layer to obtain a high-level spatiotemporal representation (feature vector) of the EEG, and a relatively stable SSVEP signal classification output result with a high signal-to-noise ratio is obtained;

[0077] Finally, an integrated learning strategy is adopted. When constructing the CNN model, different hyperparameters are used to build multiple models, and parallel training is performed on the same data set. When the SSVEP signal classification results are finally output, a "soft voting" method will be used, and the category label with the highest probability in the result is the result of this round of reasoning. This strategy enables the model to learn different types of signal features and obtain smaller variance when facing complex data sets.

[0078] In this embodiment, the electrooculogram (EOG) is an electrical activity caused by the change of potential caused by the movement of the eyeball or eyelid. The EOG waveform is recorded using the skin electrodes around the eyes. The EOG signal is used as a sign of the control switch, and its purpose is to switch the system's various control modes for multiple agents to adapt to different scene tasks and reduce eye fatigue caused by visual stimulation. The data acquisition of the electrooculogram (EOG) signal uses MindBridge. During the experiment, four electrodes are placed at the left and right corners of the eyes and the upper and lower eyelids of the participants to record the movement of the eyes. The participants are instructed to only watch the video at each SSVEP stimulation. Blinking is performed after each SSVEP stimulation block; therefore, blink detection is performed after each SSVEP stimulation block ends; in order to fully consider the interval length between blinking time and reaction time, the present invention designs the time interval between two consecutive blocks to be 2 seconds; since a blinking process takes about 0.2-0.6 seconds, then two consecutive blinking processes take 0.4-1.2 seconds, EOG is first processed by the reference electrode and low-pass filter, and is divided into 4-second non-overlapping time periods, and EOG features related to blinking are extracted from them, and then the EOG feature extraction model based on Transformer is used to identify and extract the EOG features;

[0079] The present invention uses MindBridge to collect data of electrooculogram (EOG) signals, and a series of operations of the EOG feature extractor based on Transformer. The designed EOG switch can freely and flexibly switch the control state of the system.

[0080] In this embodiment, the mapping process in the collaborative control strategy module is divided into the following three steps:

[0081] ①. EEG signals of different frequencies reflect different aspects of brain activity. This system can identify these frequency features in EEG signals, capture the user's thoughts and intentions, and classify EEG signals of different frequencies.

[0082] ②, based on the spatiotemporal convolution feature extractor to detect and classify SSVEP signals, and then based on the LSTM feature extractor to identify different EOG signals, the user's intention is determined according to different frequency components, and then the signals of different frequencies are mapped to the corresponding control commands;

[0083] ③. Map the EEG signal to the corresponding command ID and convert it into an executable operation;

[0084] At the same time, in terms of collaborative control strategy, the present invention also designs an SSVEP paradigm, including a three-level menu as shown in the figure below; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted functional commands are different; in addition, to facilitate collaborative control operations and ensure that the subjects can grasp the operation of the drone and the car at the same time, the present invention includes the transmitted images of the drone and the car during operation in the center of the GUI, providing the subjects with visual feedback of the movement of the machine and equipment, thereby enhancing their control experience.

[0085] In this embodiment, the PID control algorithm module includes three parts: proportional (P), integral (I), and differential (D): proportional control responds to error, integral eliminates steady-state error, and differential predicts future error changes; to simplify PID parameter adjustment, the AOA-HHO algorithm is used to optimize the PID parameters (Kp, Ki, Kd) to minimize the integral absolute error (IAE). The algorithm is divided into initialization and iteration stages, which respectively execute the exploration stage of AOA and the development stage of HHO, and finally find the optimal PID parameter configuration.

[0086] In this embodiment, the drone detection and tracking module uses the PP-YOLOE model for target detection, and improves the drone target detection performance by optimizing the Anchor-Free structure, Backbone & Neck, and Task Alignment Learning (TAL); the optimized PP-YOLOE system is used to detect and track the car, and the processing speed and accuracy are improved;

[0087] To deal with the situation where the drone loses the car, the drone detection and tracking module also includes an automatic tracking system based on boundary thresholds; when the car exceeds the drone's field of view, the drone will automatically adjust its speed and direction to keep the car in the center of the picture; to optimize the system response time, the MLP model is used to dynamically adjust the threshold boundary according to the car state, shorten the state mutation response time, and capture the key motion information of the car by reasonably dividing the time window, thereby improving the real-time performance of the system;

[0088] The drone detection and tracking module also has a special-shaped boundary recognition system, which enables the drone to adopt different tracking strategies in different areas, improving the robustness and flexibility of automatic tracking; using TOF sensors to measure the distance between the drone and the car, improving the accuracy and real-time performance of target tracking, and ensuring that the drone can accurately lock the position of the car and make timely adjustments.

[0089] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, characterized by: The system includes the following modules: The signal acquisition and processing module uses an 8-lead EEG cap to collect raw EEG data, selects the occipital area as the EEG data collection area, and collects the data through the device and summarizes it into a csvEEG data set; A feature extraction and classification module, which includes a feature extractor based on spatiotemporal convolution to extract features of SSVEP signals, an SSVEP detection algorithm to solve the problem of difficult target recognition in brain-computer interface system applications caused by factors such as low signal-to-noise ratio, non-stationarity, and strong individual differences of SSVEP signals, and an EOG for processing electrooculogram signals, and also includes a control instruction encoding output module; Stimulus paradigm module, used to provide the system with a practical, stable and secure human-computer interaction system interface; The collaborative control strategy module constitutes a command mapping mechanism in the system, which is used to map the user's EEG signals to the commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices can be achieved; The multi-device collaborative control module constitutes a command mapping mechanism and is used to map the user's EEG signals to the commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices can be achieved; The drone and car control algorithm module includes a PID control algorithm module and a drone detection and tracking module. The controller equipped with the PID control algorithm module is used for drone and car control, translating EEG signals into instructions, transmitting them to the control device through a mapping system, and finally realizing the action through the motor.

2. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1 is characterized in that: The signal acquisition and processing module has a built-in preprocessing module, and the processing steps of the preprocessing module are as follows: Step 1: Use a digital filter to filter the raw EEG data to remove low-frequency noise and baseline drift, usually between 0.1 Hz and 1 Hz, and remove high-frequency noise and high-frequency oscillations, usually between 30 Hz and 70 Hz; Step 2: Remove possible noise sources, including eye movement, muscle activity, and electrode noise. Use independent component analysis (ICA) technology to analyze the weight coefficients of different signals and decompose the mixed signal into non-independent components, thereby removing noise related to eye movement. Step 3: Through the IIR filter, the feedback generated by the response filter output forms multiple pulses, thereby removing the noise related to muscle activity and correcting or eliminating the noise introduced by poor electrode contact or drift.

3. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 2 is characterized in that: In the step 2, in view of the noise interference generated during the experiment, the EEG signal is denoised by combining wavelet transform with adaptive filtering, and the least mean square (LMS) adaptive filter is used to adjust the filter coefficient to adapt to the statistical characteristics of the input signal to achieve a specific signal processing goal; the multi-resolution analysis characteristics of the wavelet decompose the signal at different scales and decompose the mixed signal composed of various different frequencies intertwined into sub-signals of different frequency bands, and the signal is processed by frequency band; Wavelet transform is done through the following steps: first, the noisy signal is decomposed into different frequency bands at a certain scale according to the need, and then the frequency band where the noise is located is set to zero, or the frequency band where the useful signal is located is directly extracted for wavelet reconstruction; The least mean square (LMS) adaptive filtering denoising is done through the following steps: S1. Input signal Xn; S2. Generates output signal Yn after passing through a digital filter with adjustable parameters; S3. Compare the signal Yn with the reference signal Dn; S4. Form an error signal En.

4. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The operation steps of the SSVEP detection algorithm are as follows: First, for each EEG signal segment, GCN (graph convolutional neural network) is used to explore the dynamic and intrinsic connections between different EEG channels, that is, graph convolution operations are performed in the spatial dimension, and then a feature matrix of size N*Nf is obtained, where Nf is the number of graph convolution filters; each node in the matrix corresponds to an EEG channel, and the feature at each node is the DE feature vector extracted from the Ffb frequency band of the corresponding channel; Next, the temporal CNN is used to extract the temporal context information of the EEG samples, that is, the 2D-CNN is used to merge the feature matrices of different time periods, where the size of the convolution kernel is (Kt, 1), where Kt is the size of the convolution kernel in the time dimension; finally, the output of the temporal convolution layer is flattened and input into the dense layer to obtain a high-level spatiotemporal representation of the EEG, and obtain a relatively stable SSVEP signal classification output result with a high signal-to-noise ratio; Finally, an integrated learning strategy is adopted. When constructing the CNN model, different hyperparameters are used to build multiple models, and parallel training is performed on the same data set. When the SSVEP signal classification results are finally output, a "soft voting" method will be used, and the category label with the highest probability in the result is the result of this round of reasoning. This strategy enables the model to learn different types of signal features and obtain smaller variance when facing complex data sets.

5. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The EOG is first processed by the reference electrode and the low-pass filter and then divided into 4-second non-overlapping periods, from which EOG features related to blinking are extracted. Then, the EOG feature extraction model based on Transformer is used to identify and extract the EOG features. The model consists of six modules: input, complex spectrum representation, channel combination, encoder, multilayer perceptron, and output; Among them, the "complex spectrum representation module" converts the input EOG data into a complex spectrum representation and passes it to the channel combination module; The "channel combination module" uses multiple convolutional layers to perform weighted combination of multiple channels to obtain multiple weighted results and pass them into the encoder; The "encoder module" is an improvement on the traditional Transformer encoder. It is mainly composed of two identical and continuous sub-encoders. Each sub-encoder consists of two parts: a CNN module and a channel MLP module. In the CNN module, convolution kernels are used to extract features in the channel dimension and fuse the features of each module; while in the MLP module, linear transformation is used to extract features on each channel; The EOG features extracted above are passed into a multi-layer perceptron, which consists of two fully connected layers; this module first flattens and refines the input EOG feature data, and finally expresses the output as the probability distribution of each EOG category (single blink / double blink) through the softmax function, thereby realizing the output of the classification results of the EOG features.

6. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The mapping process in the collaborative control strategy module is divided into the following three steps: ①. EEG signals of different frequencies reflect different aspects of brain activity. This system can identify these frequency features in EEG signals, capture the user's thoughts and intentions, and classify EEG signals of different frequencies. ②, based on the spatiotemporal convolution feature extractor to detect and classify SSVEP signals, and then based on the LSTM feature extractor to identify different EOG signals, the user's intention is determined according to different frequency components, and then the signals of different frequencies are mapped to the corresponding control commands; ③. By mapping the EEG signal to the corresponding command ID, it is converted into an executable operation.

7. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The PID control algorithm module includes three parts: proportional (P), integral (I), and differential (D): proportional control responds to error, integral eliminates steady-state error, and differential predicts future error changes; to simplify PID parameter adjustment, the AOA-HHO algorithm is used to optimize the PID parameters (Kp, Ki, Kd) to minimize the integral absolute error (IAE). The algorithm is divided into initialization and iteration stages, which respectively execute the exploration stage of AOA and the development stage of HHO, and finally find the optimal PID parameter configuration.

8. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The drone detection and tracking module uses the PP-YOLOE model for target detection and improves drone target detection performance by optimizing the Anchor-Free structure, Backbone & Neck, and Task Alignment Learning (TAL); To deal with the situation where the drone loses the car, the drone detection and tracking module also includes an automatic tracking system based on boundary thresholds; when the car exceeds the boundary of the drone's field of view, the drone will automatically adjust its speed and direction to keep the car in the center of the picture; The drone detection and tracking module also has a special-shaped boundary recognition system, which enables the drone to adopt different tracking strategies in different areas, thereby improving the robustness and flexibility of automatic tracking.

Images