Multi-agent cooperative control system based on multi-modal brain-computer interface-visual tracking

Abstract

The application discloses a multi-agent cooperative control system based on a multi-modal brain-computer interface-visual tracking, which comprises the following modules: a signal acquisition and processing module, which adopts an 8-lead electroencephalogram cap to collect electroencephalogram original data, selects a part of an occipital region (PO7, PO8, P3, P4, Pz, O1, O2 and Oz) as an electroencephalogram data acquisition area, and collects EEG data; a feature extraction and classification module, which uses a space-time convolution network to extract SSVEP signal features, combines an integrated learning strategy to optimize classification accuracy, extracts EOG signals based on a Transformer model, integrates an LMS adaptive filter to combine a wavelet transform to denoise EEG signals, expands the frequency tracking range of the adaptive filter, and minimizes signal loss; a cooperative control strategy design module, which maps electroencephalogram signals into control instructions of different agents, and realizes cooperative control of unmanned aerial vehicles and trolleys; and a target detection and tracking module, which optimizes an unmanned aerial vehicle visual tracking system based on a PP-YOLOE model, and realizes accurate tracking and automatic adjustment of the trolley.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and specifically relates to a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking. Background Art

[0002] Brain-computer interface technology primarily collects EEG signals, processes and extracts features, analyzes and identifies the user's intent, and ultimately translates the EEG signals into specific control commands to control external devices, enabling interaction between the brain and the outside world. Common EEG-BCI systems include P300 potentials, motor imagery, and steady-state visual evoked potentials. Among these, the SSVEP-BCI system has become a key area of ​​BCI technology research due to its high precision, fast response, and ease of control. In 2020, Lei et al. proposed a new intelligent remote operation system based on SSVEP brain-computer interface for a mobile wall-climbing cleaning robot. The user can control the movement direction of the cleaning robot by looking at four flashing dots of different frequencies; in 2022, Guo et al. developed an SSVEP-BCI-controlled soft robotic glove system for functional rehabilitation after stroke. The user can control the two soft robotic gloves by looking at flashing dots of different frequencies marked left or right, thereby performing rehabilitation training, which has better effects than simple robotic glove rehabilitation; Zhang et al. proposed a speed modulation method for controlling the robotic arm in an SSVEP-based BCI, so that the flashing cursor and the robotic arm move synchronously, reducing the visual fatigue of the user from constantly switching the eye perspective between cursor stimulation and the robot; Chen et al. applied the Transformer model to SSVEP classification for the first time, and proposed an extended SSVEPformer based on filter bank technology. Its average classification accuracy and ITR are better than the other five commonly used baseline methods, optimizing the classification accuracy of SSVEP.

[0003] To improve the accuracy and flexibility of brain-computer interface technology, BCI systems often incorporate various auxiliary methods. EOG (Electrooculography) is a bioelectrophysiological technique used to measure and record the electrical activity of eye movements. EOG signals are relatively simple and immune to EEG noise, simplifying signal processing and improving the stability of BCI systems. Therefore, EOG is often used in conjunction with BCI technology for research.

[0004] Currently, SSVEP-based BCI systems still face several challenges in various applications: First, because SSVEP signals are easily affected by factors such as ambient light and noise, and the strength of SSVEP signals varies from user to user, the accuracy of SSVEP signal processing and differential measurement technology needs to be improved. Second, because SSVEP signals are often confused with periorbital motion signals, improving the accuracy of SSVEP signal classification is a major challenge. Furthermore, since EOG signals are easily affected by the environment and other muscle movements, and SSVEP and EOG signals easily interfere with each other, achieving accurate identification, classification, and feature extraction of the two signals is a major challenge facing SSVEP-EOG-BCI systems.

[0005] With the continuous deepening of research, BCI technology has also been more optimized and developed. Sadiq et al. proposed a new automation framework based on pre-trained convolutional neural networks (CNN) to optimize the performance of the BCI system; Abenna et al. improved the accuracy of multi-classification of EEG signals in the BCI system by combining the SCA algorithm and the LGBM classifier. Zhou et al. proposed a shared robotic arm control system based on a hybrid asynchronous brain-computer interface and computer vision, which realized the control of the robotic arm to complete precise tasks in three-dimensional space; the NOIR system proposed by Zhang et al. of Stanford University collaboratively integrated a robot learning algorithm based on a small sample strategy, which can be adapted according to the needs and behavioral habits of different users. It uses a modular idea to decode neural signals to obtain human intentions, improves the efficiency of BRI, and can efficiently complete up to 20 tasks. In the future, how to improve the robustness of BCI multimodal fusion, integrate big data and machine learning algorithms and control research will also become new challenges facing the development of BCI technology;

[0006] Currently, research on brain-controlled multi-agent systems still faces several pressing challenges: First, control commands between multiple agents are too decentralized, making collaborative control based on EEG signals inefficient. Second, the operating environments of multiple agents vary, making it difficult for different agents to adapt to the environment when operating under the same control mechanism, leading to problems such as slow collaborative system response times. Third, the real-time image transmission interface for brain-controlled multi-agent systems is limited, making it difficult to obtain information in real time. Therefore, achieving mutual coordination of decision-making among various machines and balancing the goals and needs of different machines based on BCI technology is a major research challenge.

[0007] Multi-target tracking is the core task of UAV visual tracking. UAV estimates the position, velocity and trajectory of the target by continuously taking images or videos and using tracking algorithms. There are two main strategies for multi-target tracking: Tracking by Detection (TBD) and Detection Free Tracking (DFT). Since DFT requires manual labeling, the tracking results are unstable, so the main strategy for multi-target tracking is TBD.

[0008] The main target detection algorithms of TBD are SORT and DeepSORT algorithms. SORT algorithm is proposed by Alex Bewley et al. Since the detection quality is a key factor affecting the tracking performance, the SORT algorithm can improve the tracking quality by up to 18.9%, which is 20 times faster than other trackers. DeepSORT algorithm is a multi-target tracking algorithm based on deep learning. It introduces a deep learning model, convolutional neural network, to extract appearance features and motion characteristics from detected targets, which can better handle target appearance changes and occlusion. SORT algorithm and DeepSORT algorithm both need to rely on real-time performance of target detection. YOLO detector is a single-stage target detector that combines feature extraction, regression, and classification in a convolutional network, simplifying the network. In real-time conditions, the average detection accuracy mAP reaches 63.4%, and the detection speed reaches 45 frame / s. Combining the advantages of Faster RCNN and YOLO, Liu et al. proposed SSD algorithm, which improved the running speed of Nvida Titan X to 59 frame / s, significantly better than YOLO, with mAP reaching 79.8% on VOC2007 dataset, better balancing detection accuracy and detection speed. In 2022, Sun Peng et al. proposed an improved SSD algorithm that outperformed the original SSD model by 3% in small object detection AP value, effectively solving the problem of small target detection in aerial photography. Ma et al. proposed a multi-UAV real-time tracking algorithm based on improved PaddlePaddle-YOLO (PP-YOLO) and simple online real-time tracking based on deep correlation measurement (Deep-SORT), to solve the problem of poor detection performance and unstable tracking of small targets by traditional target detection algorithms. Sun et al. proposed a lightweight target detector based on YOLO, with structure simplification and attention enhancement functions. Liu et al. designed a lightweight target detection algorithm through deep separable convolution and multiple efficient network structures, which maintains high detection accuracy while reducing model size. Xu et al. optimized PP-VOLOv2 and proposed PP-VOLPE model, which improved 1.9AP and 1.3AP compared with PP-VOLOv2 and VOLOX respectively in COCO test development, achieving higher performance target detection model.

[0009] Thanks to the continuous optimization and improvement of target detection algorithms, the accuracy of drone visual tracking has been greatly improved under ideal conditions. However, the real-time performance and precision of tracking still need to be improved, and there are still problems such as low resolution and poor quality of image transmission. While the fixed rectangular bounding box used in traditional target detection can simplify the detection task by pre-defining the target position, it lacks good generalization ability for the target's changes at different angles and cannot well adapt to the special situation of rapid changes in the vehicle's speed. Therefore, developing target detection and tracking algorithms that can adapt to various environmental changes is a major challenge in current drone visual tracking research. Summary of the Invention

[0010] The purpose of the present invention is to provide a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, so as to solve the problem that the general process of BCI technology proposed in the above background technology is EEG signal acquisition and processing, EEG signal decoding and recognition, and EEG command conversion and execution. Brain-controlled drones have been deeply studied and applied under various EEG controls such as single-modal and multi-modal brain-computer interfaces. Regarding the BCI system combining SSVEP and EOG, there are mainly problems such as insufficient accuracy in SSVEP signal and EOG signal recognition and insufficient accuracy in signal classification; current research on brain-controlled drones is still focused on the control of a single drone, and there is still little research on brain-controlled multiple drones and multiple agents. For the research on multi-agent collaborative control based on BCI technology, there are generally problems such as scattered instructions, a single control mechanism that is difficult to adapt to different operating environments, and weak generalization ability of collaborative control strategies; for target detection algorithms applied to drone visual tracking technology, current research has problems such as low resolution and poor quality of image transmission images, which make it difficult to find targets, and the real-time performance and accuracy of tracking still need to be optimized and improved. At the same time, the fixed rectangular bounding box used to predefine the target position in traditional target detection has problems such as weak generalization ability, and is difficult to adapt to special situations such as complex environments and rapid changes in target speed or angle.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, the system comprising the following modules:

[0012] The signal acquisition and processing module uses an 8-lead EEG cap to collect raw EEG data. The occipital region (PO7, PO8, P3, P4, Pz, O1, O2, Oz) is selected as the EEG data collection area with a sampling rate of 1000 Hz. Before the experiment, conductive gel is applied to the electrodes to ensure that the impedance is reduced to approximately 5 kΩ. The experimenter observes the flickering blocks on the screen to form EEG signals of the corresponding frequency. After being collected by the device, it is aggregated into a CSV EEG data set.

[0013] The feature extraction and classification module comprises a spatio-temporal convolution-based feature extractor extracting features of the SSVEP signal, an SSVEP detection algorithm solving the problem of target recognition difficulty in the brain-computer interface system application caused by low signal-to-noise ratio, non-stationary, strong individual difference of the SSVEP signal, and an EOG processing the electrooculogram, and further comprising a control instruction coding output module, for the SSVEP paradigm, comprising a three-level menu; the participant can flexibly switch between different menus through blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted function commands are different; in addition, in order to facilitate collaborative control operation and ensure that the subject can simultaneously master the operation of the unmanned aerial vehicle and the car, the application contains transmission images in the running process of the unmanned aerial vehicle and the car in the center of the GUI, providing visual feedback of the motion of the machine equipment for the subject, thereby enhancing their control experience.

[0014] The stimulation paradigm module is used to provide a human-computer interaction system interface with practicability, stability and safety for the system.

[0015] The collaborative control strategy module constitutes an instruction mapping mechanism in the system and is used to map the electroencephalogram of the user to the instructions for controlling the car and the unmanned aerial vehicle and the like, and through intelligent analysis and mapping of the electroencephalogram, adaptive control and collaborative work of different devices are realized.

[0016] The multi-device collaborative control module constitutes an instruction mapping mechanism and is used to map the electroencephalogram of the user to the instructions for controlling the car and the unmanned aerial vehicle and the like, and through intelligent analysis and mapping of the electroencephalogram, adaptive control and collaborative work of different devices are realized.

[0017] The unmanned aerial vehicle and car control algorithm module comprises a PID control algorithm module and an unmanned aerial vehicle detection and tracking module, wherein the controller carrying the PID control algorithm module is used for unmanned aerial vehicle and car control, converts the electroencephalogram into instructions, transmits the instructions to the control device through the mapping system, and finally realizes the action through the motor.

[0018] As a preferred technical solution in the application, the signal acquisition and processing module is internally provided with a preprocessing module, and the processing steps of the preprocessing module are as follows:

[0019] Step one: using a digital filter to filter the original EEG data, removing low-frequency noise and baseline drift, usually between 0.1Hz to 1Hz, removing high-frequency noise and high-frequency oscillation, usually between 30Hz to 70Hz;

[0020] Step 2: Remove possible noise sources, including eye movements, muscle activity, and electrode noise. Using independent component analysis (ICA) technology, we analyze the weight coefficients of different signals and decompose the mixed signal into independent components, thereby removing noise related to eye movements.

[0021] Step 3: Through the IIR filter, the feedback generated by the response filter output is formed into multiple pulses, thereby removing the noise related to muscle activity and correcting or eliminating the noise introduced by poor electrode contact or drift.

[0022] As a preferred technical solution of the present invention, in the step 2, a method combining wavelet transform and adaptive filtering (i.e., LMS adaptive filter) is used to denoise the EEG signal in response to noise interference generated during the experiment, and the least mean square (LMS) adaptive filter is used to adjust the filter coefficients to adapt to the statistical characteristics of the input signal to achieve a specific signal processing goal; the multi-resolution analysis characteristics of the wavelet decompose the signal at different scales with multiple resolutions, and decompose the mixed signal composed of various interwoven frequencies into sub-signals of different frequency bands, and process the signal by frequency band; the wavelet transform is carried out through the following steps: first, the noisy signal is decomposed into different frequency bands at a certain scale as needed, and then the frequency band where the noise is located is set to zero, or the frequency band where the useful signal is located is directly extracted and wavelet reconstruction is performed;

[0023] Least mean square (LMS) adaptive filtering denoising is done through the following steps:

[0024] S1. Input signal Xn;

[0025] S2 generates an output signal Yn after passing through a digital filter with adjustable parameters;

[0026] S3 compares the signal Yn with the reference signal Dn;

[0027] S4. Form an error signal En.

[0028] As a preferred technical solution of the present invention, the operating steps of the SSVEP detection algorithm are as follows:

[0029] First, for each EEG signal segment, a GCN (graph convolutional neural network) is used to explore the dynamic and intrinsic connections between different EEG channels. This involves performing graph convolution operations on the spatial dimension, and then obtaining a feature matrix of size N*Nf, where Nf is the number of graph convolution filters. Each node in the matrix corresponds to an EEG channel, and the feature at each node is the DE feature vector extracted from the Ffb frequency band of the corresponding channel.

[0030] Next, a temporal CNN is used to extract the temporal context of the EEG samples. This involves using a 2D-CNN to merge the feature matrices of different time periods, with a convolution kernel size of (Kt, 1), where Kt is the kernel size in the temporal dimension. Finally, the output of the temporal convolutional layer is flattened and fed into a dense layer to obtain a high-level spatiotemporal representation (feature vector) of the EEG, resulting in a relatively stable SSVEP signal classification output with a high signal-to-noise ratio.

[0031] Finally, an ensemble learning strategy was adopted. When constructing the CNN model, different hyperparameters were used to build multiple models, and parallel training was performed on the same dataset. When outputting the SSVEP signal classification results, a "soft voting" method was used, and the category label with the highest probability in the result was the result of this round of inference. This strategy enabled the model to learn different types of signal features and achieve smaller variance when faced with complex datasets.

[0032] As a preferred technical solution of the present invention, the EOG is first processed by a reference electrode and a low-pass filter, and then divided into 4-second non-overlapping periods, from which EOG features related to blinking are extracted. Then, a Transformer-based EOG feature extraction model is used to identify and extract the EOG features.

[0033] The model consists of six modules: input, complex spectrum representation, channel combination, encoder, multilayer perceptron and output;

[0034] Among them, the "complex spectrum representation module" converts the input EOG data into a complex spectrum representation and passes it to the channel combination module;

[0035] The "channel combination module" uses multiple convolutional layers to perform weighted combination of multiple channels to obtain multiple weighted results and pass them into the encoder;

[0036] The "encoder module" is an improvement on the traditional Transformer encoder. It mainly consists of two identical and consecutive sub-encoders. Each sub-encoder consists of two parts: a CNN module and a channel MLP module.

[0037] In the CNN module, convolution kernels are used to extract channel-dimensional features and fuse the features of each module; in the MLP module, linear transformation is used to extract features on each channel;

[0038] The extracted EOG features are passed into a multi-layer perceptron, which consists of two fully connected layers. This module first flattens and refines the input EOG feature data, and finally uses the softmax function to represent the output as a probability distribution of each EOG category (single blink / double blink), thereby outputting the classification results of the EOG features.

[0039] The present invention uses MindBridge to collect data of electrooculogram (EOG) signals. Based on a series of operations of the EOG feature extractor of Transformer, the designed EOG switch can freely and flexibly switch the control state of the system.

[0040] As a preferred technical solution in the present invention, the mapping process in the collaborative control strategy module is divided into the following three steps:

[0041] ① Based on the fact that EEG signals of different frequencies reflect different aspects of brain activity, this system captures the user's thoughts and intentions by identifying these frequency features in EEG signals, and achieves classification of EEG signals of different frequencies;

[0042] ② Based on the spatiotemporal convolutional feature extractor, the SSVEP signal is detected and classified, and then based on the LSTM feature extractor, different EOG signals are identified. The user's intention is determined based on different frequency components, and the signals of different frequencies are then mapped to corresponding control commands;

[0043] ③. By mapping the EEG signal to the corresponding command ID, it is converted into an executable operation;

[0044] At the same time, in terms of collaborative control strategy, the present invention also designed an SSVEP paradigm, including a three-level menu as shown in the figure below; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted functional commands are different; in addition, to facilitate collaborative control operations and ensure that the subjects can grasp the operation of the drone and the car at the same time, the present invention includes transmitted images of the drone and the car during operation in the center of the GUI, providing subjects with visual feedback of the movement of the machine equipment, thereby enhancing their control experience.

[0045] As a preferred technical solution in the present invention, the PID control algorithm module includes three parts: proportional (P), integral (I), and differential (D): proportional control responds to error, integral eliminates steady-state error, and differential predicts future error changes; to simplify PID parameter adjustment, the AOA-HHO algorithm is used to optimize the PID parameters (Kp, Ki, Kd) to minimize the integral absolute error (IAE). The algorithm is divided into initialization and iteration phases, executing the AOA exploration phase and the HHO development phase respectively, and ultimately finding the optimal PID parameter configuration.

[0046] As a preferred technical solution of the present invention, the drone detection and tracking module uses the PP-YOLOE model for target detection, and improves the drone target detection performance by optimizing the Anchor-Free structure, Backbone & Neck, and Task Alignment Learning (TAL). The optimized PP-YOLOE system is used to detect and track the car, with improved processing speed and accuracy.

[0047] To cope with situations where the drone loses the car, the drone detection and tracking module also includes an automatic tracking system based on boundary thresholds. When the car exceeds the drone's field of view, the drone automatically adjusts its speed and direction to keep the car in the center of the image. To optimize system response time, the MLP model dynamically adjusts the threshold boundary based on the car's state, shortening the response time to state mutations. By rationally dividing the time window to capture the car's key motion information, the system improves real-time performance.

[0048] The drone detection and tracking module also has an irregular boundary recognition system, which enables the drone to adopt different tracking strategies in different areas, improving the robustness and flexibility of automatic tracking; using a TOF sensor to measure the distance between the drone and the car, improving the accuracy and real-time performance of target tracking, ensuring that the drone can accurately lock the position of the car and make timely adjustments.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] Optimize the EEG signal preprocessing and classification and recognition scheme; based on the data preprocessing operation of traditional EEG signals, the LMS adaptive filter combined with the wavelet transform is used to denoise the EEG signal, which expands the frequency tracking range of the adaptive filter and minimizes signal loss; in the design of the multimodal brain-computer interface system, the present invention solves the problems of low signal-to-noise ratio and non-stationarity of SSVEP signals through a spatiotemporal convolution feature extractor, and uses an integrated learning strategy of "soft voting" to optimize the classification results; at the same time, for EOG signals, the present invention designs a feature extraction model of EOG signals based on the Transformer model, improves the traditional Transformer encoder, and integrates the CNN module and the channel MLP module to improve the accuracy of the extraction and classification of blink features;

[0051] Realize flexible switching of multi-agent control modes based on multimodal brain-computer interface; carry out innovative design for traditional brain-computer interface system, and in order to expand the multi-device collaborative control function, on the basis of traditional brain-computer interface system based on SSVEP, integrate EOG electrooculogram signal as the switch for the whole collaborative system to change the control mode, so as to achieve wider user participation and higher control accuracy. At the same time, considering that current academic research mainly focuses on the research field of single-device control, a multi-agent collaborative control system based on SSVEP+EOG three-level menu is designed, which organically integrates multimodal control signals with different control modes to enable multi-agents to switch between collaborative control modes and unmanned control modes. Both single-machine control mode and single-car control mode can issue accurate and reasonable control commands. When the drone and car are in different operating environments, they can adapt to different obstacles and achieve smooth and precise collaborative control of multiple devices. In addition, to optimize the collaborative control effect, the car is designed as the object to be detected and tracked. The drone provides visual tracking to identify the car's position and track it. A dual feedback interface is set up in the center of the GUI, which simultaneously presents the video streams captured by the drone and the car. This allows the control end to simultaneously grasp the real-time operation status of multiple devices and improve the overall performance of the system during the collaborative control process. Compared with the control mode of a single device, it has a wider range of uses in practical applications.

[0052] A cooperative control strategy for UAVs based on PP-YOLOE is designed. Based on the high requirements of the cooperative control system for target detection, a UAV visual tracking strategy based on PP-YOLOE is adopted, which integrates the high-performance and highly adaptable real-time target detection algorithm PP-YOLOE formed by optimized operations and applies it to the UAV system to achieve fast and accurate target recognition and tracking. In the cooperative control mode, the UAV can accurately detect the target car in real time based on this scheme. At the same time, when switching to the UAV single-control mode, the position of the target car can also be determined based on the boundary threshold set in the upper camera video stream, thereby achieving real-time autonomous tracking and improving the smoothness of cooperative control. This not only improves the autonomy and practicality of the UAV, but also provides an efficient solution for various tasks, which is of certain significance to the research in the field of UAV visual tracking.

[0053] A high degree of coordination among multiple agents in the collaborative control process is achieved; in response to the delay problem that may exist in the multi-agent collaborative control system and affect the performance of the collaborative system, a classification method based on the MLP model is introduced to dynamically adjust the boundary threshold in the drone video stream according to the current real-time state of the car, effectively alleviating the delay problem caused by the drone video stream return delay, target detection processing time and instruction transmission delay; this scheme significantly improves the response time of the collaborative control system, especially when the car speed changes sharply, by timely adjusting the threshold, the agent can quickly identify and respond to the change, thereby effectively improving the performance of the collaborative system; at the same time, in response to the problem that the common rectangular boundaries in the drone video stream have poor tracking performance when the drone and the car are facing different directions, the present invention also proposes a multi-level special-shaped recognition boundary configuration strategy, which configures different tracking strategies by judging that the tracking target is located in different areas to optimize the performance of the drone automatic tracking system, thereby improving the coordination of the drone and car collaborative control process and adapting to the multi-angle movement of the agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall system diagram of the present invention;

[0055] Figure 2 Schematic diagram of the spatiotemporal convolution module of the present invention;

[0056] Figure 3 Schematic diagram of the distribution of channels of the present invention.

[0057] Figure 4 This is the design diagram of the system control interface of the present invention DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] See also Figures 1 to 4 The present invention provides a technical solution: a multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking, which includes the following modules:

[0060] The signal acquisition and processing module uses an 8-lead EEG cap to collect raw EEG data. The occipital region (PO7, PO8, P3, P4, Pz, O1, O2, Oz) is selected as the EEG data collection area with a sampling rate of 1000 Hz. Before the experiment, conductive gel is applied to the electrodes to ensure that the impedance is reduced to approximately 5 kΩ. The experimenter observes the flickering blocks on the screen to form EEG signals of the corresponding frequency. After being collected by the device, it is aggregated into a CSV EEG data set.

[0061] A feature extraction and classification module includes a feature extractor based on spatiotemporal convolution to extract features of SSVEP signals, an SSVEP detection algorithm that solves the problem of difficult target recognition in brain-computer interface system applications caused by factors such as low signal-to-noise ratio, non-stationarity, and strong individual differences in SSVEP signals, an EOG that processes electrooculogram (EOG) signals, and a control instruction encoding and output module. For the SSVEP paradigm, a three-level menu is included; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same location, but the functional commands transmitted are different; in addition, to facilitate collaborative control operations and ensure that subjects can simultaneously grasp the operation of the drone and the car, the present invention includes transmitted images of the drone and the car during operation in the center of the GUI, providing subjects with visual feedback of the movement of the machine equipment, thereby enhancing their control experience;

[0062] The stimulation paradigm module is used to provide the system with a practical, stable and safe human-computer interaction system interface. The system interaction interface can be divided into three levels, namely the system startup interface, the system control interface and the system shutdown interface. The top layer of the system is the brain switch interface. When the user wants to use this system, he must first turn on the system switch because the system is locked by default and the drone and car are in the shutdown state. The existence of the brain switch can significantly reduce the false alarm rate of the system, especially for such applications that require robust control. Reducing or even eliminating the false trigger rate of the system is of great significance to safety and stability. After turning on the system, you can enter the system control interface. The system defaults to the collaborative control mode of the drone and the car. The user can select the instructions to be executed on the device in this control state in this interface (joint forward, joint backward, joint left turn or joint right turn). Each group of buttons is a control instruction. When the user wants to shut down the system, he needs to first trigger the button to enter the system shutdown interface on the system control interface, and then shut down the system after entering the system shutdown interface.

[0063] The system switch interface consists of a system start interface and a system shutdown interface. The label below the power button on the start interface is "Start", and the label below the shutdown interface is "Stop"; there are six groups of buttons, corresponding to the six numbers "1", "2", "3", "4", "5" and "6", of which there are five groups of pseudo-keys and only one group is the real power button, which is marked by a label below the power button; each group of buttons consists of a central square and rectangular buttons around it; among them, the buttons outside each group of buttons flash at a fixed frequency to modulate the SSVEP stimulation paradigm. This interface selects 6 frequencies, namely 7.5Hz, 9Hz, 10.5Hz, 12Hz, 13.5Hz and 15Hz; the flashing form of each group of buttons is the button background changing color, shape, etc. The shape changes from a diamond to a circle; the flashing duration is 100ms, and the time interval between two adjacent flashes is 200ms; thus, by looking at the button, an SSVEP potential can be generated; the designed system control interface; six groups of buttons, corresponding to the six letters "A", "B", "C", "D", "E" and "F", the flashing and changing frequency of the six groups of buttons are consistent with the system switch interface buttons; considering that the simultaneous flashing of the six visual stimulus blocks may cause mutual interference, the six stimulus blocks are arranged around the screen when designing the induced interface; and in order to enable the subjects to better coordinate the control of the drone and the car, and adapt to operations in different control modes, the real-time feedback of the drone and the car by the camera is displayed in the middle of the interface to achieve synchronous dual feedback;

[0064] The collaborative control strategy module constitutes a command mapping mechanism in the system, which is used to map the user's EEG signals to commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices are achieved. The command mapping mechanism mainly achieves the following three main goals: First, adaptive switching is achieved; this mechanism allows users to control different types of devices through EEG signals, and switch to control different devices in different scenarios without having to face the tedious device switching and reconfiguration process; the command mapping mechanism analyzes the user's EEG signals and intelligently maps them to the control commands of the corresponding devices, thereby achieving adaptive switching; second, collaborative control is achieved; this mechanism ensures that multiple devices can work together to complete more complex tasks, allowing users to control two devices at the same time to achieve more efficient collaborative actions; third, based on the experiment of collaborative control of drones and cars, this mechanism promotes support for multiple different types of devices to meet various application needs;

[0065] The multi-device collaborative control module constitutes a command mapping mechanism and is used to map the user's EEG signals to commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices are achieved. The command mapping mechanism mainly achieves the following three main goals: First, adaptive switching is achieved; this mechanism allows users to control different types of devices through EEG signals, and switch to control different devices in different scenarios without having to face the tedious device switching and reconfiguration process; the command mapping mechanism analyzes the user's EEG signals and intelligently maps them to the control commands of the corresponding devices, thereby achieving adaptive switching; second, collaborative control is achieved; this mechanism ensures that multiple devices can work together to complete more complex tasks, allowing users to control two devices at the same time to achieve more efficient collaborative actions; third, based on the experiment of collaborative control of drones and cars, this mechanism promotes support for multiple different types of devices to meet various application needs;

[0066] The drone and car control algorithm module includes a PID control algorithm module and a drone detection and tracking module. The controller equipped with the PID control algorithm module is used to control the drone and car, translates the EEG signals into instructions, transmits them to the control device through the mapping system, and finally realizes the action through the motor.

[0067] In this embodiment, since the EEG signals collected in the experiment are affected by various interference factors, including biological noise (such as heartbeat and eye movement), environmental noise, electrode noise, and baseline drift, the preprocessing steps generally include filtering to remove noise outside the frequency range, denoising to eliminate eye movement, muscle activity, and electrode noise, baseline correction to remove baseline drift, bad track processing, data resampling, data segment selection, data averaging to obtain the average EEG response, and finally checking and visualizing the data to ensure the quality and consistency of the signal. The signal acquisition and processing module has a built-in preprocessing module, and the processing steps of the preprocessing module are as follows:

[0068] Step 1: Filter the raw EEG data using a digital filter to remove low-frequency noise and baseline drift, typically between 0.1 Hz and 1 Hz, and high-frequency noise and oscillations, typically between 30 Hz and 70 Hz.

[0069] Step 2: Remove possible noise sources, including eye movements, muscle activity, and electrode noise. Using independent component analysis (ICA) technology, we analyze the weight coefficients of different signals and decompose the mixed signal into independent components, thereby removing noise related to eye movements.

[0070] Step 3: Through the IIR filter, the feedback generated by the response filter output is formed into multiple pulses, thereby removing the noise related to muscle activity and correcting or eliminating the noise introduced by poor electrode contact or drift.

[0071] In this embodiment, in step 2, a method combining wavelet transform and adaptive filtering (i.e., LMS adaptive filter) is used to denoise the EEG signal in response to noise interference generated during the experiment, and the least mean square (LMS) adaptive filter is used to adjust the filter coefficients to adapt to the statistical characteristics of the input signal to achieve a specific signal processing goal; the multi-resolution analysis characteristics of the wavelet decompose the signal at different scales and decompose the mixed signal composed of various interwoven frequencies into sub-signals of different frequency bands, and process the signal by frequency band; the wavelet transform is carried out through the following steps: first, the noisy signal is decomposed into different frequency bands at a certain scale as needed, and then the frequency band where the noise is located is set to zero, or the frequency band where the useful signal is located is directly extracted and wavelet reconstruction is performed;

[0072] When designing, it is not necessary to know the statistical characteristics of the input signal and noise in advance. It can gradually "understand" or estimate the required statistical characteristics during its working process, and automatically adjust its parameters based on this to achieve the best filtering effect. Once the statistical characteristics of the input signal change, it can track this change and automatically adjust the parameters to make the filter reach the best effect again.

[0073] In the experimental verification, the db5 wavelet is selected and decomposed into 5 layers. The power spectrum of the signal after denoising based on the above method becomes flatter (the flatter the power spectrum, the better the denoising effect), which proves that the method is effective and has great feasibility and flexibility.

[0074] In this embodiment, the operation steps of the SSVEP detection algorithm are as follows:

[0075] First, for each EEG signal segment, a GCN (graph convolutional neural network) is used to explore the dynamic and intrinsic connections between different EEG channels. This involves performing graph convolution operations on the spatial dimension, and then obtaining a feature matrix of size N*Nf, where Nf is the number of graph convolution filters. Each node in the matrix corresponds to an EEG channel, and the feature at each node is the DE feature vector extracted from the Ffb frequency band of the corresponding channel.

[0076] Next, a temporal CNN is used to extract the temporal context of the EEG samples. This involves using a 2D-CNN to merge the feature matrices of different time periods, with a convolution kernel size of (Kt, 1), where Kt is the kernel size in the temporal dimension. Finally, the output of the temporal convolutional layer is flattened and fed into a dense layer to obtain a high-level spatiotemporal representation (feature vector) of the EEG, resulting in a relatively stable SSVEP signal classification output with a high signal-to-noise ratio.

[0077] Finally, an ensemble learning strategy was adopted. When constructing the CNN model, different hyperparameters were used to build multiple models, and parallel training was performed on the same dataset. When outputting the SSVEP signal classification results, a "soft voting" method was used, and the category label with the highest probability in the result was the result of this round of inference. This strategy enabled the model to learn different types of signal features and achieve smaller variance when faced with complex datasets.

[0078] In this embodiment, the electrooculogram (EOG) is an electrical activity caused by the potential change caused by the movement of the eyeball or eyelid. The EOG waveform is recorded using the skin electrodes around the eye. The EOG signal serves as a sign of the control switch, and its purpose is to switch the system's various control modes for multiple agents to adapt to different scene tasks, while reducing eye fatigue caused by visual stimulation. The data acquisition of the electrooculogram (EOG) signal uses MindBridge. During the experiment, four electrodes are placed at the left and right corners of the eyes and the upper and lower eyelids of the participants to record eye movements. The participants are instructed to only observe the eye movements during each SSVEP stimulation. Blinking is performed after each SSVEP stimulation block; therefore, blink detection is performed after the end of each SSVEP stimulation block; in order to fully consider the interval length between blink time and reaction time, the present invention designs the time interval between two consecutive blocks to be 2 seconds; since the process of one blink takes about 0.2-0.6 seconds, then the process of two consecutive blinks takes 0.4-1.2 seconds. EOG is first processed by the reference electrode and low-pass filter and divided into 4-second non-overlapping periods, from which EOG features related to blinking are extracted. Then, the EOG feature extraction model based on Transformer is used to identify and extract the EOG features;

[0079] The present invention uses MindBridge to collect data of electrooculogram (EOG) signals. Based on a series of operations of the EOG feature extractor of Transformer, the designed EOG switch can freely and flexibly switch the control state of the system.

[0080] In this embodiment, the mapping process in the collaborative control strategy module is divided into the following three steps:

[0081] ① Based on the fact that EEG signals of different frequencies reflect different aspects of brain activity, this system captures the user's thoughts and intentions by identifying these frequency features in EEG signals, and achieves classification of EEG signals of different frequencies;

[0082] ② Based on the spatiotemporal convolutional feature extractor, the SSVEP signal is detected and classified, and then based on the LSTM feature extractor, different EOG signals are identified. The user's intention is determined based on different frequency components, and the signals of different frequencies are then mapped to corresponding control commands;

[0083] ③. By mapping the EEG signal to the corresponding command ID, it is converted into an executable operation;

[0084] At the same time, in terms of collaborative control strategy, the present invention also designed an SSVEP paradigm, including a three-level menu as shown in the figure below; participants can flexibly switch between different menus by blinking; the stimulation frequency of each menu is consistent at the same position, but the transmitted functional commands are different; in addition, to facilitate collaborative control operations and ensure that the subjects can grasp the operation of the drone and the car at the same time, the present invention includes transmitted images of the drone and the car during operation in the center of the GUI, providing subjects with visual feedback of the movement of the machine equipment, thereby enhancing their control experience.

[0085] In this embodiment, the PID control algorithm module includes three components: proportional (P), integral (I), and differential (D). Proportional control responds to error, integral eliminates steady-state error, and differential predicts future error changes. To simplify PID parameter adjustment, the AOA-HHO algorithm is used to optimize the PID parameters (Kp, Ki, Kd) to minimize the integral absolute error (IAE). The algorithm is divided into initialization and iteration phases, respectively executing the AOA exploration phase and the HHO development phase, ultimately finding the optimal PID parameter configuration.

[0086] In this embodiment, the drone detection and tracking module uses the PP-YOLOE model for target detection. The drone target detection performance is improved by optimizing the Anchor-Free structure, Backbone & Neck, and Task Alignment Learning (TAL). The optimized PP-YOLOE system is used to detect and track the car, with improved processing speed and accuracy.

[0087] To cope with situations where the drone loses the car, the drone detection and tracking module also includes an automatic tracking system based on boundary thresholds. When the car exceeds the drone's field of view, the drone automatically adjusts its speed and direction to keep the car in the center of the image. To optimize system response time, the MLP model dynamically adjusts the threshold boundary based on the car's state, shortening the response time to state mutations. By rationally dividing the time window to capture the car's key motion information, the system improves real-time performance.

[0088] The drone detection and tracking module also has an irregular boundary recognition system, which enables the drone to adopt different tracking strategies in different areas, improving the robustness and flexibility of automatic tracking; using a TOF sensor to measure the distance between the drone and the car, improving the accuracy and real-time performance of target tracking, ensuring that the drone can accurately lock the position of the car and make timely adjustments.

[0089] Although embodiments of the present invention have been shown and described (see the detailed description above for details), it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent collaborative control system based on multimodal brain-computer interface and visual tracking, characterized by: The system includes the following modules: The signal acquisition and processing module uses an 8-lead EEG cap to collect raw EEG data. The occipital region is selected as the EEG data collection area, and the data is collected by the device and aggregated into a csvEEG data set. A feature extraction and classification module, comprising a feature extractor based on spatiotemporal convolution to extract features of the SSVEP signal, an EOG to process the electrooculogram signal, and a control instruction encoding and output module; Stimulus Paradigm Module; The collaborative control strategy module forms a command mapping mechanism in the system, which is used to map the user's EEG signals to commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices are achieved; The multi-device collaborative control module forms a command mapping mechanism and is used to map the user's EEG signals to commands for controlling devices such as cars and drones. Through intelligent analysis and mapping of EEG signals, adaptive control and collaborative work of different devices are achieved; The drone and car control algorithm module includes a PID control algorithm module and a drone detection and tracking module. The controller equipped with the PID control algorithm module is used to control the drone and car, translates EEG signals into commands, transmits them to the control device through a mapping system, and ultimately realizes the action through the motor; The EOG is first processed by a reference electrode and a low-pass filter, and then divided into 4-second non-overlapping periods, from which EOG features related to blinking are extracted. Then, a Transformer-based EOG feature extraction model is used to identify and extract the EOG features. The model consists of six modules: input, complex spectrum representation, channel combination, encoder, multilayer perceptron and output; Among them, the "complex spectrum representation module" converts the input EOG data into a complex spectrum representation and passes it to the channel combination module; The "channel combination module" uses multiple convolutional layers to perform weighted combination of multiple channels to obtain multiple weighted results and pass them into the encoder; The "encoder module" is an improvement on the traditional Transformer encoder. It mainly consists of two identical and consecutive sub-encoders. Each sub-encoder consists of two parts: a CNN module and a channel MLP module. In the CNN module, convolution kernels are used to extract channel-dimensional features and fuse the features of each module; in the MLP module, linear transformation is used to extract features on each channel; The extracted EOG features are passed into a multi-layer perceptron, which consists of two fully connected layers. This module first flattens and refines the input EOG feature data, and finally uses the softmax function to represent the output as the probability distribution of each EOG category, thereby outputting the classification results of the EOG features.

2. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1 is characterized in that: The signal acquisition and processing module has a built-in pre-processing module, and the processing steps of the pre-processing module are as follows: Step 1: Filter the raw EEG data using a digital filter to remove low-frequency noise and baseline drift, typically between 0.1 Hz and 1 Hz, and high-frequency noise and oscillations, typically between 30 Hz and 70 Hz. Step 2: Remove possible noise sources, including eye movements, muscle activity, and electrode noise. Using independent component analysis, we analyze the weight coefficients of different signals and decompose the mixed signal into independent components, thereby removing noise related to eye movements. Step 3: Through the IIR filter, the feedback generated by the response filter output is formed into multiple pulses, thereby removing the noise related to muscle activity and correcting or eliminating the noise introduced by poor electrode contact or drift.

3. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 2, characterized in that: In step 2, the EEG signal is denoised by combining wavelet transform and adaptive filtering to address the noise interference generated during the experiment. The least mean square adaptive filter is used to adjust the filter coefficients to adapt to the statistical characteristics of the input signal to achieve a specific signal processing goal. The multi-resolution analysis characteristics of the wavelet decompose the signal at different scales and decompose the mixed signal composed of various different frequencies into sub-signals in different frequency bands, and the signal is processed by frequency band. Wavelet transform uses the following steps: first, the noisy signal is decomposed into different frequency bands at a certain scale according to the needs, and then the frequency band where the noise is located is set to zero, or the frequency band where the useful signal is located is directly extracted for wavelet reconstruction; The least mean square adaptive filtering denoising is done through the following steps: S1. Input signal Xn; S2 generates an output signal Yn after passing through a digital filter with adjustable parameters; S3 compares the signal Yn with the reference signal Dn; S4. Form an error signal En.

4. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The operating steps of the SSVEP detection algorithm are as follows: First, for each EEG signal segment, GCN is used to explore the dynamic and intrinsic connections between different EEG channels, that is, to perform graph convolution operations on the spatial dimension, and then obtain a feature matrix of size N*Nf, where Nf is the number of graph convolution filters; each node in the matrix corresponds to an EEG channel, and the feature at each node is the DE feature vector extracted from the Ffb frequency band of the corresponding channel; Next, a temporal CNN is used to extract the temporal context of the EEG samples. This involves using a 2D-CNN to merge the feature matrices of different time periods, with a convolution kernel size of (Kt, 1), where Kt is the kernel size in the temporal dimension. Finally, the output of the temporal convolutional layer is flattened and fed into a dense layer to obtain a high-level spatiotemporal representation of the EEG, resulting in a relatively stable SSVEP signal classification output with a high signal-to-noise ratio. Finally, an ensemble learning strategy was adopted. When constructing the CNN model, different hyperparameters were used to build multiple models, and parallel training was performed on the same dataset. When outputting the SSVEP signal classification results, a "soft voting" method was used, and the category label with the highest probability in the result was the result of this round of inference. This strategy enabled the model to learn different types of signal features and achieve smaller variance when faced with complex datasets.

5. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The mapping process of the multi-device collaborative control module based on the command interface is divided into the following three steps: ① Based on the fact that EEG signals of different frequencies reflect different aspects of brain activity, this system captures the user's thoughts and intentions by identifying these frequency features in EEG signals, and achieves classification of EEG signals of different frequencies; ② Based on the spatiotemporal convolutional feature extractor, the SSVEP signal is detected and classified, and then based on the LSTM feature extractor, different EOG signals are identified. The user's intention is determined based on different frequency components, and the signals of different frequencies are then mapped to corresponding control commands; ③. By mapping the EEG signal to the corresponding command ID, it is converted into an executable operation.

6. The multi-agent collaborative control system based on multimodal brain-computer interface-visual tracking according to claim 1, characterized in that: The PID control algorithm module includes three parts: proportional, integral, and differential. Proportional control responds to error, integral eliminates steady-state error, and differential predicts future error changes. To simplify PID parameter adjustment, the AOA-HHO algorithm is used to optimize the PID parameters (Kp, Ki, Kd) to minimize the absolute integral error. The algorithm is divided into initialization and iteration phases, respectively executing the AOA exploration phase and the HHO development phase, ultimately finding the optimal PID parameter configuration.

7. The multi-agent collaborative control system based on multimodal brain-computer interface and visual tracking according to claim 1, characterized in that: The drone detection and tracking module uses the PP-YOLOE model for target detection and improves drone target detection performance by optimizing the Anchor-Free structure, Backbone & Neck, and Task Alignment Learning. To handle situations where the drone loses sight of the car, the drone detection and tracking module also includes an automatic tracking system based on boundary thresholds. When the car exceeds the drone's field of view, the drone automatically adjusts its speed and direction to keep the car in the center of the frame. The drone detection and tracking module also has a special-shaped boundary recognition system, which enables the drone to adopt different tracking strategies in different areas, improving the robustness and flexibility of automatic tracking.

Images