Brain-controlled unmanned aerial vehicle group method based on brain-computer deep collaborative fusion

Abstract

The invention provides a brain-controlled unmanned aerial vehicle group method based on brain-computer deep collaborative fusion. The method comprises the following steps: firstly, designing a steady-state visual evoked brain control signal acquisition normal form, and generating a visual stimulation signal through combined frequency-phase coding; secondly, the electroencephalogram signals are preprocessed through filtering, independent component analysis and the like; then, constructing an intention decoding network with deep fusion of space-time frequency features, extracting multi-scale features of the electroencephalogram signals by using a convolutional neural network, and performing deep fusion decoding through a space-time frequency Transform module to realize accurate prediction of brain control intention; thirdly, a virtual pilot algorithm and an artificial potential field algorithm are used for flight decision making of the unmanned aerial vehicle group; and finally, displaying the flight state of the unmanned aerial vehicle in real time through a visual interface constructed by PyQt5. The method has the advantages that the brain control intention of the user can be accurately decoded, the formation control and obstacle avoidance capability of the unmanned aerial vehicle group is effectively improved, and meanwhile, a visual interface and personalized setting are provided.

Description

Technical Field

[0001] The present invention provides a brain-controlled UAV swarm method based on deep brain-computer collaborative fusion. It provides a more flexible and robust control method for UAV formation control and belongs to the fields of pattern recognition and UAV control. Background Art

[0002] Brain-controlled UAVs, as a new type of brain-computer fusion control technology, are still in the early research stage. There are problems such as few UAV control commands, low decoding accuracy of control intentions, and insufficient cooperative control ability of UAV swarms, which hinder the application of brain-controlled UAV technology in actual scenarios. Since electroencephalogram (EEG) signals are very weak and easily interfered, it poses great challenges for multi-intention high-precision decoding based on EEG signals. In existing brain-controlled systems, in order to effectively decode intention commands, it is necessary to first design a suitable EEG experimental paradigm to encode control intentions. Currently, the mainstream EEG experimental paradigms include the motor imagery paradigm, the P300 paradigm, and the steady-state visual evoked potential paradigm, etc. There are significant differences in the number of command encodings and decoding accuracies among different experimental paradigms, making it difficult to directly apply them to the brain-computer cooperation control of UAV swarms. Therefore, researching EEG experimental paradigms for multi-intention brain-controlled UAVs is the basis for realizing a high-performance brain-controlled UAV system. In addition, the reliability of the brain-controlled UAV system depends on the high-precision decoding of EEG signal intentions. Existing intention decoding models are difficult to comprehensively extract high-level intention features of EEG signals, resulting in insufficient decoding accuracy of control commands and difficulty in ensuring the reliability of the UAV system. Therefore, researching a high-precision decoding method for EEG signal intentions based on deep learning technology is the key to realizing a brain-controlled UAV system. Compared with a single UAV control system, a multi-UAV formation has the characteristics of high efficiency and the ability to cooperate to complete complex tasks, and has great theoretical research value and application value. How to achieve the formation and safe switching of UAV formation, as well as formation collision avoidance, is one of the difficulties in UAV swarm control. Therefore, researching a multi-UAV formation control decision method with high flexibility and strong robustness is the key way to improve the multi-scenario application ability of brain-controlled UAV swarms. Summary of the Invention

[0003] Aiming at the problems existing in the existing brain-controlled UAV technology, the present invention combines the brain-computer interface technology with the cooperative operation of UAV swarms, and proposes an efficient, flexible and safe brain-controlled UAV swarm system, enabling operators to directly interact with and control UAV swarms through the brain-computer interface, and achieving more precise, rapid and free operations. This system overcomes the deficiencies of traditional UAV swarm control methods limited by manual operations and preset commands. Through the brain-computer interface technology, operators can directly transmit commands to UAV swarms, achieving more precise and rapid responses, improving the application scope and efficiency of UAV swarms, and promoting the application of UAV technology in various fields.

[0004] In view of the intelligent control requirements of UAV swarms in complex environments, and aiming at the problems such as complex remote control operations of current UAVs and difficult cluster control decisions, a method for brain-controlled UAV swarms based on deep brain-computer collaborative fusion is provided. Verification experiments were carried out on 10 subjects, and the brain control signals corresponding to 14 kinds of brain control commands of each subject were recorded. The collected signals were preprocessed and divided into signal segments, which were used as the input of the neural network to train the network model to accurately identify the brain control intentions of individual subjects, and at the same time, the results were compared with classical methods. The proposed method showed excellent control performance on 10 subjects.

[0005] To achieve the above objectives, the present invention provides a method for brain-controlled UAV swarms based on deep brain-computer collaborative fusion, including the following steps:

[0006] Step 1: Design of steady-state visual evoked brain control signal acquisition paradigm:

[0007] (1) Design a periodic sine wave visual stimulus paradigm using the combined frequency-phase encoding method.

[0008] (2) The modulation range of the steady-state visual evoked stimulus frequency increases from 8 Hz at intervals of 0.5 Hz to 14.5 Hz, and the phase increases sequentially from 0 at intervals of 0.5π.

[0009] (3) Design 14 kinds of brain control commands, including 10 kinds of motion direction controls: takeoff, landing, upward movement, downward movement, left movement, backward movement, left rotation, right rotation, and 6 kinds of formation control: rectangle, circle, triangle, heart shape.

[0010] (4) For each subject, the experiment includes six rounds, and each round includes 14 experiments, corresponding to all 14 brain control commands displayed in random order.

[0011] Step 2: Preprocessing of electroencephalogram signals:

[0012] (1) Select channels for electroencephalogram signals. According to the characteristics of visual evoked stimulus signals, select channels O1, O2, Oz, PO7, PO3, POz, PO4, PO8, P7, P3, Pz, P4, P8 for signal processing.

[0013] (2) Apply a low-pass filter of 0 - 40 Hz to the signal to remove high-frequency noise in the signal.

[0014] (3) Apply a notch filter to the signal to remove power frequency interference in the signal.

[0015] (4) Resample the signal at 256 Hz.

[0016] (5) Apply the signal maximization criterion - independent component analysis method to decompose the multi-channel EEG signals, and quantitatively discriminate and remove the artifact components.

[0017] (6) Divide the signals. Using a sliding window, divide the complete EEG signals into signal segments. The length of the sliding window is set to 2 seconds with zero overlap. Each signal segment contains 512 (2s × 256Hz) sampling points.

[0018] Step 3: Construct a brain-computer intention decoding network with deep fusion of spatio-temporal-frequency features:

[0019] (1) The multi-scale convolutional spatio-temporal feature extraction layer uses convolutional blocks with multiple different kernel sizes to extract the time-domain features of different scales of the EEG signals, and uses a multi-scale attention module to weight the time-domain features of different scales to generate adaptive weights, obtaining the multi-scale spatio-temporal features of the EEG signals.

[0020] (2) Use continuous wavelet transform to convert the EEG signal segments into multi-channel time-frequency images. The multi-view attention spectrum feature extraction module uses spatial convolutional blocks, frequency convolutional blocks, and time convolutional blocks to obtain the features of three views of the multi-channel time-frequency images, and obtains the multi-view spectrum features of the EEG signals through a deep convolutional module.

[0021] (3) Adopt a spatio-temporal-frequency Transformer module to deeply fuse the multi-scale spatio-temporal features and multi-view spectrum features of the EEG signals and remove redundant information.

[0022] (4) The classification layer predicts and classifies the fused features, maps the features to the corresponding brain-computer control commands, and realizes the decoding of brain-computer intention.

[0023] Step 4: Use the multi-formation formation control decision of the drone based on the virtual navigator for formation control:

[0024] (1) After the drone control host receives the brain-computer control command, it performs multi-drone formation control based on the virtual navigator algorithm.

[0025] (2) During the formation process, use the artificial potential field algorithm to achieve autonomous collision avoidance control within the formation.

[0026] Step 5: Real-time display of the brain-computer intention decoding result and the flight state of the drone:

[0027] Build a user-friendly visualization interface based on the PyQt5 package to display the results of brain-computer intention decoding and the flight state information of the drone in real time. Present the decoding results in text and graphical forms dynamically, and display the current operating parameters of the drone, such as speed, direction, altitude, etc., through a flight trajectory map. In addition, the interface embeds an interactive function module, including:

[0028] Status Monitoring Area: Updates the decoding results and flight status data in real time to help users accurately understand the current operating conditions of the system;

[0029] Personalized Setting Options: Users can customize the interface layout and display content according to their needs, including the display mode of decoding results, the update frequency of flight status data, etc., so as to optimize the usage experience.

[0030] The main advantages of the brain-controlled UAV swarm method based on deep brain-computer collaborative fusion provided by the present invention include:

[0031] 1. The present invention comprehensively considers the common commands for UAV formation control. By designing a steady-state visual evoked brain-computer signal acquisition paradigm and combining the preprocessing of multi-channel electroencephalogram signals and a deep spatio-temporal-frequency feature fusion decoding network, it can efficiently and accurately decode the user's brain control intention. This method can not only accurately identify 10 kinds of movement directions and 6 kinds of formation control commands, but also remove redundant information through a deep learning model, thereby improving the response speed and accuracy of the brain control system.

[0032] 2. The present invention adopts a virtual navigator algorithm and an artificial potential field algorithm, effectively solving the problems of formation control and obstacle avoidance of UAV swarms during mission execution. By directly controlling the formation transformation of multiple UAVs through brain control commands, the autonomous flight and formation adjustment of UAV swarms are realized, improving the operation efficiency and safety.

[0033] 3. The present invention designs a visualization interface based on PyQt5. The system can display the decoding results of brain control commands and the flight status of UAVs in real time, including important parameters such as speed, direction, altitude, etc., and provides interactive function modules (such as emergency stop, path adjustment, etc.), enhancing the operability and safety of the system. Users can customize the interface according to their personal needs, improving the personalized experience of operation. Brief Description of the Drawings

[0034] Figure 1 It is a flowchart of the brain-controlled UAV swarm method based on deep brain-computer collaborative fusion according to an embodiment of the present invention.

[0035] Figure 2 It is a basic framework diagram of the brain control intention decoding network with deep spatio-temporal-frequency feature fusion according to an embodiment of the present invention.

[0036] Figure 3 It is a basic framework diagram of the UAV multi-formation formation control decision algorithm based on a virtual navigator according to an embodiment of the present invention.

[0037] Figure 4 It is a basic framework diagram of the brain-controlled UAV demonstration system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention proposes a method for brain-controlled drone swarms based on deep brain-computer collaborative fusion. This method comprehensively considers a variety of motion and formation commands commonly used in drone formation control, and realizes precise control of brain-controlled commands through a steady-state visual evoked experimental paradigm. Among them, a brain-controlled intention decoding network with deep fusion of temporal, spatial and frequency features is designed, which effectively improves the accuracy of brain-controlled intention decoding. Combined with a virtual navigator and an artificial potential field algorithm, precise control of the drone formation is achieved. Finally, a user-friendly visual interface is constructed through the PyQt platform to ensure the real-time and convenience of system operation. The overall structure of the present invention is as follows Figure 1 shown.

[0039] The following is a detailed description of the specific implementation of the method for controlling a swarm of drones based on deep brain-computer collaboration proposed by the present invention in conjunction with the accompanying drawings:

[0040] Step 1: Design of steady-state visually induced brain control signal acquisition paradigm. A periodic sinusoidal wave visual stimulation paradigm was designed by combining the frequency-phase encoding method to induce steady-state visual responses of EEG signals. In this process, the stimulation frequency started from 8 Hz and gradually increased at intervals of 0.5 Hz to 14.5 Hz, while the stimulation phase started from 0 and gradually increased at intervals of 0.5π. At the same time, 14 brain control instructions were designed, including 10 movement direction controls (such as take-off, landing, up, down, left, back, left rotation, right rotation, etc.) and 6 formation form controls (such as rectangle, circle, triangle, heart shape, etc.). The experimental plan arranged six rounds for each subject, each round contained 14 tasks, and each task randomly displayed a brain control instruction, so as to comprehensively evaluate the execution effect of different instructions.

[0041] Step 2: EEG signal preprocessing. First, according to the characteristics of the visual evoked stimulation signal, the processing channels of the EEG signal are selected, including O1, O2, Oz, PO7, PO3, POz, PO4, PO8, P7, P3, Pz, P4 and P8 channels. Then, a 0-40Hz low-pass filter is applied to the selected signal to remove the high-frequency noise in the signal. Next, a notch filter is applied to remove the power frequency interference. The signal is resampled to 256Hz to ensure that the sampling frequency of the signal is suitable for subsequent processing. The independent component analysis method based on the signal maximization criterion is used to decompose the multi-channel EEG signal into components, and the artifact components are quantitatively identified and removed. Finally, a sliding window is used to divide the signal data. The window length is set to 2 seconds with zero overlap. The complete EEG signal is divided into multiple signal segments, each of which contains 512 sampling points (2 seconds × 256Hz).

[0042] Step 3: Construct a brain control intention decoding network with deep fusion of temporal, spatial and frequency features. The specific structure is as follows:Figure 2 as shown

[0043] (a) First, a multi-scale convolutional spatio-temporal feature extraction layer is constructed using multiple convolutional blocks with different kernel sizes on the time scale to extract the time-domain features of different scales of the original EEG signal. In one method embodiment, convolutional kernels of sizes T / 2, T / 4, T / 8, and T / 16 are used to extract the short-term local fluctuation information and long-term dependence relationships of the original EEG signal, where T represents the length of the EEG signal time segment. Subsequently, the dependence relationships between the time-domain features of different scales are modeled through multi-scale attention, and then a fully connected feed-forward network is used to perform a linear transformation on the multi-scale time-domain features to obtain the final multi-scale spatio-temporal feature M spa , where spa represents the spatio-temporal dimension of the EEG signal. The above process can be expressed as:

[0044] F conv = Concat(σ(Conv1D(x(t), W k ))) (1),

[0045] M spa = σ(F fc (MHSA(F conv ))) (2),

[0046] where

[0047] Conv1D(·) represents one-dimensional convolution on the time scale,

[0048] x(t) represents the original EEG signal,

[0049] W k represents convolutional kernels of different scales,

[0050] Concat(·) represents matrix concatenation operation,

[0051] F conv represents multi-scale time-domain features,

[0052] MHSA(·) represents multi-scale attention operation,

[0053] F fc represents a fully connected feed-forward network,

[0054] σ(·) represents the activation function.

[0055] (b) Then, four consecutive wavelet transforms are used to perform time-frequency transformation on the EEG signal segment to obtain the multi-channel time-frequency image of the EEG signal.

[0056] Specifically, in one embodiment according to the present invention, the Daubechies mother wavelet is used for transformation. Each wavelet transformation corresponds to different time-frequency resolutions and can capture the characteristics of the signal from different scales. The main advantage of using the continuous wavelet transformation lies in its time-frequency localization ability. The traditional Fourier transform assumes that the signal is stationary, that is, the frequency components of the signal do not change with time, while the electroencephalogram (EEG) signal is usually non-stationary and contains a variety of different frequency components. The continuous wavelet transformation can analyze the signal at multiple scales, providing both the distribution of the signal in time and reflecting the changes in its frequency components. Therefore, the continuous wavelet transformation can effectively capture the instantaneous changes and short-term high-frequency characteristics in the EEG signal, and can more comprehensively analyze the spatial and time information in the EEG signal, improving the accuracy of signal recognition and classification.

[0057] An operation of continuous wavelet transformation can be expressed as:

[0058]

[0059] Where:

[0060] x(t) represents a one-dimensional EEG signal,

[0061] ψ(·) represents the mother wavelet function,

[0062] a represents the scale parameter, which is used to control the width of the wavelet,

[0063] b is the position parameter, which is used to represent the translation of the wavelet in time,

[0064] W ψ (a, b) represents the result of the wavelet transformation of the EEG signal at scale a and position b.

[0065] A multi-view attention spectrum feature extraction module is constructed. This module extracts the spatio-frequency features, spatio-temporal features, and time-frequency features of the EEG signal from the multi-channel time-frequency image through a spatial convolution block, a frequency convolution block, and a time convolution block. The processes of extracting the above-mentioned spatio-frequency features, spatio-temporal features, and time-frequency features of the EEG signal can be respectively expressed as:

[0066] G1 = σ(Φ spa (U)Φ spe (U))(4),

[0067] G2 = σ(Φ spa (U)Φ tem (U))(5),

[0068] G3 = σ(Φ spe (U)Φ tem (U))(6), where U is the multi-channel time-frequency image of the EEG signal, Φspa (·), Φ spe (·) and Φ tem (·) represents the spatial convolution block, the frequency convolution block, and the temporal convolution block respectively, spa, spe, and tem represent the spatial dimension, the frequency dimension, and the temporal dimension of the EEG signal respectively, σ(·) represents the activation function, and G1, G2, and G3 represent the spatio-frequency features of the EEG signal, the spatio-temporal features of the EEG signal, and the time-frequency features of the EEG signal respectively. Then, through the deep convolution module, the spatio-frequency features of the EEG signal, the spatio-temporal features of the EEG signal, and the time-frequency features of the EEG signal are transformed into the multi-view spectrum features M of the EEG signal spe , and this operation can be expressed as:

[0069]

[0070] where U is the multi-channel time-frequency image of the EEG signal, ⊙ represents element-wise matrix multiplication, ∑(·) represents the summation operation, and Conv(·) represents the deep convolution layer.

[0071] (c) Through sub-steps (a) and (b), the multi-scale spatio-temporal features M of the EEG signal are obtained spa and the multi-view spectrum features M spe , to establish the internal dependence relationship between these two features, according to the present invention, combined with the spatio-temporal-frequency attention mechanism, through linear transformation, the multi-scale spatio-temporal features M of the EEG signal spa and the multi-view spectrum features M of the EEG signal spe are converted into different key matrices, query matrices, and value matrices. The above operation can be expressed as:

[0072]

[0073] Then, based on the attention mechanism, the dependence attention of the spatio-temporal features and the time-frequency features is calculated

[0074]

[0075] where

[0076] the subscript spa represents the spatio-temporal features

[0077] the subscript spe represents the spectrum features

[0078] and are the transformation matrices of the linear projection

[0079] Q, K, and V represent the key matrix, the query matrix, and the value matrix respectively

[0080] d represents the vector dimension

[0081] represents K spaTranspose of the matrix,

[0082] O sts Denote the dependence attention matrix of spatio-temporal features and time-frequency features,

[0083] The subscript sts represents the fusion of EEG signal feature dimensions.

[0084] After obtaining the dependence attention of spatio-temporal features and time-frequency features, for the spatio-temporal features M of EEG signals spa and spectral features M spe Perform transformation and fusion respectively according to the dependence attention matrix O sts The process of transformation and fusion can be expressed as:

[0085] M sts = Concat(O sts ×V spa + M spa , O sts ×V spe + M spe ) (11),

[0086] where,

[0087] Concat(·) represents the matrix concatenation operation,

[0088] M sts Represents the fused spatio-temporal - time-frequency features.

[0089] Obtain the fused spatio-temporal - time-frequency features M sts .

[0090] After that, further use the multi-head attention and depth fusion layer of Transformer to construct the time - space - frequency Transformer layer, further model the temporal dependence relationship of the fused features, and obtain the final fused features Denote the expected features, the operation of the time - space - frequency Transformer layer can be expressed as:

[0091] M′ sts = MHSA(LN(M sts )) + M sts (13),

[0092]

[0093] where MHSA(·) represents the multi-head attention layer, LN(·) represents the layer normalization operation, F fc represents the fully connected layer, M′ sts represents the intermediate features generated during the processing, GELU(·) Gaussian error linear unit activation function.

[0094] (d) Finally, a classifier is constructed by cascading multiple (preferably 3) fully connected layers to classify the final fusion features obtained in sub-step (c). Perform classification on it, and map the final fusion features to the corresponding brain-computer control commands, thereby realizing the decoding of the brain-computer control intention. To verify the brain-computer control intention decoding performance of the proposed model, the present invention conducts a performance comparison with a variety of classical electroencephalogram (EEG) feature extraction models on the EEG data of ten subjects. The specific results are as follows:

[0095] Table 1 Comparison of recognition accuracy results of various EEG feature extraction models

[0096]

[0097] In the experiment of this step, different feature extraction models were used to evaluate the performance of the EEG signal classification task. The evaluation metrics used in the method included accuracy (ACC), F1 value, and AUC value. The experimental results show that the spatio-temporal-frequency feature fusion model performs excellently in all evaluation metrics, significantly exceeding other comparison models. Specifically, the accuracy (ACC) of the spatio-temporal-frequency feature fusion model reaches 91.04%, which is the most prominent among all models and significantly better than other models. In terms of the F1 value, the F1 value of the spatio-temporal-frequency feature fusion model is 0.8957, much higher than that of the deep convolutional network model (0.7638) and the recurrent neural network model (0.7642), indicating that the model can effectively integrate precision and recall when dealing with unbalanced data and has stronger classification ability. In addition, while the spatio-temporal-frequency feature fusion model achieves the best performance in the three metrics of ACC, F1 value, and AUC, its decoding time is significantly lower than that of other comparison models, further proving that the model has higher decoding efficiency on the basis of ensuring high accuracy. Therefore, the designed spatio-temporal-frequency feature fusion model combines high performance and high efficiency, and can achieve accurate and efficient decoding of brain-computer control intentions.

[0098] Step 4: In this step, according to the method of an embodiment of the present invention, a personal computer is selected as the control host of the unmanned aerial vehicle (UAV). After receiving the brain-computer control command generated in step 3, the control command is sent to each UAV through WIFI communication. During the UAV formation control process, each UAV obtains its real-time position through image positioning, and a multi-UAV formation control algorithm based on a virtual navigator is used to control the formation shape of the multi-UAV. This algorithm accurately adjusts the position of the UAV according to the brain-computer control command to achieve the required formation shape. At the same time, during the formation process, the artificial potential field algorithm is combined for collision avoidance control within the formation. By calculating the relative position and speed between UAVs, this algorithm dynamically adjusts the movement trajectory of the UAVs, effectively avoiding collisions and ensuring the stability and flight safety of the formation. The specific structure is as Figure 3 shown, and the structures of each module of the algorithm are as follows:

[0099] (a) Multi-UAV Formation Control Algorithm Based on Virtual Navigator

[0100] This method first abstracts a UAV into a particle satisfying the second-order differential equation. Its control objective is to make the position and velocity states of all UAVs within the system reach consistency and form a formation. Suppose there are N UAVs with the same dynamic characteristics that jointly constitute a multi-UAV system. After modeling each UAV as a second-order integrator model, this system can be described as:

[0101]

[0102] where u i (t) is the control input of UAV i, that is, the target acceleration, q i (t), p i (t) are the position state and velocity state of UAV i respectively, denotes the first derivative of the corresponding variable.

[0103] Then, a distributed neighbor set communication topology structure is adopted to realize the communication between a single UAV and the surrounding UAVs, enabling the UAV to adjust parameters such as attitude and speed according to the real-time positions of the neighbor set UAVs, endowing each UAV with certain intelligence and decision-making ability. After introducing the states of the neighbor set UAVs, the control equation of a single UAV can be expressed as:

[0104]

[0105] where q j (t), p j (t) respectively represent the motion states of all UAVs j∈N within the communication range of UAV i j , σ i , σ j is the position offset of the UAV in the formation, and γ is a parameter affecting the convergence speed of the formation.

[0106] Finally, the virtual navigator model is designed by the UAV control host as the reference for the motion of all UAVs, transforming the formation control problem into a trajectory tracking problem, so that the control input of the i-th UAV is affected not only by the states of the UAVs within the neighbor set but also by the speed state of the virtual navigator. At this time, the control equation can be expressed as:

[0107]

[0108] where, represents the speed state of the virtual navigator, and k is a parameter affecting the overall speed of the formation.

[0109] (b) Collision Avoidance Strategy for UAV Swarm Based on Artificial Potential Field

[0110] Assume that there is a repulsive force field between UAVs and between UAVs and obstacles. When a UAV or an obstacle approaches, the repulsive force increases, causing the UAV to be pushed away. At the same time, assume that there is an attractive force field between UAVs, making the UAVs tend to approach each other to maintain a certain formation relationship. Finally, during the process of formation generation and formation switching, this method combines the repulsive force and the attractive force to form a potential energy field. Each UAV in the system adjusts its flight direction according to the gradient of the potential energy field under the action of the repulsive forces of other UAVs in other neighbor sets, avoiding collisions with other UAVs. Each UAV can reach a consistent state according to the formation algorithm. Finally, the control input of each UAV can be expressed as:

[0111]

[0112] where φ α (·) represents the artificial potential field force acting on the UAVs in the formation. The method uses a common exponential potential function as the artificial potential field function, and ||q i -q j || represents the distance between UAV i and UAV j.

[0113] Combined with the above control strategy, effective obstacle avoidance between UAVs is achieved while the UAVs are flying in formation.

[0114] Step 5. In this step, a user-friendly visualization interface is built based on the PyQt5 package to display the results of brain control intention decoding and the flight state information of the UAVs in real time. The decoding results are dynamically presented in text and graphical ways. At the same time, the flight trajectory map is used to display the current operating parameters of the UAVs, including speed, direction, altitude, etc. In addition, the interface also embeds multiple interactive function modules, including: a status monitoring area, which is used to update the decoding results and flight state data in real time to help users accurately grasp the current operating state of the system; personalized setting options, which allow users to customize the interface layout and display content according to their needs, including the display method of the decoding results, the update frequency of the flight state data, etc., to optimize the user experience. The specific interface is as Figure 4 shown.

Claims

1. A method for constructing a brain-computer intention decoding network with deep fusion of spatio-temporal frequency features for a brain-controlled UAV swarm method based on deep brain-computer collaborative fusion, characterized in that Including: A1) First, a multi-scale convolutional spatio-temporal feature extraction layer is constructed using multiple convolutional blocks with different kernel sizes on the time scale to extract time-domain features of different scales from the raw EEG signals. Specifically, the short-term local fluctuation information and long-term dependence relationships of the raw EEG signals are extracted using convolutional kernels of four different sizes, namely T / 2, T / 4, T / 8, and T / 16, where T represents the length of the EEG signal time segment. Then, the time-domain features of the four different scales are concatenated by matrix concatenation for subsequent operations. This operation is expressed as: F conv = Concat(σ(Conv1D(x(t),W k ))) (1), A2) Subsequently, multi-scale attention is used to model the dependence relationships between time-domain features of different scales: A3) Then, a fully connected feedforward network is used to perform a linear transformation on the multi-scale time-domain features to obtain the final multi-scale spatio-temporal feature M spa , where the subscript spa represents the spatio-temporal dimension of the EEG signal, and this process is expressed as: M spa = σ(F fc (MHSA(F conv ))) (2) Where: Conv1D(·) represents one-dimensional convolution on the time scale, x(t) represents the raw EEG signal, W k represent convolutional kernels of different scales k represents the size of the convolutional kernel, Concat(·) represents the matrix concatenation operation, F conv represents multi-scale time-domain features MHSA(·) represents the multi-scale attention operation, F fc represents a fully connected feedforward network σ(·) represents the activation function, B1) Meanwhile, the time-frequency transformation of the EEG signal segment is performed using four consecutive wavelet transforms to obtain the multi-channel time-frequency image of the EEG signal. Each wavelet transform corresponds to a different time-frequency resolution to capture the spectral features of the signal from different scales. One consecutive wavelet transform operation can be expressed as: Where: x(t) represents the one-dimensional EEG signal, ψ(·) represents the mother wavelet function, a represents the scale parameter used to control the width of the wavelet, and different a values are taken for each wavelet transform. b is the position parameter used to represent the translation of the wavelet in time. W ψ (a, b) represents the wavelet transform result of the EEG signal using the wavelet of scale a at position b. B2) Construct a multi-view attention spectrum feature extraction module, which extracts the spatio-frequency features, spatio-temporal features, and time-frequency features of the EEG signal from the multi-channel time-frequency image through a spatial convolution block, a frequency convolution block, and a time convolution block. The processes of extracting the spatio-frequency features, spatio-temporal features, and time-frequency features of the above EEG signals are respectively expressed as: G1 = σ(Φ spa (U)Φ spe (U))(4), G2 = σ(Φ spa (U)Φ tem (U))(5), G3 = σ(Φ spe (U)Φ tem (U))(6), where: U is the multi-channel time-frequency image of the EEG signal, Φ spa (·), Φ spe (·) and Φ tem (·) represent a spatial convolution block, a frequency convolution block, and a temporal convolution block respectively, spa, spe, and tem respectively represent the spatial dimension, frequency dimension, and time dimension of the EEG signal, σ(·) represents the activation function, G1, G2, and G3 respectively represent the spatio-frequency features, spatio-temporal features, and time-frequency features of the EEG signal, B3) Then, through the deep convolutional module, the spatio-frequency features, spatio-temporal features, and time-frequency features of the EEG signals are transformed into the multi-view spectral features M of the EEG signals spe , and this operation is expressed as: Where: U is the multi-channel time-frequency image of the EEG signal, ⊙ represents element-wise matrix multiplication, ∑(·) represents the summation operation, Conv(·) represents the depth convolutional layer, C1) To establish the dependency relationship between the multi-scale spatio-temporal features M of electroencephalogram (EEG) signals spa and the multi-view spectral features M of EEG signals spe Combining the spatio-temporal-frequency attention mechanism, through linear transformation, the multi-scale spatio-temporal features M of EEG signals spa and the multi-view spectral features M of EEG signals spe are converted into different key matrices, query matrices and value matrices, and this operation is expressed as: C2) Then, calculate the dependence attention matrix of the multi-scale spatio-temporal features M of the EEG signals and the multi-view spectral features M of the EEG signals based on the attention mechanism: spa and the multi-view spectral features M of the EEG signals spe : Where, The subscript spa represents spatio-temporal features, The subscript spe represents spectral features, and is the transformation matrix for linear projection, Q, K, and V respectively represent the key matrix, query matrix, and value matrix, d represents the vector dimension, Denote K spa the transpose of the matrix, O sts Dependent attention matrix representing the multi-scale spatio-temporal features and multi-view spectral features of electroencephalogram signals The subscript sts represents the fusion of EEG signal feature dimensions, C3) After obtaining the dependence attention matrix O of the multi-scale spatio-temporal features and multi-view spectral features of the EEG signals sts for the multi-scale spatio-temporal features M of the EEG signals spa and the multi-view spectral features M of the EEG signals spe perform transformation and fusion respectively according to the dependence attention matrix O sts The process of this transformation and fusion is expressed as: M sts = Concat(O sts × V spa + M spa , O sts × V spe + M spe ) (11), Where, Concat(·) represents the matrix concatenation operation, M sts represents the fused spatio-temporal and time-frequency features Obtain the fused spatio-temporal and time-frequency feature M sts , After (D), the spatio-temporal-frequency Transformer layer is further constructed using the multi-head attention and deep fusion layer of the Transformer to further model the temporal dependence of the fused features and obtain the final fused features Denoting the desired features, the operation of the spatio-temporal-frequency Transformer layer can be expressed as: M′ sts = MHSA(LN(M sts )) + M sts (13) Among them, MHSA(·) represents the multi-head attention layer, LN(·) represents the layer normalization operation, F fc represents the fully connected layer, M′ sts represents the intermediate features generated during the processing, and GELU(·) is the Gaussian error linear unit activation function. E) Finally, a classifier is constructed by cascading multiple fully connected layers to classify the final fused features and map the final fused features to the corresponding brain-computer control commands, thereby realizing the decoding of the brain-computer control intention.

2. The method for constructing a brain-computer intention decoding network according to claim 1, wherein: Step E includes: constructing a classifier by cascading three fully connected layers to classify the final fused features for classification.

3. The method for constructing a brain-computer intention decoding network according to claim 1, wherein: In step B1, performing the time-frequency transformation of the EEG signal segment using four consecutive wavelet transforms includes: using the Daubechies mother wavelet for the transformation.

4. A brain-controlled UAV swarm method based on deep brain-machine collaborative fusion, characterized in that Including: Step 1. Design of the steady-state visual evoked brain-computer signal acquisition paradigm, including: Designing a periodic sine wave visual stimulation paradigm using the joint frequency-phase encoding method, The steady-state visual evoked stimulus frequency modulation range increases from 8 Hz at intervals of 0.5 Hz to 14.5 Hz, and the phase increases sequentially from 0 at intervals of 0.5π. Design 14 brain-controlled commands, including 10 motion direction controls: takeoff, landing, upward movement, downward movement, leftward movement, backward movement, left rotation, right rotation, and 6 formation control types: rectangle, circle, triangle, heart shape. For each subject, the experiment includes six rounds, and each round contains 14 experiments, corresponding to all 14 brain-controlled commands displayed in random order. Step 2. Preprocessing of electroencephalogram (EEG) signals, including: Perform channel selection on the EEG signals. According to the characteristics of the visual evoked stimulus signals, select channels O1, O2, Oz, PO7, PO3, POz, PO4, PO8, P7, P3, Pz, P4, P8 for signal processing. Apply a low-pass filter of 0 - 40 Hz to the signals to remove high-frequency noise in the signals. Apply a notch filter to the signals to remove power frequency interference in the signals. Resample the signals at 256 Hz. Adopt the signal maximization criterion - independent component analysis method to decompose the multi-channel EEG signals into components, and quantitatively discriminate and remove the artifact components. Perform data partitioning on the signals. Using a sliding window, divide the complete EEG signals into signal segments. The length of the sliding window is set to 2 seconds with zero overlap. Each signal segment contains 512 (2s × 256 Hz) sampling points. Step 3. Execute the construction method of the brain-controlled intention decoding network according to one of claims 1 - 3, thereby realizing the decoding of brain-controlled intentions. Step 4. Use the multi-formation formation control decision of the unmanned aerial vehicle (UAV) based on a virtual navigator for formation control, including: After the UAV control host receives the brain-controlled command decoded from the brain-controlled intention, it performs multi-UAV formation control based on the virtual navigator algorithm. During the formation process, use the artificial potential field algorithm to achieve autonomous collision avoidance control within the formation. Step 5. Real-time display of the brain-controlled intention decoding result and the UAV flight state, including: Build a user-friendly visualization interface based on the PyQt5 package to display the results of brain-controlled intention decoding and the flight state information of the UAV in real time. Dynamically present the decoding results in text and graphical forms, and display the current operating parameters of the UAV, including speed, direction, and altitude, through a flight trajectory map. Adopt an interface embedded interactive function module including a status monitoring area to update the decoding results and flight state data in real time, helping the user accurately understand the current operating situation of the system. Adopt a button control module to provide a set of function buttons, allowing the user to manually intervene in the UAV flight at critical moments. Set personalized setting options, enabling the user to customize the interface layout and display content, including the decoding result display method and the update frequency of flight state data, according to needs, thereby optimizing the usage experience.

5. The brain-controlled UAV swarm method based on deep brain-machine collaborative fusion according to claim 4, characterized in that: In step 1, to control the experimental irrelevant variables as much as possible and avoid excessive individual differences, the experimental subjects were selected according to the male-female ratio of 1:1, aged 20-28 years old. For each subject, the experiment included six rounds, and each round contained 14 experiments, corresponding to all 14 brain control instructions displayed in random order. Each experiment started with a 0.5-second target prompt, and the experimental subject was required to shift their gaze to the target as soon as possible; then, all the buttons began to flash simultaneously on the screen at a pre-coded frequency and phase for 5 seconds; then, before the next experiment started, the screen was blank for 0.5 seconds, and the subject was required to avoid blinking during the 5-second stimulation time. There was a 2-minute break between two consecutive experimental rounds.

6. The method for controlling a brain-controlled drone swarm based on deep brain-machine collaborative fusion according to claim 4, characterized in that: In step 1, considering the paradigm complexity and the main form of drone formation control comprehensively, 14 brain control instructions were designed, including 10 kinds of motion direction controls: takeoff, landing, upward movement, downward movement, leftward movement, backward movement, left rotation, right rotation, and 6 kinds of formation form controls: rectangle, circle, triangle, heart shape.

7. The method for controlling a brain-controlled drone swarm based on deep brain-machine collaborative fusion according to claim 4, characterized in that: In step 2, since the brain regions related to steady-state visual evoked stimuli are visual regions, to reduce redundant information and lower the computational complexity, 13 electroencephalogram channels, namely O1, O2, Oz, PO7, PO3, POz, PO4, PO8, P7, P3, Pz, P4, P8, were selected for signal processing.

8. A computer-readable storage medium storing a computer-executable program, which can enable a processor to execute the method according to any one of claims 1-7.

Images