Unmanned aerial vehicle control method and system based on EEG and fNIRS multi-modal feature fusion

Abstract

The present application relates to the technical field of unmanned aerial vehicle control, and particularly relates to an unmanned aerial vehicle control method and system based on EEG and fNIRS multi-modal feature fusion, comprising: synchronously acquiring scalp EEG and near-infrared blood oxygen signals, extracting EEG frequency band power spectrum density to construct a fast variable feature matrix, extracting oxygenated hemoglobin concentration change time series features to construct a slow variable feature matrix, and using a phase space reconstruction algorithm to compensate for the slow variable timestamp to achieve alignment; then taking the aligned slow variable as a query matrix, the fast variable as a key matrix and a value matrix, and obtaining a fusion feature matrix through attention mechanism fusion; inputting the fusion feature matrix into a double-branch network to respectively decode discrete instructions and continuous variables, splicing to generate an intention decoding instruction; finally, integrating the concentration change amount to obtain a cognitive load index, and dynamically allocating the weights of the intention decoding instruction and the bottom layer autonomous safety control law according to the cognitive load index to generate a final instruction for execution. The present application solves the problems of low control dimension and easy loss of control and crash.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a UAV control method and system based on the fusion of EEG and fNIRS multimodal features. Background Technology

[0002] In recent years, brain-computer interface (BCI) technology has been increasingly applied to the field of drone control, providing a non-contact control method for special operational scenarios. However, existing drone brain-control technologies still have shortcomings when dealing with complex and dynamic flight missions. Traditional drone brain-control links mostly use single EEG signal acquisition, which has poor noise resistance and difficulty in extracting spatial positioning information. Some improved solutions attempt to directly and physically splice EEG and blood oxygenation signals, but due to the inherent spatiotemporal misalignment between EEG signals (millisecond response) and blood oxygenation signals (second-level delay) at the physical level, simple splicing can lead to mutual interference of features. In addition, existing systems are mostly open-loop control logic, lacking quantitative monitoring and active defense mechanisms for the operator's deep cognitive fatigue state. These shortcomings collectively result in drones lacking the ability to finely adjust continuous variables in three-dimensional space. Furthermore, under the operator's mental overload, errors in feature parsing can easily lead to the forced execution of incorrect brain-control commands, ultimately causing flight trajectory errors and loss of control crashes. To address the aforementioned issues, existing technologies have failed to systematically solve the core challenges of effectively integrating spatiotemporal heterogeneous features, synchronously decoding multidimensional control commands, and closed-loop monitoring of operator physiological states. Summary of the Invention

[0003] To overcome the above shortcomings, this invention provides a UAV control method and system based on EEG and fNIRS multimodal feature fusion, aiming to improve the problems of low control dimension and easy loss of control and crash caused by the spatiotemporal misalignment of multimodal features, limited control dimension, and lack of closed-loop monitoring of physiological state in the prior art.

[0004] In a first aspect, the present invention provides the following technical solution: a UAV control method based on EEG and fNIRS multimodal feature fusion, comprising the following steps:

[0005] S1. Simultaneously acquire scalp electroencephalogram (EEG) signals and near-infrared blood oxygenation signals, and convert the near-infrared blood oxygenation signals into changes in oxyhemoglobin concentration.

[0006] S2. Extract the frequency band power spectral density of the scalp EEG signal to construct a fast variable feature matrix, and extract the temporal features of the oxyhemoglobin concentration change to construct a slow variable feature matrix; use the phase space reconstruction algorithm to compensate the timestamp of the slow variable feature matrix to achieve time alignment with the fast variable feature matrix;

[0007] S3. Map the aligned slow variable feature matrix to a query matrix, and map the aligned fast variable feature matrix to a key matrix and a value matrix. Calculate the query matrix, the key matrix, and the value matrix using an attention mechanism to obtain the fused feature matrix.

[0008] S4. Input the fused feature matrix into a dual-branch network. The first network branch decodes discrete instructions, and the second network branch decodes continuous variables. The discrete instructions and the continuous variables are then concatenated to generate an intent decoding instruction.

[0009] S5. Integrate the change in oxyhemoglobin concentration to obtain a cognitive load index, use the cognitive load index to allocate the execution weights of the intent decoding command and the underlying autonomous safety control law, and sum them to generate the final flight control command for issuance and execution.

[0010] By adopting the above technical solution, the phase space reconstruction aligns the fast and slow features and deeply integrates them through the attention mechanism, thereby decoding multi-dimensional intentions in a dual-branch manner and adaptively allocating weights based on the load, thus improving the problems of low control dimension and easy loss of control and crashes in existing technologies.

[0011] Optionally, in S1, converting the near-infrared blood oxygenation signal into a change in oxyhemoglobin concentration includes:

[0012] The changes in light attenuation after the first and second wavelengths of near-infrared light emitted by the near-infrared detection device penetrated the operator's brain tissue were obtained.

[0013] Extract the physical distance parameters between the light source and the detector, as well as the differential path factor parameters, of the near-infrared detection device;

[0014] The extinction coefficients of deoxyhemoglobin and oxyhemoglobin corresponding to the first wavelength near-infrared light and the second wavelength near-infrared light are retrieved respectively.

[0015] By combining the physical distance parameter, the differential path factor parameter, and the change in light attenuation, and by simultaneously solving the extinction coefficients of deoxyhemoglobin and oxyhemoglobin, the change in oxyhemoglobin concentration is extracted.

[0016] Optionally, in S2, extracting the frequency band power spectral density of the scalp EEG signal to construct a fast variable feature matrix, and extracting the temporal features of the change in oxyhemoglobin concentration to construct a slow variable feature matrix, includes:

[0017] The scalp EEG signal is subjected to bandpass filtering. Short-time Fourier transform is applied to the filtered scalp EEG signal to extract the frequency band power spectral density within a set frequency range. The frequency band power spectral density is then vector-concatenated according to the time dimension to construct the fast variable feature matrix.

[0018] The length of the sliding time window and the sliding step size for cutting the sequence of changes in oxyhemoglobin concentration are set. Within the sliding time window, the average concentration amplitude feature and the concentration change slope feature of the changes in oxyhemoglobin concentration are calculated respectively. The average concentration amplitude feature and the concentration change slope feature are combined to construct the slow variable feature matrix.

[0019] Optionally, in S2, compensating for the timestamps of the slow variable feature matrix using a phase space reconstruction algorithm to achieve time alignment with the fast variable feature matrix includes:

[0020] Determine the change in oxyhemoglobin concentration relative to the inherent hemodynamic delay time of the scalp electroencephalogram signal;

[0021] The embedding dimension and delay time parameter of the phase space reconstruction algorithm are set, and the slow variable feature matrix is ​​mapped to a high-dimensional phase space;

[0022] The phase shift compensation is calculated based on the hemodynamic delay time within the high-dimensional phase space;

[0023] The phase shift compensation is applied in reverse to the timestamp of each sampling point of the slow variable feature matrix to obtain the compensated slow variable feature matrix, and the timestamp of the compensated slow variable feature matrix is ​​strictly aligned with the timestamp of the fast variable feature matrix.

[0024] Optionally, in S3, the step of calculating the fused feature matrix by means of the attention mechanism, the query matrix, the key matrix, and the value matrix includes:

[0025] Extract the pre-trained first weight matrix, second weight matrix, and third weight matrix;

[0026] The query matrix is ​​obtained by multiplying the aligned slow variable feature matrix by the first weight matrix;

[0027] The aligned fast variable feature matrix is ​​multiplied by the second weight matrix and the third weight matrix respectively to obtain the key matrix and the value matrix;

[0028] Calculate the dot product of the query matrix and the transpose of the key matrix, divide the dot product by the dimension scaling factor of the feature vector, and normalize it using the Softmax function to obtain the cross-modal attention weight distribution matrix.

[0029] Perform matrix multiplication on the cross-modal attention weight distribution matrix and the value matrix to output the fusion feature matrix containing multimodal complementary information.

[0030] Optionally, in S4, the first network branch decodes discrete instructions, and the second network branch decodes continuous variables, including:

[0031] The fused feature matrix is ​​input into the hidden layer of the first network branch to extract the classification feature vector. The classification feature vector is discretized and mapped using the Softmax classification layer to output the classification probability of the UAV take-off and landing state and the classification probability of the horizontal yaw direction. The item with the highest probability is selected as the discrete command.

[0032] The fused feature matrix is ​​synchronously input into the hidden layer of the second network branch to extract regression feature vectors. The regression feature vectors are fitted using a linear regression layer to output predicted values. The predicted values ​​are mapped to the throttle depth control value for vertical climb and the pitch angle control value for horizontal displacement of the UAV, respectively, and confirmed as the continuous variables.

[0033] Optionally, in S4, concatenating the discrete instruction with the continuous variable generation intention decoding instruction includes:

[0034] According to the preset communication protocol format of the UAV underlying flight control system, extract the enumerated status bit values ​​of the discrete command mapping;

[0035] Extract the floating-point dynamic values ​​mapped from the continuous variables;

[0036] The enumerated status bit values ​​and the floating-point dynamic values ​​are encapsulated in a data frame structure, and frame header check flags and frame tail check flags are added to generate the intent decoding instruction in the standard communication data packet format.

[0037] Optionally, in S5, integrating the change in oxyhemoglobin concentration to obtain the cognitive load index includes:

[0038] Based on the spatial channel topology distribution of the near-infrared blood oxygenation signal, target oxygenated hemoglobin concentration change data belonging to the prefrontal cortex region of the brain are selected.

[0039] Set the length of the sliding integral time period used for load assessment;

[0040] Within the length of the sliding integral time period, the difference between the target oxyhemoglobin concentration change data and the resting baseline concentration data is calculated using continuous time integration.

[0041] Extract the integral results and perform maximum and minimum value normalization processing. The results mapped to the value range of zero to one are used as the cognitive load index.

[0042] Optionally, in S5, the execution weights of the intent decoding command and the underlying autonomous safety control law are allocated using the cognitive load index, and the summation is used to generate the final flight control command for issuance and execution, including:

[0043] Obtain the pre-set load sensitivity control parameters of the system;

[0044] The autonomous control weight coefficient is obtained by performing a multiplication operation between the cognitive load index and the load sensitivity control parameter;

[0045] The human control weight coefficient is calculated by subtracting the autonomous control weight coefficient from the number one.

[0046] The human control weight coefficient is multiplied by the intention decoding instruction to obtain the human control component vector.

[0047] The autonomous control weight coefficient is multiplied by the underlying autonomous security control law to obtain the autonomous defense component vector.

[0048] The final flight control command is generated by performing an element-wise summation operation on the human control component vector and the autonomous defense component vector.

[0049] Secondly, this invention provides the following technical solution: a UAV control system based on EEG and fNIRS multimodal feature fusion, comprising the following modules:

[0050] A multimodal brain signal synchronous acquisition module is used to synchronously acquire scalp electroencephalogram (EEG) signals and near-infrared blood oxygenation signals, and convert the near-infrared blood oxygenation signals into changes in oxyhemoglobin concentration.

[0051] The spatiotemporal feature cross-modal alignment module is used to extract the frequency band power spectral density of the scalp EEG signal to construct a fast variable feature matrix, and to extract the temporal features of the oxyhemoglobin concentration change to construct a slow variable feature matrix; the phase space reconstruction algorithm is used to compensate the timestamp of the slow variable feature matrix to achieve time alignment with the fast variable feature matrix;

[0052] The cross-attention feature fusion module is used to map the aligned slow variable feature matrix to a query matrix, and the aligned fast variable feature matrix to a key matrix and a value matrix. The query matrix, the key matrix, and the value matrix are calculated through an attention mechanism to obtain the fused feature matrix.

[0053] The multi-dimensional intent decoding and mapping module is used to input the fused feature matrix into a dual-branch network. The first network branch decodes discrete instructions, the second network branch decodes continuous variables, and the discrete instructions and the continuous variables are concatenated to generate intent decoding instructions.

[0054] The cognitive load adaptive flight control module is used to integrate the change in oxyhemoglobin concentration to obtain a cognitive load index, use the cognitive load index to allocate the execution weights of the intent decoding command and the underlying autonomous safety control law, and sum them to generate the final flight control command for issuance and execution.

[0055] The present invention has the following beneficial effects:

[0056] 1. This invention introduces a phase space reconstruction algorithm to perform inverse time compensation on the slow variable feature matrix reflecting hemodynamics, achieving precise alignment with the fast variable EEG features, fundamentally avoiding feature interference caused by physical signal delay; further, it utilizes a cross-attention mechanism to dynamically weight and fuse the aligned slow variable as the guide query matrix and the fast variable as the key value matrix, significantly improving the complementarity and fusion robustness of multimodal features.

[0057] 2. This invention constructs a dual-branch decoding network to synchronously decouple and output discrete state commands (such as takeoff and landing, yaw) and continuous attitude variables (such as throttle depth, pitch angle), breaking through the limitation of traditional brain control that can only output discrete switching commands, and providing technical support for UAVs to perform complex and precise flight maneuvers in three-dimensional space.

[0058] 3. This invention innovatively utilizes real-time changes in oxyhemoglobin concentration to quantify the operator's cognitive load index and dynamically allocates the execution weights of "intent decoding commands" and "underlying autonomous safety control laws" accordingly. When excessive operator load is detected, the system smoothly takes over control, forcibly initiating autonomous obstacle avoidance and hovering, constructing a closed-loop safety mechanism of "physiological monitoring - weight redistribution - active defense," fundamentally preventing crashes caused by operator fatigue leading to command errors. Attached Figure Description

[0059] Figure 1 A flowchart illustrating the UAV control method based on EEG and fNIRS multimodal feature fusion provided in this embodiment of the invention;

[0060] Figure 2A flowchart illustrating the multi-dimensional intent decoding and mapping process of a UAV control method based on EEG and fNIRS multimodal feature fusion provided in this embodiment of the invention.

[0061] Figure 3 This is an architecture diagram of an unmanned aerial vehicle (UAV) control system based on EEG and fNIRS multimodal feature fusion provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Example 1:

[0064] In a first embodiment of the present invention, the present invention provides a UAV control method based on the fusion of EEG and fNIRS multimodal features, such as... Figure 1 As shown, it includes the following steps:

[0065] S1. Simultaneously acquire scalp EEG signals and near-infrared blood oxygenation signals, and convert the near-infrared blood oxygenation signals into changes in oxyhemoglobin concentration.

[0066] Furthermore, in S1, the conversion of near-infrared blood oxygenation signals into changes in oxyhemoglobin concentration includes:

[0067] The changes in light attenuation after the first and second wavelengths of near-infrared light emitted by the near-infrared detection device penetrated the operator's brain tissue were obtained.

[0068] Extract the physical distance parameters between the light source and the detector of the near-infrared detection device, as well as the differential path factor parameters;

[0069] The extinction coefficients of deoxyhemoglobin and oxyhemoglobin corresponding to the first wavelength and the second wavelength of near-infrared light were retrieved, respectively.

[0070] By combining physical distance parameters, differential path factor parameters, and changes in light attenuation, the extinction coefficients of deoxyhemoglobin and oxyhemoglobin are solved simultaneously to extract the change in oxyhemoglobin concentration.

[0071] Specifically, the input data stream of the system's calculation unit includes the acquired light attenuation change, physical distance parameters, differential path factor parameters, and the extinction coefficients of deoxyhemoglobin and oxyhemoglobin corresponding to specific wavelengths retrieved from a preset memory. Based on these input fundamental parameters, the system performs underlying optical biological tissue scattering and absorption model calculations, and the final output data stream is a discrete data sequence of oxyhemoglobin concentration changes containing timestamp information.

[0072] The underlying algorithm for extracting changes in oxyhemoglobin concentration is strictly based on a modified Beer-Lambert law. Considering the attenuation characteristics of the first and second wavelengths of near-infrared light in brain tissue, the system establishes a system of simultaneous equations for dual-wavelength light attenuation and solves them algebraically. The specific mathematical formulas are expressed as follows:

[0073] ;

[0074] in This represents the change in oxyhemoglobin concentration output by the system. This indicates the amount of light attenuation detected after the first wavelength of near-infrared light penetrates brain tissue. This indicates the amount of light attenuation detected after the second wavelength near-infrared light penetrates brain tissue. This represents the physical distance parameter between the transmitter and receiver of a near-infrared detection device. This represents the differential path factor parameter caused by multiple scattering of photons in biological tissues. and These represent the extinction coefficients of oxyhemoglobin corresponding to the absorption of the first and second wavelengths of near-infrared light. and These represent the extinction coefficients of deoxyhemoglobin corresponding to the absorption of the first and second wavelengths of near-infrared light, respectively.

[0075] By extracting attenuated signals from dual-wavelength near-infrared light at specific wavelengths and combining them with physical distance and photon scattering path factors to solve simultaneous equations for blood oxygen concentration, this method improves upon the shortcomings of traditional single EEG control systems. These systems suffer from extremely low spatial resolution of EEG signals and are easily distorted by surface electromyography (EMG) artifacts. This avoids the flight control command parsing failure problem caused by deviations in the spatial positioning of the operator's brain's intention origin area when UAVs perform dynamic flight missions in three-dimensional space. The calculated changes in oxyhemoglobin concentration possess strong resistance to electromagnetic interference and high spatial positioning characteristics, directly providing the basic data support for slow variables required for subsequent spatiotemporal alignment of cross-attention in multimodal networks.

[0076] S2. Extract the frequency band power spectral density of the scalp EEG signal to construct a fast variable feature matrix, and extract the temporal features of the change in oxyhemoglobin concentration to construct a slow variable feature matrix; use the phase space reconstruction algorithm to compensate the timestamp of the slow variable feature matrix to achieve time alignment with the fast variable feature matrix.

[0077] Furthermore, in S2, the frequency band power spectral density of the scalp EEG signal is extracted to construct a fast variable feature matrix, and the temporal features of the change in oxyhemoglobin concentration are extracted to construct a slow variable feature matrix, including:

[0078] Bandpass filtering was performed on the scalp EEG signal. Short-time Fourier transform was applied to the filtered scalp EEG signal to extract the frequency band power spectral density within a set frequency range. The frequency band power spectral density was then vector-concatenated according to the time dimension to construct a fast variable feature matrix.

[0079] The length of the sliding time window and the sliding step size for cutting the oxyhemoglobin concentration change sequence are set. Within the sliding time window, the average concentration amplitude feature and the concentration change slope feature of the oxyhemoglobin concentration change are calculated respectively. The average concentration amplitude feature and the concentration change slope feature are combined to construct the slow variable feature matrix.

[0080] The selection of the sliding time window length L requires a balance between frequency resolution and real-time performance. According to the Fourier transform principle, frequency resolution... With time window length satisfy The main components of the fNIRS blood oxygenation signal have frequencies in the range of 0.01-0.1 Hz. Theoretically, a time window of L=100s is needed to distinguish the 0.01 Hz frequency component. Considering the system's real-time requirements and operator comfort, this embodiment uses L=60s as the preferred value, while ensuring the extraction of effective features. At this value, the frequency resolution is approximately 0.0167 Hz, which is sufficient to distinguish the main frequency components of the blood oxygenation signal.

[0081] In S2, the phase space reconstruction algorithm is used to compensate for the timestamps of the slow variable feature matrix, achieving time alignment with the fast variable feature matrix, including:

[0082] Determine the hemodynamic delay time of the change in oxyhemoglobin concentration relative to the inherent hemodynamic delay of the scalp electroencephalogram (EEG) signal;

[0083] The embedding dimension and delay time parameter of the phase space reconstruction algorithm are set to map the feature matrix of the slow variable to a high-dimensional phase space;

[0084] Specifically, the determination of the embedding dimension m of the phase space reconstruction follows Takens' embedding theorem as follows: Where d is the intrinsic dimension of the original signal (i.e., the minimum number of variables required for the system's dynamic behavior). In this invention, the intrinsic dimension of the fNIRS blood oxygenation signal has been experimentally determined to be 3-5 dimensions, reflecting 3-5 major activation modes of the prefrontal cortex. Therefore, the embedding dimension m should be 7-11 dimensions. To balance reconstruction accuracy and computational complexity, this embodiment uses m=8.

[0085] The time delay parameter τd is determined using the time delay embedding method as follows: ;

[0086] in This is the autocorrelation function. Based on the signal sampling rate and hemodynamic response characteristics of this system, The recommended value range is 50-200ms. Based on the analysis of a large amount of calibration data from subjects, this embodiment uses... .

[0087] The phase shift compensation was calculated based on the hemodynamic delay time in a high-dimensional phase space.

[0088] The phase shift compensation is applied in reverse to the timestamp of each sampling point of the slow variable feature matrix to obtain the compensated slow variable feature matrix, and the timestamp of the compensated slow variable feature matrix is ​​strictly aligned with the timestamp of the fast variable feature matrix.

[0089] Specifically, the input data stream of the system's calculation unit includes scalp EEG signals acquired at the front end and preprocessed with basic bandpass filtering, as well as the oxyhemoglobin concentration change data with time-stamped sequences generated in the previous step. After dual-link parallel extraction and phase delay compensation calculation, the final output data stream consists of fast variable feature matrices and slow variable feature matrices in an absolutely aligned time dimension.

[0090] During the feature extraction and computation process, the underlying hardware invokes two independent digital signal processing links. For the scalp EEG signal link, the system applies the Short-Time Fourier Transform (SFT) algorithm to extract the signal's frequency band power spectral density. The numerical integration equation for the underlying SFT is expressed as follows:

[0091] ;

[0092] in This indicates the solution calculated by the system at time point. With frequency point The power spectral density value of a specific frequency band at that location. This represents a continuous sequence of scalp EEG signals that has undergone filtering and noise reduction before being input to the current computation node. This represents the sliding analysis window function set by the underlying processor. The letter represents the mathematical basis upon which the complex exponential Fourier transform is based. Represents the imaginary unit. The system collects the frequency band power spectral density values ​​of each window segment in chronological order along the time axis, performs matrix row concatenation, and generates the fast variable feature matrix required for system decoding.

[0093] For the oxyhemoglobin concentration pathway, the calculation unit truncates the discrete data stream based on a fixed sliding time window length and sliding step size. Within each independent sliding time window, the system's underlying layer calculates in parallel the average concentration amplitude parameter and the concentration change slope parameter of the currently truncated data segment.

[0094] Average concentration amplitude parameter The discrete summation equation is as follows:

[0095] ;

[0096] Concentration change slope parameter The linear difference equation is as follows:

[0097] ;

[0098] in This represents the total number of discrete data sampling points contained within a single sliding time window. Representing the The actual value of the change in oxyhemoglobin concentration corresponding to each sampling point. and These represent the changes in oxyhemoglobin concentration at the end and beginning of the current sliding time window, respectively. This represents the fixed interval between adjacent sampling points of the underlying analog-to-digital converter. The extracted average concentration amplitude parameter and concentration change slope parameter are combined column-wise according to the feature dimension to construct the slow variable feature matrix.

[0099] When performing cross-modal time alignment, the system must overcome the physical barrier of spatiotemporal misalignment caused by the differences in the intrinsic properties of the two physiological signals. The underlying algorithm module establishes the inherent hemodynamic delay time of blood oxygen metabolism response lagging behind neural potential excitation. Following the dynamic Tukens theorem, the system reconstructs the slow variable characteristic matrix in one-dimensional time series form to a high-dimensional topological phase space. The reconstructed high-dimensional phase space state vector... The equation is as follows:

[0100] ;

[0101] in This represents the multidimensional phase space state vector constructed after mapping to a higher-dimensional space. Refers to the extracted feature element values ​​in the feature matrix of slow variables. The system phase space embedding dimension is defined by the algorithm. This represents the delay time parameter preset at the underlying control terminal.

[0102] The system calculates the system-level phase shift compensation within the reconstructed high-dimensional phase space according to the calibrated hemodynamic delay time metric. The underlying clock synchronizer directly applies this phase shift compensation to the timestamp label of each sampling point in the slow variable feature matrix in the reverse time flow direction. The new timestamp after compensation... Compared with the original timestamp And calculate the phase shift compensation amount It satisfies a strict linear subtraction relationship:

[0103] ;

[0104] After reverse time axis translation correction, the time reference of all feature points contained in the slow variable feature matrix is ​​completely equivalent to the potential excitation reference of the fast variable feature matrix.

[0105] At the underlying physical execution logic and system structure level, the introduction of multi-dimensional phase space mapping and inverse time translation mechanisms has clear engineering and controllability value. This architecture changes the underlying data flow path of existing brain-controlled communication systems, which relies directly on physical wiring for heterogeneous signals. It improves the technical defects of traditional solutions that cause interference between feature dimensions by mixing blood flow signals with physical delays of several seconds with millisecond-level instantaneous potential signals when sending them to the classifier. After achieving absolute physical time scale alignment, it avoids the extraction of mutually divergent spatial feature vectors by the subsequent deep fusion network when decoding multi-dimensional control intentions, thereby eliminating the problems of command stuttering and output divergence when the UAV performs continuous dynamic non-electrical variable actions such as pitch and yaw.

[0106] S3. Map the aligned slow variable feature matrix to the query matrix, and the aligned fast variable feature matrix to the key matrix and value matrix. Calculate the query matrix, key matrix, and value matrix through the attention mechanism to obtain the fused feature matrix.

[0107] Furthermore, in S3, the fused feature matrix is ​​derived by calculating the query matrix, key matrix, and value matrix through the attention mechanism, including:

[0108] Extract the pre-trained first weight matrix, second weight matrix, and third weight matrix;

[0109] The query matrix is ​​obtained by multiplying the aligned slow variable feature matrix by the first weight matrix;

[0110] The aligned fast variable feature matrix is ​​multiplied by the second weight matrix and the third weight matrix respectively to obtain the key matrix and the value matrix;

[0111] Calculate the dot product of the query matrix and the transpose of the key matrix, divide the dot product by the dimension scaling factor of the feature vector, and normalize it using the Softmax function to obtain the cross-modal attention weight distribution matrix.

[0112] Perform matrix multiplication on the cross-modal attention weight distribution matrix and the value matrix to output a fusion feature matrix containing complementary information from multiple modalities.

[0113] Specifically, the input data stream consists of the slow variable feature matrix and the fast variable feature matrix, which have been absolutely timestamped in the previous step. The system retrieves the first, second, and third weight matrices, which are pre-stored in the memory unit, in parallel for computation. After matrix projection and spatial inner product operations via cross-attention mechanism, the output data stream is a fused feature matrix containing multimodal complementary information, which directly serves as the input source for the next-level decoding mapping network.

[0114] In the process of extracting cross-modal attention weights, the computation unit first performs a linear space mapping on the input heterogeneous physiological feature matrix. The algebraic equation for generating the query matrix from the slow variable feature matrix is ​​as follows:

[0115] ;

[0116] The algebraic equations for generating the key matrix and value matrix from the fast variable characteristic matrix are as follows:

[0117] ;

[0118] ;

[0119] in This represents the query matrix generated by the network. This represents the key matrix generated by the network. This represents the value matrix generated by the network. The feature matrix represents the slow variable that completes time alignment at the input end. The feature matrix represents the fast variable that completes time alignment at the input end. This represents the first weight matrix retrieved. This represents the second weight matrix retrieved. This represents the retrieved third weight matrix.

[0120] In the feature dimensionality reduction and fusion stage, the system performs a dot product operation using the transpose of the query matrix and the key matrix, introduces a dimensionality scaling factor for the feature vectors to adjust the numerical distribution, then calls the Softmax function to perform normalization processing to obtain the cross-modal attention weight distribution matrix, and finally performs matrix multiplication with the value matrix to output the fused feature matrix. The cross-attention mechanism model is as follows:

[0121] ;

[0122] in This represents the fusion feature matrix output by the system solution. This represents the transpose matrix obtained after performing a transpose operation on the key matrix. This represents the dimension scaling factor set to prevent the network gradient from vanishing due to excessively large dot product values. This represents a normalized exponential function that maps discrete dot product values ​​to a standard probability interval of zero to one.

[0123] At the underlying physical execution and algorithmic logic architecture level, by using slow variable features reflecting the prefrontal cortex hemodynamic hysteresis state as query guide vectors to match and extract fast variable features reflecting transient neural potential excitation, this improves upon the technical shortcomings of traditional brain control systems that directly physically concatenate heterogeneous multimodal feature vectors, leading to feature interference and classifier oscillations. This avoids the problem of mis-triggered commands caused by chaotic parsing of underlying neural intent features when UAVs perform three-dimensional dynamic space flight missions. This step, at the algorithmic level, grants dominance over spatial noise resistance of blood oxygenation signals and execution control over the temporal response of EEG signals.

[0124] S4, such as Figure 2 As shown, the fused feature matrix is ​​input into a dual-branch network. The first network branch decodes discrete instructions, and the second network branch decodes continuous variables. The discrete instructions and continuous variables are then concatenated to generate the intent decoding instructions.

[0125] In this embodiment, the dual-branch network adopts a joint training architecture, which includes two independent neural network branches.

[0126] The first network branch is the discrete instruction classification branch, and its input layer reception dimension is... The fused feature matrix is ​​sequentially passed through a 128-neuron ReLU activated hidden layer, a 64-neuron ReLU activated hidden layer, and a Dropout regularization layer with an inactivation ratio of 0.3, and finally by a layer containing... The softmax output layer of each neuron generates discrete instruction classification probabilities (in this embodiment). These correspond to five discrete command categories: takeoff and landing, forward movement, backward movement, left turn, and right turn. This branch is optimized using the cross-entropy loss function.

[0127] The second network branch is a continuous variable regression branch, and its input layer also receives data in dimensions of 1000. The fused feature matrix, after passing through the same hidden layer structure and Dropout regularization layer as the first branch, is mapped by the linear activation output layer of two neurons to the throttle depth control for vertical climb and the pitch angle control for horizontal displacement of the UAV, respectively. This branch uses mean absolute error (MAE) as the loss function.

[0128] The dual-branch network uses a joint training method, and the total loss function is defined as:

[0129] ;

[0130] in For cross-entropy loss, For MAE loss, balance coefficient In this embodiment, 0.5 is used.

[0131] The training dataset was constructed as follows: synchronous EEG and fNIRS signals were collected from no fewer than 10 subjects during simulated UAV flight control tasks. Each subject performed no fewer than 500 complete flight operations (including takeoff and landing, forward, backward, left turn, right turn, and continuous attitude adjustments). Discrete command labels were determined by the task type in the experimental paradigm, while continuous variable labels were obtained from the throttle depth and pitch angle values ​​synchronously recorded by the UAV flight control system. The EEG signals were bandpass filtered from 0.5 to 40 Hz and processed for artifact removal. The fNIRS signals were converted based on the concentration change based on the modified Beer-Lambert law, and the fused feature matrix was extracted according to steps S2-S3 above and used as the network input.

[0132] The network was trained using the Adam optimizer with an initial learning rate of [missing information]. The batch size is 32 samples per batch. The training data size is no less than 5000 samples, divided into training and validation sets in an 80:20 ratio. An early stopping strategy is adopted, automatically stopping training when the validation set loss does not decrease for 10 consecutive epochs. The maximum number of training epochs is set to 200. After training, the model performance is evaluated using a reserved test set (accounting for 10% of the total dataset). The discrete instruction classification accuracy is no less than 85%, and the root mean square error of continuous variable prediction does not exceed 5% of the true range, indicating that the model has effective intent decoding capability.

[0133] Furthermore, in S4, the first network branch decodes discrete instructions, and the second network branch decodes continuous variables, including:

[0134] The fused feature matrix is ​​input into the hidden layer of the first network branch to extract the classification feature vector. The classification feature vector is discretized and mapped using the Softmax classification layer to output the classification probability of the UAV take-off and landing state and the classification probability of the horizontal yaw direction. The item with the highest probability is selected as the discrete command.

[0135] The fused feature matrix is ​​synchronously input into the hidden layer of the second network branch to extract regression feature vectors. The regression feature vectors are fitted using a linear regression layer to output predicted values. The predicted values ​​are mapped to the throttle depth control value for vertical climb and the pitch angle control value for horizontal displacement of the UAV, respectively, and confirmed as continuous variables.

[0136] In S4, the splicing of discrete instructions and continuous variable generation intention decoding instructions include:

[0137] According to the preset communication protocol format of the UAV's underlying flight control system, extract the enumerated status bit values ​​of the discrete command mapping;

[0138] Extract floating-point dynamic values ​​mapped from continuous variables;

[0139] The enumerated status bit values ​​and floating-point dynamic values ​​are encapsulated in a data frame structure, and frame header check flags and frame tail check flags are added to generate the intent decoding instruction in the standard communication data packet format.

[0140] Specifically, the input data stream to the system decoding module is the fused feature matrix extracted from the cross-modal feature fusion in the previous step. This input data is synchronously injected into two parallel independent neural network links within the system for algebraic decoupling computation. After the underlying algorithm completes multi-dimensional data separation and structure reassembly encapsulation, the output data stream is a standard data packet of intent decoding instructions conforming to the underlying hardware communication bus protocol.

[0141] For the discrete-state decoding process, the fused feature matrix is ​​first fed into the first network branch. Inside this branch, the hidden layer performs non-linear activation operations on the input matrix to extract a classification feature vector. A Softmax classification layer then maps this classification feature vector into a sequence of conditional probability distributions in the discrete state space. The probability mapping equation at the bottom layer of the classification layer is as follows:

[0142] ;

[0143] in This indicates that the network model determines that the current intent feature belongs to the first... Numerical prediction probability of discrete flight states. This indicates that the classification feature vector output by the hidden layer of the first network branch is at the 1st... The original logarithmic odds output values ​​for each dimension. This indicates the total number of categories of discrete UAV commands predefined by the underlying flight control system. is the base of the natural logarithm.

[0144] The solution unit extracts the index number corresponding to the highest probability value in the conditional probability distribution sequence and converts it into a discrete instruction containing the classification probability of UAV take-off and landing status and the classification probability of horizontal yaw direction.

[0145] For the continuous attitude decoding process, the second network branch receives the same fused feature matrix. Its internal hidden layers independently extract regression feature vectors to characterize dynamic continuous displacement. The system calls a linear regression layer to perform a multidimensional linear fitting operation on this regression feature vector, outputting a floating-point predicted value. Its underlying multivariate linear regression fitting equation is as follows:

[0146] ;

[0147] in This represents the continuous predicted numerical vector output by the solution. This represents the regression feature vector output by the hidden layer of the second network branch. This represents the weight matrix that has converged after being trained by backpropagation of the network. This represents the bias term constant of the regression network node.

[0148] The output continuous prediction vector is mapped and confirmed as the throttle depth control quantity for vertical climb and the pitch angle control quantity for horizontal displacement of the UAV, collectively referred to as continuous variables.

[0149] During the data integration and encapsulation execution phase, the communication conversion module, according to the communication protocol format preset by the underlying flight control system's physical bus, extracts the enumerated status bit values ​​generated by discrete command mapping and extracts the floating-point dynamic values ​​generated by continuous variable mapping. The underlying program concatenates the memory addresses of the enumerated status bit values ​​and the floating-point dynamic values ​​and loads them into a predefined data frame structure. The system appends a frame header check flag for message start identification and a frame tail check flag for cyclic redundancy check to both ends of the structure payload, generating an intent decoding command in the standard communication data packet format and sending it to the physical bus.

[0150] At the system network architecture level, a dual-branch parallel decoding and physical communication frame encapsulation mechanism is applied to improve the underlying technical deficiency of traditional brain-controlled classification networks, which are limited to outputting discrete switching signals in a two-dimensional plane due to their simple structure. By employing a technique of separating and decoding discrete and continuous variables and reconstructing them into standard protocol data packets, the problem of flight trajectory rigidity and abrupt action changes caused by the lack of continuous fine-tuning capability for throttle and attitude angles in three-dimensional space is avoided. This completely establishes the communication link from the brain's non-steady-state bioelectrophysiological intentions to the UAV's electromechanical servo drive signals.

[0151] S5. Integrate the change in oxyhemoglobin concentration to obtain the cognitive load index. Use the cognitive load index to allocate the execution weights of the intention decoding command and the underlying autonomous safety control law, and sum them to generate the final flight control command for issuance and execution.

[0152] Furthermore, in S5, the cognitive load index is derived by integrating the changes in oxyhemoglobin concentration, including:

[0153] Based on the spatial channel topology distribution of near-infrared blood oxygenation signals, data on the change in target oxyhemoglobin concentration that is mapped to the prefrontal cortex region of the brain were selected.

[0154] Set the length of the sliding integral time period used for load assessment;

[0155] Within the sliding integral time period, the difference between the target oxyhemoglobin concentration change data and the resting baseline concentration data is calculated by continuous time integration.

[0156] Extract the integral results and perform maximum and minimum value normalization. The results mapped to the range of zero to one are used as the cognitive load index.

[0157] In S5, the cognitive load index is used to allocate execution weights between the intent decoding command and the underlying autonomous safety control law, and the summation generates the final flight control command for issuance and execution, including:

[0158] Obtain the pre-set load sensitivity control parameters of the system;

[0159] The autonomous control weight coefficient is obtained by performing a multiplication operation between the cognitive load index and the load sensitivity control parameter;

[0160] The human control weight coefficient is calculated by subtracting the autonomous control weight coefficient from the number one.

[0161] The human control component vector is obtained by multiplying the human control weight coefficient with the intention decoding instruction execution vector;

[0162] The autonomous defense component vector is obtained by performing a vector scalar multiplication operation between the autonomous control weight coefficient and the underlying autonomous security control law.

[0163] The final flight control command is generated by summing the human control component vector and the autonomous defense component vector element by element.

[0164] Specifically, the input data stream at the bottom layer of the solution module includes the change in oxyhemoglobin concentration mapped to the prefrontal cortex region via spatial topology mapping, the intent decoding command extracted from the previous step, the resting-state baseline concentration data stored in the system storage medium, and the UAV's underlying autonomous safety control law. After data integration and weight allocation calculations, the final output data stream is the final flight control command vector issued to the UAV's servo actuators.

[0165] In the cognitive load quantification assessment stage, the system performs continuous-time integration calculations on the degree to which the target data of the prefrontal cortex deviates from the resting state, based on a preset sliding integral time period. The underlying algebraic equation for continuous-time integration is expressed as follows:

[0166] ;

[0167] in This represents the original integral value calculated within the current sliding time window. This indicates the absolute time point of the current calculation. This indicates the set length of the sliding integral time period. Indicates the integral variable Data on the change in target oxyhemoglobin concentration in the prefrontal cortex extracted at any time. This represents the resting-state baseline concentration data obtained during the calibration phase.

[0168] After extracting the raw integral values, the system calls the maximum-minimum normalization algorithm to strictly map them to the absolute value range of zero to one in order to construct a standardized cognitive load index. The normalization mapping equation is:

[0169] ;

[0170] in This represents the cognitive load index output by the system. and These represent the pre-calibrated historical maximum integration threshold and minimum baseline integration threshold, respectively.

[0171] During the dynamic weight allocation phase, the calculation unit acquires the system's preset load sensitivity control parameters and establishes a linear allocation model between human control and underlying autonomous defense. The system calculates the following multivariate linear combination equation for the final flight control commands:

[0172] ;

[0173] in This represents the final flight control command vector output by the system's integrated solution. This represents the system's preset load sensitivity control parameter, with a value range of 0 to 1. It is used to adjust the degree of influence of the cognitive load index on the weight allocation. The specific value can be calibrated through offline experiments, with the goal of maintaining subjective control as the priority when the operator's cognitive load is moderate and smoothly transitioning to autonomous control when the load is too high. This represents the received intent decoding instruction vector. This represents the autonomous defense and safe hovering control law vector built into the underlying flight control system. The product of these values ​​constitutes the autonomous control weight coefficient for determining the degree of system takeover. The difference constitutes the artificial control weight coefficient that characterizes the credibility of subjective intention.

[0174] A real-time quantification and dynamic weight allocation mechanism for cognitive load based on prefrontal cortex blood oxygen metabolism levels is introduced to improve the technical deficiency of traditional brain-controlled drone systems that only have a unidirectional open-loop action execution pathway. The system obtains the operator's deep physiological fatigue state from the underlying layer. When long-duration, high-risk operations cause the operator's mental distraction and the cognitive load index approaches its extreme value, the system, based on the aforementioned linear model, forcibly reduces the weight of human-induced erroneous commands and proportionally amplifies the takeover weight of the underlying autonomous hovering and obstacle avoidance logic. This architecture fundamentally avoids the problem of loss of control and crashes caused by the forced execution of abnormal brain-controlled commands due to the operator's deteriorating subjective state during complex dynamic operations, realizing a closed-loop active safety defense chain in human-machine co-pilot mode.

[0175] Example 2:

[0176] In the application of drones for close-range reconnaissance operations at disaster relief and hazardous chemical spill sites, operators are in a safe command vehicle and wear brain-computer interface devices to remotely control multi-rotor drones to conduct reconnaissance in the core areas of disaster zones containing a large number of building ruins and crisscrossing pipelines. In the three-dimensional environment of disaster relief, drones need to frequently perform vertical climbs, hovering fine-tuning, and multi-angle horizontal yaw to avoid obstacles in confined spaces. This requires operators to maintain a high level of mental focus and continuously issue control commands, thus exposing existing technical problems: existing pure EEG control equipment is susceptible to physiological artifacts caused by operator stress, has limited feature analysis dimensions, and mainly outputs discontinuous on / off control quantities, making it difficult to meet the drone's control requirements for continuous and fine-tuning of throttle depth and attitude angle, which can easily lead to drone stiffness and collision risks; long-endurance, intense rescue and reconnaissance missions can easily overload the operator's prefrontal cortex blood oxygen metabolism, which may induce cognitive fatigue and decreased attention. Existing open-loop unidirectional brain control links lack real-time monitoring and active defense mechanisms for the operator's underlying physiological fatigue state. When the operator issues incorrect brain control commands due to fatigue, the drone's underlying flight control system may directly execute incorrect actions, increasing the risk of loss of control and crash.

[0177] To address the aforementioned problems, this invention employs a UAV control system based on EEG and fNIRS multimodal feature fusion, the structure of which is as follows: Figure 3 As shown. The specific implementation process of this system is as follows:

[0178] The system constructs a complete closed-loop link covering physiological signal acquisition to electromechanical action execution, solving the technical defects of low brain-controlled precision and lack of active defense in UAVs. The system first simultaneously collects scalp EEG and near-infrared blood oxygenation signals to obtain heterogeneous physiological data sources with both instantaneous response and spatial positioning. Then, it extracts fast and slow variable feature matrices and uses a phase space reconstruction algorithm to perform reverse time axis translation compensation on the slow variables, eliminating spatiotemporal misalignment caused by hemodynamic lag. Further, leveraging a cross-attention mechanism, it uses the slow blood oxygenation variable as a query matrix to guide and purify the transient motion intentions of the fast EEG variables, outputting a deep fusion feature matrix. This fusion feature matrix is ​​then branched to a dual-branch network, simultaneously decoupling discrete action commands and continuous attitude variables, breaking through the physical bottleneck of traditional two-dimensional switch control. Finally, the system quantifies the cognitive load index in real time through blood oxygen concentration integration, dynamically adjusting the weights of human brain-controlled commands and underlying autonomous safety control laws. When operator cognitive fatigue is detected, it forcibly initiates autonomous obstacle avoidance and hovering takeover, achieving continuous and precise flight control and human-machine co-pilot active defense in three-dimensional space.

[0179] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for controlling a UAV based on multi-modal feature fusion of EEG and fNIRS, characterized in that, Includes the following steps: S1. Simultaneously acquire scalp electroencephalogram (EEG) signals and near-infrared blood oxygenation signals, and convert the near-infrared blood oxygenation signals into changes in oxyhemoglobin concentration. S2. Extract the frequency band power spectral density of the scalp EEG signal to construct a fast variable feature matrix, and extract the temporal features of the oxyhemoglobin concentration change to construct a slow variable feature matrix; use the phase space reconstruction algorithm to compensate the timestamp of the slow variable feature matrix to achieve time alignment with the fast variable feature matrix; Specifically, the process of using a phase space reconstruction algorithm to compensate for the timestamps of the slow variable feature matrix to achieve time alignment with the fast variable feature matrix includes: Determine the change in oxyhemoglobin concentration relative to the inherent hemodynamic delay time of the scalp electroencephalogram signal; The embedding dimension and delay time parameter of the phase space reconstruction algorithm are set, and the slow variable feature matrix is ​​mapped to a high-dimensional phase space; The phase shift compensation is calculated based on the hemodynamic delay time within the high-dimensional phase space; The phase shift compensation amount is applied in reverse to the timestamp of each sampling point of the slow variable feature matrix to obtain the compensated slow variable feature matrix, and the timestamp of the compensated slow variable feature matrix is ​​strictly aligned with the timestamp of the fast variable feature matrix. S3. Map the aligned slow variable feature matrix to a query matrix, and map the aligned fast variable feature matrix to a key matrix and a value matrix. Calculate the query matrix, the key matrix, and the value matrix using an attention mechanism to obtain the fused feature matrix. S4. Input the fused feature matrix into a dual-branch network. The first network branch decodes discrete instructions, and the second network branch decodes continuous variables. The discrete instructions and the continuous variables are then concatenated to generate an intent decoding instruction. The first network branch decodes discrete instructions, and the second network branch decodes continuous variables, including: The fused feature matrix is ​​input into the hidden layer of the first network branch to extract the classification feature vector. The classification feature vector is discretized and mapped using the Softmax classification layer to output the classification probability of the UAV take-off and landing state and the classification probability of the horizontal yaw direction. The item with the highest probability is selected as the discrete command. The fused feature matrix is ​​synchronously input into the hidden layer of the second network branch to extract regression feature vectors. The regression feature vectors are fitted using a linear regression layer to output predicted values. The predicted values ​​are mapped to the throttle depth control amount for vertical climb and the pitch angle control amount for horizontal displacement of the UAV, respectively, and confirmed as the continuous variables. S5. Integrate the change in oxyhemoglobin concentration to obtain the cognitive load index, use the cognitive load index to allocate the execution weights of the intent decoding command and the underlying autonomous safety control law, and sum them to generate the final flight control command for issuance and execution. The process of allocating execution weights between the intent decoding command and the underlying autonomous safety control law using the cognitive load index, and summing these weights to generate the final flight control command for execution, includes: Obtain the pre-set load sensitivity control parameters of the system; The autonomous control weight coefficient is obtained by performing a multiplication operation between the cognitive load index and the load sensitivity control parameter; The human control weight coefficient is calculated by subtracting the autonomous control weight coefficient from the number one. The human control weight coefficient is multiplied by the intention decoding instruction to obtain the human control component vector. The autonomous control weight coefficient is multiplied by the underlying autonomous security control law to obtain the autonomous defense component vector. The final flight control command is generated by performing an element-wise summation operation on the human control component vector and the autonomous defense component vector.

2. The method of claim 1, wherein the method is based on EEG and fNIRS multi-modal feature fusion. In S1, converting the near-infrared blood oxygenation signal into a change in oxyhemoglobin concentration includes: The changes in light attenuation after the first and second wavelengths of near-infrared light emitted by the near-infrared detection device penetrated the operator's brain tissue were obtained. Extract the physical distance parameters between the light source and the detector, as well as the differential path factor parameters, of the near-infrared detection device; The extinction coefficients of deoxyhemoglobin and oxyhemoglobin corresponding to the first wavelength near-infrared light and the second wavelength near-infrared light are retrieved respectively. By combining the physical distance parameter, the differential path factor parameter, and the change in light attenuation, and by simultaneously solving the extinction coefficients of deoxyhemoglobin and oxyhemoglobin, the change in oxyhemoglobin concentration is extracted. 3.The method of claim 1, wherein, In S2, the frequency band power spectral density of the scalp EEG signal is extracted to construct a fast variable feature matrix, and the temporal features of the oxyhemoglobin concentration change are extracted to construct a slow variable feature matrix, including: The scalp EEG signal is subjected to bandpass filtering. Short-time Fourier transform is applied to the filtered scalp EEG signal to extract the frequency band power spectral density within a set frequency range. The frequency band power spectral density is then vector-concatenated according to the time dimension to construct the fast variable feature matrix. The length of the sliding time window and the sliding step size for cutting the sequence of changes in oxyhemoglobin concentration are set. Within the sliding time window, the average concentration amplitude feature and the concentration change slope feature of the changes in oxyhemoglobin concentration are calculated respectively. The average concentration amplitude feature and the concentration change slope feature are combined to construct the slow variable feature matrix.

4. The method of claim 1, wherein the method is based on EEG and fNIRS multi-modal feature fusion. In S3, the step of calculating the fused feature matrix by means of the attention mechanism, including the query matrix, the key matrix, and the value matrix, includes: Extract the pre-trained first weight matrix, second weight matrix, and third weight matrix; The query matrix is ​​obtained by multiplying the aligned slow variable feature matrix by the first weight matrix; The aligned fast variable feature matrix is ​​multiplied by the second weight matrix and the third weight matrix respectively to obtain the key matrix and the value matrix; Calculate the dot product of the query matrix and the transpose of the key matrix, divide the dot product by the dimension scaling factor of the feature vector, and normalize it using the Softmax function to obtain the cross-modal attention weight distribution matrix. Perform matrix multiplication on the cross-modal attention weight distribution matrix and the value matrix to output the fusion feature matrix containing multimodal complementary information. 5.The method of claim 1, wherein, In S4, concatenating the discrete instruction with the continuous variable generation intention decoding instruction includes: According to the preset communication protocol format of the UAV underlying flight control system, extract the enumerated status bit values ​​of the discrete command mapping; Extract the floating-point dynamic values ​​mapped from the continuous variables; The enumerated status bit values ​​and the floating-point dynamic values ​​are encapsulated in a data frame structure, and frame header check flags and frame tail check flags are added to generate the intent decoding instruction in the standard communication data packet format. 6.The method of claim 1, wherein, In S5, the cognitive load index is derived by integrating the change in oxyhemoglobin concentration, including: Based on the spatial channel topology distribution of the near-infrared blood oxygenation signal, target oxygenated hemoglobin concentration change data belonging to the prefrontal cortex region of the brain are selected. Set the length of the sliding integral time period used for load assessment; Within the length of the sliding integral time period, the difference between the target oxyhemoglobin concentration change data and the resting baseline concentration data is calculated using continuous time integration. Extract the integral results and perform maximum and minimum value normalization processing. The results mapped to the value range of zero to one are used as the cognitive load index.

7. An unmanned aerial vehicle control system based on EEG and fNIRS multi-modal feature fusion, characterized in that, The UAV control method based on EEG and fNIRS multimodal feature fusion as described in any one of claims 1-6 includes the following modules: A multimodal brain signal synchronous acquisition module is used to synchronously acquire scalp electroencephalogram (EEG) signals and near-infrared blood oxygenation signals, and convert the near-infrared blood oxygenation signals into changes in oxyhemoglobin concentration. The spatiotemporal feature cross-modal alignment module is used to extract the frequency band power spectral density of the scalp EEG signal to construct a fast variable feature matrix, and to extract the temporal features of the oxyhemoglobin concentration change to construct a slow variable feature matrix; the phase space reconstruction algorithm is used to compensate the timestamp of the slow variable feature matrix to achieve time alignment with the fast variable feature matrix; The cross-attention feature fusion module is used to map the aligned slow variable feature matrix to a query matrix, and the aligned fast variable feature matrix to a key matrix and a value matrix. The query matrix, the key matrix, and the value matrix are calculated through an attention mechanism to obtain the fused feature matrix. The multi-dimensional intent decoding and mapping module is used to input the fused feature matrix into a dual-branch network. The first network branch decodes discrete instructions, the second network branch decodes continuous variables, and the discrete instructions and the continuous variables are concatenated to generate intent decoding instructions. The cognitive load adaptive flight control module is used to integrate the change in oxyhemoglobin concentration to obtain a cognitive load index, use the cognitive load index to allocate the execution weights of the intent decoding command and the underlying autonomous safety control law, and sum them to generate the final flight control command for issuance and execution.

Images