Unmanned aerial vehicle control system based on brain-computer interface

Abstract

The invention discloses an unmanned aerial vehicle control system based on a brain-computer interface. In the system, a near-infrared brain function imaging device obtains multi-channel fNIRS original data, noise is removed through a preprocessing module, and multi-channel fNIRS data obtained after noise removal is obtained; inputting the de-noised multi-channel fNIRS data into a diffusion-based data enhancement module, generating multi-channel fNIRS data corresponding to different motor imagery task tag categories, and performing data expansion and enhancement; a transformer-based intention recognition module performs motor imagery task recognition by extracting features of a time domain, a frequency domain and a space domain of multi-channel fNIRS data corresponding to different motor imagery task label categories, and an upper computer forms a corresponding instruction to control an unmanned aerial vehicle to move according to the recognized motor imagery task. According to the invention, the performance of cross-subject operation imagination task identification is significantly improved, and accurate control of the unmanned aerial vehicle is realized.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface control, and in particular to a UAV control system based on a brain-computer interface. Background Art

[0002] Brain-computer interface is an advanced technology that builds a system that does not require peripheral nerves and muscles, and does not require any body parts to move. Related studies have shown that the performance of blood oxygen activity in human tasks involves a large-scale distributed brain network. Researchers have successfully captured its spatial characteristics. This not only enhances the task decoding performance, but also makes it possible to use deep learning models for research. Therefore, the current trend in the development of BCI is to use the interpretability of deep learning models to explore the changes in task-state blood oxygen concentration in the brain.

[0003] As an innovative non-invasive neuroimaging technique, functional near-infrared spectroscopy not only has excellent spatial resolution, but also exhibits resistance to motion artifacts and electromagnetic interference. Therefore, it is able to capture more comprehensive spatial information. Therefore, the development of new fNIRS-based BCIs is expected to surpass the existing limitations of motor imagery task recognition performance. However, current fNIRS-based motor imagery task recognition research faces challenges such as limited recognition performance, lack of model interpretability, and small data sets. Analyzing fNIRS signals can infer the subject's thoughts, enabling people to communicate directly with external devices (drones) through brain consciousness, becoming a communication channel. With the rapid development of technology, MI (Motor Imagery)-fNIRS has been widely used in BCI research. It is assumed that these MI-fNIRS-based neural activities are masterable. In that case, it would be possible to use "imagination" to control equipment such as drones or robotic arms, and realize artificial intelligence interaction and human-machine collaborative decision-making in the military field. However, due to individual differences in subjects, coupled with the instability of brain activity and low signal-to-noise ratio, physiological signals will produce different results. These factors may limit the performance of classification. Therefore, how to extract useful features from complex fNIRS signals to improve the classification accuracy of MI-based BCI systems remains an important challenge (An asynchronous brain-controlled drone system based on a wearable display (CN107168346A)).

[0004] Deep learning models require a large amount of training data and computing resources to enable the deep network to be fully trained to prevent overfitting. However, the collection of large-scale, high-quality labeled fNIRS data is complex and expensive when factors such as time, labor and material costs, stability of the measurement system, and comfort of the subjects are taken into account. Data augmentation is a key technology to address the problems of small sample datasets and unbalanced data distribution. However, traditional data augmentation methods in image recognition, involving scaling, rotation, cropping, shifting, and other geometric transformations, are not applicable to multi-channel fNIRS data without a single target. Summary of the invention

[0005] In EEG-BCI research, empirical mode decomposition methods are used to enhance data to improve accuracy and reduce the time spent on rehabilitation training. With the development of artificial intelligence, diffusion models are used to generate artificial samples to expand training data sets and improve accuracy.

[0006] Compared with convolutional neural networks (CNNs), Transformers based on multi-head self-attention mechanisms not only have powerful global representation capabilities, but also show the ability to intelligently learn attention weight information. This demonstrates their excellent interpretability. Currently, Transformers, as a key component of large-scale language models, have achieved great success on GPT-4.0. In addition, more and more researchers have proposed Transformers-based models to enhance various visual tasks.

[0007] The purpose of the present invention is to provide a UAV control system based on brain-computer interface. The system uses the self-attention mechanism of Transformer to identify the attention weight distribution of fNIRS channels and identify the brain areas of different motor imagery tasks. It is of great significance to decode brain instructions based on deep learning models. By combining the data enhancement technology based on diffusion, the data of fNIRS motor imagery tasks is significantly enriched, thereby enhancing the performance of the model in task classification. The present invention constructs an fNIRS motor imagery task recognition framework based on a spatial transformer network, uses a convolution module to extract spatiotemporal fusion features, and learns channel attention weights based on a spatial transformer module to achieve motor imagery task decoding and further perform UAV motion control.

[0008] In the present invention, an end-to-end, high-performance fNIRS-based motor imagery task recognition framework is constructed. First, the present invention proposes a new data enhancement deep learning model for MI-fNIRS signals, and the data enhancement based on the conditional diffusion model solves the problem of insufficient fNIRS data. Secondly, based on the self-attention mechanism of Transformer, this method captures channel attention information, enabling the model to automatically understand the important brain areas for different motor imagery tasks. This significantly improves the performance of cross-subject operation imagination task recognition and realizes precise control of drones. In addition, due to the excellent spatial resolution of fNIRS and the excellent interpretability of Transformer, the present invention completes the exploration of the neural processing patterns of motor imagery tasks in more refined brain areas.

[0009] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0010] A UAV control system based on a brain-computer interface includes a near-infrared brain function imaging device and a host computer; the host computer is provided with a preprocessing module, a diffusion-based data enhancement module and a transformer-based intention recognition module; The near-infrared brain functional imaging device acquires multi-channel fNIRS raw data, removes noise through the preprocessing module, and obtains denoised multi-channel fNIRS data; the denoised multi-channel fNIRS data is input into the diffusion-based data enhancement module to generate multi-channel fNIRS data corresponding to different motor imagery task label categories, and the data is expanded and enhanced; the transformer-based intention recognition module extracts the time domain, frequency domain and spatial domain features of the multi-channel fNIRS data corresponding to different motor imagery task label categories to identify the motor imagery task, and the host computer generates corresponding instructions to control the drone to perform activities based on the identified motor imagery task.

[0011] Furthermore, the near-infrared brain functional imaging device was worn on the forehead to collect fNIRS raw data in resting and task states; The raw data of fNIRS are accompanied by interferences such as physiological noise and motion artifacts. Since the detection sensitivity of the fNIRS system is reflected in the ability to detect weak signals, these interferences will seriously affect the sensitivity of the system; The near-infrared brain function imaging device is equipped with n LED light sources and m detectors. One LED light source and one detector form a channel, that is, there are n×m channels to collect data, and the light intensity data of multiple channels is collected; The near-infrared brain function imaging device is provided with a control interface for controlling the power of the light source. The host computer adjusts the optical power of the light source through the control interface so that the intensity of the light signal received by the detector in the near-infrared brain function imaging device is as large as possible, but not oversaturated, thereby effectively increasing the intensity of the light signal and improving the signal-to-noise ratio; the near-infrared brain function imaging device will automatically convert the collected multi-channel light intensity data into multi-channel light intensity data.

[0012] Furthermore, the multi-channel fNIRS raw data are preprocessed in the preprocessing module. First, the abnormal data are eliminated, and then the noise is removed by a low-pass filter. Since the relative movement of the subject's head and the probe will cause motion artifacts, the motion artifacts are removed by discrete wavelet transform. Finally, the global oscillation is removed by the multi-range near-infrared spectral enhancement method to obtain the denoised multi-channel fNIRS data.

[0013] Furthermore, the coefficient of variation of all channels was calculated based on the PHOEBE algorithm to evaluate the stability of the raw fNIRS data collected by each channel and eliminate abnormal data; After removing abnormal data, a third-order Butterworth filter was used to remove noise; After noise removal, discrete wavelet transform (DWT) was used to decompose the fNIRS raw data into a set of high-frequency and low-frequency sub-signals, and motion artifacts were removed by setting thresholds of low-frequency coefficients and high-frequency coefficients.

[0014] Furthermore, in a near-infrared brain function imaging device, one light source corresponds to two detectors, wherein the distance between the first detector and the light source is longer than the distance between the second detector and the light source, a long-distance channel is formed between the first detector and the light source, and a short-distance channel is formed between the second detector and the light source, the information collected by the short-distance channel is global oscillation noise, and the information collected by the long-distance channel includes global oscillation noise and information on changes in deep brain signals; For the original fNIRS data after removing motion artifacts, the global oscillation is removed by multi-distance near-infrared spectroscopy enhancement method to obtain denoised multi-channel fNIRS data, as follows: Construct an adaptive filter, which is equivalent to using the short-distance channel hemodynamic changes to estimate the physiological interference associated with the induced hemodynamic change signal. Therefore, the long-distance channel signal removes the short-distance channel signal to obtain the hemodynamic response of the brain activity process to be sought; the deep brain signal change is expressed as:

[0015] in, represents the change in oxygenated hemoglobin concentration or deoxygenated hemoglobin concentration calculated from the information collected by the first detector, including physiological interference and induced hemodynamic changes, is the sampling time. is a column vector, represented by , is the adaptive filter order, For signal of Unit extension; Indicates the information obtained by the second detector or , which is mainly composed of physiological interference and serves as a reference signal for adaptive filtering; is the coefficient vector of the adaptive filter; represents the first coefficient of the adaptive filter coefficient vector w(k) at sampling time k. elements; It is the order of the adaptive filter, which defines the number of coefficients contained in the coefficient vector of the adaptive filter, that is, the complexity of the adaptive filter. It changes as the adaptive filter is iteratively updated to minimize the error between the adaptive filter output and the desired response.

[0016] Furthermore, the diffusion-based data enhancement module includes a continuous wavelet transform module, a VQ-VAE feature fusion module, and a guided diffusion module; For the input denoised multi-channel fNIRS data, the continuous wavelet transform module uses continuous wavelet transform (CWT) to transform the denoised multi-channel fNIRS data from the time domain to the time-frequency domain, thereby obtaining the time-frequency spectrum of the multi-channel fNIRS data. The VQ-VAE feature fusion module extracts and integrates the time, space and frequency features from the time-frequency spectrum of the multi-channel fNIRS data, and represents the time, space and frequency features as spatial latent vectors in a low-dimensional space. The guided diffusion module takes the spatial latent vector as input and uses the cross-attention mechanism to generate multi-channel fNIRS data corresponding to different motor imagery task label categories.

[0017] Furthermore, in the transformer-based intention recognition module, according to the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the temporal features and spatial features are extracted through the temporal convolutional network and the spatial convolutional network respectively, and the temporal features and spatial features are fused by the spatiotemporal feature fusion module to obtain the spatiotemporal features; A spatial transformer module is used to extract the global information of each channel in the multi-channel fNIRS data corresponding to different motor imagery task label categories and output the channel attention representation; Finally, the spatiotemporal features and channel attention representation are input into the fully connected layer to complete the classification.

[0018] Furthermore, both the temporal convolutional network and the spatial convolutional network include two convolutional layers and an adaptive average pooling layer, which respectively extract temporal information and spatial information according to the data of each channel in the input multi-channel fNIRS data corresponding to different motor imagery task label categories, and output a temporal feature vector and a spatial feature vector; The output temporal feature vector and spatial feature vector have the same shape; In the spatiotemporal feature fusion module, the input temporal feature vector and spatial feature vector are concatenated along the channel dimension, the concatenated vector is input into the convolution layer, and then the shape is adjusted using the adaptive average pooling layer to output the fused spatiotemporal feature vector containing local information.

[0019] Furthermore, in the spatial transformer module, the input is multi-channel fNIRS data corresponding to different motor imagery task label categories, which is used to extract the global information of the channels in the fNIRS signal, as follows: According to the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the data of each channel is globally averaged pooled according to the time series dimension, and the data in the time dimension is compressed into a single value to obtain the global representation of each channel. The global representation of each channel is regarded as a "token", similar to the word embedding in natural language processing, and an embedding vector is assigned to each channel; Add position embedding to each channel embedding vector; A learnable token-like structure is introduced. The token-like structure and the embedding vector of each channel are input into the self-attention layer. The attention weight matrix output by the self-attention layer is input into the feedforward network. The feedforward network outputs the channel attention representation vector containing the aggregated information of all channels.

[0020] Furthermore, in the fully connected layer, the fused spatiotemporal feature vector obtained by the spatiotemporal convolution module and the channel attention representation vector obtained by the spatial transformer module are concatenated according to the feature dimension to form a comprehensive feature vector; this vector combines the information from the two modules and aims to capture the spatiotemporal features and channel attention information of multi-channel fNIRS data corresponding to different motor imagery task label categories. The comprehensive feature vector is linearly transformed by weights and biases, and then nonlinearity is introduced by the nonlinear activation function ReLU to allow the model to capture more complex task classification boundaries. Finally, a probability distribution is output, indicating the probability that the multi-channel fNIRS data corresponding to different motor imagery task label categories belong to each motor imagery task category.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. Signal preprocessing and feature extraction: This paper uses continuous wavelet transform, spatiotemporal convolution and spatial transform network techniques to extract neural signal features from the time domain, spatial domain and frequency domain, thereby enhancing the recognition performance of motor imagery tasks in fNIRS signals.

[0022] 2. Data enhancement based on diffusion: This paper uses a data enhancement method based on the conditional diffusion model to make up for the problem of insufficient fNIRS data samples, which helps to improve the training effect of the deep learning model and improve the accuracy of cross-individual task recognition.

[0023] 3. Transformer-based motor imagery task recognition: This invention can automatically focus on important brain areas in different motor imagery tasks. By utilizing its spatial resolution and the interpretability of attention weight distribution, it provides the ability to explore the motor imagery task processing patterns in small brain areas, making the recognition more accurate and reliable, and improving the model's ability to recognize specific tasks. It is suitable for application scenarios of precise control of drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of data preprocessing process in an embodiment of the present invention; Figure 2 This is a workflow diagram of a UAV control system based on a brain-computer interface in an embodiment of the present invention; Figure 3 Schematic diagram of the structure of the transformer-based intent recognition module in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation of the present invention is described in detail below with reference to the accompanying drawings and examples.

[0026] Example: A UAV control system based on brain-computer interface, such as Figure 2 As shown, it includes a near-infrared brain function imaging device and a host computer; the host computer is provided with a preprocessing module, a diffusion-based data enhancement module and a transformer-based intention recognition module; The near-infrared brain functional imaging device acquires multi-channel fNIRS raw data, removes noise through the preprocessing module, and obtains denoised multi-channel fNIRS data; the denoised multi-channel fNIRS data is input into the diffusion-based data enhancement module to generate multi-channel fNIRS data corresponding to different motor imagery task label categories, and the data is expanded and enhanced; the transformer-based intention recognition module extracts the time domain, frequency domain and spatial domain features of the multi-channel fNIRS data corresponding to different motor imagery task label categories to identify the motor imagery task, and the upper computer generates corresponding instructions to control the drone to perform activities according to the identified motor imagery task.

[0027] Furthermore, the near-infrared brain functional imaging device was worn on the forehead to collect fNIRS raw data in resting and task states; The raw data of fNIRS are accompanied by interferences such as physiological noise and motion artifacts. Since the detection sensitivity of the fNIRS system is reflected in the ability to detect weak signals, these interferences will seriously affect the sensitivity of the system; The near-infrared brain function imaging device is equipped with n LED light sources and m detectors. One LED light source and one detector form a channel, that is, there are n×m channels to collect data, and the light intensity data of multiple channels is collected; The near-infrared brain function imaging device is provided with a control interface for controlling the power of the light source. The host computer adjusts the optical power of the light source through the control interface so that the intensity of the light signal received by the detector in the near-infrared brain function imaging device is as large as possible, but not oversaturated, thereby effectively increasing the intensity of the light signal and improving the signal-to-noise ratio; the near-infrared brain function imaging device will automatically convert the collected multi-channel light intensity data into multi-channel light intensity data.

[0028] In one embodiment, the near-infrared brain function imaging device includes 24 LED light sources and 32 detectors, one LED light source and one LED detector constitute one channel, and a total of 768 channels are used to collect data.

[0029] like Figure 1 As shown in the figure, the multi-channel fNIRS raw data is preprocessed in the preprocessing module. First, the abnormal data is eliminated, and then the noise is removed by a low-pass filter. Since the relative movement of the subject's head and the probe will cause motion artifacts, the motion artifacts are removed by discrete wavelet transform. Finally, the global oscillation is removed by the multi-distance near-infrared spectral enhancement method to obtain the denoised multi-channel fNIRS data.

[0030] Furthermore, based on the PHOEBE algorithm (Luca Pollonini, Heather Bortfeld, and John S.Oghalai, "PHOEBE: a method for real time mapping of optodes-scalp coupling infunctional near-infrared spectroscopy," Biomed. Opt. Express7, 5104-5119(2016)), the coefficient of variation of all channels was calculated to evaluate the stability of the fNIRS raw data collected by each channel and eliminate abnormal data; After removing abnormal data, a third-order Butterworth filter was used to remove noise; After noise removal, discrete wavelet transform (DWT) was used to decompose the fNIRS raw data into a set of high-frequency and low-frequency sub-signals, and motion artifacts were removed by setting thresholds of low-frequency coefficients and high-frequency coefficients.

[0031] Furthermore, in a near-infrared brain function imaging device, one light source corresponds to two detectors, wherein the distance between the first detector and the light source is longer than the distance between the second detector and the light source, a long-distance channel is formed between the first detector and the light source, and a short-distance channel is formed between the second detector and the light source, the information collected by the short-distance channel is global oscillation noise, and the information collected by the long-distance channel includes global oscillation noise and information on changes in deep brain signals; For the original fNIRS data after removing motion artifacts, the global oscillation is removed by multi-distance near-infrared spectroscopy enhancement method to obtain denoised multi-channel fNIRS data, as follows: Construct an adaptive filter, which is equivalent to using the short-distance channel hemodynamic changes to estimate the physiological interference associated with the induced hemodynamic change signal. Therefore, the long-distance channel signal removes the short-distance channel signal to obtain the hemodynamic response of the brain activity process to be sought; the deep brain signal change is expressed as:

[0032] in, represents the change in oxygenated hemoglobin concentration or deoxygenated hemoglobin concentration calculated from the information collected by the first detector, including physiological interference and induced hemodynamic changes, is the sampling time. is a column vector, represented by , is the adaptive filter order, For signal of Unit extension; Indicates the information obtained by the second detector or , which is mainly composed of physiological interference and serves as a reference signal for adaptive filtering; is the coefficient vector of the adaptive filter; represents the first coefficient of the adaptive filter coefficient vector w(k) at sampling time k. elements; It is the order of the adaptive filter, which defines the number of coefficients contained in the coefficient vector of the adaptive filter, that is, the complexity of the adaptive filter. It changes as the adaptive filter is iteratively updated to minimize the error between the adaptive filter output and the desired response.

[0033] Furthermore, the diffusion-based data enhancement module includes a continuous wavelet transform module, a VQ-VAE feature fusion module, and a guided diffusion module; For the input denoised multi-channel fNIRS data, the continuous wavelet transform module uses continuous wavelet transform (CWT) to transform the denoised multi-channel fNIRS data from the time domain to the time-frequency domain, thereby obtaining the time-frequency spectrum of the multi-channel fNIRS data. The VQ-VAE feature fusion module extracts and integrates the time, space and frequency features from the time-frequency spectrum of the multi-channel fNIRS data, and represents the time, space and frequency features as spatial latent vectors in a low-dimensional space. The guided diffusion module takes the spatial latent vector as input and uses the cross-attention mechanism to generate multi-channel fNIRS data corresponding to different motor imagery task label categories.

[0034] In one embodiment, a VQ-VAE feature fusion module is constructed according to the prior art (Van Den Oord A, Vinyals O. Neuraldiscrete representation learning[J]. Advances in neural informationprocessing systems, 2017, 30.), and the time-frequency spectrum of multi-channel fNIRS data is used as input. The low-dimensional information in the signal is extracted by capturing features in the spatial domain, time domain, and frequency domain, and represented as spatial latent variables in a low-dimensional latent space.

[0035] The VQ-VAE feature fusion module is adopted, which extracts and integrates time, space and frequency features from the multi-channel time-frequency spectrum and represents these features as spatial latent vectors in a low-dimensional space. VQ-VAE is a generative model that introduces vector quantization (VQ) on the basis of VAE to improve the generation quality and stability of VAE. The basic idea of ​​VQ-VAE is to map the input data to a finite discrete latent space and use a vector quantizer to quantize the continuous latent vector into a discrete latent code. The encoder is responsible for mapping the multi-channel time-frequency spectrum to a continuous low-dimensional latent space. Then, the VQ layer maps the continuous latent vector from the encoder to a discrete embedding space. The embedding space consists of a set of discrete atomic vectors, each of which represents a cluster center in the latent space. In the VQ layer, the continuous vector output by the encoder is quantized to the nearest atomic vector by finding the nearest cluster center. In addition, at last, the decoder reconstructs the time-frequency spectrum from the quantized embedding space.

[0036] In one embodiment, a guided diffusion module is constructed according to the prior art (Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processingsystems, 2021, 34: 8780-8794.), and the architecture of the guided diffusion model is adopted. The spatial latent variables are used as input, and the cross-attention mechanism is used to generate fNIRS data of a specific label category.

[0037] like Figure 3 As shown, in the transformer-based intention recognition module, according to the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the temporal features and spatial features are extracted respectively through the temporal convolutional network and the spatial convolutional network, and the temporal features and spatial features are fused by the spatiotemporal feature fusion module to obtain the spatiotemporal features; A spatial transformer module is used to extract the global information of each channel in the multi-channel fNIRS data corresponding to different motor imagery task label categories and output the channel attention representation; Finally, the spatiotemporal features and channel attention representation are input into the fully connected layer to complete the classification.

[0038] HbO data can achieve better motor intention decoding performance than HbR data. Therefore, in one embodiment, HbO data in multi-channel fNIRS data corresponding to different motor imagery task label categories is used for subsequent decoding analysis.

[0039] Furthermore, both the temporal convolutional network and the spatial convolutional network include two convolutional layers and an adaptive average pooling layer, which respectively extract temporal information and spatial information according to the data of each channel in the input multi-channel fNIRS data corresponding to different motor imagery task label categories, and output a temporal feature vector and a spatial feature vector; The output temporal feature vector and spatial feature vector have the same shape; In one embodiment, a temporal convolutional network is constructed according to the prior art (Lawhern et al., "EEGNet: A CompactConvolutional Neural Network for EEG-based Brain-Computer Interfaces," JNeural Eng, 2018), and the structure is as follows: The first convolution layer: convolution kernel size 1 8, step size 1 4. Extract the time-related features of each channel and maintain the channel dimension; The second convolution layer: the convolution kernel size is 1 4, step size is 1 2. Further extract the time series related features of fNIRS data; Adaptive average pooling layer: Through adaptive average pooling, both the time step and the spatial step are reduced to a fixed size of 4×4.

[0040] In one embodiment, a spatial convolutional network is constructed according to the prior art (Lawhern et al., "EEGNet: A CompactConvolutional Neural Network for EEG-based Brain-Computer Interfaces," JNeural Eng, 2018), and the structure is as follows: The first convolution layer: convolution kernel size 8 1, step size is 4 1. Extract spatial features between different channels and maintain the time dimension.

[0041] The second convolution layer: the convolution kernel size is 4 1, step size is 2 1. Further extract the spatial features between channels.

[0042] Adaptive average pooling layer: reduces the spatial and temporal steps to 4 4.

[0043] In the spatiotemporal feature fusion module, the input temporal feature vector and spatial feature vector are concatenated along the channel dimension, the concatenated vector is input into the convolution layer, and then the shape is adjusted using the adaptive average pooling layer to output the fused spatiotemporal feature vector containing local information.

[0044] In one embodiment, a spatial transformer module is constructed according to the prior art (A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.), and the input is multi-channel fNIRS data corresponding to different motor imagery task label categories. Combined with the prior art (Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017.), the global information of the channels in the fNIRS signal is extracted, as follows: According to the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the data of each channel is globally averaged pooled according to the time series dimension, and the data in the time dimension is compressed into a single value to obtain the global representation of each channel. The global representation of each channel is regarded as a "token", similar to the word embedding in natural language processing, and an embedding vector is assigned to each channel; Add position embedding to each channel embedding vector; A learnable token-like structure is introduced. The token-like structure and the embedding vector of each channel are input into the self-attention layer. The attention weight matrix output by the self-attention layer is input into the feedforward network. The feedforward network outputs the channel attention representation vector containing the aggregated information of all channels.

[0045] Furthermore, in the fully connected layer, the fused spatiotemporal feature vector obtained by the spatiotemporal convolution module and the channel attention representation vector obtained by the spatial transformer module are concatenated according to the feature dimension to form a comprehensive feature vector; this vector combines the information from the two modules and aims to capture the spatiotemporal features and channel attention information of multi-channel fNIRS data corresponding to different motor imagery task label categories. The comprehensive feature vector is linearly transformed by weights and biases, and then nonlinearity is introduced by the nonlinear activation function ReLU to allow the model to capture more complex task classification boundaries. Finally, a probability distribution is output, indicating the probability that the multi-channel fNIRS data corresponding to different motor imagery task label categories belong to each motor imagery task category.

[0046] The preferred embodiments of the present application disclosed above are only used to help understand the present invention and its core ideas. For those skilled in the art, according to the ideas of the present invention, there will be changes in specific application scenarios and implementation operations, and this specification should not be interpreted as limiting the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A UAV control system based on brain-computer interface, characterized in that: It includes a near-infrared brain function imaging device and a host computer; the host computer is provided with a preprocessing module, a diffusion-based data enhancement module and a transformer-based intention recognition module; The near-infrared brain functional imaging device acquires multi-channel fNIRS raw data, removes noise through the preprocessing module, and obtains denoised multi-channel fNIRS data; the denoised multi-channel fNIRS data is input into the diffusion-based data enhancement module to generate multi-channel fNIRS data corresponding to different motor imagery task label categories, and the data is expanded and enhanced; the transformer-based intention recognition module extracts the time domain, frequency domain and spatial domain features of the multi-channel fNIRS data corresponding to different motor imagery task label categories to identify the motor imagery task, and the host computer generates corresponding instructions to control the drone to perform activities based on the identified motor imagery task.

2. The UAV control system based on brain-computer interface according to claim 1 is characterized in that: The near-infrared brain functional imaging device is worn on the forehead to collect fNIRS raw data in resting and task states; The near-infrared brain function imaging device is equipped with n LED light sources and m detectors. One LED light source and one detector form a channel, that is, there are n×m channels to collect data, and the light intensity data of multiple channels is collected; The near-infrared brain function imaging device is provided with a control interface for controlling the light source power. The host computer adjusts the light power of the light source through the control interface so that the intensity of the light signal received by the detector in the near-infrared brain function imaging device is as large as possible, but not oversaturated; the near-infrared brain function imaging device will automatically convert the collected multi-channel light intensity data into multi-channel light intensity data.

3. The UAV control system based on brain-computer interface according to claim 1 is characterized in that: The multi-channel fNIRS raw data are preprocessed in the preprocessing module. First, the abnormal data are eliminated, then the noise is removed by a low-pass filter, and then the motion artifacts are removed by discrete wavelet transform. Finally, the global oscillation is removed by the multi-range near-infrared spectral enhancement method to obtain the denoised multi-channel fNIRS data.

4. The UAV control system based on brain-computer interface according to claim 3 is characterized in that: The coefficient of variation of all channels was calculated based on the PHOEBE algorithm to evaluate the stability of the fNIRS raw data collected by each channel and eliminate abnormal data; After removing abnormal data, a third-order Butterworth filter was used to remove noise; After noise removal, discrete wavelet transform (DWT) was used to decompose the fNIRS raw data into a set of high-frequency and low-frequency sub-signals, and motion artifacts were removed by setting thresholds of low-frequency coefficients and high-frequency coefficients.

5. The UAV control system based on brain-computer interface according to claim 4 is characterized in that: In the near-infrared brain function imaging device, the same light source corresponds to two detectors, wherein the distance between the first detector and the light source is longer than the distance between the second detector and the light source, the first detector and the light source form a long-distance channel, and the second detector and the light source form a short-distance channel. The information collected by the short-distance channel is global oscillation noise, and the information collected by the long-distance channel includes global oscillation noise and information on changes in deep brain signals; For the original fNIRS data after removing motion artifacts, the global oscillation is removed by multi-distance near-infrared spectroscopy enhancement method to obtain denoised multi-channel fNIRS data, as follows: Construct an adaptive filter, which is equivalent to using the short-distance channel hemodynamic changes to estimate the physiological interference associated with the induced hemodynamic change signal. Therefore, the long-distance channel signal removes the short-distance channel signal to obtain the hemodynamic response of the brain activity process to be sought; the deep brain signal change is expressed as: in, represents the change in oxygenated hemoglobin concentration or deoxygenated hemoglobin concentration calculated from the information collected by the first detector, including physiological interference and induced hemodynamic changes, is the sampling time; is a column vector, represented by , is the adaptive filter order, For signal of Unit extension; Indicates the information obtained by the second detector or , which is mainly composed of physiological interference and serves as a reference signal for adaptive filtering; is the coefficient vector of the adaptive filter; represents the first coefficient of the adaptive filter coefficient vector w(k) at sampling time k. elements; is the order of the adaptive filter.

6. The UAV control system based on brain-computer interface according to claim 1 is characterized in that: The diffusion-based data enhancement module includes a continuous wavelet transform module, a VQ-VAE feature fusion module, and a guided diffusion module; For the input denoised multi-channel fNIRS data, the continuous wavelet transform module uses continuous wavelet transform (CWT) to transform the denoised multi-channel fNIRS data from the time domain to the time-frequency domain, thereby obtaining the time-frequency spectrum of the multi-channel fNIRS data. The VQ-VAE feature fusion module extracts and integrates the time, space and frequency features from the time-frequency spectrum of the multi-channel fNIRS data, and represents the time, space and frequency features as spatial latent vectors in a low-dimensional space. The guided diffusion module takes the spatial latent vector as input and uses the cross-attention mechanism to generate multi-channel fNIRS data corresponding to different motor imagery task label categories.

7. The UAV control system based on brain-computer interface according to claim 1 is characterized in that: In the transformer-based intention recognition module, according to the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the temporal features and spatial features are extracted through the temporal convolutional network and the spatial convolutional network respectively, and the temporal features and spatial features are fused by the spatiotemporal feature fusion module to obtain the spatiotemporal features. A spatial transformer module is used to extract the global information of each channel in the multi-channel fNIRS data corresponding to different motor imagery task label categories and output the channel attention representation; Finally, the spatiotemporal features and channel attention representation are input into the fully connected layer to complete the classification.

8. The UAV control system based on brain-computer interface according to claim 7 is characterized in that: Both the temporal convolutional network and the spatial convolutional network include two convolutional layers and an adaptive average pooling layer. According to the data of each channel in the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the temporal information and spatial information are extracted respectively, and the temporal feature vector and the spatial feature vector are output; The output temporal feature vector and spatial feature vector have the same shape; In the spatiotemporal feature fusion module, the input temporal feature vector and spatial feature vector are concatenated along the channel dimension, the concatenated vector is input into the convolution layer, and then the shape is adjusted using the adaptive average pooling layer to output the fused spatiotemporal feature vector containing local information.

9. The UAV control system based on brain-computer interface according to claim 8, characterized in that: In the spatial transformer module, the input is multi-channel fNIRS data corresponding to different motor imagery task label categories, which is used to extract the global information of the channels in the fNIRS signal, as follows: According to the input multi-channel fNIRS data corresponding to different motor imagery task label categories, the data of each channel is globally averaged pooled according to the time series dimension, and the data in the time dimension is compressed into a single value to obtain the global representation of each channel. The global representation of each channel is regarded as a "token", similar to the word embedding in natural language processing, and an embedding vector is assigned to each channel; Add position embedding to each channel embedding vector; A learnable token-like structure is introduced. The token-like structure and the embedding vector of each channel are input into the self-attention layer. The attention weight matrix output by the self-attention layer is input into the feedforward network. The feedforward network outputs the channel attention representation vector containing the aggregated information of all channels.

10. The UAV control system based on brain-computer interface according to claim 9, characterized in that: In the fully connected layer, the fused spatiotemporal feature vector obtained by the spatiotemporal convolution module and the channel attention representation vector obtained by the spatial transformer module are concatenated according to the feature dimension to form a comprehensive feature vector; the comprehensive feature vector is linearly transformed through weights and biases, and then nonlinearity is introduced through the nonlinear activation function ReLU, and finally a probability distribution is output, indicating the probability that the multi-channel fNIRS data corresponding to different motor imagery task label categories belong to each motor imagery task category.

Images