Brain-controlled mechanical arm system based on augmented reality and convolutional neural network

Abstract

The invention discloses a brain-controlled mechanical arm system based on augmented reality and a convolutional neural network, and relates to the technical field of information control, and the technical scheme is characterized in that the brain-controlled mechanical arm system comprises a wireless electroencephalogram acquisition device, AR glasses, an NX2 development board, a router and a mechanical arm; the AR glasses comprise an SSVEP stimulation interface module; the NX2 development board comprises an SSVEP (Steady-State Visual Evoked Potential) electroencephalogram signal convolutional neural network processing module; the router is in communication connection with the wireless electroencephalogram collection device, the AR glasses and the NX2 development board in a wireless connection mode, and the router is connected with the mechanical arm in a wired connection mode. The system realizes wireless communication connection with the wireless electroencephalogram acquisition equipment, the AR glasses and the NX2 development board through the router, realizes wired connection with the mechanical arm, and is used for solving the problems of poor portability and low SSVEP electroencephalogram recognition accuracy of an existing system.

Description

Technical Field

[0001] The present invention relates to the field of information control technology, and more specifically, to a brain-controlled robotic arm system based on augmented reality and convolutional neural network. Background Art

[0002] With the rapid development of information technology and biomedical engineering, brain-computer interface (BCI) technology has gradually become one of the research hotspots. BCI systems enable direct communication between the human brain and external devices without relying on traditional muscle or nerve pathways. This technology has shown great potential in assisting people with disabilities to restore their motor skills and improve their quality of life. In recent years, with the advancement of augmented reality (AR) technology and its combined application with BCI systems, new development opportunities have been brought to brain control technology.

[0003] Currently, visually evoked potential-based BCI (VEP-BCI) is one of the important ways to achieve non-invasive brain control operations. Among them, steady-state visual evoked potentials (SSVEP) have attracted widespread attention due to their high information transmission rate, low training requirements and strong robustness.

[0004] However, existing SSVEP-BCI systems generally have the following problems: most existing SSVEP-BCI systems are large in size and inconvenient to carry, which limits their application in daily life; in addition, due to the weakness and complexity of electroencephalogram (EEG) signals, there are still many challenges in effectively extracting useful information and accurately classifying it, which directly affects the accuracy and reliability of the system.

[0005] Therefore, the present invention aims to provide a brain-controlled robotic arm system based on augmented reality and convolutional neural network to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a brain-controlled robotic arm system based on augmented reality and convolutional neural network. The present invention realizes wireless communication connection with a router, a wireless EEG acquisition device, AR glasses and an NX2 development board, and realizes a wired connection with the robotic arm, so as to solve the problems of poor portability of the existing system and low accuracy of SSVEP EEG recognition.

[0007] The above technical objectives of the present invention are achieved through the following technical solutions: a brain-controlled robotic arm system based on augmented reality and convolutional neural network, including a wireless EEG acquisition device, AR glasses, an NX2 development board, a router and a robotic arm;

[0008] The AR glasses include an SSVEP stimulation interface module, which is deployed on the AR glasses and induces EEG signals through stimulation of the stimulation interface;

[0009] The NX2 development board includes a SSVEP EEG signal convolutional neural network processing module, and the SSVEP EEG signal convolutional neural network processing module is used to process EEG signals collected by a wireless EEG acquisition device;

[0010] The router is connected to the wireless EEG acquisition device, the AR glasses and the NX2 development board by wireless connection, and is connected to the robotic arm by wired connection.

[0011] The present invention also provides a control method for a brain-controlled robotic arm system based on augmented reality and convolutional neural network, comprising the following steps:

[0012] S1. Use Unity to create a SSVEP stimulation paradigm, deploy the created SSVEP paradigm to the Hololens 2 AR glasses, and send SSVEP stimulation to the user through the AR glasses;

[0013] S2, receive the EEG data collected by the wireless EEG acquisition device through the NX2 development board, and use NX2 for subsequent data preprocessing, feature extraction and convolution neural network training;

[0014] S3, remove the power frequency interference from the EEG signal, and then remove the baseline drift and other noise interference;

[0015] S4, collecting the user's EEG signal, and extracting the frequency domain signal in the EEG signal using spectral entropy;

[0016] S5. Use convolutional neural network-long short-term memory network to classify the data after feature extraction;

[0017] S6. Output the obtained classification and send it to the robotic arm to realize the control of the robotic arm.

[0018] The present invention is further configured as follows: the process of extracting the frequency domain signal from the EEG signal using spectral entropy in step S4 is as follows: using fast Fourier transform to convert the preprocessed time domain SSVEP EEG signal into a frequency domain signal, extracting the main frequency components of the signal, and calculating the probability density distribution of the spectrum by normalizing the energy of each frequency component. The probability density distribution of the spectrum p iThe calculation formula is as follows:

[0019]

[0020] where |X(i)| 2 is the energy of the ith frequency component, N is the total number of frequency components, |X(j)| 2 It is the square of the modulus of the complex value of the signal at the jth frequency component obtained by Fourier transform.

[0021] The present invention is further configured as follows: the calculation formula of the spectral entropy H in step S4 is as follows:

[0022]

[0023] Among them, p i It represents the probability density of the i-th frequency component, and N represents the total number of frequency components. Spectral entropy is used to measure the complexity and randomness of the signal spectrum. The higher the spectral entropy value, the more complex the signal spectrum is.

[0024] In summary, the present invention has the following beneficial effects:

[0025] 1. The present invention adopts wireless EEG acquisition equipment and AR glasses to achieve high portability of the system. Users can easily use it in a variety of scenarios without complicated setting process, which greatly improves the practicality and convenience of the system;

[0026] 2. The present invention uses spectral entropy feature extraction and deep learning algorithm to improve the accuracy of the model: The present invention adds spectral entropy feature extraction in the preprocessing stage, extracts more representative features by performing frequency domain analysis on EEG signals, and then uses a deep learning algorithm combining convolutional neural network (CNN) and long short-term memory network (LSTM) to efficiently classify the denoised EEG signals. This method not only significantly improves the accuracy of SSVEP EEG recognition, but also maintains high-precision EEG recognition capabilities in complex environments. Within a 1s time window, the accuracy of the traditional CCA algorithm is only 55.3±10.8%, while the accuracy of this algorithm reaches 88.4±5.6%, ensuring the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic flow chart of the method for controlling a robotic arm with a brain in Embodiment 1 of the present invention;

[0028] Figure 2 It is a flowchart of data preprocessing and signal processing using deep learning in Example 2 of the present invention. DETAILED DESCRIPTION

[0029] The following is combined with Figure 1-2The present invention is described in further detail.

[0030] Embodiment 1: A brain-controlled robotic arm system based on augmented reality and convolutional neural network, comprising a wireless EEG acquisition device, AR glasses, an NX2 development board, a router and a robotic arm;

[0031] In this embodiment, the AR glasses further include an SSVEP stimulation interface module, which is deployed on the AR glasses and induces EEG signals through stimulation of the stimulation interface; the NX2 development board further includes an SSVEP EEG signal convolutional neural network processing module, which is used to process EEG signals collected by the wireless EEG acquisition device;

[0032] The router communicates with the wireless EEG acquisition device, AR glasses and NX2 development board through wireless connection, and is connected to the robotic arm through wired connection.

[0033] Embodiment 2: A control method for a brain-controlled robotic arm system based on augmented reality and convolutional neural network

[0034] 1) Use Unity to create the SSVEP stimulation paradigm, deploy the prepared SSVEP paradigm to the Hololens 2 augmented reality glasses, and send SSVEP stimulation to the user through the AR glasses.

[0035] 2) Receive the EEG data collected by the wireless EEG acquisition device through the NX2 development board, and use NX2 for subsequent data preprocessing, feature extraction, and convolutional neural network training.

[0036] 3) Remove the power frequency interference from the EEG signal, and then remove the baseline drift and other noise interference.

[0037] 4) Collect the user's EEG signal and use spectral entropy to extract the frequency domain signal in the EEG signal.

[0038] 5) Use convolutional neural network-long short-term memory network to classify the data after feature extraction.

[0039] 6) The obtained classification output is sent to the robotic arm to realize the control of the robotic arm.

[0040] Spectral entropy is a feature that measures the complexity and randomness of a signal spectrum, and can provide important information about the signal structure and pattern. The preprocessed time-domain SSVEP EEG signal is converted into a frequency-domain signal using a fast Fourier transform (FFT). FFT can quickly convert a time-domain signal into a frequency-domain signal and extract the main frequency components of the signal. In this embodiment, six SSVEP flickering stimuli are used, with a frequency range of 12.5 Hz to 15 Hz and a frequency interval of 0.5 Hz. Therefore, it is necessary to extract frequency-domain features within the range of 12.5 Hz to 15 Hz and calculate the probability density distribution p of the spectrum. i , the probability density distribution of the spectrum can be obtained by normalizing the energy of each frequency component:

[0041]

[0042] where |X(i)| 2 is the energy of the ith frequency component, N is the total number of frequency components, |X(j)| 2 It is the square of the modulus of the complex value of the signal at the jth frequency component obtained by Fourier transform.

[0043] Calculate the spectral entropy H using the entropy formula:

[0044]

[0045] Among them, p i represents the probability density of the i-th frequency component, and N represents the total number of frequency components; the spectral entropy H measures the complexity and randomness of the signal spectrum. The higher the spectral entropy value, the more complex the spectrum of the signal. In this embodiment, the calculated spectral entropy value is used as part of the feature vector. In order to further enrich the feature representation, the spectral entropy is combined with the energy features of different frequencies in the range of 12.5 Hz to 15 Hz to form a multi-feature vector:

[0046] F=[H,E 12.5 ,E 13 ,E 13.5 ,E 14 ,E 14.5 ,E 15 ]

[0047] Among them, E 12.5 ,E 13 ,E 13.5 ,E 14 ,E 14.5 ,E 15 They are the energy characteristics of frequencies of 12.5Hz, 13.0Hz, 13.5Hz, 14.0Hz, 14.5Hz and 15.0Hz respectively.

[0048] The electrode channels used in this embodiment are PO3, PO4, POz, O1, O2 and Oz, a total of 6 channels, each of which has a corresponding feature vector F i (i=1,2,3,4,5,6). Concatenate the feature vectors of all channels into a large feature vector F in order. total :

[0049] F total =[F PO3 ,F PO4 ,F POz ,F O1 ,F O2 ,F Oz ] T

[0050] For the original SSVEP EEG signal data, first extract the features of the data and use the extracted features as F total The input is provided to the CNN-LSTM network, where the original SSVEP EEG data contains N ch channels, each channel has P signal sampling points, and after feature extraction, the number of sampling points of each channel is N fc , there are T time steps.

[0051] The LSTM network is specially designed to prevent gradient explosion or gradient vanishing when a neural network with a given input and output cycles in a feedback loop. In addition, long-sequence EEG signals are also prone to gradient explosion and gradient vanishing problems during convolutional neural network training. Therefore, the long-sequence EEG signals are divided into multiple short sequences according to the time window for convolutional neural network training, and the recognition results of multiple short sequences are jointly output in combination with the LSTM network to improve the problems of gradient descent and gradient vanishing in the model.

[0052] The long-short time convolutional neural network is composed of two layers of convolutional neural network combined with one layer of LSTM network. The structure of the long-short time convolutional neural network is as follows: Figure 2 As shown, Figure 2 The final classification result is obtained by using t-SNE to visualize the six-category dataset collected independently. t 、m t+1 、m t+S-1 are the network input values ​​at time t, t+1, and t+S-1 respectively, and f t is the sequence after spatial feature extraction, h t-1 is information from the previous time period.

[0053] The CNN_LSTM network is mainly composed of four network layers: an input layer for inputting the SSVEP EEG signal data after preprocessing and feature extraction; two convolutional layers for spatial feature extraction of SSVEP EEG signals; an LSTM layer for temporal feature extraction of SSVEP EEG signals, and a fully connected output layer.

[0054] The first convolution layer uses a small 3×3 convolution kernel and the number of output channels is set to 32, which can extract rich local features from the original input data. After the convolution operation, the size of the output feature matrix is ​​kept unchanged by padding, that is, (T, 32, N ch , N fc ), the second convolution layer also uses a 3×3 convolution kernel, but the number of output channels is increased to 64, further deepening the feature extraction process, and the size of the output feature matrix remains at (T, 64, N ch , N fc ).

[0055] In this embodiment, in order to speed up the training process and reduce the impact of internal covariate shift, batch normalization technology is applied after each layer of convolution operation to ensure that the data distribution of each batch is close to the standard normal distribution. The ReLU activation function that follows it introduces nonlinear factors to improve the model's ability to capture complex patterns, while also avoiding the problem of gradient disappearance. In addition, in order to prevent the model from overfitting, a Dropout layer is configured after each group of convolution layers to randomly inactivate half of the neurons, effectively improving the generalization performance of the model.

[0056] In the back-end part of the network, the LSTM layer is responsible for processing the high-dimensional feature vector obtained through forward propagation and capturing the time series characteristics contained therein. In this embodiment, the number of hidden units of the LSTM layer is set to 500, which is sufficient to capture the temporal dynamic changes of the SSVEP signal. Finally, the fully connected layer maps the features output by the LSTM layer to an N-dimensional space, where N corresponds to the total number of target categories, thus realizing the conversion from features to category labels. This process not only simplifies the decision boundary, but also improves the accuracy of classification.

[0057] Output layer: The output layer of the network consists of K units, where K units are equal to the number of SSVEP EEG signals to be classified in the input data. The output layer is equipped with a softmax function to output the probability that a given input segment belongs to a specific category. The last layer of the network is composed of a fully connected network layer. The entire network neural network before can be used as a feature extractor, and feature classification using activation functions can also be used as the result output to output the extracted features.

[0058] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A brain-controlled robotic arm system based on augmented reality and convolutional neural network, characterized by: Including wireless EEG acquisition equipment, AR glasses, NX2 development board, router and robotic arm; The AR glasses include an SSVEP stimulation interface module, which is deployed on the AR glasses and induces EEG signals through stimulation of the stimulation interface; The NX2 development board includes a SSVEP EEG signal convolutional neural network processing module, and the SSVEP EEG signal convolutional neural network processing module is used to process EEG signals collected by a wireless EEG acquisition device; The router is connected to the wireless EEG acquisition device, the AR glasses and the NX2 development board by wireless connection, and is connected to the robotic arm by wired connection.

2. The control method of a brain-controlled robotic arm system based on augmented reality and convolutional neural network according to claim 1, characterized in that: The following steps are involved: S1. Use Unity to create a SSVEP stimulation paradigm, deploy the created SSVEP paradigm to the Hololens 2 AR glasses, and send SSVEP stimulation to the user through the AR glasses; S2, receive the EEG data collected by the wireless EEG acquisition device through the NX2 development board, and use NX2 for subsequent data preprocessing, feature extraction and convolution neural network training; S3, remove the power frequency interference from the EEG signal, and then remove the baseline drift and other noise interference; S4, collecting the user's EEG signal, and extracting the frequency domain signal in the EEG signal using spectral entropy; S5. Use convolutional neural network-long short-term memory network to classify the data after feature extraction; S6. Output the obtained classification and send it to the robotic arm to realize the control of the robotic arm.

3. The control method of a brain-controlled robotic arm system based on augmented reality and convolutional neural network according to claim 2, characterized in that: The process of extracting the frequency domain signal from the EEG signal using spectral entropy in step S4 is as follows: using fast Fourier transform to convert the preprocessed time domain SSVEP EEG signal into a frequency domain signal, extracting the main frequency components of the signal, and calculating the probability density distribution of the spectrum by normalizing the energy of each frequency component. The probability density distribution of the spectrum p i The calculation formula is as follows: where |X(i)| 2 is the energy of the ith frequency component, N is the total number of frequency components, |X(j)| 2 It is the square of the modulus of the complex value of the signal at the jth frequency component obtained by Fourier transform.

4. The control method of a brain-controlled robotic arm system based on augmented reality and convolutional neural network according to claim 2, characterized in that: The calculation formula of the spectral entropy H in step S4 is as follows: Among them, p i It represents the probability density of the i-th frequency component, and N represents the total number of frequency components. Spectral entropy is used to measure the complexity and randomness of the signal spectrum. The higher the spectral entropy value, the more complex the signal spectrum is.

Images