An asynchronous control method for a robotic arm based on SSVEP

Abstract

An asynchronous control method for a robotic arm based on SSVEP realizes the asynchronous control of the robotic arm by the user through offline training, multi-level stimulation pages, data acquisition, data preprocessing, robotic arm startup, intention recognition, feature classification, and a robotic arm control system. The system uses alpha to achieve the state control of the robotic arm system, and combines the fusion decision of the CCA coefficient and the power spectral density to realize the discrimination between the robotic arm control state and the idle state, which is more in line with the user's operation, effectively improves the efficiency of the user controlling the robotic arm through SSVEP, and increases the safety and reliability of robotic arm manipulation.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interfaces, and relates to a method for asynchronous control of a robotic arm based on SSVEP. Background Art

[0002] A brain-computer interface (BCI) system can directly detect neural activities in the brain and convert them into outputs, directly detecting a person's action thoughts from the brain and transmitting the information directly into a machine. It realizes direct communication between the brain and the computer, is a new type of interaction means, can provide a bridge for external communication for patients with limb disabilities, stroke, etc., and has important practical significance for their ability to resume normal life.

[0003] A brain-computer interface system mainly consists of units such as an electroencephalogram acquisition module, signal processing and decoding (preprocessing, feature extraction, classification and recognition), and control command output. The control methods of a brain-computer interface system are mainly divided into two types: synchronous control and asynchronous control. Most common brain-computer interface systems are synchronous control. Synchronous control requires the control signal to be sent within a specified interval, which is very inconvenient for use in real scenarios. Therefore, the research on asynchronous control systems is very necessary and is also an important bridge for the transformation of brain-computer interface systems from theory to reality.

[0004] Steady-state visual evoked potential (SSVEP) is mainly distributed in the occipital region of the occipital lobe of the brain, and is an electroencephalogram signal generated when a subject is subjected to continuous visual stimulation of an external fixed frequency. The SSVEP paradigm has become one of the most popular brain-computer interface paradigms due to its excellent electroencephalogram signal-to-noise ratio, relative immunity to electrooculogram artifacts, and the fact that the subject does not need to be pre-trained, etc. Summary of the Invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a method for asynchronous control of a robotic arm based on SSVEP, which can synchronously collect electroencephalogram signals, perform real-time classification, control state discrimination, instruction sending, and feedback control; use alpha to realize the state control of the robotic arm system, and combine the fusion decision of CCA coefficient and power spectral density to realize the discrimination between the control state and the idle state of the robotic arm, which is more in line with the user's operation, effectively improves the efficiency of the user controlling the robotic arm through SSVEP, and increases the safety and reliability of the robotic arm operation.

[0006] The technical solution adopted by the present invention to solve its technical problems is:

[0007] A method for asynchronous control of a robotic arm based on SSVEP includes the following steps:

[0008] 1) Offline training: In the offline state, collect the EEG data of the subject in the idle gaze state, the alpha wave signal, and find the optimal combination of processing channels, frequency bands, and threshold information;

[0009] 2) Multi-level stimulation page: It has multiple layers of stimulation interfaces, and there are targets flashing at different frequencies on each stimulation interface. The control system identifies the target stared at by the subject through the EEG data of the subject;

[0010] 3) Data acquisition: According to the international standard 10 / 20 of EEG electrode positions, collect the data of 9 channels in the occipital region of the brain and transmit the signals back to the processing end;

[0011] 4) Preprocessing: According to the offline training results, perform corresponding windowing segmentation on the data, select the data of the corresponding combined channels for processing, and perform band-pass filtering and notch filtering;

[0012] 5) Feature extraction: Extract the corresponding features of the collected EEG signals through the Canonical Correlation Analysis (CCA) algorithm and the combination of Power Spectral Density (PSD);

[0013] 6) Robotic arm startup: When the collected alpha wave signal reaches the threshold set in the offline experiment, start the robotic arm and display the robotic arm as being on in the stimulation interface;

[0014] 7) Intention discrimination: Identify whether the subject is in an idle state or a gaze state. If it is a gaze state, generate a control signal;

[0015] 8) Signal classification: Classify and identify the EEG signals, find the target block stared at by the subject, convert it into a control signal, and send it to the robotic arm control system;

[0016] 9) Robotic arm control: Select the robotic arm control method, convert the control command into a robotic arm motion signal, and complete the corresponding actions.

[0017] Furthermore, the process of step 2) is as follows:

[0018] A. The first layer is the selection of the working mode of the robotic arm. The working modes of the robotic arm are divided into Cartesian coordinate system control and joint angle control. Target block 1 represents Cartesian coordinate system control, target block 2 represents joint angle control, Y represents confirmation, and N represents cancellation. There will be feedback on the screen after each selection; After selecting the working mode, the confirmation button needs to be selected. If cancellation is selected, the working mode needs to be selected again. After successfully selecting the working mode, send the working mode selection signal to the robotic arm control system;

[0019] B. The second layer is the control layer of the robotic arm. The robotic arm control page in the Cartesian coordinate system working mode is shown in the figure; X+, Y+, and Z+ respectively represent the movements in the positive directions of X, Y, and Z, and X-, Y-, and Z- respectively represent the movements in the negative directions of X, Y, and Z; the green button and the red button respectively represent the on and off states of the robotic arm. When the robotic arm is on, only the green button is displayed and the red button is not displayed. When it is off, only the red button is displayed and the green button is not displayed.

[0020] C. The flashing frequencies of the target blocks in the first layer are 9Hz, 10Hz, 11Hz, and 12Hz respectively, and the flashing frequencies of the target blocks in the second layer are 8Hz, 9Hz, 10Hz, 11Hz, 12Hz, and 13Hz respectively. The stimulus is generated using the sampling sine modulation method:

[0021]

[0022] s: The brightness of the target block, where 0 represents the lowest brightness and 1 represents the maximum brightness;

[0023] f: The flashing frequency of the target block;

[0024] i: The frame number sequence number of the current screen flashing;

[0025] R: The flashing frequency of the screen, which is 60Hz in the LCD screen.

[0026] The beneficial effects of the present invention are mainly manifested in: effectively improving the efficiency of the user to control the robotic arm through SSVEP, and increasing the safety and reliability of the robotic arm operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the BCI system.

[0028] Figure 2 It is the logic flow chart of the present invention.

[0029] Figure 3 It is the first layer page of the multi-level stimulus page of the present invention.

[0030] Figure 4 It is the second layer page of the multi-level stimulus page of the present invention.

[0031] Figure 5 It is the brain electrode placement position diagram of the present invention.

[0032] Figure 6 It is the robotic arm control system diagram of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0033] The present invention will be further described below in conjunction with the drawings.

[0034] Reference Figures 1 to 6 , a method for asynchronous control of a robotic arm based on SSVEP, comprising the following steps:

[0035] 1) Offline training: In the offline state, collect the electroencephalogram (EEG) data of the subject in the idle gaze state, the alpha wave signal, and find information such as the optimal processing channel combination, frequency band, threshold, etc.;

[0036] 2) Multi-level stimulation page: Have multiple layers of stimulation interfaces, and there are targets flashing at different frequencies on each stimulation interface. The control system identifies the target stared at by the subject through the EEG data of the subject;

[0037] 3) Data acquisition: According to the international standard 10 / 20 of EEG electrode positions, collect the data of 9 channels in the occipital region of the brain and transmit the signal back to the processing end;

[0038] 4) Preprocessing: According to the offline training results, perform corresponding windowing segmentation on the data, select the data of the corresponding combined channels for processing, and perform band-pass filtering and notch filtering;

[0039] 5) Feature extraction: Extract the corresponding features of the collected EEG signals through the Canonical Correlation Analysis (CCA) algorithm and combined with the Power Spectral Density (PSD);

[0040] 6) Robotic arm startup: When the collected alpha wave signal reaches the threshold set in the offline experiment, start the robotic arm and display the robotic arm as the on state on the stimulation interface;

[0041] 7) Intention discrimination: Identify whether the subject is in the idle state or the gaze state. If it is the gaze state, generate a control signal;

[0042] 8) Signal classification: Classify and identify the EEG signals, find the target block stared at by the subject, convert it into a control signal, and send it to the robotic arm control system;

[0043] 9) Robotic arm control: Select the robotic arm control mode, convert the control command into a robotic arm motion signal, and complete the corresponding actions.

[0044] As Figure 1 shown, the BCI system mainly consists of a subject, an EEG acquisition module, an EEG signal processing and decoding module, and a control module. The present invention is a control system that collects and processes EEG signals under the SSVEP stimulation state and outputs control signals to achieve autonomous control of the robotic arm.

[0045] As Figure 2As shown in the figure, the present invention mainly consists of the following eight steps: offline training, multi-level stimulation page, data collection, data preprocessing, robotic arm startup, intention recognition, feature classification, and robotic arm control.

[0046] Before using the present invention to control the robotic arm, the subject needs to first perform offline training, process and analyze the collected data to obtain information such as the optimal EEG channel combination, sliding window length, threshold, etc., and transmit the information to the control system. After processing, it can be officially used. The multi-level stimulation page displays multiple target blocks flashing at different frequencies. In the first-level page, the subject needs to select the control mode of the robotic arm to enter the second-level stimulation page, and the subject sends commands by gazing at the target blocks. The acquisition device collects the EEG signals and sends them to the processing end. The processing end preprocesses the signals and extracts features. When the alpha wave reaches the set threshold, the robotic arm is activated. Then, the intention of the signal is discriminated. When the signal is recognized as the gazing state, the signal is allowed to be classified and recognized. The signal is classified and converted into a control signal and transmitted into the robotic arm control system. The robotic arm control system controls the robotic arm according to the control signal and the robotic arm movement mode to grasp the specified target.

[0047] The specific processes of each part of the present invention are as follows:

[0048] 1. Offline training:

[0049] A. Collect the occipital region signals of the brain under stimulation frequencies of 8Hz, 9Hz, 10Hz, 11Hz, 12Hz, and 13Hz, non-gazing state, and closed-eye state, and record them as data in the gazing state, idle state, and closed-eye state respectively.

[0050] B. In the offline experiment, the length of each experimental data is 5s. The data is windowed according to a step size of 0.1s from different lengths of 0.5 - 4s. The data with a length of 5s is windowed. For example, if it is divided into 3s windows, the data is divided into 0 - 3s, 0.1 - 3.1s, 0.2 - 3.2s... 4.1 - 5s, a total of 21 data.

[0051] C. Use the CCA algorithm to calculate the correlation coefficients in the idle state and gazing state, and record the ratio of the maximum coefficient to the second coefficient as the CCA coefficient.

[0052] D. Calculate the power spectrum values in the idle state and gazing state at the fundamental frequency and different harmonic frequencies (8 - 13Hz, 8 - 26Hz, 8 - 39Hz, 8 - 52Hz, 16 - 26Hz, 16 - 39Hz, 16 - 52Hz, 24 - 39Hz, 24 - 52Hz, 32 - 52Hz).

[0053] E. Calculate the threshold of the alpha wave in the Oz channel in the closed-eye state

[0054] F. Find the optimal experimental conditions, power spectrum, and weight coefficients of the CCA coefficient;

[0055] 2. Multi-level stimulation page:

[0056] A. As Figure 3 shown, the first layer of the multi-level stimulation page is the working mode selection of the robotic arm. The working modes of the robotic arm are divided into Cartesian coordinate system control and joint angle control. Target block 1 represents Cartesian coordinate system control, target block 2 represents joint angle control, Y represents confirmation, N represents cancellation, and feedback will be given on the screen after each selection; after selecting the working mode, the confirmation button needs to be selected. If cancellation is selected, the working mode needs to be reselected. After successfully selecting the working mode, a working mode selection signal is sent to the robotic arm control system;

[0057] B. As Figure 4 shown, the second layer of the multi-level stimulation page is the control layer of the robotic arm. The robotic arm control page in the Cartesian coordinate system working mode is shown in the figure; X+, Y+, Z+ respectively represent the movement in the positive directions of X, Y, Z, and X-, Y-, Z- respectively represent the movement in the negative directions of X, Y, Z; the green button (scattered dot button in the upper right) and the red button (dense dot button in the lower right) respectively represent the on and off states of the robotic arm. When the robotic arm is on, only the green button is displayed and the red button is not displayed. When it is off, only the red button is displayed and the green button is not displayed;

[0058] C. The flashing frequencies of each target block in the first layer are 9Hz, 10Hz, 11Hz, 12Hz respectively, and the flashing frequencies of each target block in the second layer are 8Hz, 9Hz, 10Hz, 11Hz, 12Hz, 13Hz respectively. The stimulation is generated using the sampled sine modulation method:

[0059]

[0060] s: The brightness of the target block, 0 represents the lowest brightness, and 1 represents the maximum brightness;

[0061] f: The flashing frequency of the target block;

[0062] i: The frame number sequence number of the current screen flashing;

[0063] R: The flashing frequency of the screen. In the LCD screen, it is usually 60Hz;

[0064] 3. Data acquisition: Figure 5 The following is the brain electrode placement position diagram of the present invention. According to the international brain electrode 10 / 20 standard, the brain electrodes are placed at Figure 5The 11 channels in the occipital region shown, with GND grounded in the figure;

[0065] 4. Data preprocessing: Extract the signal in the alpha discrimination state, that is, the data of the single Oz channel, perform band-pass filtering at 8 - 13 Hz, select the best channel combination data according to offline training, perform band-pass filtering at 7 - 95 Hz, and perform notch processing on it. According to the results of offline training, select whether to use Independent Component Analysis (ICA) to remove electrooculogram;

[0066] 5. Robotic arm startup: Compare the data of the single Oz channel with the threshold of the alpha wave in the closed-eye state during offline training. When the threshold is reached, send a robotic arm startup signal to the robotic arm control system;

[0067] 6. Intention recognition: Calculate the CCA coefficient and power spectrum of the best channel combination data, compare the obtained values with the threshold in offline training, and perform dimensionality reduction using the weight coefficient obtained in offline training. If the mixed coefficient is greater than 1, it is considered that the subject is staring at the target (issuing a command). If it is less than 1, it is considered that the subject is in an idle state without issuing a command;

[0068] 7. Feature classification: Use CCA to perform feature classification on the SSVEP signal. The CCA method can find the maximum correlation between two sets of variables. Denote the collected signal as X. When calculating, set the sine signal Y, f i are the 6 frequencies of the target block respectively. Calculate the correlation between X and 6 groups of sine signals Y, and find the target block frequency corresponding to the Y with the largest correlation with X, then the target stared at by the subject can be judged;

[0069]

[0070] 8. Robotic arm control system: As Figure 6 shown, the robotic arm control system includes several of the above links. When the control system is turned on, there will be continuous signal input. First, the signal needs to be discriminated. The effective discrimination signals are divided into four categories: robotic arm on / off signal, working mode selection signal, robotic arm movement control signal, end effector control signal. When the control system receives the robotic arm on signal, start the robotic arm; monitor the working mode selection signal. At this time, only receive this signal and the robotic arm state control signal. After receiving the working mode selection signal, enter the corresponding working mode; wait for the robotic arm movement control signal and the end effector control signal to be input. After receiving the signal, control the robotic arm to move and the end effector through the ROS system to grab the target object.

[0071] The asynchronous control system of the robotic arm based on SSVEP disclosed in the present invention, during the control process of the robotic arm, uses the alpha wave signal to control the state of the robotic arm, adopts the methods of dynamic windowing and dynamic channel combination, makes a weighted decision on the CCA coefficient and the power spectral density, realizes the asynchronous control of the robotic arm by SSVEP, and completes the selection of each mode of the robotic arm control system through a multi-level stimulation page. This asynchronous control system effectively improves the user's safety and reduces the economic cost, enables the user to control the robotic arm freely without having to control within the preset instruction time, can grasp the target object, and has high accuracy and real-time performance.

[0072] The content described in the embodiments of this specification is only an enumeration of the implementation forms of the inventive concept and is only for illustrative purposes. The protection scope of the present invention should not be regarded as limited to the specific forms stated in this embodiment, and the protection scope of the present invention also covers equivalent technical means that those of ordinary skill in the art can think of based on the inventive concept of the present invention.

Claims

1. An asynchronous control method for a robotic arm based on SSVEP, characterized in that, The method includes the following steps: 1) Offline training: In the offline state, collect the electroencephalogram (EEG) data of the subject in the idle gaze state, the alpha wave signal, and find the optimal combination of processing channels, frequency band, and threshold information. 2) Multi-level stimulation page: It has multiple layers of stimulation interfaces, and each stimulation interface has targets flashing at different frequencies. The control system identifies the target stared at by the subject through the EEG data of the subject. 3) Data acquisition: According to the international standard 10 / 20 for EEG electrode positions, collect the data of 9 channels in the occipital region of the brain and transmit the signal back to the processing end. 4) Preprocessing: According to the offline training results, perform corresponding window segmentation on the data, select the data of the corresponding combined channels for processing, and perform band-pass filtering and notch filtering. 5) Feature extraction: Extract the corresponding features of the collected EEG signal through the Canonical Correlation Analysis (CCA) algorithm combined with the Power Spectral Density (PSD). 6) Robotic arm activation: When the collected alpha wave signal reaches the threshold set in the offline experiment, activate the robotic arm and display the robotic arm as being in the on state on the stimulation interface. 7) Intention discrimination: Identify whether the subject is in the idle state or the gaze state. If it is the gaze state, generate a control signal. 8) Signal classification: Classify and identify the EEG signal, find the target block stared at by the subject, convert it into a control signal, and send it to the robotic arm control system. 9) Robotic arm control: Select the robotic arm control mode, convert the control command into a robotic arm motion signal, and complete the corresponding actions. The process of step 2) of the method is as follows: A. The first layer is the selection of the working mode of the robotic arm. The working modes of the robotic arm are divided into Cartesian coordinate system control and joint angle control. Target block 1 represents Cartesian coordinate system control, target block 2 represents joint angle control, Y represents confirmation, and N represents cancellation. After each selection, feedback will be given on the screen. After the working mode is selected, the confirmation button needs to be selected. If cancellation is selected, the working mode needs to be selected again. After successfully selecting the working mode, send a working mode selection signal to the robotic arm control system. B. The second layer is the control layer of the robotic arm. In the robotic arm control page in the Cartesian coordinate working mode; X+, Y+, and Z+ respectively represent the movement in the positive directions of X, Y, and Z, and X-, Y-, and Z- respectively represent the movement in the negative directions of X, Y, and Z; the green button and the red button respectively represent the on and off states of the robotic arm. When the robotic arm is on, only the green button is displayed and the red button is not displayed. When it is off, only the red button is displayed and the green button is not displayed. C. The flashing frequencies of each target block in the first layer are 9Hz, 10Hz, 11Hz, and 12Hz respectively, and the flashing frequencies of each target block in the second layer are 8Hz, 9Hz, 10Hz, 11Hz, 12Hz, and 13Hz respectively. The sampling sine modulation method is used to generate the stimulation: s: The brightness of the target block, 0 represents the lowest brightness, and 1 represents the maximum brightness. f: The flashing frequency of the target block. i: The sequence number of the current screen flashing frame. R: The flashing frequency of the screen, which is 60Hz in the LCD screen.

Images