High-frequency visual evoked potential BCI encoding and decoding method and device and mechanical arm control system

Abstract

The invention discloses a high-frequency visual evoked potential brain-computer interface encoding and decoding method and device and a mechanical arm control system. The high-frequency visual evoked potential brain-computer interface coding method comprises the steps that a plurality of instruction blocks distributed in rows and columns are displayed on a visual stimulation interface, each instruction block is jointly coded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value in the frequency-phase combination pair is greater than or equal to 30Hz. When a subject watches an instruction block, the brain induces visual evoked potentials corresponding to the rows and the columns of the instruction block respectively, so that the row number and the column number of the instruction block can be obtained through decoding instead of decoding the visual evoked potentials to be identified. The problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art is solved, so that low training cost can be consumed while high-frequency visual stimulation frequency is adopted, and the effect of improving the eye comfort of a subject is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to high-frequency signal acquisition and encoding and decoding technology of brain-computer interface, and in particular, to a high-frequency visual evoked potential brain-computer interface encoding and decoding method, device and robotic arm control system. Background Art

[0002] As researchers conduct in-depth research on electroencephalogram (EEG), non-invasive brain-computer interfaces (BCI) based on EEG establish a communication channel between the brain and the computer in a non-invasive way. In particular, brain-computer interfaces based on visually evoked potentials (VEP) have attracted more attention due to their high information transmission rate and decoding reliability. The principle of the Steady-State Visual Evoked Potentials (SSVEP) brain-computer interface is that when a user looks at a visual stimulus with a certain frequency and duration, a component consistent with the stimulus frequency or its higher harmonic frequency appears in the EEG signal. The user's intention is judged by detecting and classifying the SSVEP component in the EEG signal.

[0003] The stimulation frequency used by SSVEP-BCI is within the range of three frequency bands, including low frequency (4 Hz-12 Hz), medium frequency (12 Hz-30 Hz) and high frequency (>30 Hz). Since the amplitude and signal-to-noise ratio (SNR) of medium- and low-frequency SSVEP are significantly higher than those of high-frequency SSVEP, the stimulation frequency of SSVEP-BCI systems in the prior art is mostly concentrated in the low-frequency or medium-frequency regions. Compared with high-frequency SSVEP, medium- and low-frequency SSVEP usually brings stronger visual discomfort to subjects. Summary of the invention

[0004] The present invention provides a high-frequency visual evoked potential brain-computer interface encoding and decoding method, device and robotic arm control system, so as to achieve the effect of improving the eye comfort of the subject while consuming relatively low training cost and using high-frequency visual stimulation frequency.

[0005] In a first aspect, an embodiment of the present invention provides a high-frequency visual evoked potential brain-computer interface encoding method, comprising:

[0006] Displaying multiple instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block;

[0007] The frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz.

[0008] In a second aspect, an embodiment of the present invention further provides a high-frequency visual evoked potential brain-computer interface decoding method, corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in the first aspect, comprising:

[0009] Model each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0010] The filtered visual evoked potential to be identified X test Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential X to be recognized test The row and column numbers of the .

[0011] In a third aspect, an embodiment of the present invention further provides a high-frequency visual evoked potential brain-computer interface decoding device, comprising:

[0012] A modeling module, used to model each row instruction block and each column instruction block respectively, to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0013] The number acquisition module is used to obtain the filtered visual evoked potential X test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized test The row and column numbers of the .

[0014] In a fourth aspect, an embodiment of the present invention further provides a display device, including:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When one or more programs are executed by one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface encoding method as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present invention further provides a data acquisition and processing device, including:

[0019] one or more processors;

[0020] a storage device for storing one or more programs,

[0021] When one or more programs are executed by one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface decoding method described in the second aspect.

[0022] In a sixth aspect, an embodiment of the present invention further provides a robotic arm control system, comprising:

[0023] Display equipment for generating visual stimuli;

[0024] Data acquisition and processing equipment, used to collect and process data to obtain command signals;

[0025] A robot arm control device, used to control the movement of the robot arm according to the command signal;

[0026] Among them, the display device executes the high-frequency visual evoked potential brain-computer interface encoding method described in the first aspect, and the data acquisition and processing device executes the high-frequency visual evoked potential brain-computer interface decoding method described in the second aspect.

[0027] The technical solution of this embodiment is to display multiple instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz; when the subject looks at an instruction block, the brain induces visual evoked potentials corresponding to the rows and columns of the instruction block, respectively, so that the row number and column number of the instruction block can be decoded instead of decoding the visual evoked potential to be identified itself, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, thereby consuming lower training costs while using high-frequency visual stimulation frequencies, thereby achieving the effect of improving the eye comfort of the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the principle of steady-state visual evoked potential brain-computer interface (SSVEP-BCI) signal acquisition;

[0029] Figure 2 (a) is a schematic diagram of an instruction block corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention;

[0030] Figure 2 (b) is a schematic diagram of a visual stimulation interface corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention;

[0031] Figure 3 This is a flow chart of a high-frequency visual evoked potential brain-computer interface decoding method provided in Embodiment 2 of the present invention;

[0032] Figure 4 It is a flowchart of the rc-eTRCA algorithm;

[0033] Figure 5This is a flow chart of a high-frequency visual evoked potential brain-computer interface decoding method provided by Embodiment 3 of the present invention;

[0034] Figure 6 is a flow chart of the method for dynamic stop asynchrony detection of high-frequency visual evoked potentials;

[0035] Figure 7 A schematic diagram of the structure of a high-frequency visual evoked potential brain-computer interface decoding device provided in Embodiment 4 of the present invention;

[0036] Figure 8 A structural diagram of a data acquisition and processing device provided in Embodiment 6 of the present invention;

[0037] Fig. 9 This is a performance comparison diagram of high-frequency visual evoked potential brain-computer interface using the eTRCA algorithm and the rc-eTRCA algorithm;

[0038] FIG10( a ) is a schematic diagram of the structure of a robotic arm control system provided in Embodiment 7 of the present invention;

[0039] FIG10( b ) is a schematic diagram of a user interface presented on a display device;

[0040] Fig.11 This is the working principle diagram of the robotic arm control system. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0042] The main purpose of brain-computer interface research is to create non-muscle communication channels so that people with severe motor impairments or other healthy people in need can use this technology to communicate with and control computers. Steady-state visual evoked potential-based brain-computer interface (SSVEP-BCI) modulates EEG activity through visual channels, and then decodes the user's intention by decoding the specific modulation information in the EEG.

[0043] Figure 1 This is a schematic diagram of the principle of steady-state visual evoked potential brain-computer interface (SSVEP-BCI) signal acquisition. As shown in the figure, SSVEP-BCI signal acquisition and processing mainly includes three stages: stimulus presentation, data acquisition, and data processing and analysis. In the stimulus presentation stage, in order to make the visual stimulus well controllable, a traditional computer screen or a customized light-emitting diode (LED) and other visual stimulation interfaces are usually used. Figure 1 The eight circular icons displayed on the computer screen are eight different visual coding units, i.e., instruction blocks. When the user looks at different instruction blocks, different EEG signals can be induced at the scalp. During the data collection phase, the non-invasive brain-computer interface on the user's head collects scalp EEG. Figure 1 As shown in the frequency domain EEG signals in , different instruction blocks generate different frequency domain EEG signals. For example, instruction blocks flashing at different frequencies induce different base frequencies and multiple frequency peaks of frequency domain EEG signals. The data processing and analysis stage analyzes the instruction block that the user is looking at and outputs the user's intention corresponding to the instruction block.

[0044] As for the selection of visual coding units, when the instruction block uses medium and low frequency stimulation signals, the baseband signal-to-noise ratio of the scalp EEG reaches 15dB to 25dB, and when the instruction block uses a high frequency stimulation signal above 30Hz, the signal-to-noise ratio of the scalp EEG is less than 10dB. In other words, using high-frequency stimulation signals to encode instruction blocks, although the visual stimulation to the subjects is small, it can improve the visual comfort of the subjects, but because the scalp EEG noise interference is large and the amplitude is small, it is very difficult to train a high-frequency visual evoked potential brain-computer interface that meets the performance standards. Usually, more training data is required, and the training cost is very high. Therefore, the SSVEP-BCI in the prior art usually uses low-frequency signals for training and recognition.

[0045] Embodiment 1

[0046] A high-frequency visual evoked potential brain-computer interface encoding method, this embodiment can be applied to the case where a high-frequency signal is used to encode an instruction block, the method can be executed by a display device, and specifically includes the following steps:

[0047] Multiple instruction blocks distributed in rows and columns are displayed on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz.

[0048] Optionally, each instruction block is jointly encoded by two frequency-phase combination pairs, and the two frequency-phase combination pairs are distributed left and right.

[0049] Figure 2 (a) is a schematic diagram of the instruction block corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention. Figure 2 (b) is a schematic diagram of a visual stimulation interface corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention. Figure 2As shown in the figure, each instruction block is jointly encoded by two different frequency-phase combination pairs. The sine wave signal flashing in the instruction block is generated by the joint frequency-phase modulation method. The size of each instruction block is 120×120 pixels. Each instruction block consists of two 60×120 pixel rectangular blocks on the left and right. There is a cross between the two rectangular blocks as a focus to facilitate the subject to determine the gaze position. The left-right distribution design ensures that the flashing images generated by the bilateral rectangular blocks stimulate the left and right parts of the retina respectively, and then project the induced EEG signals to the contralateral occipital lobe area, which is conducive to distinguishing the two SSVEP frequencies.

[0050] In this embodiment, the row coding frequency and the column coding frequency are in different frequency ranges, the row coding frequency is between 30.0 Hz and 33.2 Hz, the column coding frequency is between 34.0 Hz and 36.4 Hz, and the frequency interval is 0.8 Hz (the frequency values ​​used in this embodiment are typical values, and other frequency values ​​can also be used). The sine wave of the stimulation signal is set with an initial phase, and the phase interval is 0.5π, so the two frequencies of the instruction can be separated by a filter, so that when training the recognition model, the instructions with the same frequency-phase combination pair are merged together for training.

[0051] In this embodiment, 20 (typical value) instruction blocks are arranged into a 4×5 (typical value) matrix. In actual applications, the number and arrangement of instruction blocks can be adjusted according to actual needs. Each instruction block is modulated by high-frequency dual-frequency stimulation and stimulates the left and right visual fields (a checkerboard or other dual-frequency modulation method can also be used). Among them, the instructions of each row or column share a frequency-phase combination pair, so that each row or column is modulated by a specific frequency-phase combination pair.

[0052] The present invention selects a total of 20 instruction blocks of nine frequency-phase combination pairs within the frequency range of 30Hz-36.4Hz for encoding. Based on this encoding protocol, a method based on spatial filters and template matching is used to train the model to decode the index of the row and column where the instruction corresponding to the specific visual evoked potential is located, rather than decoding the instruction itself, and then locating the specific instruction block by row and column. In this way, when training the recognition model of each row or column (spatial filter and SSVEP template), the training data of all instruction blocks in each row or column can be combined and used together, so that more training data is available when training the row recognition model and the column recognition model, thereby improving the reliability of the spatial filter and SSVEP template of the recognition model. Compared with the common method of encoding and decoding each instruction separately, this scheme achieves higher performance with less training data, and uses a limited number of frequencies to achieve encoding of more instruction blocks based on high-frequency SSVEP-BCI.

[0053] The technical solution of this embodiment is to display multiple instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz; when the subject looks at an instruction block, the brain induces visual evoked potentials corresponding to the rows and columns of the instruction block, respectively, so that the row number and column number of the instruction block can be decoded instead of decoding the visual evoked potential to be identified itself, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, thereby consuming lower training costs while using high-frequency visual stimulation frequencies, thereby achieving the effect of improving the eye comfort of the subject.

[0054] Embodiment 2

[0055] Figure 3 It is a flowchart of a high-frequency visual evoked potential brain-computer interface decoding method provided by the second embodiment of the present invention, and the decoding method can be implemented by a data acquisition and processing device. The second embodiment corresponds to the high-frequency visual evoked potential brain-computer interface encoding method described in the first embodiment. After encoding the instruction block displayed on the visual stimulation interface by the encoding method described in the first embodiment, the EEG signal generated by the user looking at the visual stimulation interface is collected, and the EEG signal is decoded. The high-frequency visual evoked potential brain-computer interface decoding method specifically includes:

[0056] S310, modeling each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0057] S320, the filtered visual evoked potential to be identified X test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized test The row and column numbers of the .

[0058] Different from the decoding strategy of "decoding by target" in the prior art, the "decoding by rows and columns" corresponding to the encoding method of the first embodiment is adopted in the second embodiment of the present application. The "decoding by target" in the prior art means that the decoding algorithm needs to model each instruction block. As the number of instruction blocks increases, the workload of modeling increases significantly. "Decoding by rows and columns" is a decoding strategy rather than a decoding algorithm. This decoding strategy can be combined with any decoding algorithm, such as Task Related Component Analysis (TRCA), Canonical Correlation Aaronnalysis (CCA) and other commonly used algorithms for SSVEP decoding. The present invention takes the TRCA algorithm as an example.

[0059] This embodiment is based on the method of spatial filter and template matching to train the high-frequency row recognition model model and the high-frequency column recognition model to decode the row number and column number of the instruction block corresponding to the visual evoked potential to be identified, and then locate the instruction block itself through the row number and column number. Compared with "decoding by target", more data can be used to train the spatial filter and SSVEP template when the number of subjects is fixed, because the training data of all instruction blocks sharing a specific frequency-phase combination pair in each row or column will be combined. The technical solution of "decoding by row and column" in this embodiment is based on the row-column ensemble TRCA (rc-eTRCA) algorithm. rc-eTRCA can obtain the spatial filter and SSVEP template of each row or column by using the training data of all instruction blocks of the corresponding row or column, thereby reducing the average training cost, where the "average training cost" refers to the average number of training times for each instruction block.

[0060] Figure 4 This is a flow chart of the rc-eTRCA algorithm. Figure 4 As shown, assuming Represents a single three-dimensional training data, namely, the training visual evoked potential χ ij . Where i represents the row number of the instruction block that the subject is looking at, j represents the column number of the instruction block, and N ch is the number of channels, N s is the number of sampling points for each test, N t is the number of training trials. ij Rearrange the rows and columns and connect the data with the same frequency-phase combination to get:

[0061]

[0062] Where Cat(·) indicates that the matrices are concatenated along the third dimension. and are the training data of the nth column and the nth row respectively. and The filter banks (FB) are used to extract SSVEP components of different frequency bands. According to the stimulation frequency of the instruction block, the column filter banks (FB) in this embodiment are col ) and line filter bank (FB row ) are set to [(m×30-2)Hz, 90Hz] and [(m×33.2)Hz, 90Hz], where m represents the subband index. Then, the column space filter W of each passband is obtained using the eTRCA algorithm.col(m) and row space filter W row(m) . And by calculating the average value of multiple trials, the column template signal of each passband is obtained and row template signals The specific calculation formula is as follows:

[0063]

[0064] Wherein, h represents the sample number of the visual evoked potential used for training.

[0065] After obtaining each row template signal, column template signal, column space filter and row space filter, the high-frequency row recognition model and high-frequency column recognition model are obtained. Next, the high-frequency row recognition model and high-frequency column recognition model that have been obtained are used to identify the visual evoked potential X test Make identification judgment. test col(m) Correlation coefficient with column template signal and X test row(m) With line template signal The correlation coefficient between them can be calculated by the following formula, where X test col(m) With X test row(m) The visual evoked potential X is identified by using a column filter bank and a row filter bank. test After filtering, we get:

[0066]

[0067] ρ(a,b) represents the Pearson correlation coefficient between a and b. The weighted sum of the squares of all sub-band correlation coefficients is defined as the classification feature:

[0068]

[0069] Where N m =2 is the number of subbands, a(m)=m -1.25 .

[0070] Take the categorical features separately and row classification features The column number and row number corresponding to the maximum value of , thereby obtaining the visual evoked potential X to be identified test Column and row numbers:

[0071]

[0072] Among them, i and j are the visual evoked potentials X to be identified. test The visual evoked potential X to be identified can be determined according to the row number i and column number j. test The corresponding instruction block τ=τ ij .

[0073] The technical solution of this embodiment is to obtain a high-frequency row recognition model and a high-frequency column recognition model by modeling each row instruction block and each column instruction block respectively; test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized test row number and column number; that is, based on the row template signal, column template signal, column space filter and row space filter, the row number and column number of the visual evoked potential to be identified are decoded (rather than decoding the visual evoked potential to be identified itself), and then the row number and column number are used to locate the specific instruction block. In this way, when training the row template signal, column template signal, column space filter or row space filter of each row or column, all the training data in each row or column can be combined and used together, so that the high-frequency recognition model has more training data available, and the reliability of the spatial filter and template signal is improved. Compared with the method of decoding each instruction block separately in the prior art, this scheme achieves higher performance with less training data, and uses limited frequency to achieve the decoding of a larger number of instruction blocks based on high-frequency SSVEP-BCI, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, and can consume low training cost while using high-frequency visual stimulation frequency, so as to achieve the effect of improving the eye comfort of the subject.

[0074] Fig. 9 This is a performance comparison diagram of high-frequency visual evoked potential brain-computer interface using the eTRCA algorithm and the rc-eTRCA algorithm, where the error bars represent standard errors and the asterisks represent the significance of the differences obtained by paired t-test (*p<0.05, **p<0.01, ***p<0.001). Fig. 9 (a) shows the comparison of recognition accuracy of the "target decoding" eTRCA algorithm and the "row and column decoding" rc-eTRCA algorithm under different data lengths when the number of training trials per instruction block is 3. Fig. 9 As shown in (a), under all data lengths in the figure, the recognition accuracy of the rc-eTRCA algorithm is higher than that of the eTRCA algorithm. Fig. 9 (b) shows the recognition accuracy of the eTRCA algorithm and the rc-eTRCA algorithm under different training trial times of the instruction block when the data length is 1.5 seconds. It can be seen that the row-column decoding strategy proposed in this embodiment significantly improves the recognition accuracy of the instruction block under less training data compared with the traditional non-row-column decoding strategy.

[0075] Embodiment 3

[0076] Figure 5This is a flowchart of a high-frequency visual evoked potential brain-computer interface decoding method provided by Example 3 of the present invention, and the decoding method can be implemented by a data acquisition and processing device. Example 3 is a further optimization of the high-frequency visual evoked potential brain-computer interface encoding method described in Example 2, specifically including:

[0077] S510 , training a classifier for distinguishing between a control state IC and a non-control state NC.

[0078] Synchronous SSVEP-BCI means that when identifying the user's visual evoked potential to be identified, it assumes that the user's EEG signal is valid information, that is, the EEG signal generated when the user is looking at a specific instruction block. However, synchronous SSVEP-BCI has the problem of limited user autonomy during human-computer interaction, that is, the user needs to stay focused at all times. Therefore, asynchronous SSVEP-BCI that can judge the user's state has more advantages in practice.

[0079] The key challenge in designing asynchronous SSVEP-BCI is how to correctly and efficiently distinguish the user's current state, that is, to determine whether the user is in an intentional control state (IC) or a non-control state (NC). The intentional control state refers to the state in which the user is looking at the instruction block, and the non-control state refers to the state in which the user is in other states, such as being in a daze or looking at objects other than the instruction block. The non-control state is also called the idle state.

[0080] In order to improve the classification efficiency and practicality of the classifier, the classifier may optionally have an adaptive window function, and the visual evoked potential X to be identified test The length of starts from the initial length and increases by a specific step size.

[0081] Since the present embodiment is designed with asynchronous SSVEP-BCI, the visual evoked potential X to be identified is input into the classifier. test It is not necessarily the EEG signal generated when the user is in the control state, but it may be the EEG signal generated when the user is in the non-control state and has nothing to do with the instruction block recognition. test When distinguishing between the control state and the non-control state, if a shorter EEG signal can be judged to be in the non-control state, there is no need to further process the subsequent EEG signal of the non-control state.

[0082] That is to say, this embodiment provides a technical solution that combines "decoding by rows and columns", "control state detection method" and "adaptive window method" to achieve dynamic stop asynchronous detection of high-frequency visual evoked potentials. The adaptive window method refers to not identifying whether the user is in a control state through a fixed-length EEG signal, but when the length of the EEG signal can reliably draw a judgment conclusion, it determines whether the EEG signal is a control state visual evoked potential. The following will explain in detail how to train a classifier that distinguishes between a control state IC and a non-control state NC. Specifically, step S510 includes using IC data of the same length and NC data of different lengths to train a classifier that distinguishes between a control state IC and a non-control state NC. The following will explain the training process of the classifier.

[0083] Figure 6 The figure is a flow chart of the method for dynamic stop asynchrony detection of high-frequency visual evoked potentials. Figure 6 As with the second embodiment, the row space filter is W row(m) , the column space filter is W col(m) , calculate the correlation coefficient between the filtered control state training visual evoked potential and the template signal:

[0084]

[0085] Where n represents the column number or row number, and h represents the sample number of the visual evoked potential used for control state training. and It can be calculated by formulas (3) and (4) in the second embodiment.

[0086] Calculate the data length as t n Visual evoked potentials Correlation coefficient with the template signal:

[0087]

[0088] As shown in formula (16) and (17), and As a feature of the classifier for distinguishing the control state from the non-control state. In order to improve the performance of the classifier, this embodiment uses non-control state training visual evoked potentials of different lengths to train the classifier, but uses control state training visual evoked potentials of the same length, thereby minimizing the time required for classifier training.

[0089]

[0090] Since the minimum data length that the EEG signal acquisition device can access online when the non-invasive brain-computer interface collects scalp EEG is 0.04s, the data length t in this embodiment is nThe range is [0.62s, 1.5s] with an interval of 0.04s. Due to the influence of visual delay in the human visual system, if the instruction block starts flashing at 0s, the EEG data can be extracted in the range of [0.14s, (0.76+n×0.04)s].

[0091] S520 , modeling each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model.

[0092] It should be noted that the execution order of step S510 and step S520 can be adjusted as needed.

[0093] S530, the filtered visual evoked potential to be identified X test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized test The row and column numbers of the .

[0094] S540, the visual evoked potential to be identified X test Input into the classifier to determine the visual evoked potential X to be identified test It is the visual evoked potential in the control state or the EEG signal in the non-control state.

[0095] According to formulas (5)-(11) in the second embodiment, the visual evoked potential X to be identified is obtained based on the rc-eTRCA algorithm in step S530. test The row number i and column number j of the visual evoked potential to be identified are then determined test The possible corresponding instruction block τ, since the visual evoked potential X to be identified has not been detected at this time test Whether it is a control state visual evoked potential, so the statement here is "possibly" corresponding to the instruction block τ.

[0096] Adaptive window method based on covariance analysis is used to determine whether the current data length is sufficient to reliably determine the visual evoked potential X to be identified. test The weighted sum of the squares of the correlation coefficients ρ is used to determine the visual evoked potential X to be identified. test The column number and row number of , that is, determine the possible corresponding instruction block τ. The visual evoked potential X to be identified test The incremental data packet length is 0.04 seconds, the initial data length t0 is 0.62 seconds, and the maximum data length t0 reaches 22 when the maximum step length is reached. x is 1.50 seconds.

[0097] t n =t n-1 +0.04,n≤22 (13)

[0098] After the data length is updated, calculate the visual evoked potential X to be identified with the latest data length test ( Figure 6 Medium X tn ) is the sum of the categorical features of the rows and columns:

[0099]

[0100] in and Calculate according to formula (7) and (8) in Example 2, and then determine by covariance analysis method With the eigenvector [t0,…,t n ]([0.62,…,1.50]). As the window length, i.e. the data length, increases until a statistically significant effect (p<0.05) appears, then the data length t n The corresponding classifier determines whether the user is in a control state or a non-control state. If it is determined that the user is in a control state at this time, the final recognition result instruction block τ is output; otherwise, the user is determined to be in a non-control state.

[0101] The technical solution of this embodiment is to train a classifier to distinguish between the control state IC and the non-control state NC; test Input into the classifier to determine the visual evoked potential X to be identified test In order to control the visual evoked potential or the non-controlled EEG signal, the implementation difficulty of the dynamic stop asynchronous detection method of high-frequency visual evoked potential was solved, the goal of improving user autonomy in the human-computer interaction process was achieved, and the training time and detection time were effectively shortened. The "high-frequency signal", "decoding by rows and columns", "control state detection method" and "adaptive window method" were combined.

[0102] Embodiment 4

[0103] Figure 7 This is a schematic diagram of the structure of a high-frequency visual evoked potential brain-computer interface decoding device provided in Embodiment 4 of the present invention. The high-frequency visual evoked potential brain-computer interface decoding device can execute the high-frequency visual evoked potential brain-computer interface decoding method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0104] A high-frequency visual evoked potential brain-computer interface decoding device, comprising:

[0105] A modeling module 701 is used to model each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0106] The number acquisition module 702 is used to obtain the filtered visual evoked potential Xtest The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized test The row and column numbers of the .

[0107] Optionally, the high-frequency visual evoked potential brain-computer interface decoding device further includes:

[0108] A classifier training module is used to train a classifier for distinguishing between a control state IC and a non-control state NC;

[0109] The control state determination module is used to determine the visual evoked potential X to be identified test Input into the classifier to determine the visual evoked potential X to be identified test It is the visual evoked potential in the control state or the EEG signal in the non-control state.

[0110] Optionally, the classifier training module is also used to use IC data of the same length and NC data of different lengths to train a classifier for distinguishing between controlled state IC and non-controlled state NC.

[0111] Embodiment 5

[0112] Figure 1 The display on the left is a schematic diagram of the structure of the display device provided in Embodiment 4 of the present invention. The display device can execute the high-frequency visual evoked potential brain-computer interface coding method provided in Embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] A display device, comprising:

[0114] one or more processors;

[0115] a storage device for storing one or more programs,

[0116] When the one or more programs are executed by the one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface encoding method as described in Example 1.

[0117] Embodiment 6

[0118] Figure 8 A structural diagram of a data acquisition and processing device provided in Embodiment 6 of the present invention is shown in FIG. Figure 8 As shown, the data acquisition and processing device includes a processor 810, a memory 820, an input device 830 and an output device 840; the number of the processor 810 in the data acquisition and processing device can be one or more. Figure 8 A processor 810 is taken as an example; the processor 810, the memory 820, the input device 830 and the output device 840 in the data acquisition and processing device can be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0119] The memory 820, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the high-frequency visual evoked potential brain-computer interface decoding method in the embodiment of the present invention (for example, the modeling module 701 and the number acquisition module 702 in the high-frequency visual evoked potential brain-computer interface decoding device). The processor 810 executes various functional applications and data processing of the data acquisition and processing device by running the software programs, instructions and modules stored in the memory 820, that is, realizing the above-mentioned high-frequency visual evoked potential brain-computer interface decoding method.

[0120] The memory 820 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 820 may further include a memory remotely arranged relative to the processor 810, and these remote memories may be connected to the data acquisition and processing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The input device 830 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the data acquisition and processing device. The output device 840 can include display devices such as display screens.

[0122] Embodiment 7

[0123] FIG10( a ) is a schematic diagram of the structure of a robotic arm control system provided by Embodiment 7 of the present invention, the robotic arm control system comprising:

[0124] Display equipment for generating visual stimuli;

[0125] Data acquisition and processing equipment, used to collect and process data to obtain command signals;

[0126] A robot arm control device, used to control the movement of the robot arm according to the command signal;

[0127] Among them, the display device executes the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1, and the data acquisition and processing device executes the high-frequency visual evoked potential brain-computer interface decoding method described in Example 2 or Example 3.

[0128] As shown in Fig. 10(a), the robot arm and the workspace (about 550 mm × 550 mm) are located on the right side of the user. The camera captures the workspace and transmits the real-time image to the display device (Alienware AW2720HF, 27 inches, 240 Hz refresh rate, 1920 × 1080 resolution), which displays the visual stimulation interface. Fig. 10(b) is a schematic diagram of the user interface presented on the display device, with 20 instruction blocks displayed on the right, and the row and column numbers corresponding to the workspace are marked on the user interface. The workspace includes a capture area and a placement area. When the user selects an instruction block in the 4 × 4 grid on the user interface, the robot arm can move to the corresponding position. In addition, the "release", "grasp", "rise" and "reset" commands marked on the user interface represent opening the robot arm gripper, closing the robot arm gripper, raising the robot arm gripper by 30 cm and resetting the robot arm, respectively.

[0129] Fig.11 This is the working principle diagram of the robot control system. The camera is used to present the interaction between the robot and other objects and the movement of the robot itself in real time. The data acquisition and processing equipment, the display equipment and the robot control equipment are connected through the USB port. The data acquisition and processing equipment communicates with the display equipment and the robot control equipment through the TCP / IP protocol. The display equipment and the data acquisition and processing equipment use the Windows 10 operating system, and the robot control equipment uses Ubuntu 16.04.

[0130] The experimental operation process of the robotic arm control system is as follows:

[0131] The experiment consists of two parts: offline experiment and online experiment. The offline experiment is used to model the high-frequency row recognition model and high-frequency column recognition model as well as the classifier used to distinguish the control state from the non-control state; the online experiment includes the instruction block recognition task and the robot arm control task, which are used to test the model and classifier built in the offline experiment. Due to the influence of visual delay in the human visual system, if the instruction block starts to flash at 0s, the EEG data can be extracted in the range of [0.14s, 1.64s]. The detailed description of the experimental process is as follows:

[0132] Offline experiment: The training visual evoked potential obtained by the offline experiment is used to verify the performance of the algorithm, and to obtain the recognition model and classifier for the subsequent online experimental training. The IC data used for training each instruction block can be 4 trials, and the NC data used for training can be 40 trials. Increasing the training trials can improve the accuracy of the row recognition model, the column recognition model and the classifier. Specifically, the offline experiment includes four groups, each group of offline experiments includes 10 NC trials and 20 IC trials, and each instruction block performs 1 IC trial. At the beginning of each trial, there is a 1-second prompt stage. The dark instruction block in Figure 10 (a) prompts the user to look at the instruction block during the IC test stage, and when the triangle in the center of the user interface becomes dark, it indicates the NC test stage, and the user should pay attention to the real-time video on the left side of the user interface. Subsequently, all instruction blocks flash simultaneously for 1.5 seconds, and the user looks at the corresponding instruction block or real-time video as required. According to the high-frequency visual evoked potential brain-computer interface decoding method of Example 3, a high-frequency row recognition model, a high-frequency column recognition model, and a classifier for distinguishing between control state and non-control state are trained.

[0133] Online experimental instruction block recognition task: A total of 40 IC tests and 22 NC tests were conducted. During the IC test, the target instruction block would turn dark for 1 second to prompt the user to fix their sight on the target instruction block, and then all instruction blocks would flash. The flashing duration was determined by the "adaptive window method" in Example 3, and the maximum duration was 1.5 seconds. During the NC test, the user turned his attention to the video on the left side of the user interface according to the instructions.

[0134] Online experimental robot arm control task: As shown in Figure 10(b), the work area includes a capture area and a placement area. Six foam blocks are randomly placed in the capture area. The user controls the robot arm control system to grab the foam blocks in the capture area and release them at the specified position in the placement area. First, the user looks at the instruction block corresponding to the foam block to be grabbed on the user interface. When the robot arm moves to the top of the foam block, the user looks at the instruction block corresponding to the "grab" instruction to grab the foam block. Then, the user looks at the instruction block corresponding to the placement area on the user interface. When the robot arm moves to the position corresponding to the instruction block, the user looks at the instruction block corresponding to the "release" instruction to put the foam block down.

[0135] Embodiment 8

[0136] Embodiment 8 of the present invention also provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the high-frequency visual evoked potential brain-computer interface encoding method described in Embodiment 1 or the high-frequency visual evoked potential brain-computer interface decoding method described in Embodiment 2.

[0137] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention, whose computer executable instructions are not limited to the method operations described above, can also execute related operations in the high-frequency visual evoked potential brain-computer interface encoding method or the high-frequency visual evoked potential brain-computer interface decoding method provided by any embodiment of the present invention.

[0138] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0139] It is worth noting that in the above-mentioned embodiment of the high-frequency visual evoked potential brain-computer interface decoding device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0140] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A high-frequency visual evoked potential brain-computer interface coding method, characterized in that: include: Displaying a plurality of instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; Wherein, the frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz.

2. A high-frequency visual evoked potential brain-computer interface decoding method, corresponding to the high-frequency visual evoked potential brain-computer interface encoding method according to claim 1, characterized in that: include: Model each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model; The filtered visual evoked potential to be identified X test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized. test The row and column numbers of the .

3. The high-frequency visual evoked potential brain-computer interface decoding method according to claim 2, characterized in that: The filtered visual evoked potential to be identified X test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized. test Before the row and column numbers, also include: Train a classifier to distinguish between control state IC and non-control state NC; The filtered visual evoked potential to be identified X test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized. test After the row and column numbers, include: The visual evoked potential X test Input into the classifier to determine the visual evoked potential X to be identified test It is the visual evoked potential in the control state or the EEG signal in the non-control state.

4. The high-frequency visual evoked potential brain-computer interface decoding method according to claim 3, characterized in that: The classifier has an adaptive window function, and the visual evoked potential X to be identified test The length of starts from the initial length and increases by a specific step size.

5. The high-frequency visual evoked potential brain-computer interface decoding method according to claim 4, characterized in that: The training of a classifier for distinguishing between a control state IC and a non-control state NC includes: IC data of the same length and NC data of different lengths are used to train a classifier that distinguishes controlled IC from uncontrolled NC.

6. A high-frequency visual evoked potential brain-computer interface decoding device, characterized in that: include: A modeling module (701) is used to model each row instruction block and each column instruction block respectively to obtain a high-frequency row identification model and a high-frequency column identification model; The number acquisition module (702) is used to obtain the filtered visual evoked potential X to be identified. test The high-frequency row recognition model and the high-frequency column recognition model are input respectively to obtain the visual evoked potential X to be recognized. test The row and column numbers of the .

7. A display device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface encoding method as described in claim 1.

8. A data acquisition and processing device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface decoding method as described in claims 2-5.

9. A robotic arm control system, characterized in that: include: Display equipment for generating visual stimuli; Data acquisition and processing equipment, used to collect and process data to obtain command signals; A robot arm control device, used to control the movement of the robot arm according to the command signal; Among them, the display device executes the high-frequency visual evoked potential brain-computer interface encoding method described in claim 1, and the data acquisition and processing device executes the high-frequency visual evoked potential brain-computer interface decoding method described in claims 2-5.

Images