High-frequency visual evoked potential BCI encoding and decoding method, device and robotic arm control system

Abstract

The present invention discloses a high-frequency visual evoked potential brain-computer interface encoding and decoding method, device and robotic arm control system. The high-frequency visual evoked potential brain-computer interface encoding method comprises: displaying a plurality of instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block being jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block corresponding to the two frequency-phase combination pairs of the instruction block; wherein the frequency value in the frequency-phase combination pair is greater than or equal to 30Hz. When a subject gazes at an instruction block, the brain induces visual evoked potentials corresponding to the rows and columns of the instruction block, respectively, so that the row number and column number of the instruction block can be decoded instead of decoding the visual evoked potential itself to be identified, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, thereby consuming lower training costs while adopting high-frequency visual stimulation frequency, achieving the effect of improving the subject's eye comfort.

Description

Technical Field

[0001] The embodiments of the present invention relate to high-frequency signal acquisition and encoding and decoding technology of brain-computer interfaces, and in particular to a high-frequency visual evoked potential brain-computer interface encoding and decoding method, device and robotic arm control system. Background Art

[0002] As researchers delve deeper into EEG (electroencephalogram, EEG) research, non-invasive brain-computer interfaces (BCI) based on EEG establish a communication channel between the brain and the computer in a non-invasive manner. In particular, BCIs based on visually evoked potentials (VEP) have attracted more attention due to their high information transmission rate and decoding reliability. The principle of the Steady-State Visual Evoked Potentials (SSVEP) BCI is that when a user looks at a visual stimulus with a certain frequency and duration, a component consistent with the stimulus frequency or its higher harmonics appears in the EEG signal. The user's intention is determined by detecting and classifying the SSVEP component in the EEG signal.

[0003] The stimulation frequency used by SSVEP-BCI is located in the three frequency bands, including low frequency (4 Hz-12 Hz), medium frequency (12 Hz-30 Hz) and high frequency (>30 Hz). Because the amplitude and signal-to-noise ratio (SNR) of low- and medium-frequency SSVEP are significantly higher than those of high-frequency SSVEP, the stimulation frequency of SSVEP-BCI systems in existing technologies is mostly concentrated in the low- and medium-frequency regions. Compared with high-frequency SSVEP, low- and medium-frequency SSVEP usually causes stronger visual discomfort to subjects. Summary of the Invention

[0004] The present invention provides a high-frequency visual evoked potential brain-computer interface encoding and decoding method, device and robotic arm control system, so as to achieve the effect of improving the eye comfort of the subject while consuming lower training costs while adopting high-frequency visual stimulation frequency.

[0005] In a first aspect, an embodiment of the present invention provides a high-frequency visual evoked potential brain-computer interface encoding method, comprising:

[0006] Displaying multiple instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block;

[0007] The frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz.

[0008] In a second aspect, an embodiment of the present invention further provides a high-frequency visual evoked potential brain-computer interface decoding method, corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in the first aspect, comprising:

[0009] Model each row instruction block and each column instruction block separately to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0010] The filtered visual evoked potential to be identified X test Input high-frequency row recognition model and high-frequency column recognition model respectively to obtain the visual evoked potential X to be identified test The row and column numbers of the .

[0011] In a third aspect, an embodiment of the present invention further provides a high-frequency visual evoked potential brain-computer interface decoding device, comprising:

[0012] A modeling module is used to model each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0013] The number acquisition module is used to obtain the filtered visual evoked potential X test Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential X to be identified test The row and column numbers of the .

[0014] In a fourth aspect, an embodiment of the present invention further provides a display device, including:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When one or more programs are executed by one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface encoding method as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present invention further provides a data acquisition and processing device, comprising:

[0019] one or more processors;

[0020] a storage device for storing one or more programs,

[0021] When one or more programs are executed by one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface decoding method described in the second aspect.

[0022] In a sixth aspect, an embodiment of the present invention further provides a robotic arm control system, comprising:

[0023] a display device for generating visual stimuli;

[0024] Data acquisition and processing equipment, used to collect and process data to obtain command signals;

[0025] A robotic arm control device, used to control the movement of the robotic arm according to a command signal;

[0026] Among them, the display device executes the high-frequency visual evoked potential brain-computer interface encoding method described in the first aspect, and the data acquisition and processing device executes the high-frequency visual evoked potential brain-computer interface decoding method described in the second aspect.

[0027] The technical solution of this embodiment is to display multiple instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value of the frequency-phase combination pair is greater than or equal to 30 Hz; when the subject looks at an instruction block, the brain induces visual evoked potentials corresponding to the rows and columns of the instruction block, respectively, so that the row number and column number of the instruction block can be decoded instead of decoding the visual evoked potential itself to be identified, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, thereby consuming lower training costs while adopting high-frequency visual stimulation frequencies, thereby achieving the effect of improving the eye comfort of the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the principle of steady-state visual evoked potential brain-computer interface (SSVEP-BCI) signal acquisition;

[0029] Figure 2 (a) is a schematic diagram of an instruction block corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention;

[0030] Figure 2 (b) is a schematic diagram of the visual stimulation interface corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention;

[0031] Figure 3 This is a flow chart of a high-frequency visual evoked potential brain-computer interface decoding method provided by the second embodiment of the present invention;

[0032] Figure 4 It is a flowchart of the rc-eTRCA algorithm;

[0033] Figure 5This is a flow chart of a high-frequency visual evoked potential brain-computer interface decoding method provided by Example 3 of the present invention;

[0034] Figure 6 is a flow chart of the method for detecting dynamic stop asynchrony of high-frequency visual evoked potentials;

[0035] Figure 7 A schematic structural diagram of a high-frequency visual evoked potential brain-computer interface decoding device provided in the fourth embodiment of the present invention;

[0036] Figure 8 A structural diagram of a data acquisition and processing device provided in Example 6 of the present invention;

[0037] Figure 9 This is a performance comparison diagram of high-frequency visual evoked potential brain-computer interface using the eTRCA algorithm and the rc-eTRCA algorithm;

[0038] FIG10( a ) is a schematic structural diagram of a robotic arm control system provided in accordance with a seventh embodiment of the present invention;

[0039] FIG10( b ) is a schematic diagram of a user interface presented on a display device;

[0040] Figure 11 This is the working principle diagram of the robotic arm control system. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0042] The main goal of brain-computer interface research is to create non-muscle communication channels so that people with severe motor impairments or other healthy people in need can use this technology to communicate with and control computers. Steady-state visual evoked potential-based brain-computer interfaces (SSVEP-BCIs) modulate EEG activity through visual channels and then decode the user's intentions by decoding the specific modulation information in the EEG.

[0043] Figure 1 This is a schematic diagram of the principle of steady-state visual evoked potential brain-computer interface (SSVEP-BCI) signal acquisition. As shown in the figure, SSVEP-BCI signal acquisition and processing mainly include three stages: stimulus presentation, data acquisition, and data processing and analysis. In the stimulus presentation stage, in order to ensure good control of visual stimulation, traditional computer screens or customized light-emitting diodes (LEDs) are usually used as visual stimulation interfaces. Figure 1 The eight circular icons displayed on the computer screen are eight different visual coding units, i.e., instruction blocks. When the user looks at different instruction blocks, different EEG signals can be induced at the scalp. During the data collection phase, the non-invasive brain-computer interface on the user's head collects scalp EEG. Figure 1 As shown in the frequency-domain EEG signals in Figure 2, different command blocks generate different frequency-domain EEG signals. For example, command blocks flashing at different frequencies induce different fundamental frequencies and peak harmonics. The data processing and analysis phase identifies the command block the user is focusing on and outputs the user's intent corresponding to that command block.

[0044] Regarding the selection of visual coding units, when the instruction block uses medium- and low-frequency stimulation signals, the fundamental frequency signal-to-noise ratio of the scalp EEG reaches 15dB to 25dB, while when the instruction block uses high-frequency stimulation signals above 30Hz, the signal-to-noise ratio of the scalp EEG is less than 10dB. In other words, using high-frequency stimulation signals to encode instruction blocks, although the visual stimulation received by the subjects is relatively small and can improve the subjects' visual comfort, the scalp EEG noise interference is relatively large and the amplitude is relatively small. Therefore, it is very difficult to train a high-frequency visual evoked potential brain-computer interface that meets the performance standards. It usually requires a lot of training data and the training cost is very high. Therefore, the SSVEP-BCI in the existing technology usually uses low-frequency signals for training and recognition.

[0045] Example 1

[0046] A high-frequency visual evoked potential brain-computer interface encoding method is disclosed. This embodiment is applicable to situations where a high-frequency signal is used to encode an instruction block. The method can be executed by a display device and specifically includes the following steps:

[0047] Multiple instruction blocks distributed in rows and columns are displayed on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz.

[0048] Optionally, each instruction block is jointly encoded by two frequency-phase combination pairs, and the two frequency-phase combination pairs are distributed left and right.

[0049] Figure 2 (a) is a schematic diagram of the instruction block corresponding to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1 of the present invention. Figure 2 (b) is a schematic diagram of the visual stimulation interface corresponding to the high-frequency visual evoked potential brain-computer interface coding method described in Example 1 of the present invention. Figure 2As shown, each instruction block is jointly encoded by two different frequency-phase combinations. The sinusoidal signal flashing in the instruction block is generated using a joint frequency-phase modulation method. Each instruction block is 120×120 pixels in size and consists of two 60×120 pixel rectangular blocks on the left and right. A cross serves as a focal point between the two blocks to facilitate the subject's determination of gaze position. This left-right distribution ensures that the flickering images generated by the bilateral blocks stimulate the left and right parts of the retina respectively, thereby projecting the evoked EEG signals to the contralateral occipital lobe area, thereby facilitating the discrimination between the two SSVEP frequencies.

[0050] In this embodiment, the row coding frequency and the column coding frequency are within different frequency ranges, with the row coding frequency being between 30.0 Hz and 33.2 Hz and the column coding frequency being between 34.0 Hz and 36.4 Hz, with a frequency interval of 0.8 Hz (the frequency values ​​used in this embodiment are typical values; other frequency values ​​may also be used). The sine wave of the stimulation signal is set with an initial phase, and the phase interval is 0.5π. Therefore, the two frequencies of the instruction can be separated by a filter, so that when training the recognition model, instructions with the same frequency-phase combination pair can be combined for training.

[0051] In this embodiment, 20 (typical) instruction blocks are arranged in a 4×5 (typical) matrix. In actual applications, the number and arrangement of instruction blocks can be adjusted according to actual needs. Each instruction block is modulated by high-frequency dual-frequency stimulation, stimulating the left and right visual fields (a checkerboard pattern or other dual-frequency modulation methods can also be used). The instructions for each row or column share a common frequency-phase combination pair, so that each row or column is modulated by a specific frequency-phase combination pair.

[0052] The present invention selects a total of 20 instruction blocks of nine frequency-phase combination pairs within the frequency range of 30Hz-36.4Hz for encoding. Based on this encoding protocol, a method based on spatial filters and template matching is used to train the model to decode the index of the row and column where the instruction corresponding to the specific visual evoked potential is located, rather than decoding the instruction itself, and then locating the specific instruction block by row and column. In this way, when training the recognition model of each row or column (spatial filter and SSVEP template), the training data of all instruction blocks in each row or column can be combined and used together, so that more training data is available when training the row recognition model and the column recognition model, thereby improving the reliability of the spatial filter and SSVEP template of the recognition model. Compared with the common method of encoding and decoding each instruction separately, this scheme achieves higher performance with less training data and uses a limited number of frequencies to encode more instruction blocks based on high-frequency SSVEP-BCI.

[0053] The technical solution of this embodiment is to display multiple instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; wherein the frequency value of the frequency-phase combination pair is greater than or equal to 30 Hz; when the subject looks at an instruction block, the brain induces visual evoked potentials corresponding to the rows and columns of the instruction block, respectively, so that the row number and column number of the instruction block can be decoded instead of decoding the visual evoked potential itself to be identified, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, thereby consuming lower training costs while adopting high-frequency visual stimulation frequencies, thereby achieving the effect of improving the eye comfort of the subject.

[0054] Example 2

[0055] Figure 3 This is a flowchart of a high-frequency visual evoked potential brain-computer interface decoding method provided by Example 2 of the present invention. This decoding method can be implemented by a data acquisition and processing device. Example 2 corresponds to the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1. After encoding the instruction block displayed on the visual stimulation interface using the encoding method described in Example 1, the EEG signal generated by the user looking at the visual stimulation interface is collected and decoded. The high-frequency visual evoked potential brain-computer interface decoding method specifically includes:

[0056] S310, modeling each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0057] S320, the filtered visual evoked potential to be identified X test Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential X to be identified test The row and column numbers of the .

[0058] Different from the decoding strategy of "decoding by target" in the prior art, the second embodiment of the present application adopts "decoding by rows and columns" corresponding to the encoding method of the first embodiment. "Decoding by target" in the prior art means that the decoding algorithm needs to model each instruction block. As the number of instruction blocks increases, the workload of modeling increases significantly. "Decoding by rows and columns" is a decoding strategy, not a decoding algorithm. This decoding strategy can be combined with any decoding algorithm, such as Task Related Component Analysis (TRCA), Canonical Correlation Aaronnalysis (CCA) and other commonly used algorithms for SSVEP decoding. The present invention takes the TRCA algorithm as an example.

[0059] This embodiment is based on the method of spatial filter and template matching to train the high-frequency row recognition model model and the high-frequency column recognition model to decode the row number and column number of the instruction block corresponding to the visual evoked potential to be identified, and then locate the instruction block itself through the row number and column number. Compared with "decoding by target", this allows more data to be used to train the spatial filter and SSVEP template when the number of subjects tested is certain, because the training data of all instruction blocks sharing a specific frequency-phase combination pair in each row or column will be combined. The technical solution of "decoding by row and column" in this embodiment is based on the row-column ensemble TRCA (rc-eTRCA) algorithm. rc-eTRCA can obtain the spatial filter and SSVEP template for each row or column by using the training data of all instruction blocks of the corresponding rows or columns, thereby reducing the average training cost, where the "average training cost" refers to the average number of training times for each instruction block.

[0060] Figure 4 This is a flow chart of the rc-eTRCA algorithm. Figure 4 As shown, assuming Represents a single three-dimensional training data, that is, the training visual evoked potential χ ij Where i represents the row number of the instruction block that the subject is looking at, j represents the column number of the instruction block, and N ch is the number of channels, N s is the number of sampling points for each test, N t is the number of training trials. ij Rearrange the rows and columns and concatenate the data with the same frequency-phase combination to obtain:

[0061]

[0062] Where Cat(·) indicates that the matrix is ​​concatenated along the third dimension. and are the training data of the nth column and nth row respectively. and Respectively represent the number of columns and rows. The filter bank (FB) is used to extract SSVEP components of different frequency bands. According to the stimulation frequency of the instruction block, the column filter bank (FB) in this embodiment col ) and row filter bank (FB row ) are set to [(m×30-2)Hz, 90Hz] and [(m×33.2)Hz, 90Hz], where m represents the subband index. Then, the column space filter W of each passband is obtained using the eTRCA algorithm.col(m) and row space filter W row(m) And by calculating the average value of multiple trials, the column template signal of each passband is obtained. and row template signals The specific calculation formula is as follows:

[0063]

[0064] Where h represents the sample number of the visual evoked potential used for training.

[0065] After obtaining each row template signal, column template signal, column space filter and row space filter, the high-frequency row recognition model and high-frequency column recognition model are obtained. Next, the high-frequency row recognition model and high-frequency column recognition model that have been obtained are used to identify the visual evoked potential X test Make identification judgment. X test col(m) Correlation coefficient with column template signal and X test row(m) With line template signal The correlation coefficient between them can be calculated by the following formula, where X test col(m) With X test row(m) The visual evoked potential X is identified by using a column filter bank and a row filter bank. test After filtering, we get:

[0066]

[0067] ρ(a,b) represents the Pearson correlation coefficient between a and b. The weighted sum of the squares of all sub-band correlation coefficients is defined as the classification feature:

[0068]

[0069] where N m =2 is the number of subbands, a(m)=m -1.25 .

[0070] Take the classification features of the columns separately and row classification features The column number and row number corresponding to the maximum value of , thereby obtaining the visual evoked potential X to be identified test Column and row numbers:

[0071]

[0072] Among them, i and j are the visual evoked potentials X to be identified test The visual evoked potential X to be identified can be determined based on the row number i and column number j. test The corresponding instruction block τ=τ ij .

[0073] The technical solution of this embodiment is to obtain a high-frequency row recognition model and a high-frequency column recognition model by modeling each row instruction block and each column instruction block respectively; the filtered visual evoked potential to be recognized X test Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential X to be identified test that is, based on the row template signal, column template signal, column spatial filter and row spatial filter, the row number and column number of the visual evoked potential to be identified are decoded (instead of decoding the visual evoked potential to be identified itself), and then the row number and column number are used to locate the specific instruction block. In this way, when training the row template signal, column template signal, column spatial filter or row spatial filter of each row or column, all the training data in each row or column can be combined and used together, so that the high-frequency recognition model has more training data available, and the reliability of the spatial filter and template signal is improved. Compared with the method of decoding each instruction block separately in the prior art, this scheme achieves higher performance with less training data, and uses a limited frequency to realize the decoding of a larger number of instruction blocks based on high-frequency SSVEP-BCI, thereby solving the problem of high training cost caused by low signal-to-noise ratio of high-frequency signals in the prior art, and can consume lower training costs while adopting high-frequency visual stimulation frequency, thereby achieving the effect of improving the eye comfort of the subject.

[0074] Figure 9 This is a performance comparison of high-frequency visual evoked potential brain-computer interface using the eTRCA algorithm and the rc-eTRCA algorithm, where the error bars represent standard errors and the asterisks represent the significance of the differences obtained by paired t-test (*p<0.05, **p<0.01, ***p<0.001). Figure 9 (a) shows the comparison of recognition accuracy of the "target decoding" eTRCA algorithm and the "row and column decoding" rc-eTRCA algorithm under different data lengths when the number of training trials for each instruction block is 3. Figure 9 As shown in (a) in the figure, the recognition accuracy of the rc-eTRCA algorithm is higher than that of the eTRCA algorithm under all data lengths. Figure 9 Figure (b) shows the recognition accuracy of the eTRCA and rc-eTRCA algorithms for instruction blocks with different numbers of training trials when the data length is 1.5 seconds. This shows that the row-column decoding strategy proposed in this embodiment significantly improves the recognition accuracy of instruction blocks with less training data compared to the traditional non-row-column decoding strategy.

[0075] Example 3

[0076] Figure 5This is a flowchart of a high-frequency visual evoked potential brain-computer interface decoding method provided by Example 3 of the present invention. This decoding method can be implemented by a data acquisition and processing device. Example 3 is a further optimization of the high-frequency visual evoked potential brain-computer interface encoding method described in Example 2, specifically including:

[0077] S510 , training a classifier for distinguishing between a control state IC and a non-control state NC.

[0078] Synchronous SSVEP-BCIs assume that the user's EEG signals are valid when identifying their visually evoked potentials (VEPs), meaning they are generated when the user is focusing on a specific instruction block. However, synchronous SSVEP-BCIs have the problem of limited user autonomy during human-computer interaction, requiring the user to maintain constant attention. Therefore, asynchronous SSVEP-BCIs, which can determine the user's state, have greater practical advantages.

[0079] The key challenge in designing an asynchronous SSVEP-BCI lies in accurately and efficiently distinguishing the user's current state, that is, determining whether the user is in an intentional control (IC) or non-control (NC) state. The intentional control state refers to when the user is focusing on the instruction block, while the non-control state refers to other states, such as being in a daze or looking at something other than the instruction block. The non-control state is also known as the idle state.

[0080] In order to improve the classification efficiency and practicality of the classifier, the classifier may optionally have an adaptive window function, and the visual evoked potential X to be identified test The length of starts from the initial length and increases by a specific step size.

[0081] Since the present embodiment designs an asynchronous SSVEP-BCI, the visual evoked potential X to be identified is input into the classifier. test It is not necessarily the EEG signal generated when the user is in the control state, but it may be the EEG signal generated when the user is in the non-control state and has nothing to do with the instruction block recognition. test When distinguishing between the control state and the non-control state, if a shorter EEG signal can be used to determine that the brain is in the non-control state, there is no need to further process the subsequent EEG signal of the non-control state.

[0082] That is to say, this embodiment provides a technical solution that combines "decoding by rows and columns", "control state detection method" and "adaptive window method" to achieve dynamic stop asynchronous detection of high-frequency visual evoked potentials. The adaptive window method refers to not using a fixed-length EEG signal to identify whether the user is in a control state, but when the length of the EEG signal can reliably draw a judgment conclusion, it determines whether the EEG signal is a control state visual evoked potential. The following will explain in detail how to train a classifier that distinguishes between control state IC and non-control state NC. Specifically, step S510 includes using IC data of the same length and NC data of different lengths to train a classifier that distinguishes between control state IC and non-control state NC. The following will explain the training process of the classifier.

[0083] Figure 6 This is a flow chart of the method for detecting dynamic stop asynchrony of high-frequency visual evoked potentials. Figure 6 As with the second embodiment, the row space filter is W row(m) , the column space filter is W col(m) , calculate the correlation coefficient between the filtered control state training visual evoked potential and the template signal:

[0084]

[0085] Where n represents the column number or row number, and h represents the sample number of the visual evoked potential used for control state training. and It can be calculated using formulas (3) and (4) in Example 2.

[0086] Calculate the data length as t n Visual evoked potentials Correlation coefficient with the template signal:

[0087]

[0088] As shown in formula (16) and (17), and As a feature of the classifier for distinguishing the control state from the non-control state. To improve the performance of the classifier, this embodiment uses non-control state training visual evoked potentials of different lengths to train the classifier, but uses control state training visual evoked potentials of the same length to train the classifier, thereby minimizing the time required for classifier training.

[0089]

[0090] Since the minimum data length that can be accessed online by the EEG signal acquisition device when the non-invasive brain-computer interface acquires scalp EEG is 0.04s, the data length t in this embodiment is nThe range of variation is [0.62s, 1.5s] with an interval of 0.04s. Due to the influence of visual delay in the human visual system, if the instruction block starts flashing at 0s, EEG data can be extracted within the range of [0.14s, (0.76+n×0.04)s].

[0091] S520 , modeling each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model.

[0092] It should be noted that the execution order of step S510 and step S520 can be adjusted as needed.

[0093] S530, the filtered visual evoked potential to be identified X test Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential X to be identified test The row and column numbers of the .

[0094] S540, the visual evoked potential to be identified X test Input into the classifier to determine the visual evoked potential X to be identified test It is the control state visual evoked potential or the non-control state EEG signal.

[0095] According to formulas (5)-(11) in the second embodiment, the visual evoked potential X to be identified is obtained based on the rc-eTRCA algorithm in step S530. test The row number i and column number j of the visual evoked potential to be identified are then determined test The corresponding instruction block τ may be, because the visual evoked potential X to be identified has not been detected at this time test Whether it is a control state visual evoked potential, so the statement here is "possibly" corresponding instruction block τ.

[0096] Adaptive window method based on covariance analysis is used to determine whether the current data length is sufficient to reliably determine the visual evoked potential X to be identified. test The weighted sum of the squares of the correlation coefficients ρ is used to determine the visual evoked potential X to be identified. test The column number and row number of , that is, determine the possible corresponding instruction block τ. test The incremental data packet length is 0.04 seconds, the initial data length t0 is 0.62 seconds, and the maximum data length t0 is 0.62 seconds when the maximum step length 22 is reached. x 1.50 seconds.

[0097] t n =t n-1 +0.04,n≤22 (13)

[0098] After the data length is updated, calculate the visual evoked potential X to be identified with the latest data length test ( Figure 6 Medium X tn )’s rows and columns:

[0099]

[0100] in and Calculate according to formula (7) and (8) in Example 2, and then determine by covariance analysis method With the eigenvector [t0,…,t n ]([0.62,…,1.50]). As the window length, i.e., the data length, increases, until a statistically significant effect (p<0.05) appears, then the data length t n The corresponding classifier determines whether the user is in a control state or a non-control state. If it is determined that the user is in a control state at this time, the final recognition result instruction block τ is output; otherwise, the user is determined to be in a non-control state.

[0101] The technical solution of this embodiment is to train a classifier to distinguish between the control state IC and the non-control state NC; test Input into the classifier to determine the visual evoked potential X to be identified test In order to control the visual evoked potential or the non-control state EEG signal, the implementation difficulty of the dynamic stop asynchronous detection method of high-frequency visual evoked potential was solved, and the goal of improving user autonomy in the human-computer interaction process was achieved. The training time and detection time were effectively shortened, and the "high-frequency signal", "row and column decoding", "control state detection method" and "adaptive window method" were combined.

[0102] Example 4

[0103] Figure 7 This is a schematic diagram of the structure of a high-frequency visual evoked potential brain-computer interface decoding device provided in Example 4 of the present invention. This high-frequency visual evoked potential brain-computer interface decoding device can execute the high-frequency visual evoked potential brain-computer interface decoding method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0104] A high-frequency visual evoked potential brain-computer interface decoding device, comprising:

[0105] Modeling module 701, used to model each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model;

[0106] The number acquisition module 702 is used to obtain the filtered visual evoked potential Xtest Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential X to be identified test The row and column numbers of the .

[0107] Optionally, the high-frequency visual evoked potential brain-computer interface decoding device also includes:

[0108] A classifier training module is used to train a classifier for distinguishing between a control state IC and a non-control state NC;

[0109] The control state determination module is used to determine the visual evoked potential X to be identified test Input into the classifier to determine the visual evoked potential X to be identified test It is the control state visual evoked potential or the non-control state EEG signal.

[0110] Optionally, the classifier training module is further configured to use IC data of the same length and NC data of different lengths to train a classifier for distinguishing between controlled state IC and non-controlled state NC.

[0111] Example 5

[0112] Figure 1 The display on the left is a schematic diagram of the structure of the display device provided by Example 4 of the present invention. The display device can execute the high-frequency visual evoked potential brain-computer interface coding method provided by Example 1 of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] A display device comprising:

[0114] one or more processors;

[0115] a storage device for storing one or more programs,

[0116] When the one or more programs are executed by the one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface encoding method as described in Example 1.

[0117] Example 6

[0118] Figure 8 A structural diagram of a data acquisition and processing device provided in Example 6 of the present invention is shown as follows: Figure 8 As shown, the data acquisition and processing device includes a processor 810, a memory 820, an input device 830 and an output device 840; the number of processors 810 in the data acquisition and processing device can be one or more. Figure 8 In the figure, a processor 810 is used as an example; the processor 810, the memory 820, the input device 830 and the output device 840 in the data acquisition and processing device can be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0119] The memory 820, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the high-frequency visual evoked potential brain-computer interface decoding method in the embodiments of the present invention (for example, the modeling module 701 and the number acquisition module 702 in the high-frequency visual evoked potential brain-computer interface decoding device). The processor 810 executes the various functional applications and data processing of the data acquisition and processing device by running the software programs, instructions, and modules stored in the memory 820, thereby implementing the above-mentioned high-frequency visual evoked potential brain-computer interface decoding method.

[0120] The memory 820 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. Furthermore, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 820 may further include a memory remotely located relative to the processor 810, and these remote memories may be connected to the data acquisition and processing device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The input device 830 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the data acquisition and processing device. The output device 840 can include a display device such as a display screen.

[0122] Example 7

[0123] FIG10( a ) is a schematic structural diagram of a robotic arm control system provided in accordance with a seventh embodiment of the present invention. The robotic arm control system includes:

[0124] a display device for generating visual stimuli;

[0125] Data acquisition and processing equipment, used to collect and process data to obtain command signals;

[0126] A robotic arm control device, used to control the movement of the robotic arm according to a command signal;

[0127] Among them, the display device executes the high-frequency visual evoked potential brain-computer interface encoding method described in Example 1, and the data acquisition and processing device executes the high-frequency visual evoked potential brain-computer interface decoding method described in Example 2 or Example 3.

[0128] As shown in Figure 10(a), the robotic arm and workspace (approximately 550 mm × 550 mm) are located to the user's right. A camera captures the workspace and transmits real-time images to a display device (Alienware AW2720HF, 27 inches, 240 Hz refresh rate, 1920 × 1080 resolution), which displays the visual stimulation interface. Figure 10(b) is a schematic diagram of the user interface displayed on the display device. Twenty command blocks are displayed on the right, with the row and column numbers corresponding to the workspace marked on the user interface. The workspace includes a capture area and a placement area. When the user selects a command block in the 4 × 4 grid on the user interface, the robotic arm moves to the corresponding position. Furthermore, the "Release," "Grab," "Raise," and "Reset" commands marked on the user interface represent opening the robotic arm gripper, closing the gripper, raising the gripper 30 cm, and resetting the robotic arm, respectively.

[0129] Figure 11 This is a diagram of the working principle of the robotic arm control system. A camera is used to display the robotic arm's interactions with other objects and its own motion in real time. The data acquisition and processing equipment, display device, and robotic arm control device are connected via USB ports. The data acquisition and processing equipment communicates with the display device and robotic arm control device via TCP / IP. The display device and data acquisition and processing equipment use the Windows 10 operating system, while the robotic arm control device uses Ubuntu 16.04.

[0130] The experimental operation process of the robotic arm control system is as follows:

[0131] The experiment consisted of two parts: offline and online. The offline experiment was used to build high-frequency row and column recognition models, as well as a classifier for distinguishing between controlled and uncontrolled states. The online experiment, which included a command block recognition task and a robotic arm control task, tested the model and classifier constructed in the offline experiment. Due to the influence of visual delay in the human visual system, if the command block starts flashing at 0 seconds, EEG data can be extracted within the range of [0.14 seconds, 1.64 seconds]. A detailed description of the experimental process is as follows:

[0132] Offline Experiments: Training visual evoked potentials (VEPs) obtained during offline experiments are used to verify the algorithm's performance and train the recognition model and classifier for subsequent online experiments. Each instruction block can be trained with 4 trials of IC data and 40 trials of NC data. Increasing the number of training trials can improve the accuracy of the row and column recognition models, as well as the classifier. Specifically, the offline experiments consisted of four groups, each containing 10 NC trials and 20 IC trials, with each instruction block undergoing one IC trial. Each trial began with a 1-second prompt phase. The dark-colored instruction block in Figure 10(a) prompted the user to focus on the instruction block during the IC trial phase. When the triangle in the center of the user interface turned dark, it indicated an NC trial phase, and the user should focus on the real-time video on the left side of the user interface. Subsequently, all instruction blocks flashed simultaneously for 1.5 seconds, and the user was instructed to focus on the corresponding instruction block or real-time video. According to the high-frequency VEP brain-computer interface decoding method of Example 3, a high-frequency row recognition model, a high-frequency column recognition model, and a classifier for distinguishing between control and non-control states were trained.

[0133] Online experimental instruction block recognition task: A total of 40 IC tests and 22 NC tests were conducted. During the IC test, the target instruction block turned dark for 1 second to prompt the user to fixate their gaze on it. Then, all instruction blocks flashed. The flashing duration was determined by the "adaptive window method" in Example 3, with a maximum duration of 1.5 seconds. During the NC test, the user was instructed to turn their attention to the video on the left side of the user interface.

[0134] Online Experimental Robotic Arm Control Task: As shown in Figure 10(b), the workspace consists of a capture zone and a placement zone. Six foam blocks are randomly placed in the capture zone. The user controls the robotic arm control system to grab a foam block in the capture zone and release it at a designated location in the placement zone. First, the user looks at the instruction block on the user interface corresponding to the foam block to be grabbed. When the robotic arm moves to the top of the foam block, the user looks at the instruction block corresponding to the "grab" instruction to pick up the foam block. Then, the user looks at the instruction block on the user interface corresponding to the placement zone. When the robotic arm moves above the position corresponding to that instruction block, the user looks at the instruction block corresponding to the "release" instruction to place the foam block.

[0135] Example 8

[0136] Embodiment 8 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the high-frequency visual evoked potential brain-computer interface encoding method described in Embodiment 1 or the high-frequency visual evoked potential brain-computer interface decoding method described in Embodiment 2.

[0137] Of course, the storage medium containing computer-executable instructions provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the high-frequency visual evoked potential brain-computer interface encoding method or the high-frequency visual evoked potential brain-computer interface decoding method provided by any embodiment of the present invention.

[0138] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0139] It is worth noting that in the embodiment of the above-mentioned high-frequency visual evoked potential brain-computer interface decoding device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0140] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A high-frequency visual evoked potential brain-computer interface decoding method, corresponding to a high-frequency visual evoked potential brain-computer interface encoding method, characterized in that: The high-frequency visual evoked potential brain-computer interface coding method comprises: Displaying a plurality of instruction blocks distributed in rows and columns on a visual stimulation interface, each instruction block is jointly encoded by two frequency-phase combination pairs, and the row and column positions of each instruction block correspond to the two frequency-phase combination pairs of the instruction block; Wherein, the frequency value in the frequency-phase combination pair is greater than or equal to 30 Hz; The high-frequency visual evoked potential brain-computer interface decoding method comprises: Model each row instruction block and each column instruction block separately to obtain a high-frequency row recognition model and a high-frequency column recognition model; Train a classifier to distinguish between controlled state IC and non-controlled state NC; The filtered visual evoked potential to be identified Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential to be identified The row and column numbers of Visual evoked potentials to be identified Input into the classifier to determine the visual evoked potential to be identified It is a control state visual evoked potential or a non-control state electroencephalogram signal; Wherein, the classifier has an adaptive window function, and the visual evoked potential to be identified The length of starts from the initial length and increases by a specific step size.

2. The high-frequency visual evoked potential brain-computer interface decoding method according to claim 1, characterized in that: The training of a classifier for distinguishing between a control state IC and a non-control state NC includes: IC data of the same length and NC data of different lengths are used to train a classifier to distinguish between controlled IC and non-controlled NC.

3. A high-frequency visual evoked potential brain-computer interface decoding device, used to execute the high-frequency visual evoked potential brain-computer interface decoding method according to claim 1 or 2, characterized in that: include: A modeling module (701) is used to model each row instruction block and each column instruction block respectively to obtain a high-frequency row recognition model and a high-frequency column recognition model; A classifier training module is used to train a classifier for distinguishing between a control state IC and a non-control state NC; The number acquisition module (702) is used to obtain the filtered visual evoked potential to be identified Input the high-frequency row recognition model and the high-frequency column recognition model respectively to obtain the visual evoked potential to be identified The row and column numbers of The control state determination module is used to identify the visual evoked potential Input into the classifier to determine the visual evoked potential to be identified It is the control state visual evoked potential or the non-control state EEG signal.

4. A data acquisition and processing device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the high-frequency visual evoked potential brain-computer interface decoding method as described in claim 1 or 2.

5. A robotic arm control system, characterized in that: include: a display device for generating visual stimuli; Data acquisition and processing equipment, used to collect and process data to obtain command signals; A robotic arm control device, used to control the movement of the robotic arm according to a command signal; Wherein, the display device executes the high-frequency visual evoked potential brain-computer interface encoding method, and the data acquisition and processing device executes the high-frequency visual evoked potential brain-computer interface decoding method described in claim 1 or 2.

Images