Self-adaptive dynamic feedback brain-computer interface training method and system based on virtual reality

Abstract

The invention relates to the technical field of brain-computer interfaces, and discloses a self-adaptive dynamic feedback brain-computer interface training method and system based on virtual reality. The training method comprises the steps that a user performs motor imagery according to a training scene presented by the VR display device; the electroencephalogram acquisition equipment acquires electroencephalogram signals; configuring a data processing module, and operating a motor imagery continuous control model to calculate characteristic parameters by taking the electroencephalogram signals as input; under the condition that the task exists, the characteristic parameters are used as input, and a dynamic self-adaptive characteristic-feedback mapping model is operated to calculate control parameters; sending the control parameters to a functional electrical stimulation device to load electrical stimulation of corresponding intensity to the muscle to obtain tactile feedback; meanwhile, the dynamic adaptive feature-feedback mapping model is dynamically updated according to latest input feature parameters; the control parameters are sent to the VR display device to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and the haptic feedback. The participation sense of the user and the training efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a virtual reality-based adaptive dynamic feedback brain-computer interface training method and system. Background Art

[0002] In recent years, motor imagery (MI) methods have been increasingly introduced into sports training. Motor imagery is a psychological process in which the brain imagines imitating specific movements without the actual execution of the movements. According to the research on the mirror neuron system and the theory of neuroplasticity, MI can strengthen the activation of motor nerves in related areas during motor nerve training, help reshape damaged motor nerve pathways or enhance healthy nerve pathways, and can be used as a training method for motor functional disorders caused by diseases such as stroke. Brain-computer interface (BCI) technology can convert the read brain activity (such as EEG signals) into control instructions for external devices. Among them, the brain-computer interface based on motor imagery can identify the user's movement intention in the brain and feedback it to the user. At the same time, functional electrical stimulation (FES) directly stimulates and activates muscles through electrodes, so that users can complete preset movements even when their autonomous movement ability is limited. Therefore, researchers proposed a closed-loop training system consisting of brain-computer interface technology and functional electrical stimulation. The system first decodes the user's movement intention, controls the functional electrical stimulation device to drive limb movement, and realizes the synchronous coupling of cortical and muscle activities. Through closed-loop neurofeedback rehabilitation training, neural pathways are rebuilt.

[0003] Combining the motor imagery brain-computer interface with functional electrical stimulation can help users with their training, but it does not solve the problem of the monotonous and boring training process. Although the use of virtual reality (VR) technology can enhance the fun and immersion, the existing training systems often use a single scene and repetitive tasks. In addition, most of the existing feedback technologies rely on pattern recognition results, and it is necessary to collect user data and establish a calibration model before training, which not only increases the time cost of system use, but also is easily restricted by the pattern recognition algorithm. In addition, the neurofeedback technology based on pattern recognition can only provide discrete control and cannot intuitively reflect the real-time state of the user's movement intention. Long-term use can easily lead to user fatigue and decreased willingness to cooperate, resulting in the user's inability to obtain the maximum benefit during the training process. More importantly, in actual neurofeedback rehabilitation training, due to the large temporal variability of the user's brain activity data (such as EEG, etc.), the model established before training is difficult to effectively match the current neural activity state.

[0004] Therefore, the existing systems and methods have the following defects: first, it is necessary to collect user data and establish a calibration model before training, which results in additional time consumption; second, the established model is difficult to accurately feedback the user's current neural activity state, resulting in a mismatch between the feedback obtained by the user during training and the current actual neural signal state, affecting the training efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide an adaptive dynamic feedback brain-computer interface training method and system based on virtual reality, which combines virtual reality technology with functional electrical stimulation to provide users with a more immersive and interesting virtual sports training scene, improve users' training motivation and sense of participation, and adopts an adaptive and dynamic feedback parameter calculation method, which can adjust the mapping relationship between characteristic parameters and control parameters according to the user's EEG signal characteristics within a current period of time, realize dynamic adjustment of training difficulty, and improve training efficiency.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] In a first aspect, the present invention provides an adaptive dynamic feedback brain-computer interface training method based on virtual reality. Before executing the training method, the user wears an electroencephalogram acquisition device, a VR display device, and a functional electrical stimulation device; wherein the training scene is pre-built in the VR display device; the training method comprises the following steps:

[0008] S1. The user performs movement imagination based on the training scene presented by the VR display device;

[0009] S2. The EEG acquisition device collects EEG signals;

[0010] S3. Configure the data processing module and execute the following sub-steps:

[0011] S30. Using the EEG signal as input, running the motor imagery continuous control model to calculate and obtain characteristic parameters;

[0012] S31. In the case of determining that there is a task, taking the characteristic parameters as input, running the dynamic adaptive characteristic-feedback mapping model to calculate and obtain the control parameters;

[0013] S4. Sending the control parameters to the functional electrical stimulation device to load the muscle with electrical stimulation of corresponding intensity to obtain tactile feedback; at the same time, the dynamic adaptive feature-feedback mapping model is dynamically updated according to the latest input feature parameters;

[0014] S5. Send the control parameters to the VR display device to present visual feedback, and the user adjusts the movement imagination based on the visual feedback and tactile feedback and then repeats S1 to S5.

[0015] As a possible implementation, the EEG signal includes a task-state EEG signal and a non-task-state EEG signal; S30 specifically includes the following sub-steps:

[0016] S300. Configure and slide the time window to continuously capture the task-state EEG signal to obtain a multi-channel task-state EEG signal;

[0017] S301. Calculate the power spectral density matrix P of multi-channel task-state EEG signals n ;

[0018] S302. Calculate the power spectral density matrix P of the non-task state EEG signal rest ;

[0019] S303. Calculate the relative power spectrum density matrix RPSD n , Among them, N is the number of sampling channels of EEG signals, Nt is the number of sampling points in the sliding time window, is the set of real numbers;

[0020] S304. Select the target frequency band related to motor imagery and calculate the relative power value RP corresponding to the target frequency band F ;

[0021] S305. According to the motor imagery action and the corresponding limb, select the corresponding channel combination, assign weights to the relative power values ​​of different channels, and obtain the characteristic parameters with the greatest difference.

[0022] As a possible implementation method, the power spectral density matrix P of the multi-channel task-state EEG signal is calculated as follows: n :

[0023]

[0024] Among them, P n represents the power spectral density matrix at time point n, represents the EEG signal captured by the sliding time window corresponding to time point n, N is the number of sampling channels of the EEG signal, Nt is the number of sampling points in the sliding time window, F(X n ) represents the EEG signal X n The Fourier transform of is the set of real numbers.

[0025] As a possible implementation method, the relative power value RP corresponding to the target frequency band is calculated by the following method: F :

[0026]

[0027] Where F represents the target frequency band, Where fs represents the sampling rate of the EEG signal, It represents the power corresponding to frequency f in the relative power spectrum density, reflecting the frequency domain-energy characteristics of the whole brain at any time period.

[0028] As a possible implementation, the characteristic parameters are calculated as follows:

[0029]

[0030] in, For the selected channel combination, is the weight corresponding to the c channel, Indicates the relative power of channel c in the target frequency band F, is the set of real numbers.

[0031] As a possible implementation, before running the dynamic adaptive feature-feedback mapping model, the training method also includes:

[0032] Before the first motor imagery task begins, a dynamic adaptive feature-feedback mapping model is constructed, which includes the following steps:

[0033] Select a feature-feedback mapping function, which is a function related to feature parameters, feature parameter thresholds, and feature parameter baselines at any time;

[0034] Calculate the baseline initial value C of the characteristic parameter B0 and the initial value of the characteristic parameter threshold C T0 , specifically including the following sub-steps:

[0035] Calculate the set of non-task state characteristic parameters before the start of the first motor imagery task

[0036] According to the set C NT The data distribution of non-task characteristic parameters within the range is used to obtain the maximum value C of non-task characteristic parameters. NTmax ∈C NT , and the third quartile Q 3NT ∈C NT ;

[0037] Q 3NT is the upper limit of the outlier of the first motor imagery task state, C NTmax is the lower limit of the outlier of the first motor imagery task state. Assuming that the characteristic parameters of the first motor imagery task state are uniformly distributed, the baseline initial value C of the characteristic parameter is determined by the following method. B0 and the initial value of the characteristic parameter threshold C T0 :

[0038]

[0039] As a possible implementation method, after the first motor imagery task is performed, the feature parameter baseline value C is iterated by the following method B and characteristic parameter threshold C T :

[0040] The characteristic parameters C obtained at any motor imagery task state Xn Add task state feature parameter set C Test and follow C Test The distribution construction value range of the characteristic parameters in is:

[0041]

[0042] Among them, max(C Test ,C Xn ) and min(C Test ,C Xn ) represent the maximum and minimum values ​​of the two respectively;

[0043] When C Test When the amount of data in the collection exceeds the set length limit, the feature parameters that are the oldest from the current time point are removed, that is, the latest feature parameters are retained.

[0044] As a possible implementation method, when dynamically iterating the characteristic parameter value range each time, a correction function f is introduced to correct C T C Xn The degree of influence, that is, C T =f[max(C Test ,C Xn )].

[0045] In a second aspect, the present invention provides an adaptive dynamic feedback brain-computer interface training system based on virtual reality, comprising an electroencephalogram acquisition device, a data processing module, a VR display device and a functional electrical stimulation device;

[0046] Among them, the training scene is pre-built in the VR display device, and the user imagines the movement according to the training scene presented by the VR display device;

[0047] The EEG acquisition device is used to collect EEG signals when the user performs motor imagery;

[0048] The data processing module is used to receive EEG signals and run a motor imagery continuous control model to calculate characteristic parameters; it is also used to use the characteristic parameters as input to run a dynamic adaptive characteristic-feedback mapping model to calculate control parameters when it is determined that there is a task;

[0049] The functional electrical stimulation device receives control parameters to load electrical stimulation of a preset intensity on the muscles to obtain tactile feedback; the VR display device is also used to receive control parameters to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and tactile feedback so that the EEG acquisition device re-acquires EEG signals.

[0050] As a possible implementation method, the EEG acquisition device includes electrodes and an EEG amplifier. The electrodes acquire the user's EEG signals, and the EEG signals are amplified and filtered by the EEG amplifier before being sent to the data processing module; the training system also includes a first control computer and a second control computer. The first control computer is used to receive control parameters, and generate a first control instruction based on the control parameters and transmit it to the VR display device. The second control computer is used to receive control parameters, and generate a second control instruction based on the control parameters and transmit it to the functional electrical stimulation device; the data processing module includes an acquisition computer, and the acquisition computer is used to calculate characteristic parameters and control parameters, and transmit the control parameters to the first control computer and the second control computer.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] 1. The adaptive dynamic feedback brain-computer interface training method based on virtual reality provided by the present invention does not require additional data collection and model training in the initial stage, thus avoiding additional consumption of time; at the same time, the present invention continuously constructs the latest mapping relationship between the control parameters and the latest feature parameter distribution during the training process, which can maximize the guarantee that the training parameters of the current system are adapted to the user status, realize dynamic adjustment of task difficulty, and improve training efficiency.

[0053] 2. The adaptive dynamic feedback brain-computer interface training method and system based on virtual reality provided by the present invention forms a complete closed-loop training system by combining BCI, FES and VR technologies. During use, the VR display device can provide task prompts for motor imagery action patterns. BCI analyzes the user's movement intention when performing the task, and contracts the corresponding muscles through functional electrical stimulation to complete the limb movements of the same mode as the motor imagery task prompted in the training system, coupling the spontaneous movement intention at the neural level with the passive movement at the muscle level, thereby improving the repair of the original neuromuscular control pathway or the construction of a new neuromuscular control pathway, and can also improve the user's sense of participation in the training process, forming an efficient feedback loop.

[0054] 3. The adaptive dynamic feedback brain-computer interface training method based on virtual reality provided by the present invention uses a data interception method of a short-interval sliding time window with a high time resolution to analyze the features related to movement intention in the EEG signals in the current time period. It does not require pattern recognition, but directly feeds back feature parameters. Therefore, its instructions are continuous rather than discrete, and can more intuitively reflect the intensity of the movement intention contained in the user's current EEG state, which is beneficial to improving the user's ability to control spontaneous EEG. Continuous control helps to obtain more real-time feedback from the user and optimize the control experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0056] Figure 1 A schematic diagram of the structure of an adaptive dynamic feedback brain-computer interface training system in virtual reality provided by an embodiment of the present invention;

[0057] Figure 2 A flow chart of a method for training a brain-computer interface based on adaptive dynamic feedback in virtual reality provided by an embodiment of the present invention;

[0058] Figure 3 A flow chart of a method for calculating characteristic parameters of a continuous control model of running motor imagery provided in an embodiment of the present invention;

[0059] Figure 4 This is a system logic structure and flow chart of a user using the training system and training method to perform training in an embodiment of the present invention.

[0060] Reference numerals

[0061] 1-EEG acquisition device, 10-electrode, 11-EEG amplifier, 2-data processing module, 20-acquisition computer, 3-VR display device, 4-functional electrical stimulation device, 5-first control computer, 6-second control computer. DETAILED DESCRIPTION

[0062] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0063] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0064] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The following at least one item (items) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (items) of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0065] The present invention aims to solve the following technical problems: First, the existing motor imagery training methods lack feedback or have a single feedback mode, the training tasks are repetitive and boring, the recognition accuracy and control accuracy of motor intentions are not high, and the existing methods need to rely on EEG signals for a long time to establish a training model, which easily leads to user fatigue, frustration, and decreased willingness to cooperate; Second, how to adaptively adjust the task difficulty and feedback intensity dynamically according to the user's current state during the training process while ensuring the training effect, thereby improving the rehabilitation efficiency.

[0066] In a first aspect, the present invention provides an adaptive dynamic feedback brain-computer interface training method based on virtual reality. Figure 1 Before executing the training method, the user wears the EEG acquisition device 1, the VR display device 3 and the functional electrical stimulation device 4; wherein the training scene is pre-built in the VR display device 3;

[0067] Exemplarily, the EEG acquisition device 1 collects the user's EEG signals, and amplifies and filters the EEG signals. The functional electrical stimulation device 4 provides electrical stimulation of corresponding intensity to the user's muscles according to the control parameters, and obtains tactile feedback. In the present invention, the VR display device 3 can use either a VR all-in-one machine or a head-mounted display. The present invention builds a life-like virtual environment based on virtual reality technology to enhance the user's experience. During the training process, the user can see his or her movements in real time from a first-person perspective in the virtual environment and obtain multi-sensory feedback. In order to further enhance the sense of immersion during the training process, the user can specify a training environment for the user or use a 6-degree-of-freedom (DOF) device during use to make the user's posture in reality consistent with the posture of the character in the virtual environment.

[0068] See also Figure 1 to Figure 2 , the training method includes the following steps:

[0069] S1. The user performs motor imagination according to the training scene presented by the VR display device 3; specifically, the VR display device 3 presents the pre-built training scene to the user in real time, and prompts the user to perform motor imagination of the corresponding limbs and actions according to the task when appropriate. The user performs motor imagination of the corresponding actions according to the prompts, such as hand grasping, flexion and extension of limbs, etc. During the motor imagination process, EEG signals are generated.

[0070] S2. The EEG acquisition device 1 collects EEG signals;

[0071] As an example, scalp electrodes 10 are used to collect EEG signals generated during user training in real time, and the EEG signals are amplified and filtered by an EEG amplifier 11 .

[0072] S3. Configure data processing module 2 and execute the following sub-steps:

[0073] S30. Using the EEG signal as input, running the motor imagery continuous control model to calculate and obtain characteristic parameters;

[0074] As a possible implementation method, the EEG signal includes a task-state EEG signal and a non-task-state EEG signal; see Figure 3 , S30 specifically includes the following sub-steps:

[0075] S300. Configure and slide the time window to continuously capture the task-state EEG signal to obtain a multi-channel task-state EEG signal;

[0076] S301. Calculate the power spectral density matrix P of multi-channel task-state EEG signals n ;

[0077] As a possible implementation method, the power spectral density matrix P of the multi-channel task-state EEG signal is calculated as follows: n :

[0078]

[0079] Among them, P n represents the power spectral density matrix at time point n, represents the EEG signal captured by the sliding time window corresponding to time point n, N is the number of sampling channels of the EEG signal, Nt is the number of sampling points in the sliding time window, F(X n ) represents the EEG signal X n The Fourier transform of .

[0080] S302. Calculate the power spectral density matrix P of the non-task state EEG signal rest ;

[0081] As an example, the power spectral density matrix P of the non-task state EEG signal is calculated by the same method as that of calculating the power spectral density matrix of the multi-channel task state EEG signal. rest , P rest It reflects the change of the time-domain-energy characteristics of the EEG signal at the current time point relative to the non-task state.

[0082] S303. Calculate the relative power spectrum density matrix RPSD n , Among them, N is the number of sampling channels of EEG signals, Nt is the number of sampling points in the sliding time window, is the set of real numbers;

[0083] S304. Select the target frequency band related to motor imagery and calculate the relative power value RP corresponding to the target frequency band F ;

[0084] As a possible implementation method, the relative power value RP corresponding to the target frequency band is calculated by the following method: F :

[0085]

[0086] Where F represents the target frequency band, Where fs represents the sampling rate of the EEG signal, It represents the power corresponding to frequency f in the relative power spectrum density, reflecting the frequency domain-energy characteristics of the whole brain at any time period.

[0087] S305. According to the motor imagery action and the corresponding limb, select the corresponding channel combination, assign weights to the relative power values ​​of different channels, and obtain the characteristic parameters with the greatest difference.

[0088] Exemplarily, using a 10-20 channel combination, the difference between the C4 channel and the C3 channel is selected when imagining left-hand movement (the weight of C4 is 1, and the weight of C3 is -1), and the difference between the C3 channel and the C4 channel is selected when imagining right-hand movement (the weight of C4 is -1, and the weight of C3 is 1).

[0089] S31. In the case of determining that there is a task, taking the characteristic parameters as input, running the dynamic adaptive characteristic-feedback mapping model to calculate and obtain the control parameters;

[0090] As a possible implementation, the characteristic parameters are calculated as follows:

[0091]

[0092] in, For the selected channel combination, is the weight corresponding to the c channel, Indicates the relative power of channel c in the target frequency band F, is the set of real numbers.

[0093] As a possible implementation, before running the dynamic adaptive feature-feedback mapping model, the training method also includes:

[0094] Before the first motor imagery task begins, a dynamic adaptive feature-feedback mapping model is constructed, which includes the following steps:

[0095] Select a feature-feedback mapping function, which is a function related to feature parameters, feature parameter thresholds, and feature parameter baselines at any time;

[0096] As a possible implementation, the feature-feedback mapping function may be a linear mapping relationship or a nonlinear mapping relationship. For example, when the feature-feedback mapping function is a linear mapping relationship, its expression is as follows:

[0097]

[0098] Among them, Fb is the control parameter, C B is the characteristic parameter baseline, C T is the characteristic parameter threshold, C i is the characteristic parameter at any time, Indicates the maximum step length of the controlled object, for example: the intensity of FES or the movement range of the virtual character in the system.

[0099] By establishing a mapping relationship from characteristic parameters to control parameters Fb, continuous control can be achieved, that is, any characteristic parameter can be directly converted into a corresponding control parameter.

[0100] Calculate the baseline initial value C of the characteristic parameter B0 and the initial value of the characteristic parameter threshold C T0 , specifically including the following sub-steps:

[0101] Calculate the set of non-task state characteristic parameters before the start of the first motor imagery task For example, before the first motor imagery task begins, the user is in a non-task state, and the EEG acquisition device continuously acquires non-task state EEG signals and segments them through a sliding time window. When the first motor imagery task begins, the EEG acquisition device calculates the EEG signals in the most recent period of time, for example, 5 seconds before the imagination task begins, using the same feature extraction method as that for extracting task state feature parameters, to obtain a set of non-task state feature parameters.

[0102] According to the set C NT The data distribution of non-task characteristic parameters within the range is used to obtain the maximum value C of non-task characteristic parameters. NTmax ∈C NT , and the third quartile Q 3NT ∈C NT ;

[0103] Q 3NT is the upper limit of the outlier of the first motor imagery task state, C NTmax is the lower limit of the outlier of the first motor imagery task state. Assuming that the characteristic parameters of the first motor imagery task state are uniformly distributed, the baseline initial value C of the characteristic parameter is determined by the following method. B0 and the initial value of the characteristic parameter threshold C T0 :

[0104]

[0105] As an example, considering that the state of the characteristic parameters of the task state in actual use may be different from the characteristic parameters of the task state calculated There is a big difference, so a larger value range should be set. A and C NTmax As characteristic parameters C i The upper and lower quartiles of the value range can be obtained by i The value range is Therefore, the characteristic parameter baseline can be expressed as:

[0106]

[0107] The characteristic parameter threshold can be expressed as:

[0108]

[0109] During use, the feature-feedback mapping constructed by the prior art only performs pattern recognition based on the template signal measured in advance or the trained model, resulting in low recognition accuracy and control accuracy for the user's actual movement intention. The present invention proposes to continuously collect feature parameters during the training process, and based on the distribution of the feature parameters, correct and iterate the feature-feedback mapping relationship, so that the feedback has higher real-time and control accuracy, which can greatly improve the user experience.

[0110] As a possible implementation method, after the first motor imagery task is performed, the feature parameter threshold C is iterated by the following method: T and the characteristic parameter baseline value C B :

[0111] The characteristic parameters C obtained at any motor imagery task state Xn Add task state feature parameter set C Test and follow C Test The distribution construction value range of the characteristic parameters in is:

[0112]

[0113] Among them, max(C Test ,C Xn ) means C Test and C Xn The maximum value, min(C Test ,C Xn ) means C Test and C Xn The minimum value in ;

[0114] When C Test When the amount of data in the collection exceeds the set length limit, the feature parameters that are the oldest from the current time point are removed, that is, the latest feature parameters are retained to achieve dynamic iteration of the feature parameters.

[0115] As a possible implementation method, when dynamically iterating the characteristic parameter value range each time, a correction function f is introduced to correct C T C Xn The degree of influence, that is, C T =f[max(C Test ,C Xn )].

[0116] As an example, different forms of correction functions can be selected according to different task objectives. The correction function can contain constants or functions related to variables such as task times and time. For example, in order to enhance motivation, C TThe value is greater than the existing C Test The maximum value C in the set max , that is, when C Xn With the existing C max When they are equal, the maximum feedback value still cannot be obtained. At this time, the correction function f can be:

[0117]

[0118] Where k is a constant greater than 1, and t is the time required to add C Xn After the collection C Test The maximum value in .

[0119] For example, in order to provide incentives and keep the control difficulty at a low level, Xn Greater than C T Time limit C T The growth rate of to ensure the stability of control feedback, at this time, the correction function f can be:

[0120]

[0121] Where k is a constant less than 1, and t is the time to add C Xn After the collection C Test The maximum value in .

[0122] For example, to set a goal for each training, you can T Add an upper limit and adjust its growth rate according to the number of current tasks. At this time, the correction function can be a function related to the number of current tasks:

[0123]

[0124] Among them C goal is the target parameter for this training, N is the total number of tasks, and t is the number of tasks added to C Xn After the collection C Test The maximum value in .

[0125] It should be explained that the type of correction function actually adopted does not affect the essence of the present invention.

[0126] S4. Sending the control parameters to the functional electrical stimulation device to load the muscle with electrical stimulation of corresponding intensity to obtain tactile feedback; at the same time, the dynamic adaptive feature-feedback mapping model is dynamically updated according to the latest input feature parameters;

[0127] Exemplarily, the functional electrical stimulation device applies electrical stimulation of corresponding intensity to the muscles according to the size of the control parameter, and achieves the same movement pattern as the movement task in the training scenario by applying electrical stimulation.

[0128] S5. Send the control parameters to the VR display device to present visual feedback. The user adjusts the movement imagination based on the visual feedback and tactile feedback, and then repeats S1 to S5 until the task is completed. During the training process, the present invention continuously constructs the latest mapping relationship between the control parameters and the latest feature parameter distribution, which can maximize the adaptation of the training parameters of the current system to the user status, realize dynamic adjustment of task difficulty, and improve training efficiency.

[0129] In the second aspect, the present invention provides an adaptive dynamic feedback brain-computer interface training system based on virtual reality, see Figure 1 , including an EEG acquisition device 1, a data processing module 2, a VR display device 3 and a functional electrical stimulation device 4;

[0130] Among them, the training scene is pre-built in the VR display device 3, and the user performs movement imagination according to the training scene presented by the VR display device 3;

[0131] The EEG acquisition device 1 is used to collect EEG signals when the user performs motor imagery;

[0132] The data processing module 2 is used to receive EEG signals and run the motor imagery continuous control model to calculate and obtain characteristic parameters; it is also used to use the characteristic parameters as input to run the dynamic adaptive feature-feedback mapping model to calculate and obtain control parameters when it is determined that there is a task;

[0133] The functional electrical stimulation device 4 receives control parameters to load electrical stimulation of a preset intensity on the muscles to obtain tactile feedback; the VR display device 3 is also used to receive control parameters to present visual feedback. The user adjusts the motor imagery based on the visual feedback and tactile feedback so that the EEG acquisition device re-acquires EEG signals.

[0134] See also Figure 1 As a possible implementation method, the EEG acquisition device 1 includes an electrode 10 and an EEG amplifier 11. The electrode 10 acquires the user's EEG signal, and the EEG amplifier 11 amplifies and filters the EEG signal, converts it into a digital signal and sends it to the data processing module 2; the training system also includes a first control computer 5 and a second control computer 6. The first control computer 5 is used to receive control parameters, and generate a first control instruction based on the control parameters and transmit it to the VR display device 3. The second control computer 6 is used to receive control parameters, and generate a second control instruction based on the control parameters and transmit it to the functional electrical stimulation device 4; the data processing module 2 includes an acquisition computer 20, and the acquisition computer 20 is used to calculate characteristic parameters and control parameters, and transmit the control parameters to the first control computer 5 and the second control computer 6.

[0135] The present invention forms a complete closed-loop training system by combining BCI, FES and VR technologies. During use, the VR display device can provide task prompts for motor imagery action patterns. BCI analyzes the user's movement intention when performing the task, and contracts the corresponding muscles through functional electrical stimulation to complete the limb movements of the same mode as the motor imagery task prompted by the training system, coupling the spontaneous movement intention at the neural level with the passive movement at the muscle level, thereby improving the repair of the original neuromuscular control pathway or the construction of a new neuromuscular control pathway, and can also improve the user's sense of participation in the training process, forming an efficient feedback loop.

[0136] In order to facilitate a clear understanding of the technical solution of the present invention, it is further elaborated below in conjunction with specific embodiments.

[0137] See also Figure 4 , which is a system logic structure and flow chart for users to use the training system and training method provided by the present invention for training. In this embodiment, the training system is composed of an EEG acquisition device, an integrated VR glasses machine, a functional electrical stimulation device and a host computer. The data processing module is deployed in the host computer, and the motor imagery continuous control model and the dynamic adaptive feature-feedback mapping model are integrated in the data processing module. The host computer and the integrated VR glasses transmit the training scene through VR streaming technology, and the EEG acquisition device and the host computer transmit EEG data through the TCP / IP protocol.

[0138] The system allows users to simulate the control of an automatically moving wheelchair and immersively roam in a virtual environment built by VR. During the roaming process, when faced with tasks such as turning and avoiding obstacles, users need to imagine grasping with their hands to achieve turning and deceleration. The system uses a random sequence to generate a certain number of tasks. When all tasks are completed, the training ends.

[0139] Before use, the user needs to wear the EEG acquisition device, the VR glasses and the functional electrical stimulation device. The processing flow for a specific task includes the following steps:

[0140] S10. Collecting non-task state EEG signals, preprocessing and A / D converting the non-task state EEG signals;

[0141] After the system is started, the EEG acquisition device collects EEG signals containing the user's movement intention, completes analog signal processing such as denoising and amplification, and A / D conversion steps. During this process, the user remains relaxed.

[0142] S11. Complete feature extraction and storage of non-task state EEG signals;

[0143] The data processing module deployed on the host computer runs the motor imagery continuous control model, calculates the characteristic parameters of the non-task state EEG signal, and saves them in the parameter set of the non-task state EEG signal.

[0144] S12. The VR glasses all-in-one device presents visual task prompts to the user to guide the user to perform motor imagination;

[0145] When a task is started in the system, the VR glasses all-in-one machine prompts the user with the current scene information and text prompts to prompt the user to perform the current action. Based on the task prompts, the user continues to simulate and perform this action in the brain. For example: imagine grasping with the right hand to make the wheelchair turn left.

[0146] S13. Collecting task-state EEG signals, preprocessing the task-state EEG signals and performing A / D conversion;

[0147] The EEG acquisition device collects EEG signals containing the user's movement intention, and completes analog signal processing such as denoising and amplification and A / D conversion steps.

[0148] S14. Calculate characteristic parameters and control parameters;

[0149] The data processing module deployed on the host computer runs the motor imagery continuous control model, calculates the characteristic parameters of the task-state EEG signal, and passes the characteristic parameters to the dynamic adaptive feature-feedback mapping model. The system needs to determine whether the current task is the first task. If it is the first task, the non-task-state EEG signal characteristic parameter set is called to construct the feature-feedback parameter mapping; otherwise, the currently constructed feature-feedback parameter mapping is called to return the control parameters.

[0150] S15. Feedback the execution result of the exercise intention to the user;

[0151] The VR glasses all-in-one machine will give feedback to the user on the execution results of the movement intention according to the control parameters. For example, turn left to a certain angle. The functional electrical stimulation device will apply electrical stimulation of corresponding intensity to the corresponding limbs of the user according to the control parameters. The user can adjust his or her motor imagination strategy based on these feedbacks to better complete the task.

[0152] S16. Iteratively update the dynamic adaptive feature-feedback mapping model;

[0153] The data processing module deployed in the host computer saves the current task state parameters in the task state EEG parameter set. If the EEG characteristic parameters in the set have reached the upper limit at this time, the earliest stored data in the set is deleted to ensure that the parameters in the set are the latest. After the saving is completed, the data processing module calculates the baseline value and threshold value of the characteristic parameter according to the preset correction function and the new parameter set, completes the iteration of the dynamic adaptive feature-feedback mapping model, and saves the new dynamic adaptive feature-feedback mapping model in the data processing module. It should be noted that steps S15 and S16 can run in parallel.

[0154] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0155] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the present invention and its equivalents.

Claims

1. A virtual reality-based adaptive dynamic feedback brain-computer interface training method, characterized in that: Before executing the training method, the user wears an EEG acquisition device, a VR display device and a functional electrical stimulation device; wherein the training scene is pre-built in the VR display device; the training method comprises the following steps: S1. The user performs movement imagination based on the training scene presented by the VR display device; S2. The EEG acquisition device collects EEG signals; S3. Configure the data processing module and execute the following sub-steps: S30. Using the EEG signal as input, running the motor imagery continuous control model to calculate and obtain characteristic parameters; S31. In the case of determining that there is a task, taking the characteristic parameters as input, running the dynamic adaptive characteristic-feedback mapping model to calculate and obtain the control parameters; S4. Sending the control parameters to the functional electrical stimulation device to load the muscle with electrical stimulation of corresponding intensity to obtain tactile feedback; at the same time, the dynamic adaptive feature-feedback mapping model is dynamically updated according to the latest input feature parameters; S5. Send the control parameters to the VR display device to present visual feedback, and the user adjusts the movement imagination based on the visual feedback and tactile feedback and then repeats S1 to S5.

2. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 1 is characterized in that: The EEG signals include task-state EEG signals and non-task-state EEG signals; S30 specifically includes the following sub-steps: S300. Configure and slide the time window to continuously capture the task-state EEG signal to obtain a multi-channel task-state EEG signal; S301. Calculate the power spectral density matrix P of multi-channel task-state EEG signals n ; S302. Calculate the power spectral density matrix P of the non-task state EEG signal rest ; S303. Calculate the relative power spectrum density matrix RPSD n , Among them, N is the number of sampling channels of EEG signals, Nt is the number of sampling points in the sliding time window, is the set of real numbers; S304. Select the target frequency band related to motor imagery and calculate the relative power value RP corresponding to the target frequency band F ; S305. According to the motor imagery action and the corresponding limb, select the corresponding channel combination, assign weights to the relative power values ​​of different channels, and obtain the characteristic parameters with the greatest difference.

3. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 2 is characterized in that: The power spectral density matrix P of the multi-channel task-state EEG signal is calculated as follows: n : Among them, P n represents the power spectral density matrix at time point n, represents the EEG signal captured by the sliding time window corresponding to time point n, N is the number of sampling channels of the EEG signal, Nt is the number of sampling points in the sliding time window, F(X n ) represents the EEG signal X n The Fourier transform of is the set of real numbers.

4. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 3 is characterized in that: The relative power value RP corresponding to the target frequency band is calculated by the following method F : Where F represents the target frequency band, Where fs represents the sampling rate of the EEG signal, It represents the power corresponding to frequency f in the relative power spectrum density, reflecting the frequency domain-energy characteristics of the whole brain at any time period. is the set of real numbers.

5. The method for training a brain-computer interface with adaptive dynamic feedback based on virtual reality according to claim 4, characterized in that: The characteristic parameters are calculated by the following method: in, For the selected channel combination, is the weight corresponding to the c channel, Indicates the relative power of channel c in the target frequency band F, is the set of real numbers.

6. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 1 is characterized in that: Before running the dynamic adaptive feature-feedback mapping model, the training method further includes: Before the first motor imagery task begins, a dynamic adaptive feature-feedback mapping model is constructed, which includes the following steps: Selecting a feature-feedback mapping function, wherein the feature-feedback mapping function is a function related to a feature parameter, a feature parameter threshold, and a feature parameter baseline at any time; Calculate the baseline initial value C of the characteristic parameter B0 and the initial value of the characteristic parameter threshold C T0 , specifically including the following sub-steps: Calculate the set of non-task state characteristic parameters before the start of the first motor imagery task According to the set C NT The data distribution of non-task characteristic parameters within the range is used to obtain the maximum value C of non-task characteristic parameters. NTmax ∈C NT , and the third quartile Q 3NT ∈C NT ; Q 3NT is the upper limit of the outlier of the first motor imagery task state, C NTmax is the lower limit of the outlier of the first motor imagery task state. Assuming that the characteristic parameters of the first motor imagery task state are uniformly distributed, the baseline initial value C of the characteristic parameter is determined by the following method. B0 and the initial value of the characteristic parameter threshold C T0 :

7. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 6 is characterized in that: After the first motor imagery task is performed, the feature parameter threshold C is iterated by the following method: T and the characteristic parameter baseline value C B : The characteristic parameters C obtained at any motor imagery task state Xn Add task state feature parameter set C Test and follow C Test The distribution construction value range of the characteristic parameters in is: Among them, max(C Test ,C Xn ) and min(C Test ,C Xn ) represent the maximum and minimum values ​​of the two respectively; When C Test When the amount of data in the collection exceeds the set length limit, the feature parameters that are the oldest from the current time point are removed, that is, the latest feature parameters are retained.

8. The method for training a brain-computer interface with adaptive dynamic feedback based on virtual reality according to claim 7, characterized in that: In each dynamic iteration of the characteristic parameter value range, the correction function f is introduced to correct C T C Xn The degree of influence, that is, C T =f[max(C Test ,C Xn )].

9. An adaptive dynamic feedback brain-computer interface training system based on virtual reality, characterized in that: Including EEG acquisition equipment, data processing module, VR display equipment and functional electrical stimulation equipment; The training scene is pre-built in the VR display device, and the user performs motion imagination according to the training scene presented by the VR display device; The EEG acquisition device is used to collect EEG signals when the user performs motor imagery; The data processing module is used to receive EEG signals and run a motor imagery continuous control model to calculate characteristic parameters; and is also used to use the characteristic parameters as input to run a dynamic adaptive feature-feedback mapping model to calculate control parameters when it is determined that there is a task. The functional electrical stimulation device receives control parameters to load electrical stimulation of a preset intensity on the muscles to obtain tactile feedback; the VR display device is also used to receive control parameters to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and tactile feedback so that the EEG acquisition device re-acquires EEG signals.

10. The adaptive dynamic feedback brain-computer interface training system based on virtual reality according to claim 9, characterized in that: The EEG acquisition device includes electrodes and an EEG amplifier. The electrodes acquire the user's EEG signals, and the EEG signals are amplified and filtered by the EEG amplifier before being sent to the data processing module. The training system also includes a first control computer and a second control computer. The first control computer is used to receive control parameters, and generate a first control instruction based on the control parameters and transmit it to the VR display device. The second control computer is used to receive control parameters, and generate a second control instruction based on the control parameters and transmit it to the functional electrical stimulation device. The data processing module includes an acquisition computer, and the acquisition computer is used to calculate characteristic parameters and control parameters, and transmit the control parameters to the first control computer and the second control computer.

Images