Adaptive Dynamic Feedback Brain-Computer Interface Training Method and System Based on Virtual Reality

Abstract

The present invention relates to the technical field of brain-computer interfaces, and discloses an adaptive dynamic feedback brain-computer interface training method and system based on virtual reality. The training method includes: a user performing motor imagery according to a training scenario presented by a VR display device; an electroencephalogram acquisition device acquiring electroencephalogram signals; configuring a data processing module, using the electroencephalogram signals as inputs, and running a motor imagery continuous control model to calculate characteristic parameters; in the case of determining that there is a task, using the characteristic parameters as inputs, and running a dynamic adaptive feature-feedback mapping model to calculate control parameters; sending the control parameters to a functional electrical stimulation device to apply electrical stimulation of a corresponding intensity to muscles to obtain tactile feedback; at the same time, the dynamic adaptive feature-feedback mapping model is dynamically updated according to the latest input characteristic parameters; sending the control parameters to the VR display device to present visual feedback, and the user adjusting the motor imagery based on the visual feedback and the tactile feedback. The present invention can improve the user's sense of participation and training efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interfaces, and particularly to an adaptive dynamic feedback brain-computer interface training method and system based on virtual reality. Background Art

[0002] In recent years, the Motor Imagery (MI) method has been increasingly introduced into sports training. Motor imagery is a mental process in which the brain imagines and imitates specific actions without actual action execution. According to the research on the mirror neuron system and the theory of neural plasticity, MI can strengthen the activation of motor nerves in related regions during the process of motor nerve training, contribute to the remodeling of damaged motor nerve pathways or the enhancement of healthy nerve pathways, and can be used as a training means for motor functional disorders caused by diseases such as stroke. Brain-Computer Interface (BCI) technology can convert the detected brain activities (such as electroencephalogram signals, EEG) into control instructions for external devices. Among them, the brain-computer interface based on motor imagery can recognize the movement intention in the user's brain and feedback it to the user. At the same time, Functional Electrical Stimulation (FES) directly stimulates and activates muscles through electrodes, enabling users to complete preset actions even when their autonomous movement ability is limited. Therefore, researchers have proposed a closed-loop training system composed of brain-computer interface technology and functional electrical stimulation. This system first decodes the user's movement intention, controls the functional electrical stimulation device to drive limb movement, and realizes the synchronous coupling of cortical and muscle activities. Through closed-loop neurofeedback rehabilitation training, nerve pathways are reconstructed.

[0003] Combining the motor imagery brain-computer interface with functional electrical stimulation can help users in training, but it does not solve the problem of the single and boring training process. Although the use of Virtual Reality (VR) technology can enhance the interest and immersion, existing training systems often use a single scenario and repetitive tasks. In addition, most existing feedback technologies rely on the results of pattern recognition, and it is necessary to collect user data and establish a calibration model before training, which not only increases the time cost of system use but is also easily restricted by pattern recognition algorithms. Moreover, the neurofeedback technology based on pattern recognition can only provide discrete control and cannot intuitively reflect the real-time state of the user's movement intention. Long-term use is likely to cause user fatigue and a decrease in cooperation willingness, resulting in the user being unable to obtain the maximum benefit during the training process. More importantly, in actual neurofeedback rehabilitation training, due to the large time variability of the user's brain activity data (such as EEG, etc.), the model established before training is difficult to effectively match the current neural activity state.

[0004] Therefore, the existing systems and methods have the following defects: First, it is necessary to collect user data before training and establish a calibration model, resulting in additional time consumption; second, the established model is difficult to accurately reflect the user's current neural activity state, resulting in a mismatch between the feedback obtained by the user during training and the current actual neural signal state, affecting the training efficiency. Summary of the Invention

[0005] The object of the present invention is to provide a virtual reality-based adaptive dynamic feedback brain-computer interface training method and system, which combines virtual reality technology and functional electrical stimulation to provide a more immersive and interesting virtual motion training scenario for users, improve the user's training motivation and sense of participation, and adopt an adaptive and dynamic feedback parameter calculation method, which can adjust the mapping relationship between feature parameters and control parameters according to the electroencephalogram signal characteristics of the user within a certain period of time, realize the dynamic adjustment of training difficulty, and improve the training efficiency.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In the first aspect, the present invention provides a virtual reality-based adaptive dynamic feedback brain-computer interface training method. Before executing the training method, the user wears an electroencephalogram acquisition device, a VR display device, and a functional electrical stimulation device; wherein, a training scenario is pre-built in the VR display device; the training method includes the following steps:

[0008] S1. The user performs motor imagery according to the training scenario presented by the VR display device.

[0009] S2. The electroencephalogram acquisition device acquires electroencephalogram signals.

[0010] S3. Configure a data processing module and execute the following sub-steps:

[0011] S30. Using the electroencephalogram signal as input, run a motor imagery continuous control model to calculate and obtain feature parameters.

[0012] S31. When it is determined that there is a task, using the feature parameters as input, run a dynamic adaptive feature-feedback mapping model to calculate and obtain control parameters.

[0013] S4. Send the control parameters to the functional electrical stimulation device to apply an electrical stimulation of corresponding intensity to the muscles to obtain tactile feedback; at the same time, the dynamic adaptive feature-feedback mapping model is dynamically updated according to the latest input feature parameters.

[0014] S5. Send the control parameters to the VR display device to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and tactile feedback and then repeats S1 to S5.

[0015] As a possible implementation, the electroencephalogram (EEG) signals include task-related EEG signals and non-task-related EEG signals; S30 specifically includes the following sub-steps:

[0016] S300. Configure and slide a time window to continuously intercept from the task-related EEG signals to obtain multi-channel task-related EEG signals;

[0017] S301. Calculate the power spectral density matrix P of the multi-channel task-related EEG signals n ;

[0018] S302. Calculate the power spectral density matrix P of the non-task-related EEG signals rest ;

[0019] S303. Calculate the relative power spectral density matrix RPSD n , where N is the number of sampling channels of the EEG signals, Nt is the number of sampling points within the sliding time window, is the set of real numbers;

[0020] S304. Select the target frequency band related to motor imagery, and calculate the relative power value RP corresponding to the target frequency band F ;

[0021] S305. According to the motor imagery action and the corresponding limb, select the corresponding channel combination, assign weights to the relative power values of different channels, and obtain the characteristic parameters with the greatest difference.

[0022] As a possible implementation, the power spectral density matrix P of the multi-channel task-related EEG signals is calculated in the following way n :

[0023]

[0024] where P n represents the power spectral density matrix at time point n, represents the EEG signals intercepted by the sliding time window corresponding to time point n, N is the number of sampling channels of the EEG signals, Nt is the number of sampling points within the sliding time window, F(X n ) represents the Fourier transform of the EEG signal X n and is the set of real numbers.

[0025] As a possible implementation, the relative power value RP corresponding to the target frequency band is calculated by the following method F :

[0026]

[0027] where F represents the target frequency band, Among them, \(f_s\) represents the sampling rate of the electroencephalogram signal, represents the power corresponding to the frequency \(f\) in the relative power spectral density, reflecting the frequency-domain energy characteristics of the whole brain at any time period.

[0028] As a possible implementation, the characteristic parameters are calculated by the following method:

[0029]

[0030] Among them, is the selected channel combination, is the weight corresponding to the \(c\)th channel, represents the relative power of the \(c\)th channel in the target frequency band \(F\), is the set of real numbers.

[0031] As a possible implementation, before running the dynamic adaptive feature-feedback mapping model, the training method further includes:

[0032] Before the start of the first motor imagery task, construct a dynamic adaptive feature-feedback mapping model, which specifically includes the following steps:

[0033] Select a feature-feedback mapping function, which is a function related to the characteristic parameters, characteristic parameter thresholds, and characteristic parameter baselines at any time;

[0034] Calculate the initial value \(C_{b}\) of the characteristic parameter baseline B0 and the initial value \(C_{t}\) of the characteristic parameter threshold T0 , which specifically includes the following sub-steps:

[0035] Calculate and obtain the set \(C_{n}\) of non-task-state characteristic parameters before the start of the first motor imagery task

[0036] According to the data distribution of the non-task-state characteristic parameters in the set \(C_{n}\) NT obtain the maximum value \(C_{max}\) of the non-task-state characteristic parameters NTmax \(\in C_{n}\) NT , and the third quartile \(Q_{3}\) 3NT \(\in C_{n}\) NT ;

[0037] Taking \(Q_{3}\) 3NT as the upper limit of outliers for the first motor imagery task state and \(C_{max}\) NTmax as the lower limit of outliers for the first motor imagery task state, and assuming that the characteristic parameters of the first motor imagery task state are uniformly distributed, determine the initial value \(C_{b}\) of the characteristic parameter baseline B0 and the initial value \(C_{t}\) of the characteristic parameter threshold T0 :

[0038]

[0039] As a possible implementation, after the first motor imagery task is executed, the baseline value C of the feature parameter is iterated by the following method B and the feature parameter threshold C T :

[0040] The feature parameter C obtained at any moment of the motor imagery task state is added Xn to the set C of task-state feature parameters Test and the value range is constructed according to the distribution of the feature parameters in C Test :

[0041]

[0042] where max(C Test , C Xn ) and min(C Test , C Xn ) represent the maximum and minimum values of the two respectively;

[0043] When the number of data in the C Test set exceeds the set length limit, the feature parameter that is the oldest from the current time point is removed, that is, the latest feature parameter is retained.

[0044] As a possible implementation, when dynamically iterating the value range of the feature parameter each time, a correction function f is introduced to correct the influence degree of C T affected by C Xn , that is, C T = f[max(C Test , C Xn )].

[0045] In a second aspect, the present invention provides a virtual reality-based adaptive dynamic feedback brain-computer interface training system, including an electroencephalogram acquisition device, a data processing module, a VR display device, and a functional electrical stimulation device;

[0046] wherein, a training scenario is pre-built in the VR display device, and the user performs motor imagery according to the training scenario presented by the VR display device;

[0047] The electroencephalogram acquisition device is used to acquire electroencephalogram signals when the user performs motor imagery;

[0048] The data processing module is used to receive the electroencephalogram signals and run a continuous motor imagery control model to calculate and obtain feature parameters; it is also used to, when it is determined that there is a task, use the feature parameters as inputs and run a dynamic adaptive feature-feedback mapping model to calculate and obtain control parameters;

[0049] The functional electrical stimulation device receives control parameters to apply electrical stimulation of a preset intensity to muscles and obtain tactile feedback; the VR display device is also used to receive control parameters to present visual feedback, and the user adjusts motor imagery based on the visual feedback and tactile feedback to enable the electroencephalogram acquisition device to re-acquire electroencephalogram signals.

[0050] As a possible implementation, the electroencephalogram acquisition device includes electrodes and an electroencephalogram amplifier. The electrodes acquire the user's electroencephalogram signals, and after amplifying and filtering the electroencephalogram signals through the electroencephalogram amplifier, send them to the data processing module; the training system also includes a first control computer and a second control computer. The first control computer is used to receive control parameters and generate a first control instruction based on the control parameters and transmit it to the VR display device. The second control computer is used to receive control parameters and generate a second control instruction based on the control parameters and transmit it to the functional electrical stimulation device; the data processing module includes an acquisition computer, and the acquisition computer is used to calculate feature parameters and control parameters and transmit the control parameters to the first control computer and the second control computer.

[0051] Compared with the prior art, the beneficial effects produced by the present invention are as follows:

[0052] 1. The method for training a virtual reality-based adaptive dynamic feedback brain-computer interface provided by the present invention does not require additional data acquisition and model training in the initial stage, avoiding additional consumption of time; at the same time, in the training process, the present invention continuously constructs the latest mapping relationship between the latest feature parameter distribution and the control parameters, which can maximize the guarantee that the training parameters of the current system are adapted to the user's state, realize dynamic adjustment of the task difficulty, and improve the training efficiency.

[0053] 2. The method and system for training a virtual reality-based adaptive dynamic feedback brain-computer interface provided by the present invention form a complete closed-loop training system by combining BCI, FES, and VR technologies. During use, the VR display device can provide task prompts for the motor imagery action mode. BCI analyzes the user's motor intention when performing the task, and through functional electrical stimulation, causes the corresponding muscles to contract to complete limb movements in the same mode as the motor imagery task prompted in the training system, coupling the spontaneous motor intention at the neural level with the passive movement at the muscle level, improving the repair of the original nerve-muscle control pathway or the construction of a new nerve-muscle control pathway, and being able to improve the user's sense of participation during the training process, forming an efficient feedback loop.

[0054] 3. The adaptive dynamic feedback brain-computer interface training method based on virtual reality provided by the present invention uses a data interception method with a short-interval sliding time window having a high time resolution to analyze the features related to the movement intention in the EEG signals within the current time period. Instead of pattern recognition, it directly feeds back the feature parameters. Therefore, its instructions are continuous rather than discrete, and can more intuitively reflect the intensity of the movement intention contained in the user's current EEG state, which is beneficial to improving the user's control ability of spontaneous EEG. Moreover, continuous control helps to obtain more real-time feedback from the user and optimize the control experience. Description of the Drawings

[0055] The drawings described herein are used to provide a further understanding of the present invention and form a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0056] Figure 1 is a schematic structural diagram of an adaptive dynamic feedback brain-computer interface training system based on virtual reality provided by an embodiment of the present invention;

[0057] Figure 2 is a flowchart of an adaptive dynamic feedback brain-computer interface training method based on virtual reality provided by an embodiment of the present invention;

[0058] Figure 3 is a flowchart of a method for calculating feature parameters by running a motor imagery continuous control model provided by an embodiment of the present invention;

[0059] Figure 4 is a system logic structure and flowchart for a user to train using the training system and training method of the present invention in an embodiment of the present invention.

[0060] Reference Numerals

[0061] 1 - EEG acquisition device, 10 - electrode, 11 - EEG amplifier, 2 - data processing module, 20 - acquisition computer, 3 - VR display device, 4 - functional electrical stimulation device, 5 - first control computer, 6 - second control computer. Detailed Embodiments

[0062] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds and do not limit their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first" and "second" do not necessarily mean different.

[0063] It should be noted that in the present invention, words such as "exemplary" or "for example" are used to give examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, or B exists alone, where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. The following at least one (item) or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b or c may represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b and c may be single or multiple.

[0065] The present invention aims to solve the following technical problems: First, the existing motor imagery training methods lack feedback or have a single feedback mode, the training tasks are repetitive and boring, the recognition accuracy and control accuracy of motor intentions are not high, and the existing methods need to rely on long-term electroencephalogram signals to establish a training model, which easily causes user fatigue, frustration and a decrease in the willingness to cooperate; Second, on the premise of ensuring the training effect, how to adaptively adjust the task difficulty and feedback intensity dynamically during the training process according to the user's current state, so as to improve the rehabilitation efficiency.

[0066] In a first aspect, the present invention provides an adaptive dynamic feedback brain-computer interface training method based on virtual reality. Refer to Figure 1 , before executing the training method, the user wears an electroencephalogram acquisition device 1, a VR display device 3 and a functional electrical stimulation device 4; wherein, a training scenario is pre-built in the VR display device 3;

[0067] Exemplarily, the electroencephalogram (EEG) acquisition device 1 acquires the user's EEG signals, amplifies and filters the EEG signals. The functional electrical stimulation device 4 provides electrical stimulation of corresponding intensity to the user's muscles according to the control parameters and obtains tactile feedback. In the present invention, the VR display device 3 can use either a VR all-in-one machine or a head-mounted display. Based on virtual reality technology, the present invention constructs a life-like virtual environment to enhance the user's experience. During the training process, the user can see their own actions in real time from the first-person perspective in the virtual environment and obtain multi-sensory feedback. To further enhance the immersion during the training process, during use, the user's real-world posture can be made consistent with the posture of the character in the virtual environment by specifying a training environment for the user or using a six-degree-of-freedom (DOF) device.

[0068] See Figures 1 to 2 , the training method includes the following steps:

[0069] S1. The user performs motor imagery according to the training scenario presented by the VR display device 3; specifically, the VR display device 3 presents the pre-constructed training scenario to the user in real time and prompts the user to perform motor imagery of corresponding limbs and actions according to the task at an appropriate time. The user performs motor imagery of corresponding actions according to the prompt. For example, grasping with the hand, flexion and extension of the limbs, etc. During the motor imagery process, EEG signals are generated.

[0070] S2. The EEG acquisition device 1 acquires EEG signals;

[0071] As an example, scalp electrodes 10 are used to acquire the EEG signals generated by the user during training in real time, and the EEG amplifier 11 amplifies and filters the EEG signals.

[0072] S3. Configure the data processing module 2 and execute the following sub-steps:

[0073] S30. Using the EEG signals as input, run the motor imagery continuous control model to calculate and obtain characteristic parameters;

[0074] As a possible implementation, the EEG signals include task-state EEG signals and non-task-state EEG signals; See Figure 3 , S30 specifically includes the following sub-steps:

[0075] S300. Configure and slide a time window to continuously intercept from the task-state EEG signals to obtain multi-channel task-state EEG signals;

[0076] S301. Calculate the power spectral density matrix P of the multi-channel task-state EEG signals n ;

[0077] As a possible implementation, the power spectral density matrix P of the multi-channel task-state EEG signals is calculated in the following way n :

[0078]

[0079] where P n represents the power spectral density matrix at time point n, represents the EEG signal intercepted by the sliding time window corresponding to time point n, N is the number of sampling channels of the EEG signal, Nt is the number of sampling points within the sliding time window, and F(X n ) represents the Fourier transform of the EEG signal X n .

[0080] S302. Calculate the power spectral density matrix P of the non-task-state EEG signals rest ;

[0081] As an example, the power spectral density matrix P of the non-task-state EEG signals is obtained by using the same method as that for calculating the power spectral density matrix of the multi-channel task-state EEG signals rest , and P rest reflects the change of the time-domain - energy characteristics of the EEG signal at the current time point relative to the non-task state.

[0082] S303. Calculate the relative power spectral density matrix RPSD n , where N is the number of sampling channels of the EEG signal, Nt is the number of sampling points within the sliding time window, is the set of real numbers;

[0083] S304. Select the target frequency band related to motor imagery, and calculate the relative power value RP corresponding to the target frequency band F ;

[0084] As a possible implementation, the relative power value RP corresponding to the target frequency band is calculated by the following method F :

[0085]

[0086] where F represents the target frequency band, where fs represents the sampling rate of the EEG signal, represents the power corresponding to the frequency f in the relative power spectral density, and reflects the frequency-domain - energy characteristics of the whole brain at any time period.

[0087] S305. According to the motor imagery action and the corresponding limb, select the corresponding channel combination, and assign weights to the relative power values of different channels to obtain the characteristic parameters with the largest difference.

[0088] Exemplarily, using a 10 - 20 channel combination, when identifying left - hand motor imagery, the difference between the C4 channel and the C3 channel is selected (the weight of C4 is 1, and the weight of C3 is - 1), and when imagining right - hand movement, the difference between the C3 channel and the C4 channel is selected (the weight of C4 is - 1, and the weight of C3 is 1).

[0089] S31. When it is determined that there is a task, using the characteristic parameter as the input, run the dynamic adaptive feature - feedback mapping model to calculate and obtain the control parameter;

[0090] As a possible implementation, the characteristic parameter is calculated by the following method:

[0091]

[0092] where, is the selected channel combination, is the weight corresponding to the c channel, represents the relative power of the c channel in the target frequency band F, is the set of real numbers.

[0093] As a possible implementation, before running the dynamic adaptive feature - feedback mapping model, the training method further includes:

[0094] Before the start of the first motor imagery task, construct the dynamic adaptive feature - feedback mapping model, which specifically includes the following steps:

[0095] Select a feature - feedback mapping function, which is a function related to the characteristic parameter, the characteristic parameter threshold, and the characteristic parameter baseline at any time;

[0096] As a possible implementation, the feature - feedback mapping function can be a linear mapping relationship or a non - linear mapping relationship. Exemplarily, when the feature - feedback mapping function is a linear mapping relationship, its expression is as follows:

[0097]

[0098] where, Fb is the control parameter, C B is the characteristic parameter baseline, C T is the characteristic parameter threshold, C i is the characteristic parameter at any time, represents the maximum step size of the controlled object, for example: the intensity of FES or the action amplitude of the virtual character in the system, etc.

[0099] By establishing the mapping relationship from the characteristic parameter to the control parameter Fb, continuous control can be achieved, that is, for any characteristic parameter, it can be directly converted into a corresponding control parameter.

[0100] Calculate the initial value C of the characteristic parameter baseline B0 and the initial value C of the characteristic parameter threshold T0 , which specifically includes the following sub-steps:

[0101] Calculate and obtain the set of non-task state characteristic parameters before the start of the first motor imagery task Exemplarily, before the start of the first motor imagery task, the user is in a non-task state all the time. The EEG acquisition device continuously acquires non-task state EEG signals and segments them through a sliding time window. When the first motor imagery task starts, the EEG acquisition device calculates the EEG signals in the most recent period of time, for example, 5 s before the start of the imagery task, through the same feature extraction method as that for extracting task state characteristic parameters, and obtains a set of non-task state characteristic parameters

[0102] According to the data distribution of the non-task state characteristic parameters in the set C NT obtain the maximum value C of the non-task state characteristic parameters NTmax ∈C NT , and the third quartile Q 3NT ∈C NT ;

[0103] Take Q 3NT as the upper outlier limit of the first motor imagery task state and C NTmax as the lower outlier limit of the first motor imagery task state. At the same time, assuming that the characteristic parameters of the first motor imagery task state are uniformly distributed, determine the initial value C of the characteristic parameter baseline B0 and the initial value C of the characteristic parameter threshold T0 :

[0104]

[0105] As an example, considering that the state of the characteristic parameters in the task state during actual use may be quite different from the calculated characteristic parameters of the task state , a relatively large value range should be set. Therefore, take C A and C NTmax as the upper and lower quartiles of the value range of the characteristic parameter C i respectively, and the value range of C i can be obtained as Therefore, the characteristic parameter baseline can be expressed as:

[0106]

[0107] The characteristic parameter threshold can be expressed as:

[0108]

[0109] In the process of using the feature-feedback mapping constructed by the prior art, pattern recognition is only based on the pre-determined template signal or the trained model, resulting in low recognition accuracy and control accuracy for the user's true motion intention. The present invention proposes to continuously collect feature parameters during the training process, and based on the distribution of the feature parameters, correct and iterate the feature-feedback mapping relationship, so that the feedback has higher real-time performance and control accuracy, and can well improve the user experience.

[0110] As a possible implementation, after the first motor imagery task is executed, the feature parameter threshold C T and the feature parameter baseline value C B are iterated by the following method:

[0111] The feature parameter C Xn obtained at any moment of the motor imagery task state is added to the task-state feature parameter set C Test , and the value range is constructed according to the distribution of the feature parameters in C Test :

[0112]

[0113] Among them, max(C Test , C Xn ) represents the maximum value in C Test and C Xn , and min(C Test , C Xn ) represents the minimum value in C Test and C Xn ;

[0114] When the number of data in the C Test set exceeds the set length upper limit, the feature parameter that is the longest distance from the current time point is removed, that is, the latest feature parameter is retained to achieve dynamic iteration of the feature parameters.

[0115] As a possible implementation, when dynamically iterating the value range of the feature parameters each time, a correction function f is introduced to correct the influence degree of C T affected by C Xn , that is, C T = f[max(C Test , C Xn )].

[0116] As an example, different forms of correction functions can be selected according to different task objectives. The correction function can include constants or functions related to variables such as the number of tasks and time. For example, in order to enhance motivation, C TThe value is greater than the existing C Test the maximum value of C in the set max , that is, when C Xn is equal to the existing C max , the maximum feedback value still cannot be obtained. At this time, the correction function f can be:

[0117]

[0118] where k is a constant greater than 1, and t is the maximum value in the set C Xn after adding C Test .

[0119] For another example, in order to provide incentives and keep the control difficulty at a low level, the growth rate of C Xn can be restricted when it is greater than C T to ensure the stability of the control feedback. At this time, the correction function f can be: T where k is a constant less than 1, and t is the maximum value in the set C

[0120]

[0121] after adding C Xn Test .

[0122] For yet another example, a target can be set for each training. The upper limit of C T can be increased and its growth rate can be adjusted according to the number of current tasks. At this time, the correction function can be a function related to the number of current tasks n:

[0123]

[0124] where C goal is the target parameter for this training, N is the total number of tasks, and t is the maximum value in the set C Xn after adding C Test .

[0125] It should be noted that the type of correction function actually adopted does not affect the essence of the present invention.

[0126] S4. Send the control parameter to the functional electrical stimulation device to apply electrical stimulation with corresponding intensity to the muscle to obtain tactile feedback; at the same time, the dynamic adaptive feature - feedback mapping model is dynamically updated according to the latest input feature parameters;

[0127] Exemplarily, the functional electrical stimulation device applies electrical stimulation with corresponding intensity to the muscle according to the magnitude of the control parameter, and realizes the same motion pattern as the motion task of the training scenario by applying electrical stimulation.

[0128] ​S5. Send the control parameters to the VR display device to present visual feedback. The user adjusts the motor imagery based on the visual feedback and tactile feedback and then repeats S1 to S5 until the task is completed. In the training process of the present invention, the latest mapping relationship between the latest feature parameter distribution and the control parameters is continuously constructed, which can maximize the adaptation of the training parameters of the current system to the user state, realize the dynamic adjustment of the task difficulty, and improve the training efficiency.

[0129] In a second aspect, the present invention provides a virtual reality-based adaptive dynamic feedback brain-computer interface training system. Refer to Figure 1 , which includes an electroencephalogram acquisition device 1, a data processing module 2, a VR display device 3, and a functional electrical stimulation device 4;

[0130] Among them, a training scenario is pre-built in the VR display device 3, and the user performs motor imagery according to the training scenario presented by the VR display device 3;

[0131] The electroencephalogram acquisition device 1 is used to acquire electroencephalogram signals when the user performs motor imagery;

[0132] The data processing module 2 is used to receive the electroencephalogram signals and run a motor imagery continuous control model to calculate feature parameters; it is also used to, when it is determined that there is a task, use the feature parameters as inputs and run a dynamic adaptive feature-feedback mapping model to calculate control parameters;

[0133] The functional electrical stimulation device 4 receives the control parameters to apply a preset intensity of electrical stimulation to the muscles to obtain tactile feedback; the VR display device 3 is also used to receive the control parameters to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and tactile feedback to enable the electroencephalogram acquisition device to re-acquire electroencephalogram signals.

[0134] Refer to Figure 1 , as a possible implementation, the electroencephalogram acquisition device 1 includes an electrode 10 and an electroencephalogram amplifier 11. The electrode 10 acquires the electroencephalogram signals of the user, and after amplifying and filtering the electroencephalogram signals through the electroencephalogram amplifier 11, converts them into digital signals and sends them to the data processing module 2; the training system further includes a first control computer 5 and a second control computer 6. The first control computer 5 is used to receive the control parameters and generate a first control instruction based on the control parameters and transmit it to the VR display device 3. The second control computer 6 is used to receive the control parameters and generate a second control instruction based on the control parameters and transmit it to the functional electrical stimulation device 4; the data processing module 2 includes an acquisition computer 20, and the acquisition computer 20 is used to calculate the feature parameters and control parameters and transmit the control parameters to the first control computer 5 and the second control computer 6.

[0135] By combining BCI, FES, and VR technologies, the present invention forms a complete closed-loop training system. During use, the VR display device can provide task prompts for the motor imagery action pattern. The BCI analyzes the user's motor intention when performing tasks, and through functional electrical stimulation, the corresponding muscles contract to complete limb movements in the same pattern as the motor imagery tasks prompted in the training system. This couples the spontaneous motor intention at the neural level with the passive movement at the muscle level, improving the repair of the original neural-muscular control pathway or the construction of a new neural-muscular control pathway, and can enhance the user's sense of participation during training, forming an efficient feedback loop.

[0136] To facilitate a clear understanding of the technical solution of the present invention, the following further elaborates with specific embodiments.

[0137] See Figure 4 , which is the system logic structure and flowchart for the user to train using the training system and training method provided by the present invention. In this embodiment, the training system consists of an electroencephalogram (EEG) acquisition device, a VR all-in-one headset, a functional electrical stimulation device, and a host computer. The data processing module is deployed in the host computer, and the motor imagery continuous control model and the dynamic adaptive feature-feedback mapping model are integrated in the data processing module. The host computer and the VR all-in-one headset transmit the training scenario through VR streaming technology, and the EEG acquisition device and the host computer transmit the EEG data through the TCP / IP protocol.

[0138] The user can simulate controlling an automatically moving wheelchair and immerse themselves in roaming in the virtual environment built by VR. During the roaming process, in the face of task events such as turning and avoiding obstacles, the user needs to imagine grasping actions with the hand to achieve turning and deceleration. The system uses a random sequence to generate a certain number of tasks, and when all tasks are completed, the training ends.

[0139] Before use, the user needs to wear the EEG acquisition device, the VR all-in-one headset, and the functional electrical stimulation device. The processing flow for a specific task includes the following steps:

[0140] S10. Collect non-task-state EEG signals, perform preprocessing and A / D conversion on the non-task-state EEG signals;

[0141] After the system starts, the EEG acquisition device collects EEG signals containing the user's motor intention, and completes steps such as analog signal processing including denoising and amplification and A / D conversion. During this process, the user remains relaxed.

[0142] S11. Complete the feature extraction and preservation of the non-task-state EEG signals;

[0143] The data processing module deployed on the host computer runs the motor imagery continuous control model, calculates the characteristic parameters of the non-task electroencephalogram (EEG) signals, and saves them in the parameter set of the non-task EEG signals.

[0144] S12. The VR all-in-one machine presents visual task prompts to the user to guide the user to perform motor imagery.

[0145] When a task is started in the system, the VR all-in-one machine uses the current scene information and text prompts to inform the user of the action to be performed. Based on the task prompts, the user continuously simulates the execution of this action in the brain. For example, imagine grasping with the right hand to turn the wheelchair to the left.

[0146] S13. Collect the task EEG signals, and perform preprocessing and A / D conversion on the task EEG signals.

[0147] The EEG acquisition device collects the EEG signals containing the user's motor intention, and completes steps such as denoising, amplification and other analog signal processing and A / D conversion.

[0148] S14. Perform operations on the characteristic parameters and control parameters.

[0149] The data processing module deployed on the host computer runs the motor imagery continuous control model, calculates the characteristic parameters of the task EEG signals, and transfers the characteristic parameters to the dynamic adaptive feature-feedback mapping model. The system needs to determine whether the current is the first task. If it is the first task, it calls the set of characteristic parameters of the non-task EEG signals to construct the feature-feedback parameter mapping; otherwise, it calls the already constructed feature-feedback parameter mapping and returns the control parameters.

[0150] S15. Feedback the execution result of the motor intention to the user.

[0151] The VR all-in-one machine feeds back the execution result of the motor intention to the user according to the control parameters. For example, turn left by a certain angle. The functional electrical stimulation device applies corresponding intensity electrical stimulation to the corresponding limb of the user according to the control parameters. The user can adjust their motor imagery strategy based on these feedbacks to better complete the task.

[0152] S16. Iteratively update the dynamic adaptive feature-feedback mapping model.

[0153] The data processing module deployed in the host computer saves the current task state parameters in the task state EEG parameter set. If the EEG feature parameters in the set have reached the upper limit at this time, the earliest stored data in the set is deleted to ensure that the parameters in the set are the latest. After the saving is completed, the data processing module calculates the baseline value and threshold of the feature parameters according to the preset correction function and the new parameter set, completes the iteration of the dynamic adaptive feature-feedback mapping model, and saves the new dynamic adaptive feature-feedback mapping model in the data processing module. It should be noted that steps S15 and S16 can run in parallel.

[0154] Although the present invention has been described in connection with various embodiments, however, in the process of implementing the claimed invention, those skilled in the art can understand and realize other variations of the disclosed embodiments by viewing the drawings, the disclosure content, and the like. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality of cases. A single processor or other unit can implement several functions listed in the specification. Certain measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0155] Although the present invention has been described in connection with specific features and their embodiments, it is obvious that various modifications and combinations can be made without departing from the spirit and scope of the present invention. Accordingly, the present specification and the drawings are only exemplary descriptions of the present invention and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the present invention and its equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. An adaptive dynamic feedback brain-computer interface training method based on virtual reality, characterized in that Before performing the training method, the user wears an electroencephalogram (EEG) acquisition device, a virtual reality (VR) display device, and a functional electrical stimulation device; wherein, a training scenario is pre-built in the VR display device; the training method includes the following steps: S1. The user performs motor imagery according to the training scenario presented by the VR display device. S2. The EEG acquisition device acquires EEG signals. S3. Configure a data processing module and perform the following sub-steps: S30. Using the EEG signals as input, run a motor imagery continuous control model to calculate and obtain characteristic parameters, specifically including the following sub-steps: S300. Configure and slide a time window to continuously intercept from the task-state EEG signals to obtain multi-channel task-state EEG signals. S301. Calculate the power spectral density matrix of multi-channel task-state EEG signals ; S302. Calculate the power spectral density matrix of the non-task-state EEG signals ; S303. Calculate the relative power spectral density matrix , ; , where is the number of sampling channels of the EEG signal, is the number of sampling points within the sliding time window, is the set of real numbers; S304. Select the target frequency band related to motor imagery and calculate the relative power value corresponding to the target frequency band ; S305. According to the motor imagery actions and the corresponding limbs, select the corresponding channel combinations, assign weights to the relative power values of different channels, and obtain the characteristic parameters with the greatest difference. S31. When it is determined that there is a task, using the characteristic parameters as input, run a dynamic adaptive feature-feedback mapping model to calculate and obtain control parameters. S4. Send the control parameters to the functional electrical stimulation device to apply electrical stimulation of corresponding intensity to the muscles to obtain tactile feedback; meanwhile, the dynamic adaptive feature-feedback mapping model is dynamically updated according to the latest input characteristic parameters. S5. Send the control parameters to the VR display device to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and tactile feedback and then repeats S1 to S5.

2. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 1, wherein Calculate the power spectral density matrix of multi-channel task-state EEG signals in the following way : Among them, represents the power spectral density matrix at time point n, represents the EEG signal intercepted by the sliding time window corresponding to time point n, is the number of sampling channels of the EEG signal, is the number of sampling points within the sliding time window, represents the EEG signal of the Fourier transform, is the set of real numbers.

3. The method for training a brain-computer interface with adaptive dynamic feedback based on virtual reality according to claim 2, wherein The relative power value corresponding to the target frequency band is calculated by the following method : Among them, represents the target frequency band, , where fs represents the sampling rate of the EEG signal, represents the power corresponding to the frequency f in the relative power spectral density, reflecting the frequency domain-energy characteristics of the whole brain at any time period, is the set of real numbers.

4. The method for training a brain-computer interface with adaptive dynamic feedback based on virtual reality according to claim 3, wherein, The characteristic parameters are calculated by the following method: Among them, is the selected channel combination, is the weight corresponding to the c channel, represents the relative power of the c channel in the target frequency band and is the set of real numbers.

5. The adaptive dynamic feedback brain-computer interface training method based on virtual reality according to claim 1, characterized in that, Before running the dynamic adaptive feature-feedback mapping model, the training method further includes: Before the start of the first motor imagery task, construct a dynamic adaptive feature-feedback mapping model, specifically including the following steps: Select a feature-feedback mapping function, which is a function related to the characteristic parameters, characteristic parameter thresholds, and characteristic parameter baselines at any time. Calculate the initial baseline value C of the characteristic parameter B0 and the initial threshold value C of the characteristic parameter T0 , which specifically includes the following sub-steps: Calculate and obtain a set of non-task state feature parameters before the start of the first motor imagery task ; According to the set of the data distribution of the non-task state characteristic parameters, obtain the maximum value of the non-task state characteristic parameters , and the third quartile ; Taking as the upper limit of outliers for the first motor imagery task state and as the lower limit of outliers for the first motor imagery task state, and assuming that the characteristic parameters of the first motor imagery task state are uniformly distributed, the initial value C of the characteristic parameter baseline is determined in the following manner B0 and the initial value C of the characteristic parameter threshold T0 : 。 6. The method for training a brain-computer interface with adaptive dynamic feedback based on virtual reality according to claim 5, characterized in that After the first motor imagery task is executed, the feature parameter threshold C is iterated by the following method T and the feature parameter baseline value C B : The feature parameter C obtained at any moment of the movement imagination task state Xn is added to the task state feature parameter set C Test , and the value range is constructed according to the distribution of the feature parameters in C Test : Among them, and respectively represent the maximum value and the minimum value of the two; When C Test When the number of data in the set exceeds the set length limit, the feature parameter that is the oldest from the current time point is removed, that is, the latest feature parameter is retained.

7. The method for training a virtual reality-based adaptive dynamic feedback brain-computer interface according to claim 6, wherein When dynamically iterating the value range of the feature parameters each time, introduce the correction function f to correct C T affected by C Xn the degree of influence, that is .

8. An adaptive dynamic feedback brain-computer interface training system based on virtual reality, which is used to implement the adaptive dynamic feedback brain-computer interface training method according to any one of claims 1 to 7, characterized in that The adaptive dynamic feedback brain-computer interface training system includes an EEG acquisition device, a data processing module, a VR display device, and a functional electrical stimulation device. Among them, a training scenario is pre-built in the VR display device, and the user performs motor imagery according to the training scenario presented by the VR display device. The EEG acquisition device is used to acquire EEG signals when the user performs motor imagery. The data processing module is used to receive the EEG signals, run a motor imagery continuous control model to calculate and obtain characteristic parameters; it is also used to, when it is determined that there is a task, use the characteristic parameters as input and run a dynamic adaptive feature-feedback mapping model to calculate and obtain control parameters. The functional electrical stimulation device receives the control parameters to apply electrical stimulation of a preset intensity to the muscles to obtain tactile feedback; the VR display device is also used to receive the control parameters to present visual feedback, and the user adjusts the motor imagery based on the visual feedback and tactile feedback to enable the EEG acquisition device to re-acquire EEG signals.

9. The adaptive dynamic feedback brain-computer interface training system based on virtual reality according to claim 8, wherein The electroencephalogram (EEG) acquisition device includes electrodes and an EEG amplifier. The electrodes collect the user's EEG signals, and after amplifying and filtering the EEG signals through the EEG amplifier, the signals are sent to the data processing module. The training system further includes a first control computer and a second control computer. The first control computer is configured to receive control parameters and generate a first control instruction based on the control parameters for transmission to the VR display device. The second control computer is configured to receive control parameters and generate a second control instruction based on the control parameters for transmission to the functional electrical stimulation device. The data processing module includes an acquisition computer, which is configured to calculate feature parameters and control parameters and transmit the control parameters to the first control computer and the second control computer.

Images