Rehabilitation training auxiliary method and system based on brain-computer interface and virtual reality

Abstract

The embodiment of the present invention provides a rehabilitation training auxiliary method and system based on brain-computer interface and virtual reality, belonging to the field of medical care technology. The method includes: determining the target motion mode of the target user according to the user device of the target user; when the target motion mode is the rehabilitation training mode, obtaining the target motion imagination decoding model of the target user from the cloud server; obtaining the first EEG data of the target user under the first motion imagination task, and using a preset filter to perform bandpass filtering on the first EEG data to obtain the first filtered data; performing motion imagination decoding on the first filtered data according to the target motion imagination decoding model to obtain the target motion imagination action of the target user; determining the first target operation of the virtual device of the target user according to the target motion imagination action and determining the second target operation of the rehabilitation device of the target user according to the target motion imagination action; performing rehabilitation training on the target user according to the first target operation and the second target operation.

Description

Technical Field

[0001] The present invention relates to the field of medical care technology, and in particular to a rehabilitation training auxiliary method and system based on brain-computer interface and virtual reality. Background Art

[0002] Rehabilitation training for stroke patients under the guidance of rehabilitation physicians has become an important way to help stroke patients recover their motor function. At the same time, rehabilitation training technology regulated by brain-computer interface has also shown significant effects in motor function recovery training. However, traditional physical rehabilitation methods usually perform mechanized and repetitive movements under the guidance of physicians. This method is easily affected by the physician's subjective consciousness, and patients may have resistance due to lack of participation, thus affecting the rehabilitation effect. The decoding algorithms of existing brain-computer interface-guided motor imagery rehabilitation methods are often too simple or single, making it difficult to accurately capture the patient's motor intentions. Complex models rely on high computing resources, making it difficult to achieve widespread promotion and application, and it is also difficult to ensure the real-time feedback performance of the system. In addition, most of the current brain-computer interfaces that use motor imagery for movement guidance and rehabilitation training have failed to fully consider the feedback effects of vision and touch on motor imagery decoding. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a rehabilitation training auxiliary method and system based on brain-computer interface and virtual reality, aiming to solve the problem that the decoding algorithm of the brain-computer interface-guided motor imagery rehabilitation method in the related technology is often too simple or single, and it is difficult to accurately capture the patient's movement intention.

[0004] In a first aspect, an embodiment of the present invention provides a rehabilitation training auxiliary method based on a brain-computer interface and virtual reality, comprising:

[0005] Determine a target motion mode corresponding to the target user according to the selection of the target user in the user device;

[0006] When the target movement mode is a rehabilitation training mode, obtaining a target movement imagery decoding model corresponding to the target user from a cloud server;

[0007] Acquire first EEG data corresponding to the target user under a first motor imagery task, and perform bandpass filtering on the first EEG data using a preset filter to obtain first filtered data;

[0008] Performing motor imagery decoding on the first filtered data according to the target motor imagery decoding model to obtain a target motor imagery action corresponding to the target user;

[0009] Determine a first target operation of the virtual device corresponding to the target user according to the target motor imagery action and determine a second target operation of the rehabilitation device corresponding to the target user according to the target motor imagery action;

[0010] Rehabilitation training is performed on the target user according to the first target operation and the second target operation.

[0011] In a second aspect, an embodiment of the present invention provides a rehabilitation training auxiliary system based on a brain-computer interface and virtual reality, comprising:

[0012] A mode determination module, used to determine a target motion mode corresponding to the target user according to the selection of the target user in the user device;

[0013] A model acquisition module, used for obtaining a target movement imagery decoding model corresponding to the target user from a cloud server when the target movement mode is a rehabilitation training mode;

[0014] A data acquisition module, used to acquire first EEG data corresponding to the target user under the first motor imagery task, and perform bandpass filtering on the first EEG data using a preset filter to obtain first filtered data;

[0015] A data decoding module, configured to perform motion imagery decoding on the first filtered data according to the target motion imagery decoding model to obtain a target motion imagery action corresponding to the target user;

[0016] An operation determination module, configured to determine a first target operation of a virtual device corresponding to the target user according to the target motor imagery action and to determine a second target operation of a rehabilitation device corresponding to the target user according to the target motor imagery action;

[0017] A rehabilitation training module is used to perform rehabilitation training on the target user according to the first target operation and the second target operation.

[0018] The embodiment of the present invention provides a rehabilitation training auxiliary method and system based on brain-computer interface and virtual reality, the method:

[0019] According to the user device used by the target user, the target motion mode suitable for the target user is determined. When the target motion mode is set to the rehabilitation training mode, the target motion imagination decoding model specially provided for the target user is obtained from the cloud server. The target motion imagination decoding model can be used to decode the EEG data of the target user more accurately, significantly improving the recognition accuracy of the user's motion imagination action; then the first EEG data collected by the target user in the first motion imagination task is obtained, and the first EEG data is bandpass filtered to remove noise and interference. The first filtered data is obtained after processing by a preset filter, thereby eliminating the frequency band irrelevant to the motion imagination and retaining the frequency band related to the motion imagination, and then the target motion imagination decoding model is used to decode the first filtered data to obtain the target motion imagination action, and then the first target operation of the virtual device and the second target operation of the rehabilitation device are determined according to the target motion imagination action. This step makes the operation between the virtual training and the actual rehabilitation device coordinated and consistent, ensuring the integrity of the training program. The first target operation and the second target operation are applied to the rehabilitation training process, and a comprehensive and integrated training experience is achieved through the effective linkage of the virtual device and the rehabilitation device. This linkage can not only improve the interactivity of the training, but also enhance the user's sense of participation and enthusiasm. It also solves the problem of brain-computer interface-guided motor imagery rehabilitation methods in related technologies, whose decoding algorithms are often too simple or single, making it difficult to accurately capture the patient's motor intentions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a rehabilitation training auxiliary method based on brain-computer interface and virtual reality provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a model structure of a target motion imagery decoding model provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the network structure of a temporal convolutional neural network provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of two stationary symmetrical virtual hands presented on a virtual screen of a VR device provided by an embodiment of the present invention;

[0025] Figure 5A schematic diagram of presenting a second prompt symbol on a virtual screen of a VR device provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram showing a first target hand performing a grasping operation in a virtual screen of a VR device provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of a scenario of a rehabilitation training auxiliary method based on a brain-computer interface and virtual reality provided in an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of the module structure of a rehabilitation training auxiliary system based on brain-computer interface and virtual reality provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] The embodiment of the present invention provides a method and system for assisting rehabilitation training based on brain-computer interface and virtual reality. The method and system for assisting rehabilitation training based on brain-computer interface and virtual reality can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0033] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1A flowchart of a rehabilitation training assistance method based on brain-computer interface and virtual reality is provided in an embodiment of the present invention.

[0035] like Figure 1 As shown, the rehabilitation training auxiliary method based on brain-computer interface and virtual reality includes steps S101 to S106.

[0036] Step S101: determining a target motion mode corresponding to the target user according to a selection made by the target user in a user device.

[0037] Exemplarily, the target user is a user who needs rehabilitation training, and the user device corresponding to the target user is a user terminal connected to a rehabilitation device or a virtual device, or is any one of the rehabilitation device or the virtual device. For example, a client of a virtual reality hand movement rehabilitation training guidance paradigm is established using a user device, and then the target user selects a movement mode on the client, thereby obtaining a target movement mode corresponding to the target user according to the client.

[0038] Exemplarily, the target motion mode may be any one of a rehabilitation training mode or a motion calibration mode, or may be other modes. This application does not impose any specific restrictions and may be selected according to actual needs.

[0039] Step S102: When the target movement mode is a rehabilitation training mode, a target movement imagery decoding model corresponding to the target user is obtained from a cloud server.

[0040] Exemplarily, when the target motion mode is the rehabilitation training mode, a model query is performed from the cloud server according to the user identifier corresponding to the target user to obtain the target motion imagery decoding model corresponding to the target user. For example, the target motion imagery decoding model is downloaded from the cloud server via the FTP protocol to start the rehabilitation training mode.

[0041] Step S103: acquiring first EEG data corresponding to the target user under the first motor imagery task, and performing bandpass filtering on the first EEG data using a preset filter to obtain first filtered data.

[0042] Exemplarily, the first motor imagery task is a rehabilitation action that the target user needs to perform. The target user can determine the first motor imagery task according to his or her own needs, and then the EEG acquisition system acquires the first EEG data corresponding to the target user under the first motor imagery task.

[0043] Exemplarily, after obtaining the first EEG data, a preset filter is used to perform bandpass filtering on the first EEG data to obtain first filtered data.

[0044] For example, the filter is a sixth-order Butterworth filter, and the rehabilitation training mode is performed in 3 blocks, each block includes 24 trials, and each first motor imagery task appears randomly, and then the first EEG data of the target user performing the first motor imagery task is randomly collected, and then the first EEG data is subjected to 4-32 Hz bandpass filtering using a sixth-order Butterworth filter to obtain the first filtered data.

[0045] Step S104: performing motion imagery decoding on the first filtered data according to the target motion imagery decoding model to obtain a target motion imagery action corresponding to the target user.

[0046] Exemplarily, the target motor imagery decoding model is used to perform motor imagery decoding on the first filtered data, thereby identifying the target motor imagery action corresponding to the target user under the first EEG data.

[0047] Step S105: determining a first target operation of the virtual device corresponding to the target user according to the target motor imagery action and determining a second target operation of the rehabilitation device corresponding to the target user according to the target motor imagery action.

[0048] Exemplarily, the virtual device may be a VR device, and the rehabilitation device may be a real rehabilitation equipment. Then, a first target operation corresponding to the target motion imagination action is displayed in the VR device, and a second target operation corresponding to the target motion imagination action is presented in the real rehabilitation equipment.

[0049] For example, if the target movement imagination action is left-hand grasping, the first target operation corresponding to the left-hand grasping displayed in the VR device is grasping by the virtual left hand, and the second target operation corresponding to the left-hand grasping in the real rehabilitation equipment is grasping by the left hand in the rehabilitation equipment.

[0050] Step S106: performing rehabilitation training on the target user according to the first target operation and the second target operation.

[0051] Exemplarily, the target user obtains a first target operation in the virtual device to perform a rehabilitation action prompt, and then performs rehabilitation training assistance according to a second target operation in the rehabilitation device, so that the target user performs rehabilitation training.

[0052] In some embodiments, the method also includes: when the target motion mode is the motion calibration mode, or the target motion mode is the rehabilitation training mode, but the target user is a new user, obtaining second EEG data corresponding to the target user under a second motor imagery task; and obtaining the target motor imagery decoding model corresponding to the target user based on the second EEG data and the second motor imagery task.

[0053] Exemplarily, when the target motion mode is the motion calibration mode, it indicates that the target motion imagination decoding model corresponding to the target user does not meet the actual needs of the user when performing running decoding, and the target motion imagination decoding model corresponding to the target user needs to be calibrated. When the target motion mode is the rehabilitation training mode, but the target user is a new user, the target motion imagination decoding model corresponding to the target user does not exist in the cloud server. Even if the target motion mode selected by the target user is the rehabilitation training mode, the target motion mode corresponding to the target user needs to be switched to the motion calibration mode, and then the target motion imagination decoding model corresponding to the target user is obtained.

[0054] Exemplarily, the second motor imagery task is any one of the rehabilitation movements that the target user needs to perform, and then the second EEG data corresponding to the target user under the second motor imagery task is obtained according to the EEG acquisition device, so as to train the decoding model according to the second EEG data and the second motor imagery task to obtain the target motor imagery decoding model corresponding to the target user.

[0055] For example, the target user is asked to perform a second motor imagery task for 3 blocks, each block contains 24 trials, including 12 second running imagined movements of left hand grasping and right hand grasping, and each second motor imagery task appears randomly, and the second EEG data of the target user performing the second motor imagery task is collected.

[0056] Specifically, in the motion calibration mode, the EEG signal decoding results of the target user are calibrated more accurately, which lays a more reliable foundation for subsequent rehabilitation training. When the target user is a new user, by collecting their EEG data, a target motor imagery decoding model specifically for the target user can be generated, thereby significantly improving the accuracy of subsequent decoding.

[0057] In some embodiments, the target motor imagery decoding model includes: a dynamic time attention mechanism module, a multi-scale spatiotemporal feature extraction module, a Transformer module, and an output module. The target motor imagery decoding model corresponding to the target user is obtained according to the second EEG data and the second motor imagery task, including: using the filter to perform bandpass filtering on the second EEG data to obtain second filtered data, and uploading the second filtered data and the second motor imagery task to the cloud server using the FTP protocol, so that the cloud server stores the second filtered data and the second motor imagery task in the cloud database; using the dynamic time attention mechanism module to perform signal weighting on the second filtered data to obtain the first intermediate data; using the multi-scale spatiotemporal feature extraction module to perform multi-scale spatiotemporal feature extraction on the first intermediate data to obtain second intermediate data; using the Transformer module to perform high-level feature extraction on the second intermediate data to obtain third intermediate data; using the output module to predict the imagined movement type of the third intermediate data to obtain a predicted type; using a loss function to perform cross entropy loss calculation on the predicted type and the second movement imagination task to obtain a loss value; when the loss value is less than or equal to a preset value, obtaining the target movement imagination decoding model; when the loss value is greater than the preset value, continuing to use the second filtered data and the second movement imagination task to perform model training until the target movement imagination decoding model is obtained.

[0058] For example, Figure 2 As shown, the target motion imagery decoding model includes a dynamic time attention mechanism module, a multi-scale spatiotemporal feature extraction module, a Transformer module, and an output module.

[0059] Exemplarily, the second EEG data is bandpass filtered using a filter to obtain second filtered data. For example, the filter is a sixth-order Butterworth filter, and then the second EEG data is bandpass filtered at 4-32 Hz using a sixth-order Butterworth filter to obtain second filtered data.

[0060] Exemplarily, the second EEG data collected from the target user is uploaded to the cloud server using the FTP protocol and saved in the cloud database.

[0061] Exemplarily, the time window size is set, that is, the time period range for processing the second filtered data. A weight matrix is ​​defined in the dynamic time attention mechanism module to weight the data at different time points. These weight values ​​will determine the contribution of each time point to the final result. The dynamic time attention mechanism module is used to weight each time point in the second filtered data according to the set time window and weight matrix, and then the weighted data is summarized to obtain the first intermediate data.

[0062] Exemplarily, the multi-scale spatiotemporal feature extraction module extracts features at different scales and times from the first intermediate data. Determine the convolution scale to be used, each convolution scale corresponds to a convolution kernel of a different size. The multi-scale spatiotemporal feature extraction module usually needs to define multiple convolution scales to extract features of different granularities. In addition, it is also necessary to set a time window for analyzing the temporal changes of the first intermediate data. Thus, the multi-scale spatiotemporal feature extraction module is applied to the first intermediate data. The module will perform convolution, filtering or other feature extraction operations on the first intermediate data through different convolution scales to capture the feature information of the data at different convolution scales and different time and space to obtain the second intermediate data.

[0063] Exemplarily, the input parameters of the Transformer module are determined, including the embedding dimension, the number of attention heads, the number of layers, etc. The second intermediate data is converted into a representation suitable for input to the Transformer module through the embedding layer. In the Transformer module, the self-attention mechanism will allow the model to dynamically adjust the weights according to the features at different positions in the input sequence. By calculating the relationship between each feature, the model can identify important features and contextual information. After the self-attention mechanism, the Transformer module usually includes a feedforward neural network layer to further process and nonlinearly transform the extracted features. This step helps to increase the expressive power of the model. Layer normalization and residual connections are applied to enhance the stability and learning ability of the model, and finally, through the last layer of the Transformer module, high-level features are extracted to obtain the third intermediate data.

[0064] Exemplarily, the output module predicts the imagined motion type of the third intermediate data to obtain the predicted type, and then uses the loss function to calculate the cross entropy loss of the predicted type and the label corresponding to the second motion imagination task to obtain the loss value.

[0065] Exemplarily, when the loss value is less than or equal to a preset value, the current model parameters meet the requirements, thereby obtaining a target motor imagery decoding model; when the loss value is greater than the preset value, the current model parameters do not meet the requirements, and it is necessary to continue to use the second filtering data and the second motor imagery task for model training until the target motor imagery decoding model is obtained.

[0066] Specifically, the dynamic time attention mechanism module is used to perform signal weighting on the second filter data to extract more accurate features; the multi-scale spatiotemporal feature extraction module is used to further extract rich features from different scales and times; then, the Transformer module performs high-level extraction of these features to obtain a deeper data representation. Finally, the output module predicts the motion type of these high-level features and evaluates them through the loss function to optimize the target motor imagery decoding model, thereby ensuring high-precision data processing and efficient model training, which helps to improve the decoding accuracy and application effect of motor imagery tasks.

[0067] In some embodiments, the use of the dynamic time attention mechanism module to perform signal weighting on the second filtered data to obtain the first intermediate data includes: using the dynamic time attention mechanism module to determine the number of channels and the first convolution filling method corresponding to the second filtered data, and determining the first convolution kernel according to the number of channels; using the dynamic time attention mechanism module to perform a convolution operation on the second filtered data according to the first convolution filling method and the first convolution kernel to obtain a first feature map; using the dynamic time attention mechanism module to reshape the first feature map to obtain a second feature map and a third feature map; performing matrix multiplication of the second feature map and the third feature map and then using the Softmax function to obtain the attention weight matrix; multiplying the attention weight matrix and the second filtered data to obtain a first weighted EEG signal; using the dynamic time attention mechanism module to determine a scaling factor, and using the scaling factor to perform dynamic time attention adjustment on the first weighted EEG signal to obtain a second weighted EEG signal; using the dynamic time attention mechanism module to add the second weighted EEG signal and the first feature map to implement a residual operation to obtain the first intermediate data.

[0068] Exemplarily, the number of channels corresponding to the second filter data and the first convolution filling method are determined by using the dynamic time attention mechanism module, so as to determine the first convolution kernel according to the number of channels. For example, the size of the first convolution kernel is (1, C), C is the number of channels of the second filter data, the number of the first convolution kernel is 16, the strides are (1, 1), and the first convolution filling method is the same.

[0069] Exemplarily, a dynamic time attention mechanism module is used to perform a convolution operation on the second filter data according to a first convolution filling method and a first convolution kernel to obtain a first feature map.

[0070] Exemplarily, the dynamic temporal attention mechanism module is used to reshape the first feature map to obtain the second feature map and the third feature map, and then the second feature map and the third feature map are matrix multiplied, and the Softmax function is used to obtain the attention weight matrix of the product of the second feature map and the third feature map, and then the attention weight matrix and the second filtered data are multiplied to obtain the first weighted EEG signal.

[0071] For example, the second feature map is , the third characteristic diagram is , and then the weight value corresponding to the i-th row and j-th column in the attention weight matrix W is obtained according to the following formula:

[0072]

[0073] in, represents the i-th second feature map, represents the jth third feature map, represents the natural exponential function, Softmax represents the Softmax function, T represents the number corresponding to the third feature map, represents the pth third feature map.

[0074] Exemplarily, a dynamic temporal attention mechanism module is used to determine a scaling factor to control dynamic temporal attention, and the scaling factor is multiplied by the first weighted EEG signal to adjust the dynamic temporal attention to obtain a second weighted EEG signal.

[0075] Exemplarily, the second weighted EEG signal and the first feature map are added together using a dynamic time attention mechanism module to implement a residual operation, thereby obtaining first intermediate data.

[0076] For example, the first intermediate data is obtained according to the following formula:

[0077]

[0078] in, represents the first intermediate data, represents the scaling factor, represents the attention weight matrix, represents the second filtering data, represents the first weighted EEG signal, represents the second weighted EEG signal, Represents the first feature map.

[0079] Specifically, through the processing of the dynamic time attention mechanism module, it is possible to weight the time dimension and highlight some important time points, thereby optimizing the convolution operation and generating high-quality feature maps. Reshaping and matrix multiplication of the feature map and calculating the attention weight matrix can effectively extract important information from the EEG signal. Subsequently, through weighting, dynamic adjustment and residual operation of the attention weight matrix, the signal processing is further refined to obtain more accurate and meaningful first intermediate data. This process not only improves the accuracy of feature extraction, but also enhances the model's ability to model time series data, which helps to improve the effect and reliability of the second EEG data analysis.

[0080] In some embodiments, the use of the multi-scale spatiotemporal feature extraction module to perform multi-scale spatiotemporal feature extraction on the first intermediate data to obtain second intermediate data includes: using the multi-scale spatiotemporal feature extraction module to determine multiple second convolution kernels corresponding to different frequency information and different time information, and determining the number of convolution kernels corresponding to the convolution layer and the second convolution filling method; performing convolution processing on the first intermediate data according to the multiple second convolution kernels, the number of convolution kernels and the second convolution filling method to obtain multiple fourth feature maps; splicing the multiple fourth feature maps along the depth dimension to obtain a fifth feature map; using a depth-separable convolution layer to perform spatial feature extraction on the fifth feature map to obtain first spatial feature information; using an average pooling layer to perform pooling processing on the first spatial feature information to obtain second spatial feature information; using a time feature extraction layer to perform time feature extraction and average pooling processing on the second spatial feature information to obtain second intermediate data.

[0081] Exemplarily, a multi-scale spatiotemporal feature extraction module is used to determine multiple second convolution kernels corresponding to different frequency information and different time information, and to determine the number of convolution kernels and the second convolution filling method corresponding to the convolution layer, so that the first intermediate data is convolved according to the multiple second convolution kernels, the number of convolution kernels, and the second convolution filling method to obtain multiple fourth feature maps; finally, the multiple fourth feature maps are spliced ​​along the depth dimension to obtain the fifth feature map.

[0082] For example, three second convolution kernel sizes are (32, 1), (16, 1), and (64, 1), the second convolution filling method is the same, and a two-dimensional convolution layer with 8 convolution kernels is used to extract frequency information above 4 Hz, 8 Hz, and 16 Hz and time information of 0.256 s, 0.128 s, and 0.064 s along the time dimension. Then, the three fourth feature maps corresponding to the convolution of the three scales are spliced ​​along the depth to obtain the fifth feature map.

[0083] Exemplarily, a depth-wise separable convolutional layer is used to perform spatial feature extraction on the fifth feature map to obtain first spatial feature information; an average pooling layer is used to perform pooling processing on the first spatial feature information to obtain second spatial feature information; and a temporal feature extraction layer is used to perform temporal feature extraction and average pooling processing on the second spatial feature information to obtain second intermediate data.

[0084] For example, a depth-separable convolution layer with a convolution kernel size of (1, C) and a convolution channel D of 2 is used to extract the spatial features of the fifth feature map to obtain the first spatial feature information, and then an average pooling layer with a pooling size of (1, 8) is used to downsample the first spatial feature information to 32Hz to obtain the second spatial feature information. In order to further deepen the extraction of temporal features, a convolution layer with a convolution kernel size of (16, 1), strides of (1, 1), 32 convolution kernels, and the same padding method is finally used to extract temporal features, and an average pooling layer with a pooling size of (1, 8) is used to reduce the dimension of the output spatiotemporal features to obtain the second intermediate data.

[0085] Specifically, by determining the convolution kernels under different frequency and time information through the multi-scale spatiotemporal feature extraction module, it is possible to ensure that the details of the data are effectively captured at multiple scales, thereby comprehensively analyzing the data. The multiple feature maps generated by the convolution process provide data views from different angles, and splicing these feature maps can integrate rich information. Using a depth-separable convolution layer for spatial feature extraction helps to reduce computational complexity while maintaining the details of spatial features. Further processing through the average pooling layer and the temporal feature extraction layer can effectively reduce the data dimension and extract key spatial and temporal features, thereby obtaining more accurate and useful second intermediate data. This processing flow enhances the ability to analyze complex data and improves the performance of the model in processing and understanding time series data.

[0086] In some embodiments, the use of the Transformer module to perform high-level feature extraction on the second intermediate data to obtain third intermediate data includes: using a temporal convolutional neural network to obtain a global dependency feature corresponding to the second filtered data; adding and fusing the global dependency feature and the second intermediate data to obtain fused feature data; performing layer normalization on the fused feature data to obtain a normalized sixth feature map; capturing key features and dependency relationships of the sixth feature map through a multi-head attention mechanism to obtain a seventh feature map; adding the seventh feature map and the second intermediate data to perform a residual operation to obtain third intermediate data.

[0087] Exemplarily, the Transformer module performs more abstract encoding from the multi-scale spatiotemporal feature extraction module to obtain higher-level features in the feature sequence. First, the second filtered data is feature extracted using a temporal convolutional neural network to obtain the corresponding global dependency features, and then the global dependency features are added and fused with the second intermediate data to obtain fused feature data. Since the output of each point in the temporal convolutional neural network is determined by the features of the previous t-1 time points, it is necessary to regard the output of the temporal convolutional neural network as a positional encoding, and then perform layer normalization on the fused feature data to obtain a normalized sixth feature map. Among them, the network structure of the temporal convolutional neural network is as follows: Figure 3 shown.

[0088] Exemplarily, a multi-head attention mechanism is used to capture the key features and dependencies of the sixth feature map to obtain the seventh feature map, so that the model can more comprehensively capture the key feature information and dependencies in the feature sequence by dispersing attention, where the dimensions of Key, Query and Value are 8 and the number of heads is 2.

[0089] Exemplarily, the seventh feature map is added to the second intermediate data to perform a residual operation to achieve hierarchical feature fusion, thereby obtaining the third intermediate data.

[0090] Specifically, by extracting global dependency features through a temporal convolutional neural network, long-distance dependencies in the second intermediate data can be captured, which is particularly important for time series data. Fusion of global dependency features with the second intermediate data helps to combine global information with local features, thereby enhancing the comprehensiveness and richness of data representation. Through layer normalization, the training process can be stabilized and the convergence speed and stability of the model can be improved. The multi-head attention mechanism further optimizes the feature capture capability and can capture key features and dependencies from different angles and scales. Finally, by combining the seventh feature map with the second intermediate data through a residual operation, the detailed information of the shallow features can be supplemented into the deep features, so that the fused features have sufficient semantic information for classification tasks, while accelerating the convergence of the model, making the third intermediate data more accurate and useful.

[0091] In some embodiments, determining the first target operation of the virtual device corresponding to the target user according to the target movement imagination action and determining the second target operation of the rehabilitation device corresponding to the target user according to the target movement imagination action include: presenting two symmetrical virtual hands at the display position corresponding to the virtual device, and displaying a first prompt symbol in the middle of the two virtual hands; determining the second prompt symbol according to the target movement imagination action, and after prompting the target user to continuously stare at the first prompt symbol for a first time, replacing the first prompt symbol with the second prompt symbol; determining the first target hand from the two virtual hands according to the second prompt symbol, and after the second prompt symbol is presented for a second time, causing the first target hand to perform a grasping action according to a third time in the virtual device; determining the second target hand corresponding to the rehabilitation device according to the target movement imagination action, if the first target hand and the second target hand are consistent, providing visual feedback for the luminous operation of the second target hand for a fourth time, and using the auxiliary grasping movement of the second target hand for a fifth time; if the first target hand and the second target hand are inconsistent, skipping the fourth time and the fifth time, that is, the second target hand does not perform the luminous operation, and the second target hand does not perform any auxiliary grasping movement.

[0092] Exemplarily, two symmetrical virtual hands are presented at the display position corresponding to the virtual device, and a first prompt symbol is displayed in the middle of the two virtual hands; a second prompt symbol is determined according to the target movement imagination action, and after the target user is prompted to continuously stare at the first prompt symbol for a first time, the first prompt symbol is replaced with the second prompt symbol.

[0093] Exemplarily, a first target hand is determined from two virtual hands according to a second prompt symbol, and after the second prompt symbol is presented for a second time, the first target hand performs a grasping action according to a third time in the virtual device.

[0094] For example, two symmetrical virtual hands are first presented on the virtual screen of the VR device, such as Figure 4 As shown in the figure, there is a "+" in the middle of the two virtual hands, that is, the "+" is the first prompt symbol. Before the guided experiment begins, the target user needs to stare at the "+" in the VR device for 1 second, that is, 1 second is the first time, then the "+" disappears, and then an arrow indicating left or right will randomly appear at the same position of the "+", that is, the arrow indicating left or right is the second prompt symbol. The direction of the arrow is used to prompt the target user's behavior in the next trial, that is, to autonomously imagine the hand in the direction indicated by the arrow to perform a grasping action such as Figure 5As shown, the hand in the direction indicated by the self-imagined arrow is determined as the first target hand, so that after the second prompt symbol is presented for the second time, the first target hand performs a grasping action in the virtual device for 4-6 seconds, that is, 4-6 seconds is the third time. Each 1-2 seconds is a cycle, and a grasping movement is completed. Each trial requires the target user to complete n (n can be set manually) grasping movements synchronously according to the grasping rhythm of the virtual hand, such as Figure 6 shown.

[0095] Exemplarily, the second target hand corresponding to the rehabilitation device is determined according to the target movement imaginary action. For example, if the target movement imaginary action is left hand grasping, the second target hand is a virtual left hand, and if the target movement imaginary action is right hand grasping, the second target hand is a virtual right hand. If the first target hand and the second target hand are consistent, the light-emitting operation of the second target hand for the fourth time is visually fed back, and the auxiliary grasping movement of the second target hand for the fifth time is used.

[0096] For example, if the first target hand and the second target hand are consistent, then 0.5s of visual feedback consistent with the actual task is presented, and 1-2s of tactile feedback consistent with the actual motor imagery task is performed to complete an assisted grasping. The form of visual feedback is to make the unilateral hand of the actual task glow, and the tactile feedback is to use the mechanical force of the rehabilitation glove to assist the hand in grasping movement. Among them, the fourth time is 0.5s, and the fifth time is 1-2s.

[0097] Exemplarily, if the first target hand and the second target hand are inconsistent, the fourth time and the fifth time are skipped, that is, the second target hand does not perform the light-emitting operation, and the second target hand does not perform any auxiliary grasping movement.

[0098] Specifically, by presenting symmetrical virtual hands and the first prompt symbol at the display position, the user can be guided to concentrate and clarify the movement target. The second prompt symbol is updated according to the target movement imagination action to ensure that the target user's attention is always focused on the current task and accurately guide its movement imagination process. Displaying the first target hand and performing the grasping action through a virtual device can help users better understand and imitate the target action in actual operation, thereby improving the effect of movement imagination. Adjusting the visual feedback and auxiliary grasping movement according to the consistency of the first target hand and the second target hand, this personalized feedback mechanism can enhance the pertinence and effectiveness of training. If the first target hand and the second target hand are consistent, visual feedback and auxiliary movement are performed to help users better master the movement; if they are inconsistent, these feedback and auxiliary operations are skipped to avoid invalid training. Such steps can effectively improve the user's sense of participation and training effect, while avoiding unnecessary interference, which helps to improve the efficiency and accuracy of the rehabilitation and training process.

[0099] In some embodiments, obtaining the second EEG data corresponding to the target user under the second motor imagery task includes: presenting two symmetrical virtual hands at the display position corresponding to the virtual device, and displaying the first prompt symbol at the middle position of the two virtual hands; determining the third prompt symbol according to the second motor imagery task, and prompting the target user to continuously stare at the first prompt symbol for the first time, and then replacing the first prompt symbol with the third prompt symbol; determining the third target hand from the two virtual hands according to the third prompt symbol, and after the third prompt symbol is presented for the second time, causing the third target hand to perform a grasping action according to the third time in the virtual device; determining the fourth target hand corresponding to the rehabilitation device according to the second motor imagery task, and providing visual feedback for the luminous operation of the fourth target hand for the fourth time, and at the same time, using the fourth target hand for the auxiliary grasping movement for the fifth time; while performing the second motor imagery task, collecting the second EEG data corresponding to the target user using an EEG device.

[0100] Exemplarily, in the motion calibration mode, two symmetrical virtual hands are presented at the display position corresponding to the virtual device, and a first prompt symbol is displayed in the middle of the two virtual hands. The third prompt symbol is determined according to the second motor imagery task, and after the target user is prompted to stare at the first prompt symbol for a first time, the first prompt symbol is replaced with the third prompt symbol.

[0101] For example, two symmetrical virtual hands are presented first, with a “+” in the middle of the virtual hands, that is, the “+” is the first prompt symbol. Before the guided experiment starts, the user is required to stare at the “+” in the VR device for 1 second, and then an arrow will appear, that is, the arrow is the third prompt symbol, indicating that the target user needs to imagine the hand in the direction indicated by the arrow to perform a grasping action, and the third prompt symbol will be presented for 1 second.

[0102] Exemplarily, the third target hand is determined from the two virtual hands according to the third prompt symbol, that is, the virtual hand pointed to by the third prompt symbol is the third target hand, for example, if the third prompt symbol points to the left, the virtual hand on the left is the third target hand, and if the third prompt symbol points to the right, the virtual hand on the right is the third target hand. After the third prompt symbol is presented for the second time, the third target hand performs a grasping action according to the third time in the virtual device, and the fourth target hand corresponding to the rehabilitation device is determined according to the second motor imagery task, so as to provide visual feedback for the luminous operation of the fourth target hand for the fourth time, and at the same time, the auxiliary grasping movement of the fourth target hand for the fifth time is used, and then when performing the second motor imagery task, the second EEG data corresponding to the target user is collected using the EEG device.

[0103] For example, the third prompt symbol is presented for 1 second, and then the corresponding third target hand appears to display the grasping movement, which is presented for 4-6 seconds, and a grasping movement is completed every 1-2 seconds. Each trial requires the target user to synchronously complete n (n can be set manually) grasping movements according to the grasping rhythm of the third target hand. Then, the fourth target hand corresponding to the rehabilitation device is determined according to the second motor imagery task. For example, if the target corresponding to the second motor imagery task is the left hand to grasp, the left hand in the rehabilitation device is determined as the fourth target hand, and then the light-emitting operation for the fourth time of 0.5 seconds is performed for visual feedback, and the auxiliary grasping movement of the fourth target hand for the fifth time of 1-2 seconds is performed for tactile feedback, that is, the rehabilitation glove corresponding to the fourth target hand assists in the grasping movement, wherein the grasping movement lasts for 1-2 seconds and completes one grasp. Therefore, when performing the second motor imagery task, the second EEG data corresponding to the target user is collected using the EEG device.

[0104] It should be noted that when collecting the second EEG data corresponding to the target user during the training process, the target user is not required to make adjustments based on the feedback. In fact, the visual feedback and tactile feedback during the training process are all programmed, and the visual feedback and tactile feedback are consistent with the third prompt symbol, with the purpose of prompting the target user to form a correct connection between motor imagery and subsequent rehabilitation movements.

[0105] Specifically, displaying symmetrical virtual hands and dynamically changing prompt symbols through a virtual device can attract the user's attention and enhance their participation and interest in the training or rehabilitation process. The transition from the first prompt symbol to the third prompt symbol provides the user with a clear and unambiguous visual guide, which helps the user to better understand and perform the second motor imagery task. The third target hand is determined according to the third prompt symbol, and the grasping action is simulated in the virtual device, which provides the user with a visual reference, which helps them to more accurately imitate and perform the corresponding actions in actual operation. The fourth target hand of the rehabilitation device performs luminous operations and assists in grasping movements, providing the user with immediate visual and kinesthetic feedback, which helps them perceive the correctness of their own actions. While performing the second motor imagery task, the EEG device is used to collect the user's second EEG data, which provides a rich source of information for subsequent data analysis and processing, and helps to more deeply understand the user's neural activity and motor imagery process, thereby optimizing the training or rehabilitation program.

[0106] In some embodiments, after performing rehabilitation training on the target user according to the first target operation and the second target operation, the method further includes: transmitting the first EEG data and the first motor imagery task to the cloud server via the FTP protocol to expand the basic database; determining the model accuracy corresponding to the target motor imagery decoding model according to the first motor imagery task and the target motor imagery action; when the model accuracy is less than or equal to a preset threshold, obtaining data within a preset time from the basic database to update the model parameters of the target motor imagery decoding model, obtaining the updated target motor imagery decoding model, and storing the updated target motor imagery decoding model to the cloud server.

[0107] Exemplarily, the first EEG data and the first motor imagery task collected in the rehabilitation training mode are uploaded to the cloud server via the FTP protocol to expand the basic database.

[0108] Exemplarily, a real motor imagery action corresponding to the first motor imagery task is obtained, and then the real motor imagery action and the target motor imagery action obtained by decoding the first EEG data by the target motor imagery decoding model are compared for consistency, so as to determine the model accuracy corresponding to the target motor imagery decoding model.

[0109] Exemplarily, the preset threshold can be set manually. When the model accuracy is greater than the preset threshold, the model accuracy of this rehabilitation training is output; when the model accuracy is less than or equal to the preset threshold, the data within the preset time is obtained from the basic database to update the model parameters of the target motor imagery decoding model, thereby obtaining an updated target motor imagery decoding model, and then storing the updated target motor imagery decoding model in the cloud server.

[0110] For example, when the model accuracy is less than or equal to the preset threshold, a parameter update instruction is sent to the cloud server. When the cloud server listens to the parameter update instruction, it starts to update the model parameters of the target motor imagery decoding model. The update of the target motor imagery decoding model mainly uses the most recently uploaded EEG data to fine-tune the parameters. First, the learning rate of all layers outside the fully connected layer is set to 0.0001 and the fully connected layer of the original target motor imagery decoding model is reinitialized. The learning rate of the fully connected layer is set to 0.01, and 500 rounds of training are performed. The trained model parameters are saved to the cloud server.

[0111] Specifically, by transmitting the first EEG data and the first motor imagery task to the cloud server to expand the basic database, more sample data can be accumulated. These data help the model to better learn and adapt to different motor imagery tasks, thereby improving the generalization ability of the decoding model. The accuracy of the target motor imagery decoding model is regularly evaluated according to the first motor imagery task and the target motor imagery action. If the model accuracy is found to be lower than the preset threshold, the model parameters are updated through the data in the basic database, which can effectively improve the performance of the model. This dynamic update mechanism ensures that the decoding model can be continuously optimized with the increase of time and data, thereby improving the performance of the target motor imagery decoding model.

[0112] Please refer to Figure 7 , Figure 7 A schematic diagram of a scene for implementing the rehabilitation training auxiliary method based on brain-computer interface and virtual reality provided in this embodiment, as shown in FIG. Figure 7As shown, the target motion mode corresponding to the target user is determined according to the user device corresponding to the target user; when the target motion mode is the rehabilitation training mode, the target motion imagination decoding model corresponding to the target user is obtained from the cloud server; the first EEG data corresponding to the target user under the first motion imagination task is obtained, and the first EEG data is bandpass filtered by using a preset filter to obtain the first filtered data; the first filtered data is subjected to motion imagination decoding according to the target motion imagination decoding model to obtain the target motion imagination action corresponding to the target user; two symmetrical virtual hands are presented at the display position corresponding to the virtual device, and a first prompt symbol is displayed at the middle position of the two virtual hands; a second prompt symbol is determined according to the target motion imagination action, and the target user is prompted After the user continuously stares at the first prompt symbol for the first time, the first prompt symbol is replaced with the second prompt symbol; the first target hand is determined from the two virtual hands according to the second prompt symbol, and after the second prompt symbol is presented for the second time, the first target hand performs a grasping action according to the third time in the virtual device; the second target hand corresponding to the rehabilitation device is determined according to the target movement imagination action, if the first target hand and the second target hand are consistent, the second target hand is continuously illuminated for the fourth time for visual feedback, and the second target hand is continuously assisted by the grasping action of the fifth time; if the first target hand and the second target hand are inconsistent, the fourth time and the fifth time are skipped, that is, the second target hand does not perform the luminous operation, and the second target hand does not perform any auxiliary grasping movement. Thus, rehabilitation training is performed on the target user according to the first target operation and the second target operation. When the target movement mode is the movement calibration mode, or the target movement mode is the rehabilitation training mode, but the target user is a new user, the second EEG data corresponding to the target user under the second movement imagination task is obtained; the target movement imagination decoding model corresponding to the target user is obtained according to the second EEG data and the second movement imagination task. In addition, the first EEG data and the first motor imagery task need to be transferred to the cloud server through the FTP protocol to expand the basic database; the model accuracy corresponding to the target motor imagery decoding model is determined according to the first motor imagery task and the target motor imagery action; when the model accuracy is less than or equal to the preset threshold, the data within the preset time is obtained from the basic database to update the model parameters of the target motor imagery decoding model, and the updated target motor imagery decoding model is obtained, and the updated target motor imagery decoding model is stored in the cloud server, thereby solving the existing brain-computer interface-guided motor imagery rehabilitation method, whose decoding algorithm is often too simple or single, and it is difficult to accurately capture the patient's motor intention. Moreover, when using the motor imagery brain-computer interface for movement guidance and rehabilitation training, the feedback effect of vision and touch on motor imagery decoding can be fully considered, further helping the target user to better conduct rehabilitation training.

[0113] See also Figure 8 , Figure 8A rehabilitation training auxiliary system 200 based on a brain-computer interface and virtual reality is provided in an embodiment of the present application. The rehabilitation training auxiliary system 200 based on a brain-computer interface and virtual reality includes a mode determination module 201, a model acquisition module 202, a data acquisition module 203, a data decoding module 204, an operation determination module 205, and a rehabilitation training module 206, wherein the mode determination module 201 is used to determine the target motion mode corresponding to the target user according to the selection of the target user in the user device; the model acquisition module 202 is used to obtain the target motion imagination decoding model corresponding to the target user from the cloud server when the target motion mode is the rehabilitation training mode; the data acquisition module 203 is used to obtain the target motion imagination decoding model corresponding to the target user The first EEG data corresponding to the target user under the first motor imagery task is obtained, and the first EEG data is band-pass filtered by a preset filter to obtain first filtered data; a data decoding module 204 is used to perform motor imagery decoding on the first filtered data according to the target motor imagery decoding model to obtain the target motor imagery action corresponding to the target user; an operation determination module 205 is used to determine the first target operation of the virtual device corresponding to the target user according to the target motor imagery action and determine the second target operation of the rehabilitation device corresponding to the target user according to the target motor imagery action; a rehabilitation training module 206 is used to perform rehabilitation training on the target user according to the first target operation and the second target operation.

[0114] In some embodiments, the rehabilitation training auxiliary system 200 based on brain-computer interface and virtual reality further performs:

[0115] When the target motion mode is the motion calibration mode, or the target motion mode is the rehabilitation training mode but the target user is a new user, obtaining second EEG data corresponding to the target user under a second motor imagery task;

[0116] The target motor imagery decoding model corresponding to the target user is obtained according to the second EEG data and the second motor imagery task.

[0117] In some embodiments, the target motor imagery decoding model includes: a dynamic time attention mechanism module, a multi-scale spatiotemporal feature extraction module, a Transformer module, and an output module. The rehabilitation training auxiliary system 200 based on brain-computer interface and virtual reality performs, in the process of obtaining the target motor imagery decoding model corresponding to the target user according to the second EEG data and the second motor imagery task:

[0118] Using the filter to perform bandpass filtering on the second EEG data to obtain second filtered data, and uploading the second filtered data and the second motor imagery task to the cloud server using the FTP protocol, so that the cloud server stores the second filtered data and the second motor imagery task in a cloud database;

[0119] Using the dynamic time attention mechanism module to perform signal weighting on the second filtered data to obtain first intermediate data;

[0120] Using the multi-scale spatio-temporal feature extraction module to perform multi-scale spatio-temporal feature extraction on the first intermediate data to obtain second intermediate data;

[0121] Using the Transformer module to perform high-level feature extraction on the second intermediate data to obtain third intermediate data;

[0122] Using the output module to predict the imagined motion type of the third intermediate data to obtain a predicted type;

[0123] Using a loss function to perform cross entropy loss calculation on the prediction type and the second motor imagery task to obtain a loss value;

[0124] When the loss value is less than or equal to a preset value, the target motor imagery decoding model is obtained;

[0125] When the loss value is greater than the preset value, the second filtered data and the second motor imagery task are continued to be used for model training until the target motor imagery decoding model is obtained.

[0126] In some embodiments, the rehabilitation training auxiliary system 200 based on brain-computer interface and virtual reality performs, during the process of using the dynamic time attention mechanism module to perform signal weighting on the second filtered data to obtain the first intermediate data:

[0127] Determine the number of channels and the first convolution filling method corresponding to the second filter data by using the dynamic time attention mechanism module, and determine the first convolution kernel according to the number of channels;

[0128] Using the dynamic time attention mechanism module to perform a convolution operation on the second filter data according to the first convolution filling method and the first convolution kernel to obtain a first feature map;

[0129] Reshape the first feature map using the dynamic time attention mechanism module to obtain a second feature map and a third feature map;

[0130] After performing matrix multiplication of the second feature map and the third feature map, the attention weight matrix is ​​obtained using the Softmax function;

[0131] Multiplying the attention weight matrix and the second filtered data to obtain a first weighted EEG signal;

[0132] Determine a scaling factor using the dynamic time attention mechanism module, and use the scaling factor to perform dynamic time attention adjustment on the first weighted EEG signal to obtain a second weighted EEG signal;

[0133] The dynamic time attention mechanism module is used to add the second weighted EEG signal and the first feature map to implement a residual operation, so as to obtain the first intermediate data.

[0134] In some embodiments, the rehabilitation training auxiliary system 200 based on brain-computer interface and virtual reality performs, during the process of extracting multi-scale spatiotemporal features from the first intermediate data using the multi-scale spatiotemporal feature extraction module to obtain the second intermediate data:

[0135] Determine a plurality of second convolution kernels corresponding to different frequency information and different time information by using the multi-scale spatiotemporal feature extraction module, and determine the number of convolution kernels corresponding to the convolution layer and the second convolution filling method;

[0136] Performing convolution processing on the first intermediate data according to the plurality of second convolution kernels, the number of the convolution kernels, and the second convolution filling method to obtain a plurality of fourth feature maps;

[0137] Performing splicing processing on the plurality of fourth feature maps along the depth dimension to obtain a fifth feature map;

[0138] Performing spatial feature extraction on the fifth feature map using a depthwise separable convolutional layer to obtain first spatial feature information;

[0139] Using an average pooling layer to perform pooling processing on the first spatial feature information to obtain second spatial feature information;

[0140] The time feature extraction layer is used to perform time feature extraction and average pooling processing on the second spatial feature information to obtain second intermediate data.

[0141] In some embodiments, the rehabilitation training auxiliary system 200 based on brain-computer interface and virtual reality performs, during the process of extracting advanced features from the second intermediate data using the Transformer module to obtain the third intermediate data:

[0142] Obtaining a global dependency feature corresponding to the second filtered data using a temporal convolutional neural network;

[0143] Performing feature fusion on the global dependency feature and the second intermediate data to obtain fused feature data;

[0144] Performing layer normalization processing on the fused feature data to obtain a normalized sixth feature map;

[0145] Capturing key features and dependency relationships of the sixth feature map through a multi-head attention mechanism to obtain a seventh feature map;

[0146] The seventh feature map and the second intermediate data are added to perform a residual operation to obtain third intermediate data.

[0147] In some implementations, the operation determination module 205, during the process of determining the first target operation of the virtual device corresponding to the target user according to the target motor imagery action and determining the second target operation of the rehabilitation device corresponding to the target user according to the target motor imagery action, performs:

[0148] Presenting two symmetrical virtual hands at the display position corresponding to the virtual device, and displaying a first prompt symbol in the middle of the two virtual hands;

[0149] Determine a second prompt symbol according to the target movement imagination action, and after prompting the target user to continue to gaze at the first prompt symbol for a first time, replace the first prompt symbol with the second prompt symbol;

[0150] Determine a first target hand from the two virtual hands according to the second prompt symbol, and after the second prompt symbol is presented for a second time, cause the first target hand to perform a grasping action according to a third time in the virtual device;

[0151] The second target hand corresponding to the rehabilitation device is determined according to the target motion imagination action. If the first target hand and the second target hand are consistent, the luminous operation of the second target hand for a fourth time is given visual feedback, and the auxiliary grasping movement of the second target hand for a fifth time is used; if the first target hand and the second target hand are inconsistent, the fourth time and the fifth time are skipped, that is, the second target hand does not perform the luminous operation, and the second target hand does not perform any auxiliary grasping movement.

[0152] In some embodiments, the rehabilitation training auxiliary system 200 based on brain-computer interface and virtual reality performs the following steps during the process of acquiring the second EEG data corresponding to the target user under the second motor imagery task:

[0153] Presenting two symmetrical virtual hands at the display position corresponding to the virtual device, and displaying the first prompt symbol in the middle of the two virtual hands;

[0154] Determine a third prompt symbol according to the second motor imagery task, and after prompting the target user to continue to gaze at the first prompt symbol for the first time, replace the first prompt symbol with the third prompt symbol;

[0155] Determine a third target hand from the two virtual hands according to the third prompt symbol, and after the third prompt symbol is presented for the second time, cause the third target hand to perform a grasping action according to the third time in the virtual device;

[0156] Determine a fourth target hand corresponding to the rehabilitation device according to the second motor imagery task, and provide visual feedback for the light-emitting operation of the fourth target hand for the fourth time, and at the same time, use the auxiliary grasping movement of the fourth target hand for the fifth time;

[0157] When performing the second motor imagery task, the second EEG data corresponding to the target user is collected using an EEG device.

[0158] In some implementations, the rehabilitation training module 206, after performing rehabilitation training on the target user according to the first target operation and the second target operation, further executes:

[0159] Transmitting the first EEG data and the first motor imagery task to the cloud server via the FTP protocol to expand the basic database;

[0160] Determine the model accuracy corresponding to the target motor imagery decoding model according to the first motor imagery task and the target motor imagery action;

[0161] When the model accuracy is less than or equal to a preset threshold, data within a preset time is obtained from the basic database to update the model parameters of the target motion imagination decoding model, the updated target motion imagination decoding model is obtained, and the updated target motion imagination decoding model is stored in the cloud server.

[0162] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the rehabilitation training assistance system 200 based on brain-computer interface and virtual reality described above can refer to the corresponding process in the aforementioned embodiment of the rehabilitation training assistance method based on brain-computer interface and virtual reality, and will not be repeated here.

[0163] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of the rehabilitation training auxiliary methods based on brain-computer interface and virtual reality provided in the description of the embodiment of the present invention.

[0164] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0165] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0166] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0167] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A rehabilitation training auxiliary method based on brain-computer interface and virtual reality, characterized in that: The method comprises: Determine a target motion mode corresponding to the target user according to the selection of the target user in the user device; When the target movement mode is a rehabilitation training mode, obtaining a target movement imagery decoding model corresponding to the target user from a cloud server; When the target motion mode is the motion calibration mode, or the target motion mode is the rehabilitation training mode and the target user is a new user, obtaining second EEG data corresponding to the target user under a second motor imagery task; obtaining the target motor imagery decoding model corresponding to the target user according to the second EEG data and the second motor imagery task; Acquire first EEG data corresponding to the target user under a first motor imagery task, and perform bandpass filtering on the first EEG data using a preset filter to obtain first filtered data; Performing motor imagery decoding on the first filtered data according to the target motor imagery decoding model to obtain a target motor imagery action corresponding to the target user; Determine a first target operation of the virtual device corresponding to the target user according to the target motor imagery action and determine a second target operation of the rehabilitation device corresponding to the target user according to the target motor imagery action; performing rehabilitation training on the target user according to the first target operation and the second target operation; The target motor imagery decoding model includes: a dynamic time attention mechanism module, a multi-scale spatiotemporal feature extraction module, a Transformer module, and an output module. The target motor imagery decoding model corresponding to the target user is obtained according to the second EEG data and the second motor imagery task, including: Using the preset filter to perform bandpass filtering on the second EEG data to obtain second filtered data, and uploading the second filtered data and the second motor imagery task to the cloud server using the FTP protocol, so that the cloud server stores the second filtered data and the second motor imagery task in a cloud database; Using the dynamic time attention mechanism module to perform signal weighting on the second filtered data to obtain first intermediate data; Using the multi-scale spatio-temporal feature extraction module to perform multi-scale spatio-temporal feature extraction on the first intermediate data to obtain second intermediate data; Using the Transformer module to perform high-level feature extraction on the second intermediate data to obtain third intermediate data; Using the output module to predict the imagined motion type of the third intermediate data to obtain a predicted type; Using a loss function to perform cross entropy loss calculation on the prediction type and the second motor imagery task to obtain a loss value; When the loss value is less than or equal to a preset value, the target motor imagery decoding model is obtained; When the loss value is greater than the preset value, the second filtered data and the second motor imagery task are continued to be used for model training until the target motor imagery decoding model is obtained; Wherein, determining a first target operation of a virtual device corresponding to the target user according to the target motor imagery action and determining a second target operation of a rehabilitation device corresponding to the target user according to the target motor imagery action include: Presenting two symmetrical virtual hands at the display position corresponding to the virtual device, and displaying a first prompt symbol in the middle of the two virtual hands; Determine a second prompt symbol according to the target movement imagination action, and after prompting the target user to continue to gaze at the first prompt symbol for a first time, replace the first prompt symbol with the second prompt symbol; Determine a first target hand from the two virtual hands according to the second prompt symbol, and after the second prompt symbol is presented for a second time, cause the first target hand to perform a grasping action according to a third time in the virtual device; The second target hand corresponding to the rehabilitation device is determined according to the target motion imagination action. If the first target hand and the second target hand are consistent, the luminous operation of the second target hand for a fourth time is given visual feedback, and the auxiliary grasping movement of the second target hand for a fifth time is used; if the first target hand and the second target hand are inconsistent, the fourth time and the fifth time are skipped, that is, the second target hand does not perform the luminous operation, and the second target hand does not perform any auxiliary grasping movement.

2. The method according to claim 1, characterized in that The step of using the dynamic time attention mechanism module to perform signal weighting on the second filtered data to obtain first intermediate data includes: Determine the number of channels and the first convolution filling method corresponding to the second filter data by using the dynamic time attention mechanism module, and determine the first convolution kernel according to the number of channels; Using the dynamic time attention mechanism module to perform a convolution operation on the second filter data according to the first convolution filling mode and the first convolution kernel to obtain a first feature map; Reshape the first feature map using the dynamic time attention mechanism module to obtain a second feature map and a third feature map; After performing matrix multiplication of the second feature map and the third feature map, the attention weight matrix is ​​obtained using the Softmax function; Multiplying the attention weight matrix and the second filtered data to obtain a first weighted EEG signal; Determine a scaling factor using the dynamic time attention mechanism module, and use the scaling factor to perform dynamic time attention adjustment on the first weighted EEG signal to obtain a second weighted EEG signal; The dynamic temporal attention mechanism module is used to add the second weighted EEG signal and the first feature map to implement a residual operation and obtain the first intermediate data.

3. The method according to claim 1, characterized in that The step of extracting multi-scale spatio-temporal features from the first intermediate data using the multi-scale spatio-temporal feature extraction module to obtain second intermediate data includes: Determine a plurality of second convolution kernels corresponding to different frequency information and different time information by using the multi-scale spatiotemporal feature extraction module, and determine the number of convolution kernels corresponding to the convolution layer and the second convolution filling method; Performing convolution processing on the first intermediate data according to the plurality of second convolution kernels, the number of the convolution kernels, and the second convolution filling method to obtain a plurality of fourth feature maps; Performing splicing processing on the plurality of fourth feature maps along the depth dimension to obtain a fifth feature map; Performing spatial feature extraction on the fifth feature map using a depthwise separable convolutional layer to obtain first spatial feature information; Using an average pooling layer to perform pooling processing on the first spatial feature information to obtain second spatial feature information; The time feature extraction layer is used to perform time feature extraction and average pooling processing on the second spatial feature information to obtain second intermediate data.

4. The method according to claim 1, characterized in that: The step of extracting high-level features from the second intermediate data using the Transformer module to obtain third intermediate data includes: Obtaining a global dependency feature corresponding to the second filtered data using a temporal convolutional neural network; Performing feature fusion on the global dependency feature and the second intermediate data to obtain fused feature data; Performing layer normalization processing on the fused feature data to obtain a normalized sixth feature map; Capturing key features and dependency relationships of the sixth feature map through a multi-head attention mechanism to obtain a seventh feature map; The seventh feature map and the second intermediate data are added to perform a residual operation to obtain third intermediate data.

5. The method according to claim 1, characterized in that Acquiring second EEG data corresponding to the target user under a second motor imagery task, including: Presenting two symmetrical virtual hands at the display position corresponding to the virtual device, and displaying the first prompt symbol in the middle of the two virtual hands; Determine a third prompt symbol according to the second motor imagery task, and after prompting the target user to continue to gaze at the first prompt symbol for the first time, replace the first prompt symbol with the third prompt symbol; Determine a third target hand from the two virtual hands according to the third prompt symbol, and after the third prompt symbol is presented for the second time, cause the third target hand to perform a grasping action according to the third time in the virtual device; Determine a fourth target hand corresponding to the rehabilitation device according to the second motor imagery task, and provide visual feedback for the light-emitting operation of the fourth target hand for the fourth time, and at the same time, use the auxiliary grasping movement of the fourth target hand for the fifth time; When performing the second motor imagery task, the second EEG data corresponding to the target user is collected using an EEG device.

6. The method according to any one of claims 1 to 5, characterized in that: After performing rehabilitation training on the target user according to the first target operation and the second target operation, the method further includes: Transmitting the first EEG data and the first motor imagery task to the cloud server via the FTP protocol to expand the basic database; Determine the model accuracy corresponding to the target motor imagery decoding model according to the first motor imagery task and the target motor imagery action; When the model accuracy is less than or equal to a preset threshold, data within a preset time is obtained from the basic database to update the model parameters of the target motion imagination decoding model, the updated target motion imagination decoding model is obtained, and the updated target motion imagination decoding model is stored in the cloud server.

7. A rehabilitation training auxiliary system based on brain-computer interface and virtual reality, characterized in that: include: A mode determination module, used to determine a target motion mode corresponding to the target user according to the selection of the target user in the user device; A model acquisition module is used to obtain a target motor imagery decoding model corresponding to the target user from a cloud server when the target motor mode is a rehabilitation training mode; when the target motor mode is a motion calibration mode, or the target motor mode is the rehabilitation training mode and the target user is a new user, obtain the second EEG data corresponding to the target user under a second motor imagery task; The target motor imagination decoding model corresponding to the target user is obtained according to the second EEG data and the second motor imagination task; wherein the target motor imagination decoding model includes: a dynamic time attention mechanism module, a multi-scale spatiotemporal feature extraction module, a Transformer module, and an output module, wherein the target motor imagination decoding model corresponding to the target user is obtained according to the second EEG data and the second motor imagination task, including: performing bandpass filtering on the second EEG data using a preset filter to obtain second filtered data, and uploading the second filtered data and the second motor imagination task to the cloud server using the FTP protocol, so that the cloud server stores the second filtered data and the second motor imagination task in the cloud database; using the dynamic time attention mechanism model The block performs signal weighting on the second filtered data to obtain the first intermediate data; uses the multi-scale spatiotemporal feature extraction module to perform multi-scale spatiotemporal feature extraction on the first intermediate data to obtain the second intermediate data; uses the Transformer module to perform high-level feature extraction on the second intermediate data to obtain the third intermediate data; uses the output module to predict the imagined movement type on the third intermediate data to obtain the predicted type; uses the loss function to perform cross entropy loss calculation on the predicted type and the second movement imagination task to obtain the loss value; when the loss value is less than or equal to the preset value, the target movement imagination decoding model is obtained; when the loss value is greater than the preset value, the second filtered data and the second movement imagination task are continued to be used for model training until the target movement imagination decoding model is obtained; A data acquisition module, used to acquire first EEG data corresponding to the target user under the first motor imagery task, and perform bandpass filtering on the first EEG data using the preset filter to obtain first filtered data; A data decoding module, configured to perform motion imagery decoding on the first filtered data according to the target motion imagery decoding model to obtain a target motion imagery action corresponding to the target user; An operation determination module is used to determine a first target operation of a virtual device corresponding to the target user according to the target movement imagination action and a second target operation of a rehabilitation device corresponding to the target user according to the target movement imagination action; wherein, the first target operation of the virtual device corresponding to the target user according to the target movement imagination action and the second target operation of the rehabilitation device corresponding to the target user according to the target movement imagination action include: presenting two symmetrical virtual hands at a display position corresponding to the virtual device, and displaying a first prompt symbol at a middle position of the two virtual hands; determining a second prompt symbol according to the target movement imagination action, and prompting the target user to continuously stare at the first prompt symbol for a first time, and then displaying the first prompt symbol. The first target hand is replaced with the second prompt symbol; the first target hand is determined from the two virtual hands according to the second prompt symbol, and after the second prompt symbol is presented for a second time, the first target hand is made to perform a grasping action according to a third time in the virtual device; the second target hand corresponding to the rehabilitation device is determined according to the target movement imagination action, and if the first target hand and the second target hand are consistent, the light-emitting operation of the second target hand for a fourth time is given visual feedback, and the auxiliary grasping movement of the second target hand for a fifth time is used; if the first target hand and the second target hand are inconsistent, the fourth time and the fifth time are skipped, that is, the second target hand does not perform the light-emitting operation, and the second target hand does not perform any auxiliary grasping movement; A rehabilitation training module is used to perform rehabilitation training on the target user according to the first target operation and the second target operation.

Images