Brain-computer interaction mirror therapy rehabilitation training system based on augmented reality
By using augmented reality technology and brain-computer interfaces to monitor and provide feedback on the patient's brainwave signals, the problem of unclear participation in traditional mirror therapy is solved, achieving personalized closed-loop rehabilitation training and promoting the recovery of neurological function.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-13
- Publication Date
- 2026-03-13
AI Technical Summary
Traditional mirror therapy lacks brain state monitoring, making it unclear how much patients participate in the training process and difficult to achieve effective closed-loop feedback. Furthermore, traditional rehabilitation methods lack personalized adaptability, resulting in limited training effects.
The rehabilitation training system employs augmented reality-based brain-computer interface mirror therapy, which includes a healthy limb movement acquisition unit, an augmented reality-based mirror presentation unit, an EEG signal acquisition unit, an EEG signal recognition unit, and an audio output unit. It monitors the patient's attention, fatigue, and motor area activation through EEG signals, and provides real-time feedback to promote engagement.
It enabled the active participation of patients, provided immediate closed-loop feedback, enhanced the effectiveness of rehabilitation training, and promoted neural reorganization and functional reconstruction.
Smart Images
Figure CN121647702A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of rehabilitation training technology, and more specifically to a brain-computer interface mirror therapy rehabilitation training system based on augmented reality. Background Technology
[0002] Stroke is a common disease caused by hemorrhagic or ischemic brain injury, characterized by high incidence and high disability rate. Against the backdrop of an aging population, the number of stroke cases continues to rise. Epidemiological studies show that there are 2 million new stroke patients in China every year, one every 12 seconds, making the social demand for related rehabilitation treatment particularly urgent.
[0003] There are many sequelae of stroke, the most common being speech impairment, cognitive impairment, and motor dysfunction, especially hand motor dysfunction, which is slow to recover from, difficult to treat, and has a poor prognosis, making it a challenging problem in clinical rehabilitation. The basic principle of motor function rehabilitation is to stimulate the activation of the brain's motor cortex and the remodeling of neuroplasticity through repetitive, regular sensory input and motor output, helping patients gradually recover their impaired motor functions. However, traditional rehabilitation methods mainly rely on physical therapy and passive training, which often fail to fully stimulate the patient's initiative. The training methods are often monotonous, lacking individualized adaptation, and thus the rehabilitation training effect is limited.
[0004] Mirror therapy (MT), also known as mirror visual feedback (MVF), is a neurorehabilitation technique based on the principle of visual stimulation. Traditional mirror therapy uses the physical properties of a plane mirror to project the movement of the patient's unaffected limbs onto the affected side's visual field in real time, creating the visual illusion of normal movement. This unique treatment utilizes the human brain's visual-motor integration mechanism, inducing neural plasticity through visual illusions to promote motor function recovery. However, because the mirror and the unaffected limb are exposed, patients may peek at their unaffected side, and the presence of the mirror makes them subjectively aware that it is a mirror image, thus reducing the therapeutic effect. To address this, some researchers have constructed a monocular vision-based mirror therapy system. This system uses a camera to capture the movements of the unaffected limbs, processes the image, and presents the mirrored image to the patient, while the patient simultaneously imagines the movements of the affected limb. However, current mirror therapy lacks brain state monitoring, making it unclear how much patient participation is involved during training and hindering effective closed-loop feedback. Summary of the Invention
[0005] The purpose of this invention is to provide a brain-computer interface mirror therapy rehabilitation training system based on augmented reality, including: a healthy limb movement acquisition unit, an augmented reality mirror presentation unit, an electroencephalogram (EEG) signal acquisition unit, an EEG signal recognition unit, and an audio output unit.
[0006] The healthy limb movement acquisition unit is used to acquire images of normal limb movements in the upper or lower limbs of stroke patients.
[0007] The augmented reality-based mirror presentation unit is used to mirror and flip the limb movement images acquired by the healthy limb movement acquisition unit, and present the processed limb movement images to the stroke patient based on augmented reality technology.
[0008] Stroke patients generate corresponding motor imagery when they fixate on processed images of limb movements.
[0009] The EEG signal acquisition unit is used to acquire EEG signals when stroke patients are imagining actions.
[0010] The EEG signal identification unit is used to identify the EEG signals acquired by the EEG signal acquisition unit and obtain the EEG identification result.
[0011] The audio output unit outputs corresponding audio based on the EEG recognition results.
[0012] The augmented reality-based mirror presentation unit controls the start and stop of the system based on EEG recognition results.
[0013] Furthermore, the healthy limb motion acquisition unit uses a camera to acquire RGB images of normal limb movements in real time.
[0014] The normal upper limb movements include clenching a fist, interlocking fingers, interlocking palms, flexing and extending a single finger, and flexing and extending the upper arm.
[0015] Normal lower limb movements include knee flexion and extension, and ankle flexion and extension.
[0016] Furthermore, the augmented reality-based mirror rendering unit includes an image processing module and an augmented reality module.
[0017] The image processing module is used to mirror and flip the limb movement images acquired by the healthy limb movement acquisition unit to obtain the processed limb movement images.
[0018] The augmented reality module uses augmented reality technology to present processed images of limb movements to stroke patients.
[0019] The augmented reality module controls the start and stop of the system based on EEG recognition results.
[0020] The augmented reality module includes augmented reality glasses that can replicate a computer screen.
[0021] Furthermore, the image processing module performs mirror flipping processing on the limb movement image as follows:
[0022] A1 mirrors and flips the limb movement images acquired by the healthy limb movement acquisition unit to obtain flipped limb movement images.
[0023] A2 divides the flipped limb movement image into discrete images at preset time intervals to obtain several limb movement images.
[0024] A3 adds a background color to the body movement images so that the background colors of two body movement images in adjacent time periods are different, and plays the body movement images at a fixed frame rate to obtain the processed body movement video.
[0025] Furthermore, the image processing module performs mirror flipping processing on the limb movement image as follows:
[0026] B1 mirrors and flips the limb movement images acquired by the healthy limb movement acquisition unit to obtain flipped limb movement images.
[0027] B2 splits the flipped limb movement image into discrete images at preset time intervals, resulting in several limb movement images.
[0028] B3 adds a semi-transparent square to the limb movement part in each even-indexed limb movement image and plays the limb movement image at a fixed frame rate to obtain the processed limb movement video.
[0029] Furthermore, the sampling frequency of the EEG signal acquisition unit is greater than 250Hz.
[0030] Furthermore, the areas acquired by the EEG signal acquisition unit include the prefrontal cortex, visual cortex, and motor cortex.
[0031] Furthermore, the steps by which the EEG signal identification unit identifies the EEG signals acquired by the EEG signal acquisition unit are as follows:
[0032] C1 extracts EEG signals from the prefrontal cortex, visual cortex, and motor cortex using a sliding window, and then performs bandpass filtering on the extracted EEG signals to obtain filtered EEG signals.
[0033] C2 calculates the power ratio of the β band to the α band based on the filtered EEG signal from the prefrontal cortex to obtain attention-related state assessment features. It then determines whether these features exceed a preset attention threshold; if so, it sets the concentration score accordingly. If not, then the focus score will be adjusted. .
[0034] C3 calculates the relative power change in the α1 band of the filtered EEG signal from the prefrontal cortex and determines whether the relative power change exceeds a preset threshold. If so, the fatigue score is set accordingly. If not, then set the fatigue score. .
[0035] C4 calculates the µ-β rhythmic inhibition index on the filtered EEG signal from the motor cortex and determines whether the inhibition index is greater than a preset threshold. If so, it sets the activation score of the motor cortex. If not, then set the score of the activated motor area. .
[0036] C5 evaluates the intensity of feature-evoked SSVEP on the filtered EEG signal from the visual cortex and determines whether the intensity of feature-evoked SSVEP exceeds a preset intensity threshold. If so, the intensity of the SSVEP in the visual cortex is set accordingly. If not, then set the SSVEP score of the evoked visual area. .
[0037] The EEG identification results obtained by C6 calculation are shown below:
[0038] (1)
[0039] In the formula, This indicates the results of the electroencephalogram (EEG) identification. This indicates the score for concentration. This indicates the fatigue score. This indicates the score for the activated motor area. This indicates the SSVEP score in the evoked visual area.
[0040] Furthermore, the β band is 13Hz-30Hz.
[0041] The α band is 8Hz-13Hz.
[0042] The α1 frequency band is 8Hz-10Hz.
[0043] The µ-β rhythm is 8Hz-26Hz.
[0044] The µ-β rhythmic inhibition index is shown below:
[0045] (2)
[0046] In the formula, This represents the µ-β rhythmic inhibition index. This represents the energy within the µ-β rhythm. This represents the energy within the µ and β rhythms prior to limb movement.
[0047] The methods for evaluating the intensity of SSVEP induced by features include canonical correlation analysis and frequency domain feature extraction.
[0048] The formula for calculating the intensity of feature-induced SSVEP using canonical correlation analysis is as follows:
[0049] (3)
[0050] In the formula, x is the filtered EEG signal, and y is a template function composed of sine and cosine functions with the image acquisition frequency as the frequency. , These are the linear projection vectors of the EEG signal x and the template function y, respectively. This represents the correlation coefficient. E represents the expectation operator. The superscript T indicates transpose.
[0051] The formula for calculating the intensity of feature-induced SSVEP using frequency domain feature extraction is as follows:
[0052] (4)
[0053] (5)
[0054] In the formula, Indicates the frequency component index. The first frequency domain signal represents the second frequency domain signal. Each frequency component. Indicates the first Frequency components The amplitude value. Indicates the sampling point index. This indicates the total number of sampling points. This represents the nth sampling point. This represents the complex exponential basis function.
[0055] Furthermore, the audio output unit includes speakers and headphones.
[0056] The audio output unit outputs audio messages including "I'm tired," "Training is over," and "Please concentrate."
[0057] When EEG identification results When the audio output unit is fatigued and training ends, the augmented reality-based mirror rendering unit stops the system from running.
[0058] When EEG identification results At this time, the audio output unit outputs "Please concentrate," and the augmented reality-based mirror presentation unit pauses the presentation of body movement images.
[0059] When EEG identification results At that time, the augmented reality-based mirroring unit continues to display images of body movements, while the audio output unit stops outputting audio.
[0060] The technical effects of this invention are undeniable. The system provided by this invention provides subjects with real-time mirrored limb movement videos through an augmented reality-based mirror presentation unit. When subjects watch the video, SSVEP can be induced. The system collects the subjects' EEG signals through an EEG signal acquisition unit and monitors the activation status of the subjects' prefrontal cortex, motor cortex, and visual cortex in real time based on the EEG signal recognition unit. When insufficient participation of the subjects is detected, voice prompts and temporary videos are used to encourage subjects to increase their participation during training, thereby establishing a closed-loop system and promoting neural reorganization and functional reconstruction.
[0061] The brain-computer interface (BCI) technology used in this invention breaks through the limitations of traditional damaged neuromuscular pathways, can more effectively mobilize the patient's active participation, provide immediate and closed-loop reinforcement feedback, and provide the possibility of active training for patients with severe functional impairment, showing good rehabilitation prospects.
[0062] In addition, augmented reality (AR) technology enhances users' perception of the real world by overlaying digital elements onto the real environment, providing a new visual presentation method for mirror therapy. Attached Figure Description
[0063] Figure 1 A schematic diagram of an augmented reality-based brain-computer interface mirror therapy rehabilitation training system;
[0064] Figure 2 This is a schematic diagram of the image processing module in an augmented reality-based mirror rendering unit, illustrating the processing method 1.
[0065] Figure 3 This is a schematic diagram of the image processing module in the augmented reality-based mirror rendering unit, illustrating processing method 2. Detailed Implementation
[0066] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.
[0067] Example 1:
[0068] See Figures 1 to 3 The augmented reality-based brain-computer interface mirror therapy rehabilitation training system includes: a healthy limb movement acquisition unit, an augmented reality-based mirror presentation unit, an EEG signal acquisition unit, an EEG signal recognition unit, and an audio output unit.
[0069] The healthy limb movement acquisition unit is used to acquire images of normal limb movements in the upper or lower limbs of stroke patients.
[0070] The augmented reality-based mirror presentation unit is used to mirror and flip the limb movement images acquired by the healthy limb movement acquisition unit, and present the processed limb movement images to the stroke patient based on augmented reality technology.
[0071] Stroke patients generate corresponding motor imagery when they fixate on processed images of limb movements.
[0072] The EEG signal acquisition unit is used to acquire EEG signals when stroke patients are imagining actions.
[0073] The EEG signal identification unit is used to identify the EEG signals acquired by the EEG signal acquisition unit and obtain the EEG identification result.
[0074] The audio output unit outputs corresponding audio based on the EEG recognition results.
[0075] The augmented reality-based mirror presentation unit controls the start and stop of the system based on EEG recognition results.
[0076] Example 2:
[0077] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the main technical contents of which are described in Example 1, further wherein the healthy limb movement acquisition unit uses a camera to acquire RGB images of normal limb movements in real time.
[0078] The normal upper limb movements include clenching a fist, interlocking fingers, interlocking palms, flexing and extending a single finger, and flexing and extending the upper arm.
[0079] Normal lower limb movements include knee flexion and extension, and ankle flexion and extension.
[0080] Example 3:
[0081] The main technical contents of the augmented reality-based brain-computer interface mirror therapy rehabilitation training system are described in any one of Embodiments 1 and 2. Furthermore, the augmented reality-based mirror presentation unit includes an image processing module and an augmented reality module.
[0082] The image processing module is used to mirror and flip the limb movement images acquired by the healthy limb movement acquisition unit to obtain the processed limb movement images.
[0083] The augmented reality module uses augmented reality technology to present processed images of limb movements to stroke patients.
[0084] The augmented reality module controls the start and stop of the system based on EEG recognition results.
[0085] The augmented reality module includes augmented reality glasses that can replicate a computer screen.
[0086] Example 4:
[0087] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, whose main technical contents are described in any one of Examples 1 to 3, further includes the following steps for the image processing module to perform mirror flipping processing on the limb movement images:
[0088] A1 mirrors and flips the limb movement images acquired by the healthy limb movement acquisition unit to obtain flipped limb movement images.
[0089] A2 divides the flipped limb movement image into discrete images at preset time intervals to obtain several limb movement images.
[0090] A3 adds a background color to the body movement images so that the background colors of two body movement images in adjacent time periods are different, and plays the body movement images at a fixed frame rate to obtain the processed body movement video.
[0091] Example 5:
[0092] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, whose main technical contents are described in any one of embodiments 1 to 4, further includes the following steps for the image processing module to perform mirror flipping processing on the limb movement images:
[0093] B1 mirrors and flips the limb movement images acquired by the healthy limb movement acquisition unit to obtain flipped limb movement images.
[0094] B2 splits the flipped limb movement image into discrete images at preset time intervals, resulting in several limb movement images.
[0095] B3 adds a semi-transparent square to the limb movement part in each even-indexed limb movement image and plays the limb movement image at a fixed frame rate to obtain the processed limb movement video.
[0096] Example 6:
[0097] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the main technical contents of which are described in any one of Examples 1 to 5, further wherein the sampling frequency of the EEG signal acquisition unit is greater than 250Hz.
[0098] Example 7:
[0099] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the main technical contents of which are described in any one of Examples 1 to 6, further wherein the areas acquired by the EEG signal acquisition unit include the prefrontal cortex, visual cortex, and motor cortex.
[0100] Example 8:
[0101] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, whose main technical contents are described in any one of embodiments 1 to 7, further includes the following steps for the EEG signal identification unit to identify the EEG signals acquired by the EEG signal acquisition unit:
[0102] C1 extracts EEG signals from the prefrontal cortex, visual cortex, and motor cortex using a sliding window, and then performs bandpass filtering on the extracted EEG signals to obtain filtered EEG signals.
[0103] C2 calculates the power ratio of the β band to the α band based on the filtered EEG signal from the prefrontal cortex to obtain attention-related state assessment features. It then determines whether these features exceed a preset attention threshold; if so, it sets the concentration score accordingly. If not, then the focus score will be adjusted. .
[0104] The preset attention threshold ranges from 0.3 to 0.5.
[0105] C3 calculates the relative power change in the α1 band of the filtered EEG signal from the prefrontal cortex and determines whether the relative power change exceeds a preset threshold. If so, the fatigue score is set accordingly. If not, then set the fatigue score. The preset change threshold ranges from 0.01 to 0.2.
[0106] C4 calculates the µ-β rhythmic inhibition index on the filtered EEG signal from the motor cortex and determines whether the inhibition index is greater than a preset threshold. If so, it sets the activation score of the motor cortex. If not, then set the score of the activated motor area. The preset index threshold value ranges from -1 to -1.5.
[0107] C5 evaluates the intensity of feature-evoked SSVEP on the filtered EEG signal from the visual cortex and determines whether the intensity of feature-evoked SSVEP exceeds a preset intensity threshold. If so, the intensity of the SSVEP in the visual cortex is set accordingly. If not, then set the SSVEP score of the evoked visual area. .
[0108] The preset intensity thresholds include the correlation coefficient intensity threshold and the spectral amplitude intensity threshold.
[0109] The correlation coefficient strength threshold ranges from 0.4 to 0.6.
[0110] The threshold value for the spectral amplitude intensity ranges from 1μV to 2μV.
[0111] The EEG identification results obtained by C6 calculation are shown below:
[0112] (1)
[0113] In the formula, This indicates the results of the electroencephalogram (EEG) identification. This indicates the score for concentration. This indicates the fatigue score. This indicates the score for the activated motor area. This indicates the SSVEP score in the evoked visual area.
[0114] Example 9:
[0115] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the main technical contents of which are described in any one of Examples 1 to 8, further wherein the β band is 13Hz-30Hz.
[0116] The α band is 8Hz-13Hz.
[0117] The α1 frequency band is 8Hz-10Hz.
[0118] The µ-β rhythm is 8Hz-26Hz.
[0119] The µ-β rhythmic inhibition index is shown below:
[0120] (2)
[0121] In the formula, This represents the µ-β rhythmic inhibition index. This represents the energy within the µ-β rhythm. This represents the energy within the µ and β rhythms prior to limb movement.
[0122] The methods for evaluating the intensity of SSVEP induced by features include canonical correlation analysis and frequency domain feature extraction.
[0123] The formula for calculating the intensity of feature-induced SSVEP using canonical correlation analysis is as follows:
[0124] (3)
[0125] In the formula, x is the filtered EEG signal, and y is a template function composed of sine and cosine functions with the image acquisition frequency as the frequency. , These are the linear projection vectors of the EEG signal x and the template function y, respectively. This represents the correlation coefficient. E represents the expectation operator. The superscript T indicates transpose.
[0126] The formula for calculating the intensity of feature-induced SSVEP using frequency domain feature extraction is as follows:
[0127] (4)
[0128] (5)
[0129] In the formula, Indicates the frequency component index. The first frequency domain signal represents the second frequency domain signal. Each frequency component. Indicates the first Frequency components The amplitude value. Indicates the sampling point index. This indicates the total number of sampling points. This represents the nth sampling point. This represents the complex exponential basis function.
[0130] Example 10:
[0131] The augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the main technical contents of which are described in any one of embodiments 1 to 9, and further, the audio output unit includes a speaker and headphones.
[0132] The audio output unit outputs audio messages including "I'm tired," "Training is over," and "Please concentrate."
[0133] When EEG identification results When the audio output unit is fatigued and training ends, the augmented reality-based mirror rendering unit stops the system from running.
[0134] When EEG identification results At this time, the audio output unit outputs "Please concentrate," and the augmented reality-based mirror presentation unit pauses the presentation of body movement images.
[0135] When EEG identification results At that time, the augmented reality-based mirroring unit continues to display images of body movements, while the audio output unit stops outputting audio.
[0136] Example 11:
[0137] See Figures 1 to 3 The augmented reality-based brain-computer interface mirror therapy rehabilitation training system mainly includes: a healthy limb movement acquisition unit, an augmented reality-based mirror presentation unit, an electroencephalogram (EEG) signal acquisition unit, an EEG signal recognition unit, and an audio output unit.
[0138] The signal output terminal of the healthy limb movement acquisition unit is connected to the signal input terminal of the augmented reality-based mirror presentation unit; the signal output terminal of the EEG signal acquisition unit is connected to the signal input terminal of the EEG signal recognition unit; and the signal output terminal of the EEG signal recognition unit is connected to the signal input terminal of the augmented reality-based mirror presentation unit and the input terminal of the audio output unit.
[0139] The healthy limb movement acquisition unit includes: an image acquisition module, used to acquire images reflecting normal limb movements of the subject's upper or lower limbs, and output them to the augmented reality-based mirror presentation unit;
[0140] The augmented reality-based mirror presentation unit includes an image processing module and an augmented reality module. The image processing module is used to mirror and flip the acquired RGB image of limb movement and process the image so that steady-state visual evoked potentials (SSVEPs) can be induced when the subject gazes at it. At the same time, the module receives signals from the EEG signal recognition unit in real time. The augmented reality module is used to present the movement video processed by the image processing module.
[0141] The EEG signal acquisition unit includes: an EEG signal acquisition module, used to acquire the subject's EEG signals and send them to the EEG signal recognition unit;
[0142] The EEG signal identification unit includes: an EEG signal identification module, used to analyze the collected EEG signals of the subject, analyze the subject's attention concentration, fatigue level, intensity of brain feature induction in the visual area and intensity of activation in the motor area, and send the identification results to the augmented reality-based mirror presentation unit.
[0143] The audio output unit includes: an audio output module for outputting audio.
[0144] In the aforementioned augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the image acquisition module uses a camera to capture RGB images of normal limb movements in real time.
[0145] In the aforementioned augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the image processing module mirrors and flips the acquired RGB images of limb movements and processes the images. Processing method one: adding a color or grayscale change to the background color of the image so that the background colors or grayscale of two adjacent images are different, and playing them at a fixed frame rate; Processing method two: adding a semi-transparent square to the limb movement part in the image every other image, and playing it at a fixed frame rate.
[0146] In the aforementioned augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the augmented reality module is used to present the images output by the image processing module, and can be augmented reality glasses with the function of replicating a computer screen.
[0147] In the aforementioned augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the sampling frequency of the EEG signal acquisition module exceeds 250Hz.
[0148] In the aforementioned augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the EEG signal identification is configured as follows:
[0149] Regions of interest are pre-defined based on differences in brain regions, including the prefrontal cortex, visual cortex, and motor cortex;
[0150] The system receives the EEG signals, extracts them using a sliding window, performs bandpass filtering, calculates the power ratio of the β band (13–30 Hz) to the α band (8–13 Hz) for the prefrontal cortex EEG signals to obtain attention-related state assessment features, calculates the relative power change of α1 (8–10 Hz) to reflect fatigue level, and compares it with a preset threshold to determine whether attention is being focused (yes: a=1, no: a=0) or fatigue is present (yes: b=-1, no: b=0). For the motor cortex EEG signals, the µ-β rhythmic inhibition index is calculated as the activation intensity of the motor cortex and compared with a preset threshold to determine whether motor cortex activation is present (yes: c=1, no: c=0). For the visual cortex EEG signals, canonical correlation analysis and frequency domain feature extraction are used to assess the intensity of feature-evoked SSVEP (Special State Verification Epidemic), and the result is compared with a preset threshold to determine whether visual cortex SSVEP is evoked (yes: d=1, no: d=0).
[0151] The result e of a+b+c+d is calculated to judge the participant's participation level. If e=-1, "Fatigue, end training" is output to the voice output module and the system stops running. If e=0 or 1, the participation level is determined to be insufficient. "Please concentrate" is output to the voice output module and a control command is output to the image processing module to pause image presentation until e≠0 and 1, then the image playback continues and the voice output stops.
[0152] In the aforementioned augmented reality-based brain-computer interface mirror therapy rehabilitation training system, the audio output module is used to play the statements to be played output by the EEG signal recognition module.
[0153] Example 12:
[0154] See Figures 1 to 3 The main technical components of the augmented reality-based brain-computer interface mirror therapy rehabilitation training system include:
[0155] Reference Figure 1 The augmented reality-based brain-computer interface mirror therapy rehabilitation training system includes a healthy limb movement acquisition unit, an electroencephalogram (EEG) signal acquisition unit, an output connection between the healthy limb movement acquisition unit and the input connection between the augmented reality-based mirror presentation unit, an output connection between the EEG signal acquisition unit and the input connection between the EEG signal recognition unit, an output connection between the EEG signal recognition unit and the audio output unit, and an input connection between the augmented reality-based mirror presentation unit.
[0156] This embodiment demonstrates hand motor function rehabilitation training, but the present invention is not limited to hand motor function rehabilitation training.
[0157] The healthy limb motion acquisition unit is an RGB camera, placed directly above the healthy hand, with the healthy hand placed on a black background table, and the image acquisition frequency F≤30fps;
[0158] The processing procedure of the image processing module in the augmented reality-based mirror rendering unit is as follows:
[0159] The acquired RGB images of limb movements can be mirrored and flipped, for example, using the fliplr function in MATLAB;
[0160] Process the mirrored image
[0161] Processing Method 1: Modify the background color. Based on the background color of the acquired image, determine the RGB range, or convert RGB to HSV. Extract the background region based on the threshold method and generate a background mask (pixels within the range are 1, otherwise 0). For complex background colors, image segmentation algorithms such as GrabCut or deep learning segmentation algorithms can be used to extract the background region and generate a background mask, and edge smoothing can be performed. Replace the 1 in the three channels of the background mask with the background color to be modified, such as gray in this implementation case (RGB=[159,160,161]).
[0162] For the images input by the healthy limb motion acquisition unit, the background color is modified every other image, such as... Figure 2 As shown, these images were presented using the Psychophysics Toolbox, with precise control over the presentation duration of each image. Each image lasted for 1 / F seconds (F is the image acquisition frequency, F≤30, and can be divided by the screen refresh rate), forming motion videos, thereby effectively inducing SSVEP.
[0163] Method 2: Add a semi-transparent square to the body movement area in one image, such as... Figure 3As shown, a red (RGB=[255,0,0]) semi-transparent square is added to the hand in the image. The process is as follows: Create a layer of the same size as the input image (inputImage_raw), initially set to all zeros; draw a square on the layer, calculate the alpha_mask of the square area, ensuring it is within the image range, fill the layer with color, and create an opacity mask: alpha=0.5 within the square area and 0 in other areas. Then, use the following formula to achieve the blended image (OutputImage) with the original image:
[0164] OutputImage=uint8(double(inputImage_raw) .* (1 - alpha_mask) + double(layer) .* alpha_mask)
[0165] Adding a semi-transparent square can also be achieved using the insertShape function in the Image Processing Toolbox of MATLAB;
[0166] These images are then presented using the Psychophysics Toolbox, with precise control over the presentation duration of each image. Each image lasts for 1 / F seconds (F is the image acquisition frequency, F≤30, and can be divided by the screen refresh rate), forming motion videos to effectively induce SSVEP.
[0167] In addition, this module continuously monitors the TCP / IP or serial communication channels for control commands to be sent, and controls the start and stop of the motion video.
[0168] The augmented reality module in the augmented reality-based mirror presentation unit projects the generated dynamic images onto the augmented reality glasses. The subject wears the augmented reality glasses, observes the limb movements in their field of vision, and imagines the corresponding movements.
[0169] The EEG signal acquisition unit, constructed according to the international standard 10 / 20 system, uses the forehead Fpz as the ground pole and the left earlobe as a reference to acquire 32 channels of EEG signals distributed on the scalp surface of the brain, including the frontal lobe, parietal lobe, occipital lobe, and temporal lobe, at a sampling frequency of 250Hz and above. These channels are Fp1, Fp2, AFz, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC4, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P3, Pz, P4, PO7, PO3, POz, PO4, PO8, O1, Oz, and O2.
[0170] The processing procedure of the EEG signal recognition unit is as follows:
[0171] The channels Fp1, Fp2, AFz, F3, Fz, and F4 are designated as the forehead area; the channels FC5, FC1, FC2, FC4, C3, Cz, C4, CP5, CP1, CP2, and CP6 are designated as the motion area; and the channels PO7, PO3, POz, PO4, PO8, O1, Oz, and O2 are designated as the visual area.
[0172] EEG signals were captured using a sliding window with a window length of 3 seconds and a sliding window interval of 0.3 seconds; the EEG signals captured by the sliding window were then bandpass filtered (3Hz to 40Hz).
[0173] The EEG signals from the three channels Fp1, Fp2, and AFz in the prefrontal cortex were processed. The Welch method was used to calculate the power spectral density of four frequency bands: 8Hz to 10Hz, 8Hz to 13Hz, 13Hz to 30Hz, and 8Hz to 30Hz, respectively, and denoted as α1, α, β, and α_β. β / α and α1 / α_β were also calculated to reflect the level of attention and fatigue, respectively. Thresholds were set based on experience and training data to determine whether attention was focused (yes: a=1, no: a=0) or fatigued (yes: b=-1, no: a=0).
[0174] For the bandpass-filtered EEG signals from the motor cortex C3, C4, and Cz, surface Laplace filtering is applied, i.e., C3 = C3 - 0.25 × (FC5 + FC1 + CP5 + CP1), C4 = C4 - 0.25 × (FC6 + FC2 + CP6 + CP2), Cz = Cz - 0.25 × (FC1 + FC2 + CP1 + CP2); then the µ-β intrarhythmic inhibition index is applied. Where P is the energy within the µ-β rhythm (8Hz to 26Hz), R is the energy within the µ and β rhythms before limb movement, and ERDindex is the inhibition index; the ERDindex of the C3, C4, or Cz channels (left upper limb movement: C4; right upper limb movement: C3; leg and foot movement: Cz) EEG signals are selected as the state characteristics of motor area activation intensity according to the limb movement; the judgment result is either yes or no activation of the motor area (yes: c=1, no: c=0).
[0175] The EEG signals from the visual field channels PO7, PO3, POz, PO4, PO8, O1, Oz, and O2 are processed. The correlation coefficients with the sine and cosine template functions based on the image acquisition frequency F and its harmonics are calculated and compared with a preset threshold. Alternatively, the spectra of each channel PO7, PO3, POz, PO4, PO8, O1, Oz, and O2 are calculated, the amplitude at the image acquisition frequency F is statistically analyzed, and the average is calculated and compared with a preset threshold. The result is then determined as whether or not SSVEP in the visual field is induced (yes: d=1, no: d=0).
[0176] The result e of a+b+c+d is used to judge the participant's participation level. If e=-1, "Fatigue, end training" is output to the voice output module and the system stops running. If e=0 or 1, the participation level is considered insufficient. "Please concentrate" is output to the voice output module, and a control command is output to the image processing module via TCP / IP communication or serial communication to pause image presentation until e≠0 and 1, at which point image playback continues and voice output stops.
[0177] The audio output unit can be an audio output device such as a speaker or headphones, used to play the sentences to be played output by the EEG signal recognition module.
[0178] In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0179] Example 13:
[0180] See Figures 1 to 3 The augmented reality-based brain-computer interface mirror therapy rehabilitation training system mainly includes: a healthy limb movement acquisition unit, an augmented reality-based mirror presentation unit, an electroencephalogram (EEG) signal acquisition unit, an EEG signal recognition unit, and an audio output unit.
[0181] The signal output terminal of the healthy limb movement acquisition unit is connected to the signal input terminal of the augmented reality-based mirror presentation unit; the signal output terminal of the EEG signal acquisition unit is connected to the signal input terminal of the EEG signal recognition unit; and the signal output terminal of the EEG signal recognition unit is connected to the signal input terminal of the augmented reality-based mirror presentation unit and the input terminal of the audio output unit.
[0182] The healthy limb movement acquisition unit includes: an image acquisition module, used to acquire images reflecting normal limb movements of the subject's upper or lower limbs, and output them to the augmented reality-based mirror presentation unit;
[0183] The augmented reality-based mirror presentation unit includes an image processing module and an augmented reality module. The image processing module is used to mirror and flip the acquired RGB image of limb movement and process the image so that steady-state visual evoked potentials (SSVEPs) can be induced when the subject gazes at it. At the same time, the module receives signals from the EEG signal recognition unit in real time. The augmented reality module is used to present the movement video processed by the image processing module.
[0184] The EEG signal acquisition unit includes: an EEG signal acquisition module, used to acquire the subject's EEG signals and send them to the EEG signal recognition unit;
[0185] The EEG signal identification unit includes: an EEG signal identification module, used to analyze the collected EEG signals of the subject, analyze the subject's attention concentration, fatigue level, intensity of brain feature induction in the visual area and intensity of activation in the motor area, and send the identification results to the augmented reality-based mirror presentation unit.
[0186] The audio output unit includes: an audio output module for outputting audio.
[0187] The augmented reality-based mirror rendering unit can induce steady-state visual evoked potentials in the visual field and is configured as follows:
[0188] After mirroring and flipping the RGB image of the limb movement input by the image acquisition module, the image is processed. The processing method can be to add a color or grayscale change to the background color of the image so that the background colors or grayscale of two adjacent images are different, or to add a semi-transparent square to the limb movement part of the image every other image, and play it at a fixed frame rate of ≤30Hz.
[0189] The EEG signal recognition unit is configured as follows:
[0190] Regions of interest are pre-defined based on differences in brain regions, including the prefrontal cortex, motor cortex, and visual cortex;
[0191] Attention-related state assessment features were obtained by calculating the power ratio of the β band (13–30 Hz) to the α band (8–13 Hz) based on the EEG signals of the prefrontal cortex, and fatigue was reflected by the relative power change of α1 (8–10 Hz). The µ-β intrarhythmic inhibition index was calculated as the activation intensity of the motor cortex based on the EEG signals of the motor cortex. The intensity of feature-evoked SSVEP was evaluated by using correlation analysis and frequency domain feature extraction on the EEG signals of the visual cortex. The participation of the subjects was evaluated based on the comprehensive judgment results of the three brain regions, and control commands were output to the image processing module to start and stop the presentation of motion images and the speech output module to output speech based on the participation status.
Claims
1. A brain-computer interface-based mirror therapy rehabilitation training system based on augmented reality, characterized in that: include: The unit includes a healthy limb movement acquisition unit, an augmented reality-based mirror presentation unit, an EEG signal acquisition unit, an EEG signal recognition unit, and an audio output unit. The healthy limb movement acquisition unit is used to acquire images of normal limb movements of the upper or lower limbs of stroke patients. The augmented reality-based mirror presentation unit is used to mirror and flip the limb movement images captured by the healthy limb movement acquisition unit, and present the processed limb movement images to the stroke patient based on augmented reality technology. Stroke patients generate corresponding motor imagery when they fixate on processed images of limb movements. The EEG signal acquisition unit is used to acquire EEG signals when stroke patients are imagining actions. The EEG signal identification unit is used to identify the EEG signals acquired by the EEG signal acquisition unit and obtain the EEG identification result. The audio output unit outputs the corresponding audio based on the EEG recognition results; The augmented reality-based mirror presentation unit controls the start and stop of the system based on EEG recognition results.
2. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 1, characterized in that, The healthy limb movement acquisition unit uses a camera to capture RGB images of normal limb movements in real time; The normal upper limb movements include clenching a fist, interlocking fingers, interlocking palms, flexing and extending a single finger, and flexing and extending the upper arm; Normal lower limb movements include knee flexion and extension, and ankle flexion and extension.
3. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 1, characterized in that, The augmented reality-based mirror rendering unit includes an image processing module and an augmented reality module; The image processing module is used to mirror and flip the limb movement images acquired by the healthy limb movement acquisition unit to obtain the processed limb movement images. The augmented reality module uses augmented reality technology to present processed images of limb movements to stroke patients. The augmented reality module controls the system's start and stop based on EEG recognition results; The augmented reality module includes augmented reality glasses that can replicate a computer screen.
4. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 3, characterized in that, The image processing module performs mirror flipping on the limb movement image as follows: A1 mirrors and flips the limb motion images acquired by the healthy limb motion acquisition unit to obtain flipped limb motion images. A2 divides the flipped limb movement image into discrete images at preset time intervals to obtain several limb movement images. A3 adds a background color to the body movement images so that the background colors of two body movement images in adjacent time periods are different, and plays the body movement images at a fixed frame rate to obtain the processed body movement video.
5. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 3, characterized in that, The image processing module performs mirror flipping on the limb movement image as follows: B1 mirrors and flips the limb motion images acquired by the healthy limb motion acquisition unit to obtain flipped limb motion images. B2 breaks down the flipped limb movement image into discrete images at preset time intervals to obtain several limb movement images. B3 adds a semi-transparent square to the limb movement part in each even-indexed limb movement image and plays the limb movement image at a fixed frame rate to obtain the processed limb movement video.
6. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 1, characterized in that, The sampling frequency of the EEG signal acquisition unit is greater than 250Hz.
7. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 1, characterized in that, The areas acquired by the EEG signal acquisition unit include the prefrontal cortex, visual cortex, and motor cortex.
8. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 7, characterized in that, The steps by which the EEG signal identification unit identifies the EEG signals acquired by the EEG signal acquisition unit are as follows: C1 extracts EEG signals from the prefrontal cortex, visual cortex, and motor cortex using a sliding window, and then performs bandpass filtering on the extracted EEG signals to obtain filtered EEG signals. C2 calculates the power ratio of the β band to the α band based on the filtered EEG signal from the prefrontal cortex to obtain attention-related state assessment features. It then determines whether these features exceed a preset attention threshold; if so, it sets the concentration score accordingly. If not, then the focus score will be adjusted. ; C3 calculates the relative power change in the α1 band of the filtered EEG signal from the prefrontal cortex and determines whether the relative power change exceeds a preset threshold. If so, the fatigue score is set accordingly. If not, then set the fatigue score. ; C4 calculates the µ-β rhythmic inhibition index on the filtered EEG signal from the motor cortex and determines whether the inhibition index is greater than a preset threshold. If so, it sets the activation score of the motor cortex. If not, then set the score of the activated motor area. ; C5 evaluates the intensity of feature-evoked SSVEP on the filtered EEG signal from the visual cortex and determines whether the intensity of feature-evoked SSVEP exceeds a preset intensity threshold. If so, the intensity of the SSVEP in the visual cortex is set accordingly. If not, then set the SSVEP score of the evoked visual area. ; The EEG identification results obtained by C6 calculation are shown below: （1） In the formula, This indicates the results of the electroencephalogram (EEG) identification. This indicates the score for concentration. Indicates fatigue score; This indicates the score for activating the motor area; This indicates the SSVEP score in the evoked visual area.
9. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 8, characterized in that, The β band is 13Hz-30Hz; The α frequency band is 8Hz-13Hz; The α1 frequency band is 8Hz-10Hz; The µ-β rhythm is 8Hz-26Hz; The µ-β rhythmic inhibition index is shown below: （2） In the formula, Indicates the µ-β rhythmic inhibition index; This represents the energy within the µ-β rhythm; This represents the energy within the µ-rhythm and β-rhythm prior to limb movement; The methods for evaluating the intensity of SSVEP induced by features include canonical correlation analysis and frequency domain feature extraction. The formula for calculating the intensity of feature-induced SSVEP using canonical correlation analysis is as follows: （3） In the formula, x is the filtered EEG signal, and y is a template function composed of sine and cosine functions with the image acquisition frequency as the frequency. , These are the linear projection vectors of the EEG signal x and the template function y, respectively. Represents the correlation coefficient; E represents the expectation operator; the superscript T indicates transpose; The formula for calculating the intensity of feature-induced SSVEP using frequency domain feature extraction is as follows: （4） （5） In the formula, Indicates the frequency component index. The first frequency domain signal represents the second frequency domain signal. One frequency component; Indicates the first Frequency components The amplitude value; Indicates the sampling point index; Indicates the total number of sampling points; This represents the nth sampling point; This represents the complex exponential basis function.
10. The augmented reality-based brain-computer interface mirror therapy rehabilitation training system according to claim 8, characterized in that, The audio output unit includes speakers and headphones; The audio output unit outputs audio messages including "I'm tired," "Training is over," and "Please concentrate." When EEG identification results When the audio output unit outputs "fatigue" and training ends, the augmented reality-based mirror rendering unit stops the system from running. When EEG identification results When the audio output unit outputs "Please concentrate," the augmented reality-based mirror presentation unit pauses the presentation of the body movement image. When EEG identification results At that time, the augmented reality-based mirroring unit continues to display images of body movements, while the audio output unit stops outputting audio.