Post-stroke cognitive impairment rehabilitation training system and method based on mixed reality technology

Abstract

This invention relates to the fields of medical rehabilitation and digital healthcare, specifically to a rehabilitation training system and method for post-stroke cognitive impairment based on mixed reality technology. The system includes: a mixed reality interactive terminal for constructing a virtual-real integrated training scenario and capturing user interactions; a cognitive assessment module; a personalized training plan generation module; an immersive mixed reality training task library; and an adaptive adjustment and feedback engine. The method includes: immersive cognitive assessment, generation of personalized training plans, execution of mixed reality training tasks, real-time data analysis and dynamic adjustment of difficulty, provision of immediate feedback, and generation of training reports. The beneficial effects are: this invention achieves integrated assessment and training, improves the ecological validity, engagement, and personalization of training, and effectively improves the efficiency and effectiveness of rehabilitation training through a closed-loop adaptive mechanism.

Description

Technical Field

[0001] This invention relates to the fields of medical rehabilitation and digital healthcare, specifically to an intelligent system and method that combines mixed reality (MR) technology for cognitive assessment and rehabilitation training of patients with post-stroke cognitive impairment (PSCI). Background Technology

[0002] Post-stroke cognitive impairment (PSCI) is a common and serious complication of stroke, severely impacting patients' quality of life and social function. Traditional PSCI rehabilitation training largely relies on static assessments using neuropsychological scales (such as MMSE and MoCA), followed by cognitive training through paper-and-pencil tasks, computer-based two-dimensional programs, or manual guidance. This approach has the following significant drawbacks:

[0003] 1. Disconnect between assessment and training: Assessment is usually conducted before training or at long intervals, which cannot reflect changes in cognitive state during training in real time and makes it difficult to dynamically adjust training parameters.

[0004] 2. Monotonous and boring training methods: Traditional training tasks lack immersion and fun, resulting in low patient participation and poor compliance, which affects the rehabilitation effect.

[0005] 3. Lack of ecological validity: The training scenarios are disconnected from daily life situations, making it difficult to effectively transfer training results to real-life skills.

[0006] 4. Insufficient personalization and adaptability: Training programs are mostly based on static settings of the initial assessment, making it difficult to adaptively adjust the difficulty according to the patient's real-time performance.

[0007] In recent years, virtual reality (VR) technology has begun to be introduced into cognitive rehabilitation. However, its completely closed virtual environment severs the connection between patients and the real world, making it difficult to generalize training skills to the real environment. Mixed reality (MR) technology, by seamlessly integrating virtual information with the real world, offers the possibility of building a rehabilitation environment with high ecological validity. However, among existing technologies, there is still no systematic solution that deeply integrates MR technology with artificial intelligence decision-making to form a closed-loop, adaptive, and personalized rehabilitation training for PSCI. Summary of the Invention

[0008] To overcome the technical deficiencies in existing technologies and effectively solve the technical problems in the background art, this invention provides a rehabilitation training system and method for post-stroke cognitive impairment based on mixed reality technology. This system can achieve integrated assessment and training, gamification and personalization of the training process, adaptive adjustment of training difficulty, and real-time feedback, thereby effectively improving the rehabilitation training effect, participation, and ecological validity of PSCI patients. The specific technical solution is as follows:

[0009] This invention provides a rehabilitation training system for post-stroke cognitive impairment based on mixed reality technology, comprising:

[0010] 1. Mixed Reality Interactive Terminal: Includes an MR head-mounted display device for presenting users with a virtual training environment that integrates real-world scenes; and an interactive perception module for capturing users' gestures, voice, and eye movement information in real space to enable natural interaction between users and virtual training content.

[0011] 2. User Information Management Module: Used to store and manage user identity information, medical history, and previous assessment and training records.

[0012] 3. Cognitive Assessment Module: This module guides users through a standardized cognitive function screening process in a mixed reality environment. It utilizes pre-stored neuropsychological scales (such as MMSE and MR immersive adaptations of MoCA), records the user's response process, reaction time, and behavioral performance via a mixed reality interactive terminal, and automatically calculates assessment scores to determine the user's impaired cognitive domains (such as attention, memory, executive function, and visuospatial ability) and their severity.

[0013] 4. Personalized Training Plan Generation Module: Connected to the cognitive assessment module, this module matches and generates a targeted, personalized initial training plan from the training task library based on the assessment results. This plan clearly defines the cognitive domain of the training objective, the sequence of training tasks, the initial difficulty level, and the training duration.

[0014] 5. Immersive Mixed Reality Training Task Library: This library stores multiple immersive interactive training tasks designed for different cognitive domains and capable of running in mixed reality environments. For example:

[0015] Attention / Execution Function Training Tasks: Virtual Supermarket Shopping Task (requires finding specific items from a list among mixed goods while suppressing distractions), Virtual Kitchen Safety Task (requires quickly responding to multiple stove switches and turning off a designated dangerous stove).

[0016] Memory training tasks: Virtual living room item memory task (after a virtual character places an item, the user is required to point out or retrieve the virtual item at the corresponding location in the real living room), and virtual path memory and reproduction task.

[0017] Visual spatial ability training tasks: Superimpose virtual building blocks in a real room and require users to imitate and assemble them; or perform tasks such as clock drawing and 3D route planning in a mixed reality environment.

[0018] 6. Adaptive Adjustment and Feedback Engine: Connected to the mixed reality interactive terminal and the immersive mixed reality training task library. This engine is used for:

[0019] Real-time analytics: Real-time analysis of user behavior data (accuracy, reaction time, task completion strategy).

[0020] Dynamic difficulty adjustment: Based on the analysis results, dynamically adjust the difficulty parameters of the current training task (such as the number of distractors, task complexity, and time limit). For example, automatically increase the difficulty when the user performs consistently well; automatically decrease the difficulty or suggest a break when the user makes frequent mistakes.

[0021] Multimodal feedback: In mixed reality environments, virtual digital human guides provide voice encouragement and guidance; visual effects (such as highlighting correct choices and task progress bars) provide real-time behavioral feedback.

[0022] 7. Central Processing and Control Server: Used to coordinate and control the operation of the above modules, process data streams, execute training logic, and store complete training process data.

[0023] This invention also provides a rehabilitation training method for post-stroke cognitive impairment based on mixed reality technology, applied to the above-mentioned system. The method includes the following steps:

[0024] S1: Users log in to the system through a mixed reality interactive terminal, and the user information management module retrieves their historical records.

[0025] S2: If the user is a new user or needs to be reassessed, the cognitive assessment module will be activated to guide the user to complete an immersive cognitive function screening in a mixed reality environment and generate an assessment report that includes the damaged cognitive domains and their extent.

[0026] S3: The personalized training plan generation module selects matching training tasks from the immersive mixed reality training task library based on the evaluation report and combines them to generate an initial personalized training plan.

[0027] S4: Guided by a virtual digital human, the user begins performing training tasks in a mixed reality environment. The system captures user interactions through a mixed reality interactive terminal.

[0028] S5: The adaptive adjustment and feedback engine integrates and analyzes behavioral data in real time during task execution.

[0029] S6: Based on real-time analysis results, the adaptive adjustment and feedback engine dynamically adjusts the difficulty parameters of the current task and provides instant feedback through virtual digital humans, visual effects, and other means.

[0030] S7: After a single training task or program is completed, the system automatically generates a training report, including task completion status, difficulty change curve, behavioral indicators, and updates and stores it in the user information management module.

[0031] S8: Based on the periodic reassessment results and long-term training data, the system optimizes subsequent training plans through algorithms, forming a closed-loop rehabilitation process of "assessment-training-monitoring-adjustment-reassessment".

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] 1. Integrated assessment and training with high ecological validity: Standardized cognitive assessment is integrated into a mixed reality environment, and training tasks are carried out based on real spaces (such as the user's own living room or kitchen), which greatly improves the ecological validity of training and facilitates the transfer of skills to daily life.

[0034] 2. Highly personalized and adaptive: Based on initial assessment and real-time multimodal data, the system can dynamically adjust the training program to achieve truly individualized adaptive rehabilitation and maximize training efficiency.

[0035] 3. Enhanced immersion and engagement: By using mixed reality technology, tedious cognitive tasks are transformed into immersive, gamified interactive experiences, significantly improving patients' motivation to participate and their training adherence.

[0036] 4. Achieve closed-loop rehabilitation management: A complete digital closed loop has been constructed, from screening, personalized program development, immersive training, real-time adjustment to effect evaluation, providing innovative tools for the long-term, scientific and efficient rehabilitation management of PSCI. Attached Figure Description

[0037] Figure 1 This is a block diagram of the modular composition of the system of the present invention.

[0038] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1: System Composition

[0041] like Figure 1 As shown, the rehabilitation training system of the present invention mainly includes the following modules:

[0042] The mixed reality interactive terminal uses a Microsoft HoloLens 2 device, whose built-in SLAM (Simultaneous Localization and Mapping) technology can achieve 3D reconstruction and stable spatial anchoring of the user's real training environment (such as a family living room).

[0043] The cognitive assessment module incorporates a modified version of the MoCA (Montreal Cognitive Assessment) scale. Users can complete the "visuospatial skills (cube)" test by grasping and placing virtual objects with gestures, and complete the "word fluency" test by interacting with the virtual digital human through voice. The system automatically records and scores the results.

[0044] The "Virtual Kitchen Safety Task" in the immersive mixed reality training task library targets attention and executive function: a virtual stove interface is overlaid on the user's real kitchen stove area, and multiple virtual stove burners randomly light up with "danger" red lights. The user needs to quickly and accurately turn off the designated target stove burner by tapping it in the air, while suppressing the reaction to non-target stove burners.

[0045] The adaptive adjustment and feedback engine algorithm is configured as follows: If the user responds correctly for three consecutive trials with a reaction time below the threshold, the system automatically increases the number of interfering elements (increasing the difficulty). If the user makes consecutive mistakes, the virtual digital human will provide a voice prompt saying "Looks a bit challenging, let's relax a bit," and automatically reduce the number of interfering elements or extend the reaction time window (reducing the difficulty).

[0046] Example 2: Method Flow

[0047] like Figure 2 As shown, the training method is carried out in the following steps:

[0048] Mr. Li, a patient with mild executive function impairment following a left basal ganglia stroke, is using this system for the first time.

[0049] S1-S2: Mr. Li wore HoloLens 2 and completed the MR version of MoCA assessment. The system's automatic analysis showed that his scores in "Alternating Line Test" and "Continuous Subtraction of 7" were low, confirming impaired executive function.

[0050] S3: The system generates an initial training plan for it, with the core task being "virtual supermarket shopping" (training working memory and inhibitory control in the execution function), and the initial difficulty being: 3 items on the shopping list and 12 kinds of goods on the shelf.

[0051] S4-S7: Training begins. Mr. Li needs to use gestures to grab the correct items and add them to a virtual shopping cart from a virtual list floating in front of him, placed on a virtual shelf superimposed on his real living room bookshelf. The system monitors his grabbing accuracy and decision-making time in real time. In the first few rounds, Mr. Li performs well. The system gradually increases the list to 5 items and the number of items to 15. Later, when the system detects an increase in his error rate (which may indicate cognitive fatigue), it automatically adjusts the difficulty back to a 4-item list and provides encouragement from a virtual digital human: "You've persisted for a long time, doing very well. Let's make a slight adjustment and continue."

[0052] S8: After 30 minutes of training, the system generates a report: average task accuracy 85%, reaction time reduced by 15% compared to the initial level. The report is stored in Mr. Li's electronic medical record.

[0053] S9: One week later, the system suggested a short MR assessment sub-test, and the results showed an improvement in the speed of the "connection test". Based on this, the system fine-tuned the training plan for the following week, introducing more complex "dual-task" training (such as memorizing a list while performing simple mental arithmetic).

[0054] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A rehabilitation training system for post-stroke cognitive impairment based on mixed reality technology, characterized in that: include: Mixed reality interactive terminals are used to present a training environment that blends the virtual and real worlds and capture users' natural interactive behaviors. The user information management module is used to manage users' personal and historical rehabilitation data; The cognitive assessment module is used to guide and complete standardized cognitive function screening in a mixed reality environment and output cognitive domain impairment assessment results. The personalized training plan generation module generates a targeted initial training plan based on the evaluation results; An immersive mixed reality training task library stores interactive training tasks that cover multiple cognitive domains and run in a mixed reality environment. An adaptive adjustment and feedback engine is used to analyze user interaction behavior data in real time and dynamically adjust the difficulty of training tasks accordingly, while providing real-time feedback through a mixed reality environment. The central processing and control server is used to coordinate and control the operation and data flow of various modules.

2. The post-stroke cognitive impairment rehabilitation training system based on mixed reality technology according to claim 1, characterized in that: The adaptive adjustment and feedback engine performs dynamic difficulty adjustment, including automatically increasing or decreasing the amount of interference information in the training task, the complexity of the task sequence, or adjusting the task time limit parameters based on the accuracy of the user's behavior, the cognitive load or fatigue level reflected in the reaction time.

3. The post-stroke cognitive impairment rehabilitation training system based on mixed reality technology according to claim 1, characterized in that: The immersive mixed reality training task library includes the following task types: virtual environment selective attention tasks for training attention and executive functions, virtual space object memory and retrieval tasks for training working memory, and virtual-real combined 3D structure imitation or path planning tasks for training visuospatial abilities.

4. The post-stroke cognitive impairment rehabilitation training system based on mixed reality technology according to claim 1, characterized in that: The multimodal feedback provided by the mixed reality interactive terminal includes: voice guidance and encouragement presented by a virtual digital human, and visual effects enhancement based on the correctness of user operations.

5. A method for rehabilitation training of post-stroke cognitive impairment based on mixed reality technology, applied to the post-stroke cognitive impairment rehabilitation training system based on mixed reality technology as described in any one of claims 1-4, characterized in that: Includes the following steps: Step 1: Verify user identity and load information through a mixed reality interactive terminal; Step 2: Conduct an immersive cognitive function assessment in a mixed reality environment and obtain a cognitive function assessment report; Step 3: Based on the evaluation report, generate a personalized initial mixed reality training plan; Step 4: Guide the user to perform tasks in the training program within a mixed reality environment; Step 5: Collect user task interaction data in real time; Step Six: Analyze the behavioral data and adjust the difficulty of the current training task dynamically in real time accordingly; Step 7: In a mixed reality environment, provide users with immediate feedback based on the analysis results; Step 8: After training is completed, generate and save a report on the training process and results.

6. The post-stroke cognitive impairment rehabilitation training method based on mixed reality technology according to claim 5, characterized in that: The real-time dynamic adjustment of difficulty specifically includes: increasing the task difficulty when the user's behavior is consistently better than the preset threshold; and decreasing the task difficulty or triggering a rest prompt when the user's behavior declines.

7. The post-stroke cognitive impairment rehabilitation training method based on mixed reality technology according to claim 5, characterized in that: It also includes the step of periodically performing cognitive reassessment, and optimizing and iterating the user's personalized training plan based on the reassessment results and long-term training data, forming a closed-loop rehabilitation management process of assessment-training-adjustment.

Images