A cognitive rehabilitation assessment training system based on multi-modal AI

Abstract

The application relates to the technical field of artificial intelligence, and discloses a cognitive rehabilitation evaluation training system based on a multi-modal AI, which comprises an AI cognitive evaluation module, a rehabilitation scheme generation module, a somatosensory interactive rehabilitation training module and a data management module. Through multi-modal AI dialogue evaluation technology, the core dimensions of cognitive impairment screening can be covered, not only the scale text answer is focused on, but also multi-modal features such as voice speed, pause hesitation degree and the like are included, automatic evaluation is realized, the evaluation result is more accurate in reflecting the cognitive state of a user, and a scientific basis is provided for personalized rehabilitation scheme generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically discloses a cognitive rehabilitation assessment and training system based on multimodal AI. Background Technology

[0002] Cognitive rehabilitation assessment and training refers to a non-pharmacological intervention process for patients with mild cognitive impairment, Alzheimer's disease, and post-stroke cognitive impairment. Through systematic cognitive function assessment and targeted training, it aims to slow cognitive decline and improve daily living abilities. Traditional methods often rely on manual administration of assessment scales (such as MMSE and MoCA) and one-on-one rehabilitation guidance. This is not only time-consuming and labor-intensive, but also highly subjective, difficult to scale up, and monotonous in training format, resulting in low patient compliance. Furthermore, it cannot meet the needs of home rehabilitation and personalized, continuous monitoring.

[0003] In the prior art, patent document CN117204852A discloses a "method, device and system for comprehensive cognitive assessment and rehabilitation training of cognitive impairment", which includes: generating rendering results based on the training scene and displaying them on the display screen; converting the position of the marker into the first coordinate in the reference coordinate system; displaying the three-dimensional model corresponding to the marker on the display screen; capturing the eye movement signals of the user's eyes and converting the gaze point coordinates into the second coordinate in the reference coordinate system; capturing the position information of the user's hand and converting it into the third coordinate in the reference coordinate system; and determining the user's visual reaction time and motor reaction time.

[0004] The patent document with announcement number CN111816295B discloses a "system and method for screening, assessing and training cognitive impairment based on the Internet", which includes a rapid screening module for obtaining rapid screening data to achieve rapid preliminary screening of completely healthy individuals, individuals suspected of having mild to moderate cognitive impairment, and individuals suspected of having severe cognitive impairment; a remote consultation module for scheduling remote consultations for individuals suspected of having mild to moderate cognitive impairment, selecting and pushing auxiliary assessment plans, and obtaining the final remote consultation data; and an offline outpatient appointment module for providing offline outpatient appointment windows for individuals with mild to moderate cognitive impairment confirmed after remote consultation.

[0005] While existing technologies have enabled internet-based tiered screening and assessment of cognitive impairment, along with remote rehabilitation, allowing for early intervention to slow cognitive decline, and the integration of eye movements, gestures, and augmented reality to assess multi-dimensional cognitive and hand-eye coordination abilities, thus improving the scientific rigor and efficiency of training and reducing the cost of rehabilitation consumables, current technologies still rely on manual application of measurement scales. This results in low assessment efficiency, high subjectivity, and a high risk of missed diagnoses. Furthermore, rehabilitation training is often monotonous, leading to poor patient compliance. Assessment and training are disconnected, and personalized plans cannot be automatically matched. The recommendation algorithms are black-box and uninterpretable, making it difficult for physicians to review and adjust them. Data collection is coarse-grained and lacks effective filtering. Moreover, the reliance on dedicated hardware and independent apps makes large-scale home deployment costly and impractical. Summary of the Invention

[0006] The main technical problem solved by this invention is to provide a cognitive rehabilitation assessment and training system based on multimodal AI, which can solve the problems mentioned in the background art.

[0007] To address the aforementioned technical problems, according to one aspect of the present invention, more specifically, a cognitive rehabilitation assessment and training system based on multimodal AI is provided, comprising: an AI cognitive assessment module, a rehabilitation plan generation module, a somatosensory interactive rehabilitation training module, and a data management module; The AI ​​cognitive assessment module collects multimodal speech data to complete conversational assessment and scoring of cognitive abilities; The rehabilitation program generation module generates personalized cognitive rehabilitation training programs based on the cognitive assessment results and the user's ability profile. The somatosensory interactive rehabilitation training module uses human posture recognition to conduct somatosensory rehabilitation training that coordinates cognition and movement. The data management module collects data from the entire evaluation and training process through the SDK, and then performs data storage and analysis after filtering.

[0008] Furthermore, the AI ​​cognitive assessment module includes: a voice interaction module, an intelligent assessment module, and a cognitive risk grading module; Voice interaction module: Provides ASR speech recognition, collects multimodal speech features such as user voice responses, speech rate, pause duration, and speech hesitation, and converts them into text data; Intelligent assessment module: Based on the MMSE scale, it performs conversational cognitive assessment and integrates voice features and text answers to complete the assessment and scoring; Cognitive Risk Classification Module: Based on the assessment and scoring results, users' cognitive impairment is classified into mild, moderate, and severe levels.

[0009] Furthermore, the rehabilitation plan generation module includes: an interpretable rule engine module, a user profiling module, and a plan review module; Explainable rules engine module: Reads cognitive assessment results, user ability profiles, training tags and historical training data, and generates rehabilitation plans according to preset medical rules; User profile module: Stores basic user information, cognitive ability level, training preferences, and rehabilitation progress data; The treatment plan review module provides an interactive interface for physicians to manually modify, review, and confirm rehabilitation plans.

[0010] Furthermore, the somatosensory interactive rehabilitation training module includes: a somatosensory recognition module, a somatosensory game module, and a motion feedback module; Motion recognition module: Based on the MediaPipe algorithm, it extracts 2D key points of the human body through a regular RGB camera to complete posture capture and motion recognition; Motion-sensing game module: Developed using Unity WebGL, featuring a cognitive-motor dual-task rehabilitation game, and supports direct operation in a browser; Motion feedback module: Recognizes the compliance of user's body movements and outputs real-time motion correction prompts.

[0011] Furthermore, the data management module includes: an SDK acquisition module, a data filtering module, and a storage and analysis module; SDK data acquisition module: Real-time acquisition of fine-grained behavioral data such as reaction time, movement accuracy, limb trajectory, difficulty changes, and recognition confidence during motion-sensing training; Data filtering module: It adopts a two-layer serial filtering mechanism, which sequentially performs data quality verification and training content adaptation filtering; Storage and analysis module: The filtered data is structured and stored in the database to complete the statistical analysis and trend analysis of rehabilitation data.

[0012] Furthermore, it also includes operational support modules: front-end display module, back-end service module, and communication interface module; Front-end presentation module: Built on Next.js, React, and TypeScript, it supports fully voice-enabled accessibility and visualization of rehabilitation data; Backend service module: It adopts the NestJS framework, combined with PostgreSQL database and Redis cache to realize business scheduling, access control and data reading and writing; Communication interface module: Enables bidirectional encrypted communication between modules through HTTPS RESTful API and JavaScript bridge interface.

[0013] Furthermore, the system adopts a lightweight Web architecture, supports multi-browser compatibility, and can be deployed and used without installing a separate app or dedicated motion-sensing hardware.

[0014] The beneficial effects of the cognitive rehabilitation assessment and training system based on multimodal AI of the present invention are as follows: Through multimodal AI dialogue assessment technology, it can cover the core dimensions of cognitive impairment screening, not only focusing on the text answers of the scale, but also incorporating multimodal features such as speech rate, pauses and hesitations, to achieve automated assessment, so that the assessment results can more accurately reflect the user's cognitive state and provide a scientific basis for the generation of personalized rehabilitation plans. In addition, through the closed-loop linkage of assessment and training and the technology of interpretable rule engine, the assessment results are seamlessly transformed into somatosensory training programs, breaking down the barriers of the traditional rehabilitation process and improving patient rehabilitation compliance and clinical adaptability. Meanwhile, by employing hardware-free RGB motion sensing recognition, fine-grained data acquisition, and dual-layer filtering technology, home-based cognitive and motor training can be achieved using ordinary cameras, ensuring data quality and training safety. Combined with remote physician intervention and data visualization, this greatly improves the efficiency, scientific rigor, and accessibility of home-based rehabilitation management for patients. Attached Figure Description

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific implementation methods.

[0016] Figure 1 This is a schematic diagram of the system module architecture; Figure 2 This is a diagram illustrating the evaluation results and intelligent matching. Figure 3 A diagram illustrating patient record creation and informed consent procedures; Figure 4 Edit a diagram for the physician's rehabilitation plan; Figure 5 A diagram illustrating the preparation for Unity training; Figure 6 This is a diagram illustrating real-time feedback collected from the game. Figure 7 This is a diagram illustrating the training results and historical records. Figure 8 A diagram showing the overall overview of physician and patient details; Figure 9 This is a diagram illustrating the login and role entry points; Figure 10 This is a diagram illustrating the patient's training today. Figure 11 This is a diagram illustrating security, privacy, and auditing. Detailed Implementation

[0017] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0018] According to one aspect of the invention, such as Figures 1-11 As shown, a cognitive rehabilitation assessment and training system based on multimodal AI is provided, including: an AI cognitive assessment module that collects multimodal speech data and completes conversational assessment and scoring of cognitive abilities. This module includes: Voice interaction module: Provides ASR speech recognition, collects multimodal speech features such as user voice responses, speech rate, pause duration, and speech hesitation, and converts them into text data; First, the system is built on Python / FastAPI to provide an asynchronous high-concurrency voice interaction service. The front end initiates voice acquisition requests through an HTTPS encrypted interface. ASR speech recognition receives the user's voice stream in real time and completes the sentence-by-sentence transcription from speech to text. Simultaneously, it extracts multimodal feature parameters such as speech rate, pause duration, and speech hesitation from the speech signal and binds the text data and speech features according to timestamps to form a multimodal dataset. Then, the voice interaction module adopts a fully voice-guided interaction logic, which reads the MMSE scale assessment questions through voice, without the need for text operation, which is suitable for patients with cognitive impairment with low literacy ability. At the same time, it automatically filters out environmental noise and short invalid input, and only retains valid response voice data. Finally, the processed multimodal dataset is synchronized to the backend service module in real time, temporarily stored in the Redis cache, and pushed to the intelligent evaluation module to ensure the real-time and continuous transmission of evaluation data.

[0019] Intelligent assessment module: Based on the MMSE scale, it performs conversational cognitive assessment and integrates voice features and text answers to complete the assessment and scoring; The system loads the Spark Big Model to simulate the professional physician assessment logic. According to the fixed dimensions and question sequence of the MMSE scale, it pushes assessment questions to the user in rounds through the voice interaction module. After receiving the multimodal dataset, it performs correctness verification and semantic analysis on the text answers. Then, the intelligent assessment module constructs a fusion scoring model, which uses the text answer score as the base score and incorporates speech features such as abnormal speech rate, long pauses, and frequent hesitations into the auxiliary score according to preset medical weights. The weighted calculation generates the final assessment score, replacing the traditional single text scoring mode and improving the accuracy of the assessment.

[0020] Cognitive Risk Classification Module: Based on the assessment and scoring results, users' cognitive impairment is classified into mild, moderate, and severe levels; Specifically, the system presets medically compliant cognitive impairment grading thresholds, matches the total score output by the intelligent assessment module with the thresholds, determines mild cognitive impairment if the score is above the first threshold, moderate cognitive impairment if the score is between the first and second thresholds, and severe cognitive impairment if the score is below the second threshold, thus completing the automated grading. The grading results will also be used to generate a structured assessment report, annotating scores for each cognitive dimension, voice feature-assisted scoring, risk level, and other information. After being encrypted on the backend, the report will be stored in a PostgreSQL database, providing standardized data for the generation of rehabilitation plans (e.g., Figure 2 (As shown).

[0021] The rehabilitation plan generation module generates personalized cognitive rehabilitation training plans based on cognitive assessment results and user ability profiles. This module includes: Explainable rules engine module: Reads cognitive assessment results, user ability profiles, training tags and historical training data, and generates rehabilitation plans according to preset medical rules; First, a rule parsing and execution service is built based on the NestJS backend framework. It retrieves cognitive risk grading results, user ability profile tags, somatosensory rehabilitation game training tags, and historical training completion rate, movement accuracy, training interruption records, and other data from the PostgreSQL database. The multi-dimensional data is loaded into the rule operation unit and accurately matched and logically operated according to the cognitive rehabilitation medical rule library preset by clinicians. The rule library contains standardized medical configuration parameters such as daily training duration, rehabilitation game type, difficulty gradient, training frequency, and training cycle corresponding to mild, moderate, and severe cognitive impairment. Then, the rules engine filters and sorts the somatosensory rehabilitation game resources based on the matching results, automatically removing game projects that are not compatible with the user's cognitive risk level, have poor historical training effects, or are beyond the user's ability. Combining the user's cognitive shortcomings (such as memory, attention, executive ability, and orientation), it generates a complete personalized rehabilitation plan that includes a daily training task list, game combination schemes, dynamic difficulty parameters, and phased rehabilitation goals. The entire plan generation process follows interpretable logic, and each training recommendation corresponds to a clear medical rule basis, with no black-box algorithm decision-making process. The final personalized rehabilitation plan is encrypted via HTTPS and then pushed to a Redis cache for high-speed temporary storage. It is simultaneously transmitted to the front-end display module and the plan review module. At the same time, the core data of the plan is structured and written into a PostgreSQL database to achieve persistent storage and fast cross-module access to the rehabilitation plan, ensuring the security and traceability of the plan data.

[0022] User profile module: Stores basic user information, cognitive ability level, training preferences, and rehabilitation progress data; First, a standardized user profile data storage structure is built based on the PostgreSQL relational database. The front-end display module collects basic information from the user registration phase, including core medical and identity data such as age, gender, type of cognitive impairment, past medical history, and medication history. This completes the initial entry, encrypted storage, and permission binding of basic user information (e.g., ...). Figure 3(as shown) Then, it connects in real time with the assessment results of the AI ​​cognitive assessment module and the training statistics of the data management module to automatically update dynamic data such as the user's cognitive ability level score, rehabilitation game preference type, training completion progress, training frequency, and ability improvement trend, forming a complete user ability profile that deeply integrates static basic information and dynamic rehabilitation data. Finally, the real-time updated profile data is synchronously pushed to the interpretable rule engine module and the backend service module, providing unified user data support for personalized rehabilitation plan generation, rehabilitation effect analysis, and remote physician intervention. At the same time, based on the backend permission control mechanism, only licensed physicians and system administrators are authorized to access and retrieve the core user profile data, ensuring user data privacy and security.

[0023] The rehabilitation plan review module provides an interactive interface for physicians to manually modify, review, and confirm rehabilitation plans. Specifically, a dedicated visual interactive interface for physicians is built based on the front-end display module. The interface clearly displays all content of the rehabilitation plan awaiting review, including training task details, game configuration parameters, difficulty level, training cycle, rule matching criteria, user profile suitability, and other information. The interface supports mouse operation, touch operation, and full voice-assisted operation, and is compatible with multiple terminal devices such as physician office computers and tablets (e.g., ...). Figure 4 (as shown) Next, physicians can manually adjust the rehabilitation plan through an interactive interface, modifying core parameters such as the type of training game, the duration of a single training session, difficulty levels, and the duration of training cycles. All adjustments are synchronized to the NestJS backend service in real time. The system automatically records the modified content, modification time, modified physician information, and modified medical basis, generating an unalterable review and modification log (e.g., Figure 11 (As shown), it meets the medical compliance retention requirements; Finally, after the doctor confirms that the plan meets the user's rehabilitation needs, they click the review and confirmation button. The system pushes the approved final rehabilitation plan to the user's front-end display module. At the same time, the review results, modification logs, and final plan data are encrypted and stored in the PostgreSQL database, completing the plan review closed loop. If the review fails, the system returns the plan to the interpretable rule engine module, marks modification suggestions, and drives the engine to regenerate an adapted plan.

[0024] The somatosensory interactive rehabilitation training module utilizes human posture recognition to conduct somatosensory rehabilitation training that integrates cognitive and motor coordination. This module includes: Motion recognition module: Based on the MediaPipe algorithm, it extracts 2D key points of the human body through a regular RGB camera to complete posture capture and motion recognition; The motion recognition module relies on MediaPipe's lightweight human posture recognition algorithm to build a visual perception unit. It calls the real-time video stream from the user terminal's ordinary RGB camera through the communication interface module. It does not require dedicated motion-sensing hardware such as depth cameras or motion-sensing handles. The video stream is first pre-processed at the frame level to complete environmental noise filtering, human body contour segmentation, and background interference removal, accurately locking the effective recognition area of ​​the user's whole body. The MediaPipe algorithm analyzes the pre-processed video frames one by one, extracting key 2D skeletal points such as the head, shoulders, elbows, wrists, hips, knees, and ankles of the human body. It calculates quantitative parameters such as spatial coordinates, movement speed, limb angles, and displacement trajectories of each key point in real time, transforming the user's limb movements into standardized digital features. It accurately identifies the types of movements required for rehabilitation training, such as raising hands, stretching, turning, stepping, swinging arms, and bending knees, while outputting movement recognition confidence data in real time. Finally, the extracted key point coordinates, motion features, recognition confidence, and other data are synchronously transmitted to the motion-sensing game module and motion feedback module via the JavaScript bridge interface. At the same time, the raw posture data is pushed to the SDK acquisition module of the data management module, providing underlying support for fine-grained behavior data acquisition.

[0025] Motion-sensing game module: Developed using Unity WebGL, featuring a cognitive-motor dual-task rehabilitation game, and supports direct operation in a browser; First, based on the Unity engine, the WebGL version was compiled and developed. Evidence-based cognitive-motor dual-task rehabilitation games such as Tai Chi simulation, virtual table tennis, fast numerical calculation, item classification, and graphic matching were packaged into a lightweight program that can be directly loaded by the browser. It is seamlessly connected with the front-end display module through the communication interface module, and is compatible with multiple brands and versions of browsers. There is no need to download a separate APP or install plugins. It is also suitable for low-configuration computers, tablets and other terminal devices for elderly users. Next, a personalized rehabilitation plan, reviewed by a physician, is retrieved from the backend service module via a JavaScript bridge interface. This plan analyzes configuration parameters such as game difficulty, single training session duration, task type, cognitive training dimensions, and exercise intensity, automatically loading game scenarios and task content (e.g., [missing information]) that match the user's cognitive impairment level and motor abilities. Figure 5 As shown, the motion features transmitted by the motion recognition module are converted into in-game character control commands, enabling real-time control of the game process by body movements; Finally, the cognitive-motor dual training logic was strictly implemented during operation. Cognitive tasks such as memory, attention, calculation, and spatial orientation were simultaneously superimposed on physical movement training. Reaction time, movement accuracy, and task completion data were recorded in real time throughout the training process. Fine-grained training data was reported to the data management module in real time via a JavaScript bridge interface. After each training session, a training summary was automatically generated and pushed to the front-end display module (e.g., ...). Figure 6 (As shown).

[0026] Motion feedback module: Recognizes the compliance of user's body movements and outputs real-time motion correction prompts; Specifically, it receives the coordinates of key human body points, movement trajectory, and recognition confidence data transmitted by the body sensing recognition module in real time, and compares the user's actual movements with the standard movement templates built into the rehabilitation game in a precise dimension-by-dimensional comparison. It calculates quantitative indicators such as movement compliance rate, posture deviation value, trajectory error rate, and movement accuracy, and quickly determines whether the limb movements meet the rehabilitation training standards. It also includes generating real-time correction prompts in both visual and voice modes based on the comparison results. The front-end display module presents movement problems in a visual form, such as dynamic arrow guidance, highlighting of standard postures, and red marking of deviation areas. Simultaneously, it broadcasts concise and easy-to-understand adjustment instructions (such as "raise the left arm to shoulder height", "slow down the movement speed", "increase the turning angle to 45°"). The prompts are adapted to the comprehension ability of patients with cognitive impairment. The correction information is output synchronously with the game progress without delay, helping users to correct their movements in real time and ensuring the standardization and effectiveness of rehabilitation training.

[0027] The data management module collects data from the entire evaluation and training process via the SDK, filters the data, and then stores and analyzes it. This module includes: SDK data acquisition module: Real-time acquisition of fine-grained behavioral data such as reaction time, movement accuracy, limb trajectory, difficulty changes, and recognition confidence during motion-sensing training; Firstly, the SDK data collection module is integrated into the front-end display module and the Unity WebGL motion-sensing game runtime environment in the form of lightweight embedded code. It establishes a low-latency, highly stable real-time data transmission link through the communication interface module. It runs in the background without the user's awareness throughout the motion-sensing training and AI evaluation process. It synchronously captures all-dimensional fine-grained behavioral data during the user's training process at a fixed collection frequency, including reaction time, action accuracy, limb movement trajectory coordinates, dynamic adjustment value of training difficulty, camera recognition confidence, training interruption reasons, task completion progress, and cognitive task answering results. At the same time, it connects with the AI ​​cognitive evaluation module to obtain the raw data of the entire evaluation process. All collected data are accurately bound to the user's unique identity identifier according to a unified timestamp, forming a standardized and structured raw data sequence. Next, the raw data undergoes preliminary standardization processing to unify data units, data types, and encoding formats. Invalid and redundant data such as null values, duplicate values, and garbled values ​​generated during the collection process are removed. Missing key data items are marked and completed to ensure data integrity and consistency. After processing, the raw data is pushed to the data filtering module in real time to provide a complete and standardized data source for subsequent data cleaning and screening. Finally, a multi-threaded parallel acquisition architecture is adopted to support data acquisition tasks for concurrent training by multiple users. The asynchronous non-blocking transmission mechanism reduces the impact on system performance. The entire acquisition process strictly follows medical data security standards, anonymizes sensitive user data, and does not collect or transmit personal information unrelated to rehabilitation training, thus ensuring the compliance of the data acquisition process and the security of user privacy.

[0028] Data filtering module: It adopts a two-layer serial filtering mechanism, which sequentially performs data quality verification and training content adaptation filtering; First, the first layer of data quality verification and filtering is initiated to conduct a full-dimensional screening of the raw data transmitted by the SDK acquisition module. This process sequentially completes the following steps: deduplication of duplicate data, removal of extreme abnormal values, discarding of data with camera recognition confidence levels below a preset medical threshold, automatic identification and removal of invalid training segments, and verification of the legality of user data access authorization. Only training and evaluation data that meet medical data quality standards, are effectively collected, and are authorized in compliance with regulations are retained, ensuring the accuracy, validity, and legality of the data entering the database from the source. Then, the second layer of training content adaptation filtering is performed. Based on core parameters such as user cognitive risk level, ability profile, historical training performance, and rehabilitation plan configuration, the effective data after the first layer of filtering is matched and verified. Training data that is not compatible with the user's cognitive impairment level, limb motor ability, and rehabilitation stage is removed, and compliant data that fully fits the personalized rehabilitation plan is selected to avoid invalid or incompatible data interfering with the rehabilitation effect analysis and subsequent plan optimization. Finally, the clean data after the two-layer serial filtering is packaged and encapsulated in a unified manner, and classified and organized according to data type, collection time, user identifier, and training scenario. Detailed data filtering logs are generated simultaneously, which fully record information such as filtering rules, the number of filtered data, the proportion of valid data, and the reasons for abnormal data. The filtered compliant data and the filtering logs are transmitted together to the storage and analysis module to achieve full traceability and verification of the data filtering process.

[0029] Storage and analysis module: Structures and stores the filtered data into the database, and completes rehabilitation data statistics and trend analysis; Specifically, a standardized data storage architecture is built based on the PostgreSQL medical-grade relational database. The filtered compliant data is divided into four categories: user basic information, cognitive assessment data, somatosensory training data, and rehabilitation analysis results, and stored in a structured database. All medical rehabilitation data is encrypted using national cryptographic encryption algorithms, and Redis caching is used to accelerate the storage of frequently accessed user rehabilitation data and training records, which greatly improves the efficiency of data reading, calling and querying. Simultaneously, in-depth mining and quantitative analysis are conducted on the entered data, statistically analyzing core indicators such as user single training completion rate, average movement accuracy, reaction time change, cognitive task score, training difficulty adaptation, and rehabilitation cycle progress. Daily, weekly, and monthly rehabilitation progress trend reports are generated, presenting the improvement patterns of users' cognitive and motor abilities in visual formats such as line charts, bar charts, and radar charts (e.g., Figure 8 As shown in the image, the user's historical training records and results summary interface are as follows: Figure 7 As shown; Finally, the data analysis results are pushed to the front-end display module in real time for users to view intuitively, and transmitted to the back-end service module for physicians to use for remote monitoring, intervention and rehabilitation plan adjustment. This provides high-quality and reliable data support for the optimization of the interpretable rule engine, the iteration of the AI ​​cognitive assessment model, and the clinical rehabilitation effect research, and fully realizes the closed-loop management of the entire process of data collection, filtering, storage, analysis and application.

[0030] It also includes a runtime support module, providing underlying rendering, business processing, data communication, and security assurance for the entire system operation. This module includes: Front-end presentation module: Built on Next.js, React, and TypeScript, it supports fully voice-enabled accessibility and visualization of rehabilitation data; Specifically, Next.js server-side rendering (SSR) technology is used to build the core rendering logic of the page. Resource loading strategies are optimized for low-configuration computers, tablets and other terminal devices for elderly users, which greatly improves page loading speed and the stability of medical data access. Based on the React component-based development model, functions such as AI dialogue interaction, motion-sensing game entry, rehabilitation data dashboard, points incentive, and family social interaction are decomposed into independent reusable components to achieve module decoupling and rapid iteration. The TypeScript static type validation mechanism strictly constrains the medical data interaction format and interface call logic, eliminating data corruption and logic abnormalities from the code level, and laying a solid foundation for stable system operation. It supports full-process voice interaction, including voice wake-up, command control, and result broadcasting, adapting to the needs of cognitively impaired patients with low literacy abilities. The page adopts a large font, high contrast, and minimalist layout design to reduce visual cognitive load. It is compatible with all major browsers such as Chrome, Edge, and Firefox, and can be accessed normally without installing plugins. It visualizes rehabilitation progress, assessment scores, and training records in multiple formats, including line graphs, bar charts, text descriptions, and voice broadcasts, balancing user convenience with the needs of physicians for data analysis (such as...). Figure 9 , Figure 10 (As shown).

[0031] Backend service module: It adopts the NestJS framework, combined with PostgreSQL database and Redis cache to realize business scheduling, access control and data reading and writing; First, a standardized backend service system was built based on the NestJS enterprise-level modular architecture. In accordance with the medical rehabilitation business specifications, core functions such as business logic scheduling, user permission management, interface forwarding, recommendation engine execution, data ingestion scheduling, and physician backend management were broken down into independent business modules to adapt to the process control and permission compliance requirements of the medical industry. Then, the system uses a PostgreSQL relational database to store basic user information, cognitive assessment reports, motion training details, physician configuration rules, rehabilitation plans and other medical data in a structured manner. It supports complex condition queries, batch data processing and long-term secure storage. Redis memory cache is used to store hot data such as user profiles, personalized rehabilitation plans, high-frequency game configurations and session status, reducing repeated database queries, minimizing system response latency and ensuring smooth and stable access for multiple users concurrently.

[0032] Communication interface module: Enables bidirectional encrypted communication between modules through HTTPS RESTful API and JavaScript bridge interface; Specifically, an HTTPS encrypted transmission channel is established, and a standardized communication interface between the front end and the back end is designed based on the RESTful API specification. Sensitive information such as user operation data, training data, and evaluation data are encrypted throughout the transmission process, and the system has the ability to prevent theft, tampering, and leakage. When the front end calls the AI ​​service voice interface, the dialogue context is simultaneously transmitted to the back end for persistent storage, ensuring the continuity and integrity of the AI ​​cognitive evaluation process. It also includes a Unity WebGL-JS bridging communication interface, enabling bidirectional data exchange between the motion-sensing game module and the system platform. The game client obtains configuration parameters such as personalized difficulty, training duration, and task type through the bridging interface, reports fine-grained training data in real time, and uploads complete records in batches after training. All interfaces are equipped with identity verification and data signature verification mechanisms to ensure the legitimacy of the communication subject and the accuracy of data transmission, providing stable and reliable communication support for the system's lightweight web deployment and app-free deployment.

[0033] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention are also within the protection scope of the present invention.

Claims

1. A cognitive rehabilitation assessment and training system based on multimodal AI, characterized in that, include: AI cognitive assessment module, rehabilitation plan generation module, somatosensory interactive rehabilitation training module, data management module; The AI ​​cognitive assessment module collects multimodal speech data to complete conversational assessment and scoring of cognitive abilities; The rehabilitation program generation module generates personalized cognitive rehabilitation training programs based on the cognitive assessment results and the user's ability profile. The somatosensory interactive rehabilitation training module uses human posture recognition to conduct somatosensory rehabilitation training that coordinates cognition and movement. The data management module collects data from the entire evaluation and training process through the SDK, and then performs data storage and analysis after filtering.

2. The cognitive rehabilitation assessment and training system based on multimodal AI according to claim 1, characterized in that: The AI ​​cognitive assessment module includes: a voice interaction module, an intelligent assessment module, and a cognitive risk grading module; Voice interaction module: Provides ASR speech recognition, collects multimodal speech features such as user voice responses, speech rate, pause duration, and speech hesitation, and converts them into text data; Intelligent assessment module: Based on the MMSE scale, it performs conversational cognitive assessment and integrates voice features and text answers to complete the assessment and scoring; Cognitive Risk Classification Module: Based on the assessment and scoring results, users' cognitive impairment is classified into mild, moderate, and severe levels.

3. The cognitive rehabilitation assessment and training system based on multimodal AI according to claim 1, characterized in that: The rehabilitation plan generation module includes: an interpretable rule engine module, a user profiling module, and a plan review module; Explainable rules engine module: Reads cognitive assessment results, user ability profiles, training tags and historical training data, and generates rehabilitation plans according to preset medical rules; User profile module: Stores basic user information, cognitive ability level, training preferences, and rehabilitation progress data; The treatment plan review module provides an interactive interface for physicians to manually modify, review, and confirm rehabilitation plans.

4. The cognitive rehabilitation assessment and training system based on multimodal AI according to claim 1, characterized in that: The somatosensory interactive rehabilitation training module includes: a somatosensory recognition module, a somatosensory game module, and a motion feedback module; Motion recognition module: Based on the MediaPipe algorithm, it extracts 2D key points of the human body through a regular RGB camera to complete posture capture and motion recognition; Motion-sensing game module: Developed using Unity WebGL, featuring a cognitive-motor dual-task rehabilitation game, and supports direct operation in a browser; Motion feedback module: Recognizes the compliance of user's body movements and outputs real-time motion correction prompts.

5. The cognitive rehabilitation assessment and training system based on multimodal AI according to claim 1, characterized in that: The data management module includes: an SDK acquisition module, a data filtering module, and a storage and analysis module; SDK data acquisition module: Real-time acquisition of fine-grained behavioral data such as reaction time, movement accuracy, limb trajectory, difficulty changes, and recognition confidence during motion-sensing training; Data filtering module: It adopts a two-layer serial filtering mechanism, which sequentially performs data quality verification and training content adaptation filtering; Storage and analysis module: The filtered data is structured and stored in the database to complete the statistical analysis and trend analysis of rehabilitation data.

6. The cognitive rehabilitation assessment and training system based on multimodal AI according to claim 1 further includes an operation support module: a front-end display module, a back-end service module, and a communication interface module; Front-end presentation module: Built on Next.js, React, and TypeScript, it supports fully voice-enabled accessibility and visualization of rehabilitation data; Backend service module: It adopts the NestJS framework, combined with PostgreSQL database and Redis cache to realize business scheduling, access control and data reading and writing; Communication interface module: Enables bidirectional encrypted communication between modules through HTTPS RESTful API and JavaScript bridge interface.

7. The cognitive rehabilitation assessment and training system based on multimodal AI according to claim 1, characterized in that: The system adopts a lightweight web architecture, supports multi-browser compatibility, and can be deployed and used without installing a separate APP or dedicated motion-sensing hardware.

Images