A multi-modal interaction and dynamic-based cognitive impairment digital therapy system

Abstract

This invention discloses a digital therapy system for cognitive impairment based on multimodal interaction and dynamics. Through an interaction and data acquisition layer, multidimensional data is collected via gamified tasks in a gamified training environment, and preprocessed to obtain preprocessed multidimensional data. An intelligent analysis and decision-making layer is responsible for modeling, evaluating, and optimizing the preprocessed multidimensional data to generate rehabilitation strategies. An application and presentation layer provides dynamically adjusted training content, enabling objective, continuous, and fine-grained quantitative assessment of cognitive function. Based on the patient's real-time abilities and performance, truly personalized rehabilitation training content is dynamically generated and adjusted, enhancing the fun and patient participation in the training process, and improving rehabilitation efficacy and efficiency.

Description

Technical Field

[0001] This invention belongs to the field of cognitive medical technology, specifically relating to a digital therapy system for cognitive impairment based on multimodal interaction and dynamics. Background Technology

[0002] Cognitive impairment after stroke is a common and serious sequela, manifesting as damage in multiple cognitive domains such as memory, attention, and executive function, which seriously affects patients' daily living abilities, rehabilitation outcomes, and quality of life.

[0003] Traditional PSCI rehabilitation largely relies on hospitals or rehabilitation institutions, with therapists conducting one-on-one paper-and-pencil tests and manual training. This method has the following significant limitations:

[0004] 1. The evaluation methods are subjective, one-sided, and discontinuous. Limited assessment dimensions: Most of the existing clinical scales (such as MoCA and MMSE) are static questionnaire-based assessments, which are time-consuming, infrequent, and easily affected by the patient's emotions, the testing environment, and the subjectivity of the test administrator. They are difficult to comprehensively and objectively reflect the patient's true cognitive ability in complex and dynamic tasks.

[0005] Lack of quantification and sensitivity: Traditional scales have coarse scoring granularity and are not sensitive to slight changes in cognitive function or deficits in specific areas (such as processing speed), and cannot provide fine-grained data support for precise intervention.

[0006] Disconnect between assessment and training: Assessment is usually independent of the training process, making it impossible to adjust rehabilitation strategies in real time and dynamically based on training performance, thus failing to form an "assessment-intervention" closed loop.

[0007] 2. The rehabilitation intervention model is standardized and lacks personalization and fun. "One-size-fits-all" approach: Traditional rehabilitation training content and intensity are often based on a general plan, which cannot be dynamically adjusted according to each patient's specific cognitive deficit map, severity, interests and real-time performance, resulting in low rehabilitation efficiency and insufficient patient motivation.

[0008] Monotonous format and poor compliance: Training mainly consists of repetitive paper-and-pen or simple computer tasks, which are boring and difficult to maintain the patient's attention and enthusiasm for participation in the long term, seriously affecting the compliance and continuity of family rehabilitation.

[0009] High dependence on resources: High-quality rehabilitation resources are concentrated in core medical institutions in large cities, requiring patients to travel back and forth frequently, resulting in high economic and time costs and poor accessibility.

[0010] 3. The technology is in its early stages of application and lacks in-depth integration. Single-technology application: While some existing digital cognitive training products have introduced simple gamification elements or single-modal interactions (such as touch screens), they are mostly isolated sets of tasks. Their inherent evaluation logic is simple (such as using only accuracy / reaction time as indicators), and they fail to deeply integrate artificial intelligence algorithms to model and analyze multi-dimensional behavioral and physiological data.

[0011] Untapped Data Value: A large amount of behavioral time-series data and interaction trajectory data generated during the training process were not effectively collected, integrated, and analyzed. They were only used to judge whether the task was completed or not, and failed to be transformed into deeper information that reveals cognitive neural mechanisms and predicts rehabilitation trends.

[0012] Insufficient systematization and intelligence: The market lacks a systematic and intelligent digital therapy platform that can connect the entire process of "high-precision multimodal data acquisition → quantitative assessment of AI-driven capabilities → intelligent generation of personalized solutions → real-time adaptive adjustment".

[0013] With the rapid development of sensor technology, artificial intelligence, and human-computer interaction technology, their deep application in the healthcare field has become a clear trend. In the field of cognitive rehabilitation, the industry is actively exploring ways to enhance engagement through gamification and virtual reality (VR / AR), improve assessment efficiency using computerized adaptive testing, and is beginning to integrate multimodal data such as eye movement and speech. However, how to organically integrate these cutting-edge technologies to build an intelligent closed-loop system that can scientifically, accurately, dynamically, and personally serve the entire rehabilitation cycle of PSCI patients remains a gap in current technological development and an urgent clinical need. Summary of the Invention

[0014] The technical problem that the present invention needs to solve is: 1. How to achieve objective, continuous, and fine-grained quantitative assessment of cognitive function to replace or supplement traditional subjective scales.

[0015] 2. How to dynamically generate and adjust truly personalized rehabilitation training content based on the patient's real-time abilities and performance, so as to achieve "one policy for one person".

[0016] 3. How to enhance the fun and patient participation in the training process and ensure long-term adherence to home rehabilitation through natural and immersive multimodal interaction.

[0017] 4. How to seamlessly integrate assessment and intervention to build an intelligent closed loop of "data collection-analysis-decision-feedback" and improve rehabilitation efficacy and efficiency.

[0018] To address the aforementioned technical problems, the present invention provides a digital therapy system for cognitive impairment based on multimodal interaction and dynamics, comprising: The interaction and data acquisition layer is responsible for collecting multi-dimensional data through gamified tasks in the gamified training environment and preprocessing it to obtain preprocessed multi-dimensional data. The intelligent analysis and decision-making layer is responsible for modeling, evaluating, and optimizing the pre-processed multidimensional data to generate rehabilitation strategies. The application and presentation layer provides dynamically adjusted training content.

[0019] Preferably, the interaction and data acquisition layer includes: The interactive task scenario generation subunit designs serious games or virtual reality scenarios for specific cognitive domains as the gamified training environment. The behavior data collection subunit records all user actions and collects behavior data during gamified tasks. The physiological data acquisition subunit acquires physiological signals and obtains physiological data in a non-invasive manner. The visual acquisition subunit performs eye tracking to analyze fixation point, saccade path, and pupil diameter changes in order to assess attention allocation and cognitive load, and obtain visual assessment data. The auditory acquisition subunit acquires speech commands, analyzes the fluency, speed, intonation, and accuracy of the speech, evaluates language and executive functions, and obtains auditory assessment data. The data synchronization and preprocessing subunit performs timestamp alignment, filtering, denoising, and initial feature extraction on the multidimensional data to form a structured multimodal time-series data stream, resulting in preprocessed multidimensional data.

[0020] Preferably, the intelligent analysis and decision-making layer includes: The cognitive ability feature engineering sub-model extracts high-order features that are highly correlated with specific cognitive domains from preprocessed multidimensional data; The multi-dimensional cognitive profile generation sub-model maps the extracted high-order features to a multi-dimensional cognitive ability space, generating a dynamic digital cognitive ability profile for each user. The dynamic baseline modeling sub-model, based on a dynamic digital cognitive ability profile, establishes a personalized ability baseline model for each user. The personalized solution generator, based on dynamic digital cognitive ability profile and ability baseline model, prioritizes the areas with the most severe deficits in the cognitive profile, selects and combines corresponding training tasks from the intervention resource library, and adjusts the presentation of training tasks and formulates rehabilitation strategies by combining users' historical preference data on game type, difficulty and theme. The adaptive difficulty adjuster analyzes user performance in real time during a single training task and dynamically optimizes the rehabilitation strategy.

[0021] Preferably, the higher-order features include working memory capacity index, attention maintenance curve, conflict inhibition reaction time cost, and error rate growth slope over time.

[0022] Preferably, the multidimensional cognitive ability space covers core dimensions related to memory, attention, executive function, information processing speed, and visuospatial ability.

[0023] Preferably, the rehabilitation strategy includes gamified training tasks, cognitive behavioral therapy guidance, and personalized schedules of daily functional activity recommendations.

[0024] Preferably, the adaptive difficulty adjuster dynamically adjusts the task parameters to keep the task difficulty in the optimal range that matches the challenge with the ability. The task parameters include stimulus presentation speed, number of distractors, and memory load.

[0025] This invention provides a digital therapy system for cognitive impairment based on multimodal interaction and dynamics. Through an interaction and data acquisition layer, multi-dimensional data is collected via gamified tasks in a gamified training environment, and preprocessed to obtain preprocessed multi-dimensional data. An intelligent analysis and decision-making layer is responsible for modeling, evaluating, and optimizing the preprocessed multi-dimensional data to generate rehabilitation strategies. An application and presentation layer provides dynamically adjusted training content, enabling objective, continuous, and fine-grained quantitative assessment of cognitive function. Based on the patient's real-time abilities and performance, truly personalized rehabilitation training content is dynamically generated and adjusted, enhancing the fun and patient participation in the training process, and improving rehabilitation efficacy and efficiency. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the workflow of a digital therapy system for cognitive impairment based on multimodal interaction and dynamics, provided as an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0028] This invention provides a digital therapy system for cognitive impairment based on multimodal interaction and dynamics, comprising: The interaction and data acquisition layer, as the front-end perception system, is responsible for collecting multi-dimensional data in a gamified training environment through gamified tasks in a non-disruptive, accurate, and synchronous manner, and then preprocessing the data to obtain preprocessed multi-dimensional data.

[0029] The interactive task scenario generation subunit uses serious games or virtual reality (VR / AR) scenarios designed for specific cognitive domains (such as memory, attention, and executive function) as gamified training environments. Examples include virtual supermarket shopping tasks (assessing and training executive function) and rhythmic auditory stimulation tasks (assessing attention).

[0030] Multidimensional data includes behavioral data, physiological data, visual assessment data, and auditory assessment data.

[0031] The behavior data collection subunit records all user actions and collects behavior data during the task, including but not limited to the coordinates, timing, precision, response time, error type, and task completion path of clicks / touches.

[0032] The physiological data acquisition subunit integrates or connects to external sensors to non-invasively acquire physiological signals and obtain physiological data.

[0033] The visual acquisition subunit uses a camera to perform eye tracking, analyzes fixation points, saccade paths, and pupil diameter changes to assess attention allocation and cognitive load, and obtains visual assessment data.

[0034] The auditory acquisition subunit collects voice commands through a microphone, analyzes the fluency, speed, tone, and accuracy of the speech, evaluates language and executive functions, and obtains auditory assessment data.

[0035] The data synchronization and preprocessing subunit performs timestamp alignment, filtering, denoising, and initial feature extraction on multidimensional data from different sensors and interaction channels to form a structured multimodal time-series data stream, resulting in preprocessed multidimensional data.

[0036] The intelligent analysis and decision-making layer, as the core brain, is responsible for modeling, evaluating, and generating personalized solutions from pre-processed multidimensional data, forming an AI-driven digital assessment model for cognitive abilities.

[0037] The cognitive ability feature engineering sub-model extracts high-order features highly correlated with specific cognitive domains from preprocessed multidimensional data. These high-order features include: Extract the working memory capacity index and attention maintenance curve from the "N-back" task.

[0038] Extract the conflict suppression reaction time cost and the slope of error rate growth over time from the "Stroop" task.

[0039] The multi-dimensional cognitive profile generation sub-model uses machine learning or deep learning models (such as random forests and graph neural networks) to map the extracted high-order features to a multi-dimensional cognitive ability space, generating a dynamic digital cognitive ability profile for each user.

[0040] This space covers core dimensions such as memory, attention, executive function, information processing speed, and visuospatial ability, generating a dynamic digital cognitive ability profile for each user, intuitively showing their strengths and weaknesses and the severity of those weaknesses.

[0041] The dynamic baseline modeling sub-model, based on a dynamic digital cognitive ability profile, establishes a personalized ability baseline model for each user, which can be adaptively updated as the rehabilitation progresses. By comparing the deviation between real-time data and the individual baseline, the system can keenly perceive the user's intraday fluctuations (such as fatigue) and long-term trends (rehabilitation progress or regression).

[0042] The personalized solution generator, based on a dynamic digital cognitive ability profile and a baseline model, uses a deficiency-priority algorithm to target the areas with the most severe deficiencies in the cognitive profile. It selects and combines corresponding training tasks from the intervention resource library, and adjusts the presentation of training tasks by combining users' historical preference data on game types, difficulty, and themes to enhance participation motivation and develop rehabilitation strategies.

[0043] The rehabilitation strategy includes gamified training tasks, cognitive behavioral therapy guidance, and personalized schedules with recommendations for daily functional activities.

[0044] The adaptive difficulty adjuster analyzes user performance in real time during a single training task and dynamically optimizes the rehabilitation strategy.

[0045] By employing psychometrics or reinforcement learning algorithms, task parameters (such as stimulus presentation speed, number of distractors, and memory load) are dynamically adjusted to keep the task difficulty within the user's "zone of proximal development," which is the optimal range between challenge and ability, in order to maintain high engagement and optimal learning outcomes.

[0046] The application and presentation layer acts as the execution end, providing dynamically adjusted training content.

[0047] The three elements form a continuously optimized closed loop through data flow.

Claims

1. A digital therapy system for cognitive impairment based on multimodal interaction and dynamics, characterized in that, include: The interaction and data acquisition layer is responsible for collecting multi-dimensional data through gamified tasks in the gamified training environment and preprocessing it to obtain preprocessed multi-dimensional data. The intelligent analysis and decision-making layer is responsible for modeling, evaluating, and optimizing the pre-processed multidimensional data to generate rehabilitation strategies. The application and presentation layer provides dynamically adjusted training content.

2. The digital therapy system for cognitive impairment based on multimodal interaction and dynamics as described in claim 1, characterized in that, The interaction and data acquisition layer includes: The interactive task scenario generation subunit designs serious games or virtual reality scenarios for specific cognitive domains as the gamified training environment. The behavior data collection subunit records all user actions and collects behavior data during gamified tasks. The physiological data acquisition subunit acquires physiological signals and obtains physiological data in a non-invasive manner. The visual acquisition subunit performs eye tracking to analyze fixation point, saccade path, and pupil diameter changes in order to assess attention allocation and cognitive load, and obtain visual assessment data. The auditory acquisition subunit acquires speech commands, analyzes the fluency, speed, intonation, and accuracy of the speech, evaluates language and executive functions, and obtains auditory assessment data. The data synchronization and preprocessing subunit performs timestamp alignment, filtering, denoising, and initial feature extraction on the multidimensional data to form a structured multimodal time-series data stream, resulting in preprocessed multidimensional data.

3. The digital therapy system for cognitive impairment based on multimodal interaction and dynamics as described in claim 1, characterized in that, The intelligent analysis and decision-making layer includes: The cognitive ability feature engineering sub-model extracts high-order features that are highly correlated with specific cognitive domains from preprocessed multidimensional data; The multi-dimensional cognitive profile generation sub-model maps the extracted high-order features to a multi-dimensional cognitive ability space, generating a dynamic digital cognitive ability profile for each user. The dynamic baseline modeling sub-model, based on a dynamic digital cognitive ability profile, establishes a personalized ability baseline model for each user. The personalized solution generator, based on dynamic digital cognitive ability profile and ability baseline model, prioritizes the areas with the most severe deficits in the cognitive profile, selects and combines corresponding training tasks from the intervention resource library, and adjusts the presentation of training tasks and formulates rehabilitation strategies by combining users' historical preference data on game type, difficulty and theme. The adaptive difficulty adjuster analyzes user performance in real time during a single training task and dynamically optimizes the rehabilitation strategy.

4. The digital therapy system for cognitive impairment based on multimodal interaction and dynamics as described in claim 3, characterized in that, The higher-order features include the working memory capacity index, attention maintenance curve, conflict inhibition reaction time cost, and the slope of error rate growth over time.

5. The digital therapy system for cognitive impairment based on multimodal interaction and dynamics as described in claim 3, characterized in that, The multidimensional cognitive ability space encompasses core dimensions related to memory, attention, executive function, information processing speed, and visuospatial ability.

6. The digital therapy system for cognitive impairment based on multimodal interaction and dynamics as described in claim 3, characterized in that, The rehabilitation strategy includes gamified training tasks, cognitive behavioral therapy guidance, and personalized schedules with recommendations for daily functional activities.

7. The digital therapy system for cognitive impairment based on multimodal interaction and dynamics as described in claim 3, characterized in that, The adaptive difficulty adjuster dynamically adjusts task parameters to keep the task difficulty in the optimal range that matches the challenge and ability. Task parameters include stimulus presentation speed, number of distractors, and memory load.

Images