Fine exercise evaluation and rehabilitation system and method based on mixed reality technology

Abstract

The invention discloses a fine exercise assessment and rehabilitation system and method based on a mixed reality technology. The system comprises a mixed reality glasses module used for data acquisition and data analysis and providing interaction and feedback functions at the same time; the arm motion capture module is used for collecting motion data of the arm; a depth camera is arranged in the mixed reality glasses module, and a real scene data acquisition module, a virtual-real registration module, a hand data acquisition module, a hand posture recognition module, a real-time tracking module, a key node detection module, an evaluation and analysis module, a display and interaction module, a teaching module and a multi-mode feedback module are further arranged in the mixed reality glasses module. By means of the mixed reality technology, in combination with precise posture tracking, visual multi-mode guiding and optimized interactive design for the cognitive impairment patient, automatic and gamified evaluation and auxiliary rehabilitation of the fine movement ability of the upper limbs and the hands are achieved.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation medicine technology, and in particular to a fine motor assessment and rehabilitation system based on mixed reality technology. Specifically, it is a game system for automated assessment and assisted rehabilitation of upper limb fine motor abilities in patients with cognitive impairment. Background Technology

[0002] Stroke, or cerebrovascular accident for short, is an acute cerebrovascular disease caused by the sudden rupture or blockage of blood vessels in the brain, preventing blood flow and resulting in brain tissue damage. According to the World Health Organization, stroke is the second leading cause of death worldwide after heart disease and is a leading cause of disability in adults. Approximately 15 million people worldwide suffer a stroke each year, with ischemic stroke accounting for about 85% of all cases. In China, the incidence and prevalence of stroke continue to rise, becoming a major public health problem threatening national health and safety. The high incidence, high disability rate, high mortality rate, and high recurrence rate of stroke impose a heavy economic burden and psychological stress on society, families, and individuals.

[0003] Following a stroke, patients often suffer from a series of severe sequelae, significantly impacting their quality of life and self-care ability. These sequelae encompass multiple aspects, including motor dysfunction, sensory disturbances, speech impairments, cognitive impairments, and emotional disturbances. Among these, upper limb motor dysfunction is particularly common and has a significant impact on daily life, manifesting as unilateral limb weakness, limited mobility, or even complete loss of mobility, often accompanied by sensory impairment or loss. Patients require upper limb involvement in basic physiological activities such as eating, dressing, and personal hygiene. Impaired upper limb function makes it difficult for them to care for themselves, leading to severe dependence on others for care. This not only increases the caregiving burden on families and society but also negatively impacts the patient's mental health and motivation for rehabilitation.

[0004] Because the human central nervous system has extremely limited regenerative capacity after injury, neuronal necrosis caused by stroke is usually irreversible. When cerebral blood flow is interrupted or blood vessels rupture, causing cerebral ischemia and hypoxia, neurons can die within a short period of time, and their axons and synaptic connections break down. Factors such as glial scarring after injury and a lack of neurotrophic factors jointly inhibit effective nerve regeneration. Therefore, there are currently no specific drugs or surgical methods that can directly repair or regenerate necrotic nerve cells. Clinical treatment mainly focuses on promoting neuroplasticity through rehabilitation training, utilizing the brain's compensatory and reorganization capabilities to restore damaged functions as much as possible and improve the patient's quality of life.

[0005] Traditional methods of motor rehabilitation and assessment face numerous limitations in practice. Firstly, in terms of assessment, many traditional scales are not sensitive enough to the subtle motor signals in early-stage patients, especially those in the flaccid paralysis phase, exhibiting a "floor effect" and making it difficult to accurately quantify subtle functional improvements. Automated assessment methods for upper limb fine motor skills are particularly lacking, with existing automated assessments focusing primarily on lower or upper limb gross motor skills. Secondly, in terms of rehabilitation training, traditional physical therapy and occupational therapy often require high levels of muscle strength and cognitive cooperation from patients. Some stroke patients with severe cognitive impairment struggle to understand therapist instructions or accurately reproduce assessment or training movements through video demonstrations, leading to poor rehabilitation outcomes. Furthermore, while virtual reality-based rehabilitation systems enhance engagement and interactivity to some extent, some patients may experience dizziness or discomfort during use. The completely virtual nature of VR environments may also reduce patients' sense of security and realism, resulting in decreased cooperation. Finally, in the absence of a professional rehabilitation therapist, compensatory movements or abnormal muscle coordination during assessment or training are often difficult to detect and correct in a timely manner, affecting the accuracy and effectiveness of rehabilitation. Traditional rehabilitation methods also suffer from problems such as long processing time, high reliance on human resources, monotonous process, poor patient compliance, and high cost.

[0006] Mixed reality (MR) technology integrates virtual information with real-world scenarios, bringing new possibilities to post-stroke motor function assessment and rehabilitation. Compared to virtual reality (VR), MRT allows users to interact with virtual objects while seeing real-world scenes. This reduces the likelihood of dizziness during use, and the ability to perceive the real environment provides a greater sense of security, resulting in more natural and realistic interactions. MRT systems can overlay virtual guidance information related to assessment and training tasks onto the patient's real field of vision, making task guidance clearer and more explicit, reducing the excessive demands on the patient's cognitive abilities, making it particularly suitable for patients with cognitive impairments. Simultaneously, the MRT environment can provide immediate and rich positive feedback, enhancing patient engagement and rehabilitation motivation. By accurately tracking the patient's limb movements, MRT systems can also provide robust motor data feedback, providing a basis for quantitative assessment and the development of personalized rehabilitation plans. This approach, combining virtual assistance with real-world perception, holds the promise of overcoming some limitations of traditional rehabilitation and VR rehabilitation, improving the engagement, compliance, and effectiveness of rehabilitation assessments.

[0007] However, existing mixed reality-based rehabilitation systems still have shortcomings in the automated assessment of upper limb fine motor skills for specific patient groups. For example, they lack effective solutions in areas such as how to more effectively utilize mixed reality technology to reduce cognitive load, how to accurately capture and analyze complex upper limb and hand fine movements, and how to design more targeted assessment tasks and feedback mechanisms. Therefore, it is necessary to develop a system that fully leverages the advantages of mixed reality technology, specifically designed for patients with cognitive impairment, to achieve automated, gamified assessment and assisted rehabilitation of upper limb, especially hand, fine motor skills, in order to address the problems existing in current technologies. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and to provide a fine motor assessment and rehabilitation system based on mixed reality technology, which addresses the deficiencies in the fine motor assessment of the upper limbs of patients with cognitive impairment. This system can reduce cognitive load and achieve automated assessment and fun rehabilitation.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] A fine motor assessment and rehabilitation system based on mixed reality technology, suitable for patients with cognitive impairment, includes:

[0011] The mixed reality glasses module is used for data collection and analysis of all collected data, while also providing interactive and feedback functions;

[0012] The arm motion capture module is used to collect arm motion data and transmit this data to the mixed reality glasses module;

[0013] The mixed reality glasses module includes a depth camera and also includes:

[0014] The real-scene data acquisition module is used to acquire 3D point cloud data of real scenes;

[0015] The virtual-real registration module is used to identify the boundaries of planes and objects in the 3D point cloud data of real scenes and refine the relative positions between real scenes and virtual scenes.

[0016] The hand data acquisition module is used to acquire three-dimensional point cloud data of the hand.

[0017] The hand pose recognition module is used to process hand 3D point cloud data and use a deep learning model to segment and recognize it, and further obtain the 3D coordinates of key nodes of the hand skeleton.

[0018] The real-time tracking module is used to calculate the three-dimensional coordinates of key nodes of the arm skeleton based on the arm acceleration and angular velocity data collected by the arm motion capture module. Combined with the above three-dimensional coordinates of key nodes of the hand skeleton, the kinematic information of the upper limb is obtained according to the inverse kinematics algorithm. Combined with the preset upper limb model, the virtual upper limb is constructed.

[0019] The key node detection module is used to perform task analysis on preset assessment or rehabilitation movements, and judge the achievement of the movements in real time based on the preset upper limb model thresholds to trigger the next node or provide error correction feedback, and provide immediate feedback based on the detection results.

[0020] The assessment and analysis module is used to comprehensively analyze the collected arm movement data and hand 3D point cloud data using machine learning models, so as to realize accurate rehabilitation assessment, movement parameter analysis and health progress recording functions.

[0021] The display and interaction module is used to display and provide virtual interaction, voice guidance, adaptive difficulty adjustment and gaze confirmation functions;

[0022] The teaching module is used to demonstrate correct movements using virtual upper limbs, supplemented by voice prompts and progress displays;

[0023] The multimodal feedback module combines tactile, visual, and auditory feedback to present the results of the key node detection module.

[0024] Furthermore, it also includes a terminal module, which includes mobile and fixed terminals, for receiving, storing, and visualizing data. During visualization, the terminal module provides the exported data to researchers or clinicians in a standard format for subsequent quantitative analysis, model training, or generating more detailed clinical reports.

[0025] Furthermore, the arm motion capture module includes:

[0026] The arm data acquisition module includes one or more inertial measurement units, each equipped with a three-axis accelerometer and a three-axis gyroscope for acquiring the arm's acceleration and angular velocity data; the arm data acquisition module also includes a magnetometer and an infrared sensor for calibrating the data;

[0027] The arm data transmission module is used to transmit the data collected by the arm data acquisition module to the mixed reality glasses module.

[0028] Furthermore, the information acquired by the depth camera includes depth image and color image information;

[0029] The virtual-real registration module uses a random sampling consensus algorithm for plane detection, a Euclidean clustering segmentation algorithm for object boundary recognition, and an iterative nearest point algorithm for pose refinement when the judgment is not up to standard.

[0030] The hand gesture recognition module employs a deep learning model that combines convolutional neural networks and long short-term memory networks.

[0031] Furthermore, the real-time tracking module fuses the three-dimensional coordinates of key nodes of the hand bones with the three-dimensional coordinates of key nodes of the arm bones; it uses an inverse kinematics algorithm to convert the fused data into the three-dimensional spatial position and posture information of the key nodes of the upper limb in a preset coordinate system, and reconstructs the motion trajectory of the upper limb segments to obtain the kinematic information of the upper limb; it then constructs a virtual upper limb by combining a preset upper limb model, which simplifies the arm into a multi-rigid-body system composed of several links and joints.

[0032] Furthermore, the key node detection module can perform real-time feature extraction on the data collected by the arm motion capture module and the hand data acquisition module; the real-time extracted features include spatial position features and joint angle features. Spatial position features include the envelope area formed by the user's actual movement trajectory and the dwell time of the user's limb in the target area; joint angle features include the flexion angle of the elbow joint or the abduction angle of the shoulder joint.

[0033] The error correction feedback of the critical node detection module includes visual prompts for local joint angles that do not meet the standards.

[0034] Furthermore, the machine learning models in the evaluation and analysis module include random forest regression models or support vector machine classification models; the motion parameter analysis performed in the evaluation and analysis module includes muscle rehabilitation analysis, neurological rehabilitation analysis, fine motor ability analysis, and gross motor ability analysis.

[0035] Furthermore, in the display and interaction module, the adaptive difficulty adjustment function refers to automatically simplifying the difficulty of an action after a set number of consecutive failures at a certain task node, or skipping the current task or node by setting an instruction; the gaze confirmation function refers to an interaction method that confirms options based on the user's gaze duration; it also provides an augmented reality-assisted guidance function to guide the user to correctly wear the mixed reality glasses module or related sensors; and it also provides a physiological signal monitoring function to automatically pause the task when abnormal physiological indicators are detected.

[0036] Furthermore, the visual feedback of the multimodal feedback module includes highlighting key objects and progress bar prompts; the auditory feedback includes sound prompts; and the tactile feedback includes feedback from touching objects.

[0037] Preferably, the present invention also provides a method for fine motor assessment and rehabilitation, comprising:

[0038] S1. Wearing mixed reality glasses and arm motion capture module;

[0039] S2. Access the virtual-real registration module using the display and interaction module;

[0040] S3. The virtual-real registration module processes the acquired 3D point cloud data of the real scene to achieve accurate registration between the real scene and the virtual scene;

[0041] S4. The teaching module begins teaching, including virtual hand demonstration, voice stimulation, and progress display;

[0042] S5. The real scene data acquisition module and the hand data acquisition module acquire real scene 3D point cloud data and hand 3D point cloud data respectively, and the arm data acquisition module acquires arm acceleration and angular velocity information;

[0043] S6. The hand pose recognition module processes the obtained 3D point cloud data of the hand to obtain the 3D coordinates of key nodes of the hand skeleton;

[0044] S7. The real-time tracking module uses the extracted arm acceleration and angular velocity information to calculate the three-dimensional coordinates of key nodes of the arm skeleton, and combines the above three-dimensional coordinates of key nodes of the hand skeleton to obtain upper limb kinematic information for evaluation using the inverse kinematics algorithm. It can also combine the preset upper limb model to construct a virtual upper limb for recording teaching animations.

[0045] S8. The key node detection module makes a judgment based on the obtained upper limb kinematic information, detects in real time whether the movement reaches each key node in the preset movement sequence, and provides immediate feedback based on the detection results.

[0046] S9. The multimodal feedback module provides multimodal feedback based on the detection results of the key node detection module, including key object highlighting, progress bar prompts, sound prompts, and touch object feedback;

[0047] S10. The assessment and analysis module performs a comprehensive quantitative assessment based on the collected upper limb kinematic information and the results returned by the key node detection module, and completes the analysis of movement parameters and the recording of health progress.

[0048] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows:

[0049] 1. This invention achieves 3D point cloud data acquisition based on a mixed reality glasses module, thereby solving problems such as dizziness caused by traditional VR systems and difficulty in accurately capturing hand and scene information.

[0050] Depth cameras can acquire depth and color images with high precision within their effective range, enabling accurate registration between real and virtual scenes, ensuring spatial alignment accuracy, reducing dizziness, improving patient safety and cooperation, and enhancing the user's immersive experience and sense of security.

[0051] 2. The arm motion capture module in the system of this invention is equipped with an inertial measurement unit, a magnetometer, and an infrared sensor, which solves the problem that single visual tracking is easily affected by occlusion and lighting, resulting in incomplete recognition of key nodes of the upper limb skeleton.

[0052] By fusing multiple sensors (accelerometer, gyroscope, magnetometer, infrared positioning) to calculate the three-dimensional coordinates of each node of the arm in real time, and by using sensor fusion algorithms to eliminate drift and noise, we can ensure stable and continuous acquisition of arm motion information in complex environments, providing reliable data for subsequent posture recognition.

[0053] 3. The hand pose recognition module is based on a deep learning model that combines a convolutional neural network (CNN) and a long short-term memory network (LSTM) to solve the problems of difficulty in recognizing subtle hand movements and the susceptibility of traditional algorithms to background clutter interference.

[0054] By first preprocessing the original depth and color images through noise filtering and ROI extraction, and then combining convolutional neural networks and long short-term memory networks for hand region segmentation and temporal learning, the key points of finger bones and gesture changes can be accurately identified, improving the accuracy of fine motion capture. Using a deep learning model to estimate the 3D skeletal key points of the hand in real time effectively improves the accuracy and robustness of hand pose recognition, avoiding the effects of occlusion and lighting that plague traditional 2D tracking.

[0055] 4. This invention constructs a virtual upper limb using an inverse kinematics fusion algorithm, solving the problem that it is difficult to construct complete upper limb kinematic information using only single sensor data, which affects the accuracy of the assessment.

[0056] After fusing key node data of the hand and arm bones, the inverse kinematics are solved based on quaternion algorithm and Levenberg-Marquardt optimization to reconstruct the virtual upper limb. The preset upper limb model simplifies the arm into a multi-rigid-body system composed of several links and joints, which can realize real-time virtual teaching and provide accurate joint angles and spatial positions, providing a high-precision basis for evaluation and feedback.

[0057] 5. The key node detection module extracts and detects spatial location features and joint angle features in real time, solving the problems of low sensitivity of traditional rehabilitation assessment scales and severe "floor effect" for patients in the early stage of flaccid paralysis.

[0058] It extracts multimodal features such as motion trajectory envelope area, dwell time, and elbow-shoulder joint angle in real time, and uses threshold judgment to finely break down the action stage, triggering error correction feedback in a timely manner, significantly improving the sensitivity and objectivity of detecting subtle motion improvements. Through multi-dimensional quantitative evaluation of motion quality and efficiency, it achieves refined and objective functional assessment.

[0059] 6. An evaluation and analysis module based on random forest regression or SVM classification models addresses the problems of single-rule evaluation lacking personalization and multi-dimensional analysis, and difficulty in quantifying rehabilitation effects.

[0060] By leveraging machine learning models to comprehensively analyze multidimensional features such as muscle, nerve, and motor abilities, we can achieve quantitative assessments of muscle rehabilitation, neural reconstruction, and fine and gross motor skills, generating structured reports to provide data support for clinical decision-making and personalized treatment plans.

[0061] 7. The multimodal feedback module adopts a multimodal feedback method of visual highlighting + progress bar + voice prompts + tactile feedback to solve the problems of weak response to single feedback channels, easy comprehension and operation difficulties for patients with cognitive impairment.

[0062] This module combines visual, auditory, and tactile multi-sensory feedback. Visually, it highlights key objects or target areas, and a dynamic progress bar intuitively displays the completion status. The voice prompts are concise and clear, and the tactile indicators enhance the interactive experience, reduce cognitive load, and improve completion rate, cooperation, and training interest.

[0063] 8. Adaptive difficulty adjustment and gaze confirmation interaction address the issues of limited instruction comprehension and loss of confidence among patients with cognitive impairment due to repeated failures. The benefits are reflected in the system automatically simplifying actions or allowing skipping after a certain number of failed detection steps; and option confirmation based on gaze duration, eliminating cumbersome gesture operations and improving system usability and patient compliance.

[0064] 9. The coordinated use of the arm motion capture module and the hand posture recognition module integrates spatial and angular data, overcomes the blind spots of a single sensor, achieves seamless tracking of the entire upper limb movement, significantly improves the accuracy of kinematic reconstruction, and solves the problems of incomplete data and evaluation errors caused by tremors.

[0065] 10. The collaborative use of the virtual-real registration module and the multimodal feedback module first ensures the alignment of the virtual and real scenes through the Random Sampling Consensus Algorithm (RANSAC) + ICP, and then superimposes multi-sensory prompts in the correct coordinate system to make the error correction feedback intuitive and accurate, thus solving the misoperation and safety hazards caused by virtual guidance deviation.

[0066] 11. The combination of the key node detection module and adaptive difficulty adjustment not only provides visual / voice correction prompts after real-time determination of non-compliance, but also automatically adjusts the task difficulty based on the patient's performance and provides a skip option. This effectively avoids the frustration caused by repeated failures and ensures the continuous progress of rehabilitation. It also allows patients at different stages of rehabilitation to receive appropriate challenges and feedback.

[0067] 12. This invention uses a machine learning evaluation model based on multi-source motion and behavior data to achieve automated quantitative evaluation of fine motor functions and output a visual report, thereby improving the objectivity and scientific nature of the evaluation.

[0068] 13. This invention employs multi-dimensional data acquisition, encrypted transmission, and structured storage to meet clinical safety and privacy protection requirements. It can also export illustrated PDF reports for family members and original data packages for professionals, facilitating collaborative rehabilitation and efficacy tracking by interdisciplinary teams.

[0069] In summary, this invention balances perceptual accuracy, intuitive feedback, and ease of use for patients in fine motor assessment and gamified rehabilitation, and is particularly suitable for automated assessment and assisted training of upper limb fine motor function in patients with cognitive impairment. Attached Figure Description

[0070] Figure 1 A schematic diagram showing the wearing positions of the mixed reality glasses module and the arm motion capture module for the user;

[0071] Figure 2 Specific application scenarios of the mixed reality glasses module worn by patients and corresponding scene diagrams within the mixed reality glasses module;

[0072] Figure 3 This is a schematic diagram of the specific components of a fine motor assessment and rehabilitation system.

[0073] Figure 4 This is a schematic diagram of the overall workflow of a fine motor assessment and rehabilitation system. Detailed Implementation

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0075] Example 1

[0076] This embodiment provides a fine motor assessment and rehabilitation system based on mixed reality technology. See Figure 1 and Figure 4The system comprises a mixed reality glasses module 1 and an arm motion capture module 2. The user wears the core mixed reality glasses module 1, which integrates a front-facing depth camera to capture the user's hand's 3D movement trajectory in real time, identify 3D information of the real operating environment, and provide an immersive mixed reality teaching and interactive interface from a first-person perspective. The depth camera must have sufficient accuracy and frame rate to acquire hand depth images and color / infrared images to support subsequent accurate hand pose estimation and gesture recognition algorithms. The mixed reality glasses can clearly and stably overlay virtual information onto the user's real field of vision, providing a visually suitable environment for interaction.

[0077] See Figure 3 The arm motion capture module 2 includes an arm data acquisition module and an arm data transmission module. The arm data acquisition module includes one or more inertial measurement units (IMUs). IMUs are worn on key segments of the upper limb to achieve a comprehensive assessment of the entire upper limb's movement. Each IMU integrates a three-axis accelerometer and a three-axis gyroscope to monitor the posture and movement of the corresponding upper limb segment in real time and calculate the three-dimensional coordinates of key nodes in the arm skeleton. This data is fused with data from the mixed reality glasses module to reconstruct a complete virtual upper limb. The arm data acquisition module also integrates a magnetometer and an infrared sensor. The infrared sensor is used to collect information about the relative position to a fixed reference point in the environment for data calibration.

[0078] The mixed reality glasses module also includes modules for real-scene data acquisition, virtual-real registration, hand data acquisition, hand posture recognition, real-time tracking, key node detection, evaluation and analysis, display and interaction, teaching, and multimodal feedback. Specifically:

[0079] The real-scene data acquisition module is used to acquire 3D point cloud data of real scenes, including depth images and color image information acquired based on a depth camera;

[0080] The virtual-real registration module is used to identify plane and object boundaries in the 3D point cloud data of the real scene and refine the relative position between the real scene and the virtual scene. The virtual-real registration module uses the random sampling consistency algorithm for plane detection and the Euclidean clustering segmentation algorithm for object boundary recognition. When the judgment is not up to standard, the iterative nearest point algorithm is used for pose refinement.

[0081] The hand data acquisition module is used to acquire 3D point cloud data of the hand, including raw depth images and color image data of the hand obtained based on a depth camera;

[0082] The hand pose recognition module first preprocesses the raw image data acquired by the hand data acquisition module, including noise filtering and region of interest extraction. Then, using a deep learning model combining convolutional neural networks and long short-term memory networks, it identifies and segments the hand region from the preprocessed image data, and further estimates the hand's position coordinates in 3D space, as well as the angles and orientations of each finger joint. This module outputs accurate 3D coordinates of key hand skeletal points and overall gesture information, providing foundational data for subsequent motion analysis and interaction.

[0083] The real-time tracking module integrates the 3D coordinates and angles of the complete upper limb from the hand and arm data acquisition modules. Through inverse kinematics algorithms, it drives a pre-defined upper limb model for subsequent teaching and evaluation. The pre-defined upper limb model consists of a skinned mesh and an internal skeletal hierarchy, including basic driving skeletons (shoulder joint skeleton, upper arm skeleton, elbow joint skeleton, forearm skeleton, and wrist joint skeleton) and the complete hand skeleton.

[0084] Furthermore, the inverse kinematics algorithm calculates the angles of each joint based on the known desired position and posture of the hand pose recognition module and the length parameters of each link in the upper limb. In the preset upper limb model, the arm is simplified into a multi-rigid-body system, consisting of the shoulder joint (i=1), elbow joint (i=2), wrist joint (i=3), and links. The end point is the endpoint of the entire kinematic chain, namely the base of the third metacarpal bone of the hand (the projection point of the palm). The end point coordinate system is the hand coordinate system, and the base coordinate system serves as the starting point, which is the shoulder joint coordinate system. The specific inverse kinematics solution can be obtained using analytical methods or numerical optimization methods, as detailed below:

[0085] (1) Definition of coordinate system: Establish a base coordinate system To the end coordinate system The transformation relationship, Represents the transformation matrix between adjacent coordinate systems (from joint i-1 to joint i):

[0086]

[0087] in : is the joint number ( For the shoulder joint, For the elbow joint, (for the wrist joint) Joint angle, The length of the link. For the connecting rod torsion angle, This is the link offset.

[0088] (2) Inverse position solution: Let the coordinates of the base of the third metacarpal bone of the hand in the preset upper limb model be in the base coordinate system. Joint angle satisfy:

[0089]

[0090] Length from shoulder joint to elbow joint Length from elbow to wrist joint. The distance from the wrist joint to the base of the third metacarpal bone in the hand;

[0091] (3) Steps for solving using the analytical method:

[0092] a) Solving for the shoulder joint angle :

[0093]

[0094] in This represents the shoulder and elbow offset.

[0095] b) Solving the elbow joint angle Constructing planar geometric relationships:

[0096]

[0097] in , , r is the projection distance of the base of the third metacarpal bone of the hand onto the xOy plane, R is the planar distance, and φ is the angle between the planes.

[0098] c) Solving for the wrist joint angle :

[0099]

[0100] Where |P E | represents the distance from the base of the third metacarpal bone of the hand to the origin of the base coordinate system.

[0101] (4) Numerical optimization method: Define the objective function:

[0102]

[0103] in, Represents the forward kinematic function. This indicates the desired basal position of the third metacarpal bone of the hand. The damping factor, This is the nominal value of the joint angle.

[0104] The Levenberg-Marquardt algorithm is used for iterative solution:

[0105]

[0106] in It is the identity matrix. For Jacobian matrices, The damping factor, For the desired pose, θ k Let be the joint angle vector of the k-th iteration. By reconstructing the complete upper limb posture, the fine motor assessment and rehabilitation system can generate an upper limb model synchronized with the user's upper limb movements or a preset model within a mixed reality environment for movement teaching. For example, it can display a standard virtual arm to guide the user through specific movements or provide real-time feedback on the accuracy of the user's own movements.

[0107] The key node detection module analyzes preset assessment or rehabilitation movements to break down complex, continuous actions into a series of ordered key movement stages or key posture nodes. The system monitors the user's motion data in real time, acquired by the arm motion capture module and hand posture recognition module. After 3D skeletal tracking is initiated and joint coordinate system calculations are completed, the system performs real-time feature extraction and judges the achievement of the movement based on preset upper limb model thresholds to trigger the next node or provide error correction feedback. Immediate feedback is provided based on the detection results; error correction feedback includes visual prompts for unmet local joint angles in the display and interaction module, or allowing the user to retry the current node within a limited number of attempts.

[0108] Furthermore, the features extracted in real time by the fine motor assessment and rehabilitation system are mainly divided into spatial position features and joint angle features:

[0109] Spatial location features are used to calculate the envelope area formed by the user's actual movement trajectory and the dwell time of the user's limbs within a specific target area.

[0110] Joint angle features are used to monitor the angles of key joints in real time, such as the flexion angle of the elbow joint or the abduction angle of the shoulder joint. These feature values ​​or combinations thereof are used to determine the user's current stage of motion or whether a certain key posture action node has been reached.

[0111] The joint angle feature calculation model is as follows:

[0112] elbow flexion angle Calculation: Let the upper arm vector be... Forearm vector , where P i Represents the coordinates of the center point of joint i.

[0113]

[0114] Shoulder abduction angle Calculation (coronal projection method): First, center the shoulder joint... elbow joint Projecting onto the coronal plane (YZ plane) yields the projection vector. :

[0115]

[0116] shoulder flexion angle Calculation (Sagittal Projection Method): First, project the motion trajectory onto the sagittal XZ plane to obtain the projection components. :

[0117]

[0118] Wrist pronation / supination angle calculation: Establishing a local coordinate system:

[0119]

[0120] Palm normal vector Projection in the local coordinate system:

[0121]

[0122] When the system detects that the user has completed a key action in the current stage or reached a key posture node, it automatically triggers the corresponding feedback mechanism. For example, it may play a guiding animation for the next stage in the mixed reality environment, update the task objective, provide voice prompts or visual highlights to guide the user to complete the entire sequence of actions sequentially. This action decomposition and phased feedback based on key point detection helps reduce the difficulty for patients with cognitive impairment to understand and perform complex actions, and can provide more refined motor performance assessments.

[0123] The assessment and analysis module is used to comprehensively analyze the collected data on the spatial position features and joint angle features of arm and hand movements using machine learning models, so as to realize accurate rehabilitation assessment, movement parameter analysis, and health progress recording functions. The machine learning models include random forest regression models or support vector machine classification models. The movement parameter analysis performed in the assessment and analysis module includes muscle rehabilitation analysis, neurological rehabilitation analysis, fine motor ability analysis, and gross motor ability analysis.

[0124] The display and interaction module displays and provides virtual interaction, voice guidance, adaptive difficulty adjustment, and gaze confirmation functions. The adaptive difficulty adjustment function automatically simplifies the difficulty of actions after a set number of consecutive failures at a task node, or allows skipping the current task or node via a set command. The gaze confirmation function is an interaction method that confirms options based on the user's gaze duration. The module also provides augmented reality-assisted guidance to instruct users on the correct wearing of mixed reality glasses or related sensors. Furthermore, it provides physiological signal monitoring to automatically pause the task upon detecting abnormal physiological indicators.

[0125] The teaching module is used to demonstrate correct movements using a virtual upper limb, supplemented by voice stimulation and progress display. Specifically, a semi-transparent or highlighted virtual hand model is displayed in the patient's field of vision. This virtual hand demonstrates the target movement or the next stage of the movement in a standard, correct posture and at a correct speed. The patient can learn and perform the movements by imitating the virtual hand.

[0126] The multimodal feedback module combines tactile, visual, and auditory feedback to present the results of the key node detection module. These three feedback mechanisms can be used simultaneously, individually, or any two together. Visual feedback includes highlighting key objects and progress bar prompts; auditory feedback includes sound cues; and tactile feedback includes feedback from touching objects. Based on these multiple assistance and feedback mechanisms, presented in a mixed reality environment, the cognitive load on patients with cognitive impairment during motor assessment and rehabilitation training can be reduced, while their participation and task completion accuracy can be improved.

[0127] Specifically:

[0128] Voice prompts provide clear and concise voice instructions and prompts for different stages of an action, key action points, or common error patterns. For example, "Please raise your arm," "Please straighten your fingers," "Insufficient range of motion," etc.

[0129] Key object highlighting is used in assessment tasks involving interaction with objects (such as grasping or placing objects). The system highlights or dynamically indicates the target object or area that needs to be manipulated in the mixed reality view to guide the patient's attention.

[0130] Progress bars are used for continuous actions or multi-step tasks. The system displays an intuitive progress bar or completion indicator to let the patient know the current progress of the task, providing immediate feedback and encouragement. These multimodal assistive devices (their specific presentation methods can be found in...) Figure 2 Working in tandem, it provides patients with intuitive, easy-to-understand, and multi-sensory auxiliary guidance information, effectively assisting them in completing assessments and training.

[0131] Example 2

[0132] This embodiment, based on the fine motor assessment and rehabilitation system in Embodiment 1, focuses on describing the real-scene data acquisition module and the virtual-real registration module. To ensure accurate alignment and stable overlay of virtual information with the real scene in the mixed reality environment, a real-scene calibration technique is provided. This technique aims to establish a digital model of the user's physical space and accurately determine the real-time position and posture of the mixed reality glasses (i.e., the user's viewing angle) within it. Specifically:

[0133] System initialization: Start the depth camera driver and read the preset calibration parameters, including the camera intrinsic parameter matrix K (containing the focal length f). x ,f y and principal point c x ,c y ), and the initial extrinsic parameter estimates of the depth camera relative to the mixed reality glasses module. .

[0134] Data Acquisition and Preprocessing: Depth images and RGB color images are synchronously acquired from a depth camera at a certain frame rate using a real-scene data acquisition module. A 3D point cloud is generated based on this raw data. Specifically, pixel coordinates are first converted to 3D spatial coordinates through coordinate mapping, and then voxel filtering downsampling can be performed to reduce the data volume. Subsequently, noise removal is performed on the point cloud data, for example, using statistical outlier removal algorithms and bilateral filtering algorithms to smooth edges and preserve details.

[0135] Scene segmentation: A random sample consensus algorithm is used for plane detection to identify major planes in the scene, such as horizontal and vertical planes. The validity of detected planes is determined; if an invalid plane is detected, the user is prompted to perform forced manual calibration. Point cloud portions that fail to fit to a plane are considered suspicious object sets.

[0136] Object boundary extraction: Euclidean clustering is used to identify objects in the remaining unfitted point cloud after scene segmentation. Different objects are distinguished by setting the neighborhood radius and the minimum number of cluster points. For each segmented object cluster, its axis-aligned bounding box and convex hull are calculated, and the shape preservation of the convex hull fitting is used as a criterion for validity judgment.

[0137] Coordinate system transformation: The point cloud, planar parameters, and object boundary information obtained in the camera coordinate system are transformed using a homogeneous transformation matrix. Transform to the world coordinate system. Further, map the information in the world coordinate system to a coordinate system and representation compatible with the virtual engine, such as generating the corresponding transformation matrix.

[0138] Boundary determination and refinement: Logical verification is performed on the identified plane and object boundaries. For example, it checks whether the angle between the normal vectors of the tabletop and the wall is close to 90 degrees, and whether the height offset between the bottom of the object and the supporting plane is within a reasonable range. If the verification fails, the iterative closest point algorithm is initiated for pose refinement, such as adjusting the estimated camera extrinsic parameters. Alternatively, the pose of an object relative to a plane can be optimized. The maximum number of ICP iterations can be set to, for example, 50. The algorithm is then checked to determine if the ICP algorithm has converged. If it has, the relevant pose parameters are updated; if it has not converged, a retry prompt is given or a hardware error alarm is issued.

[0139] Calibration Output and Application: After calibration, a configuration file containing the calibration results is generated. This file includes planar parameters, object AABB coordinates, and the final optimized camera extrinsic matrix. The mixed reality engine loads this configuration file to achieve accurate virtual-real occlusion effects and dynamic collider generation. Through this calibration technique, virtual evaluation elements can be accurately presented in the patient's real field of vision and establish correct spatial interactions with real objects.

[0140] Example 3

[0141] This embodiment is based on the fine motor assessment and rehabilitation system in Embodiment 1, focusing on the description of the key node detection module. During the user's execution of specific assessment or rehabilitation movements, it detects in real time whether the user has reached each key node in the preset movement sequence and provides immediate feedback based on the detection results, forming a closed-loop guidance and assessment process, as detailed below:

[0142] 3D skeleton tracking and calculation: After the system starts 3D skeleton tracking, it first performs real-time joint coordinate system calculation, updates the state of each joint through shoulder-elbow-wrist chain forward kinematic calculation, and determines the precise position of the base of the third metacarpal bone of the hand, thereby calculating the real-time coordinates of each joint of the upper limb.

[0143] Real-time multimodal feature extraction: As mentioned above, the system extracts spatial location features and joint angle features from motion data in real time.

[0144] Multimodal feature fusion and completion assessment: A weighted decision model is used to fuse the extracted features. For example, spatial location features and joint angle features are assigned weights. The weighted features are then fused into a comprehensive completion score, which is compared with a preset threshold to determine whether the current action meets the standard.

[0145] Judgment Logic and Feedback Trigger: If the overall completion score reaches or exceeds the threshold, the system determines that the user has accurately completed the action of the current key node. At this time, positive feedback is triggered, such as the currently playing instructional video or animation fading out smoothly, and the system unlocks and triggers the guidance content for the next target key node.

[0146] If the overall completion score fails to reach the threshold, the system enters error correction mode. At this time, targeted feedback may be provided, such as visual prompts in the display and interaction modules using AR arrows to highlight substandard joint angles, guiding the user to correct their actions. The system may also allow users to revert to their previous actions and attempt to complete the current node again, but the maximum number of replays is usually limited.

[0147] Scene switching and parameter reset: When a key node is completed, or when a new action task needs to be switched under specific conditions, the system loads a new action template. This may include rendering a new target trajectory heatmap in a mixed reality environment, or displaying a new safe range of motion for joints. Simultaneously, the system resets counters or state parameters related to kinematic analysis, preparing for the evaluation of the next action or task. This cycle of refined detection and real-time feedback based on key nodes effectively guides patients to gradually complete complex movements and provides detailed process data for automated evaluation.

[0148] Example 4

[0149] This embodiment, based on the fine motor assessment and rehabilitation system in Embodiment 1, focuses on describing the display and interaction module. This module fully considers the difficulties and special needs that patients with cognitive impairment may encounter when using electronic devices, and specifically designs a simple, intuitive, and easy-to-use user interface and corresponding interaction logic. Key design principles and functional phases include:

[0150] System preparation phase: Before the user officially begins, the system can perform environmental checks and device self-tests. Clear voice prompts guide the user through the preparation process, and an easy-to-use ready button is provided.

[0151] Device Wearing Guidance: To ensure correct device wearing, the system provides AR-assisted guidance. For example, a semi-transparent green outline of the glasses' wearing position is displayed in the display and interaction module.

[0152] Virtual Environment Entry and Navigation: Upon entering the system, users are presented with a minimalist main interface. All interactive buttons are large and can be supplemented with visual cues such as dynamic breathing light effects to attract user attention. Voice guidance is supported, with latency detection for user voice commands and a semantically simplified command pattern.

[0153] Intelligent Calibration System: Before the task begins, the system performs intelligent calibration. During the hand calibration phase, it guides the user to assume a specific posture and performs tolerance checks on joint angles. During the environmental registration phase, the system enhances the edges of key objects in the field of view by highlighting them and utilizes multi-object spatial mapping technologies such as ICP to ensure registration accuracy. Positive feedback is given upon successful calibration; if calibration fails, only the abnormal or uncalibrated areas are highlighted to avoid information overload.

[0154] Evaluation of task engine interaction:

[0155] Affected side selection: For unilateral upper limb tasks, the affected side can be automatically selected through gaze interaction.

[0156] Teach loop: During action demonstration, complex actions are broken down and demonstrated at a slower pace. At key action nodes, the teach animation is briefly frozen to allow the user to observe and understand.

[0157] Execution monitoring and feedback: Real-time monitoring of deviations between the user's movement trajectory and the standard trajectory, issuing a warning when the deviation exceeds a dynamically adjusted threshold. Simultaneously, it can integrate physiological signal monitoring; when abnormal physiological indicators are detected, the system can automatically pause the task to ensure user safety.

[0158] Adaptive logic: If a user fails a set number of times consecutively at a certain task node, the system can automatically simplify the difficulty of the action or provide a voice "skip" command, allowing the user to skip the current difficult task or node.

[0159] Report Generation and Presentation: Upon completion of the task, the assessment results are presented in a visual summary format. For example, a star-shaped rating system using a five-dimensional radar chart visually displays the user's performance across different ability dimensions, and animations can be used to compare the current assessment results with the previous ones, highlighting trends of improvement. The assessment report offers multimodal output, generating a visually appealing and easy-to-understand PDF report for family members, as well as clinical data packages for further analysis by professionals. The UI design and interaction logic are designed to minimize the operational threshold and cognitive load for patients, enabling them to use the system independently or with minimal assistance for assessment and rehabilitation training.

[0160] Example 5

[0161] This embodiment, based on the fine motor assessment and rehabilitation system in Embodiment 1, focuses on describing the arm data acquisition module, arm data transmission module, hand data acquisition module, hand posture recognition module, and terminal module. Through multi-dimensional analysis of data acquisition, processing, transmission, and export, it demonstrates that this system not only performs motor assessment but also simultaneously acquires, processes, securely transmits, and ultimately exports various related data from the assessment process. This facilitates further development of clinical research, efficacy analysis, and personalized rehabilitation plans. Specifically, it includes:

[0162] Data collection:

[0163] Hand joint data: The raw data obtained from the hand posture recognition module is converted into joint angles represented by Euler angles, and motion characteristics such as the velocity of the base of the third metacarpal bone and the velocity of each finger are calculated.

[0164] Spatial coordinate data: including the 6-DOF positioning and pose information of the mixed reality glasses module itself, as well as optional environmental feature point clouds.

[0165] Timestamp synchronization: To ensure the consistency of multi-source data, the network time protocol is used to align the timestamps of each sensor data stream, and an interpolation algorithm is used to compensate for potential transmission delays.

[0166] Data compression: To reduce the bandwidth pressure of wireless transmission, appropriate compression algorithms are used for different types of data. For example, travel coding can be used for eye-tracking data with high repetition, differential coding can be used for continuously changing joint angle data, and octree structures can be used for compression of 3D point cloud data.

[0167] Security Encryption: To protect user data privacy, encryption is performed before data transmission. End-to-end encryption is performed using the AES-256-bit standard, and data integrity is verified using a hash algorithm.

[0168] Channel selection and reliability assurance: Select the appropriate transmission protocol based on the data type. For data with high real-time requirements but tolerating a small amount of packet loss, UDP protocol with forward error correction mechanism can be used; for data requiring high reliability, TCP protocol can be used for retransmission assurance.

[0169] Data packet segmentation and optimization: Adaptive maximum transmission unit (MTBF) fragmentation technology is employed to optimize packet size based on the current wireless network conditions. Different quality of service (QoS) priority markers can be set for data packets to ensure priority transmission of critical data.

[0170] Transmission status monitoring and adaptive adjustment: Real-time monitoring of wireless transmission status. When the packet loss rate exceeds the preset threshold, an alarm can be triggered or the data transmission resolution can be automatically reduced. At the same time, the data transmission bit rate can be dynamically adjusted according to the available bandwidth.

[0171] Terminal module reception, storage, and visualization:

[0172] The terminal module can be a fixed PC, PAD, mobile phone, etc., and can be connected to the mixed reality glasses module wirelessly / wiredly. The mixed reality glasses module is equipped with a corresponding wireless / wired transmission module.

[0173] Data unpacking and reassembly: After receiving the data packet, the terminal module decrypts, decompresses, and verifies its integrity. A sliding window buffering mechanism aligns the timelines of different data streams and merges multi-source data streams, including eye-tracking, hand-tracking, and spatial data.

[0174] Persistent storage: Raw sensor data can be saved in binary archive format, while processed and structured data is stored in a relational database.

[0175] Real-time visualization: The system can provide the exported data to researchers or clinicians in a standard format for subsequent quantitative analysis, model training, or to generate more detailed clinical reports.

[0176] Example 6

[0177] This embodiment, based on the fine motor assessment and rehabilitation system in Embodiment 1, focuses on describing the assessment and analysis module. This module, based on the comprehensive data obtained from the aforementioned modules, enables the automated assessment of the fine motor function of the patient's upper limbs and hands. The assessment algorithm comprehensively considers multiple dimensions such as the quality, efficiency, and accuracy of the movement, as detailed below:

[0178] (1) Integration and Feature Extraction of Raw Data Streams: First, the synchronous data streams from the arm motion capture module and the depth camera within the mixed reality glasses module are integrated. Based on this integrated data, feature parameters for clinical functional assessment are extracted. For assessment dimensions similar to the FMA-UE (Fugl-Meyer Assessment-Upper Extremity) scale, the following features are extracted:

[0179] (2) Separate movement pattern characteristics: Analyze the movement coordination among the joints of the shoulder, elbow, wrist and fingers. For example, when performing a specific movement, quantify whether other joints have unnecessary linkage (i.e., degree of coordinated movement), or whether there are typical abnormal coordination patterns such as finger flexion accompanied by wrist and palmar flexion.

[0180] (3) Reflex activity related characteristics: Although there is no direct electromyographic signal, rapid involuntary movements of the limbs can be analyzed through high-frequency motion data, or the presence of rhythmic oscillations similar to spastic tremors and their energy can be identified through spectral analysis of the movement trajectory.

[0181] (4) Coordination and speed characteristics: For alternating reciprocating motion, calculate its motion cycle, speed, and rhythm consistency. For target-oriented motion, analyze its trajectory smoothness, number of corrections, and stabilization time or correction time after reaching the target.

[0182] (5) Analysis of arm and hand movement quality:

[0183] Motion smoothness and efficiency: Based on the motion trajectory data of the arm and hand, a motion smoothness index is calculated. The efficiency of completing an effective movement per unit time is analyzed.

[0184] Joint range of motion and accuracy: Calculate the maximum range of motion of major joints such as the shoulder, elbow, and wrist during task execution. Assess the accuracy of the limb extremities in reaching the target position or completing the target posture, such as distance error to the target point / region, posture angle error, etc.

[0185] Hand manipulation precision: Based on the hand skeleton key point trajectory obtained by the depth camera, the stability of fingertip trajectory, the ability to make small adjustments to movement, and the degree of coordination between different fingers are analyzed when simulating the manipulation of small objects or performing fine gestures.

[0186] (6) Multi-dimensional evaluation fusion and model prediction:

[0187] Machine learning model application: The extracted multi-dimensional kinematic feature parameters are used as input and evaluated using a pre-trained machine learning model.

[0188] Comprehensive assessment of motor quality: Combining multiple indicators such as calculated motor smoothness, efficiency, ROM, accuracy, and hand dexterity, the overall motor quality of the patient is comprehensively assessed using weighted scoring or fuzzy logic methods.

[0189] Abnormal movement pattern labeling: Based on the feature analysis results, the system automatically identifies and labels typical abnormal patterns that patients exhibit during movement, such as excessive trunk compensation, insufficient range of motion of joints, discontinuous movement, and abnormal coordination.

[0190] (7) Structured output of clinical reports:

[0191] Digital mapping of clinical scales such as FMA-UE: Integrating the predicted scores of machine learning models with key kinematic parameters related to each item of FMA-UE obtained by direct calculation (e.g., the quantitative value of reflex activity features is mapped to the "reflex" item 0-2 points, wrist stability parameters such as deviation standard deviation are mapped to the "wrist stability" item 0-2 points, and coordination parameters such as target overshoot times or completion time are mapped to the "coordination" item 0-2 points) to generate digital clinical scale-style reports.

[0192] Kinematic parameter dashboard: Displays the patient's performance and changes over time on key kinematic indicators (such as joint ROM, motion smoothness, trajectory accuracy, task completion time, and hand fine motor skills) in an intuitive manner (e.g., heatmaps, radar charts, progress bars, or trend graphs). The system ultimately generates a comprehensive assessment report including the aforementioned quantitative indicators, model prediction scores, abnormal pattern alerts, and visual charts, providing objective and detailed data support for clinical diagnosis, rehabilitation plan adjustments, and efficacy tracking.

[0193] Example 7

[0194] This embodiment provides a fine motor assessment and rehabilitation method based on the above-mentioned fine motor assessment and rehabilitation system, including:

[0195] S1. Wearing mixed reality glasses and arm motion capture module;

[0196] S2. Perform real-scene calibration and virtual-real scene registration to achieve accurate registration between real and virtual scenes; specifically:

[0197] First, real-world scene calibration and virtual-real registration are performed. This process is crucial for ensuring the accurate and stable fusion of virtual information with the real scene. The core of this process involves: system initialization, loading the necessary mixed reality glasses module runtime framework and camera driver, and reading the inherent parameters and initial pose estimation of the depth camera. Next, the depth camera acquires raw depth images and color or infrared image data of the user's environment in real time. After preprocessing, this data generates 3D point cloud data for subsequent analysis. Then, the system uses robust estimation algorithms to analyze the point cloud to accurately identify and segment the main planes in the scene, such as desktops and walls, providing a benchmark for the placement and interaction of virtual objects. For point cloud portions that fail to fit to the main planes, segmentation algorithms such as Euclidean clustering are used to further extract potential real-world object boundaries and calculate their geometric descriptions, such as axis-aligned bounding boxes or convex hulls. After extracting this crucial geometric information, the system unifies the information obtained in the camera coordinate system to the world coordinate system through precise coordinate transformation and further maps it to the virtual environment coordinate system used by the mixed reality glasses module. After the conversion, to further improve registration accuracy, the system performs boundary judgment and logic verification, and employs algorithms such as the iterative nearest point algorithm to finely optimize the pose of the mixed reality glasses or the relative pose of virtual objects with respect to the real scene, ensuring a high degree of consistency in geometric relationships. Finally, the output calibration results are saved and loaded into the mixed reality glasses module to achieve accurate virtual-real occlusion effects and the generation of dynamic physical colliders based on the real scene. Through this meticulous calibration and registration process, the system ensures that the mixed reality glasses module can accurately perceive the real 3D environment scene in which the user is located, i.e., the real scene's 3D point cloud data. This provides a solid foundation for the accurate placement of all subsequent virtual elements and the natural interaction between virtual and real objects, effectively supporting core interactive needs such as immersive teaching and accurate evaluation.

[0198] S3. Begin evaluation and measurement using the display and interaction modules;

[0199] S4. The teaching module begins teaching, including virtual hand demonstration, voice stimulation, and progress display;

[0200] S5. The depth camera acquires 3D point cloud data of the real scene and 3D point cloud data of the hand, while the arm data acquisition module acquires arm acceleration and angular velocity information;

[0201] S6. The hand pose recognition module processes the acquired 3D point cloud data of the hand to obtain the 3D coordinates of key nodes of the hand bones. During specific assessment or rehabilitation tasks, the system uses the front-facing depth camera of the mixed reality glasses to perform real-time hand pose and orientation recognition. This depth camera acquires depth images and color or infrared images of the user's hand in real time. After necessary image preprocessing, it is analyzed using a deep learning model (such as a deep learning model based on convolutional neural networks and long short-term memory networks) to accurately resolve the 3D spatial coordinates of more than 20 key bone points of the user's hand and the overall gesture.

[0202] S7. The aforementioned high-precision three-dimensional spatial information of the hand is the foundation for achieving fine motion assessment and natural interaction. Simultaneously, to capture the overall movement of the upper limb (arm) other than the hand, one or more inertial measurement units worn on the user's arm collect its angular velocity and linear acceleration data in real time. This data is processed using sensor fusion algorithms and kinematic calculations to obtain the three-dimensional coordinates of key nodes in the arm skeleton. Combined with the aforementioned three-dimensional coordinates of key nodes in the hand skeleton, complete upper limb kinematic information is obtained using an inverse kinematics algorithm. This information, combined with a pre-set upper limb model, is then used to construct a virtual upper limb.

[0203] Through precise inverse kinematics solving, the system can not only accurately calculate the real-time state of each joint in the user's upper limbs, but also drive a pre-set upper limb model to generate a natural, biomechanically sound sequence of movements that precisely matches the task objective. This sequence is then presented to the user in a mixed reality environment from a user-friendly and easily understood perspective. For example, a patient might see a semi-transparent virtual hand or arm demonstrating a grasping or lifting motion in front of their field of vision. This clear and synchronized virtual movement instruction presented in a mixed reality environment significantly reduces the difficulty for patients with cognitive impairments to understand and imitate assessment or rehabilitation movements, effectively guiding their attention to key motor elements and achieving the core goal of reducing cognitive load through immersive instruction using mixed reality technology.

[0204] S8. The key node detection module makes a judgment based on the obtained complete upper limb kinematic information, detects in real time whether the movement reaches each key node in the preset movement sequence, and provides immediate feedback based on the detection results;

[0205] S9. The multimodal feedback module provides multimodal feedback based on the detection results of the key node detection module, including key object highlighting, progress bar prompts, sound prompts, and touch object feedback;

[0206] S10. The assessment and analysis module performs a comprehensive quantitative assessment based on the collected complete upper limb kinematic information and the results returned by the key node detection module, and completes the analysis of movement parameters and the recording of health progress.

[0207] Preferably, addressing the common cognitive impairment in early stroke patients, which makes it difficult for them to understand and execute complex instructions, a core strategy of this embodiment is to decompose complex assessment or rehabilitation sequence of movements into a series of more manageable key movement stages or nodes. The system achieves this goal through a closely coordinated movement stage identification and dynamic feedback module and a key node detection and feedback loop module, constructing a structured, feedback-driven interactive process designed to facilitate patients' understanding, execution, and learning of movements, thereby supporting neural remodeling. Specifically, after the user's upper limb and hand 3D skeletal tracking is initiated and real-time joint coordinate system calculation is completed, the system extracts key feature parameters from the user's movement data in real time for the current task stage. These feature parameters are mainly divided into spatial position features and joint angle features. Subsequently, these real-time extracted multimodal feature values ​​are fused through a preset weighted decision model to calculate a comprehensive current movement completion score. This score is compared with a preset target threshold for the current key node. If the user's movement performance causes the comprehensive completion score to reach or exceed the preset threshold, the system determines that the user has accurately and sufficiently completed the movement requirements of the current key node. At this point, the system will trigger positive feedback, such as the currently playing instructional video or animation fading out smoothly, and the system will automatically unlock and trigger the guidance content for the next key objective node. Conversely, when the overall completion score fails to reach the preset threshold, indicating that the user's current action has not met the standard, the system will enter error correction and assistance mode. In this mode, more targeted feedback may be provided, such as using augmented reality arrows, highlighting incorrect body parts, or visualizing the difference between the ideal and actual trajectories in the display and interaction modules to provide intuitive prompts for the user's substandard joint angles or movement paths, guiding the user to actively correct their actions. The system may also allow users to rewind and retry to complete the current node's requirements within a limited number of attempts. When a key objective node is successfully completed, or when switching to a new action task is necessary due to the user's inability to complete the action, the system will smoothly load a new action template. This may include rendering a new target motion trajectory heatmap in a mixed reality environment or displaying a new safe range of motion for the joints. Simultaneously, the biomechanical feature counters or state parameters related to the task will be reset, preparing for accurate evaluation of the next action or task.

[0208] Taking a typical upper limb function assessment task, "reaching-grasping-moving-placing," as an example, the system first guides the user to perform the initial key step of "reaching out." By monitoring the spatial characteristics of the user's arm extension and the angle characteristics of the shoulder and elbow joints in real time, the system determines whether the user has successfully completed the "reaching out" action. Once the target is met, the system smoothly transitions to guiding the user to the next key step, "grasping," and updates the virtual teaching content and feedback mechanism accordingly. This process is repeated until the entire multi-step task sequence is completed. At each stage, the system provides comprehensive and clear guidance through a multimodal assistance and feedback module. This includes, but is not limited to:

[0209] Virtual upper limb teaching: Clearly and accurately displaying standard movement patterns and the target posture or trajectory for the next stage in the user's field of vision.

[0210] Concise and clear voice commands and prompts: Provide real-time voice feedback based on the current task stage, user performance, or common error patterns. For example, prompt "Please reach forward" at the beginning stage, provide positive feedback such as "Great, please proceed to the next step: hold the cube" after successfully completing a node, or provide corrective prompts such as "Please raise it a little higher" or "Please keep your shoulders relaxed" when the user's movement is insufficient or when compensation occurs.

[0211] Visual highlighting of key objects or areas: When assessment or rehabilitation tasks involve interaction with specific virtual or real objects, the system will highlight the target object or target area that the user's limbs should reach by flashing, changing color, or adding dynamic directional arrows on the screen. This will effectively guide the user's attention and reduce the cognitive load of finding and locating the target.

[0212] Real-time feedback on task progress and completion: Visual elements such as dynamically filled progress bars, gradually increasing star ratings, or continuously accumulating scores allow users to understand their current task completion status and overall performance in real time and intuitively, thereby enhancing their sense of participation, accomplishment, and motivation to continue. This design, which breaks down complex tasks into manageable stages and provides precise guidance through clear, immediate, and multi-sensory feedback, is one of the effective strategies of this invention for addressing the rehabilitation needs of patients with cognitive impairment. It aims to promote patients' in-depth understanding, precise execution, and effective learning of target actions through structured task presentation and rich sensory input, thereby supporting the core goal of promoting neural remodeling through a closed-loop system of "perception-assessment-interaction-rehabilitation."

[0213] Users interact with the system through a user interface and logic specifically designed for patients with cognitive impairment. This design is consistent throughout the entire usage process: from initial system preparation, to AR-assisted device wearing guidance and haptic feedback verification, to entering the minimalist main interface featuring large buttons and dynamic breathing light effects. Throughout this process, the system supports semantically simplified voice guidance. Before starting a task, users undergo an intelligent calibration process, including verification of hand posture and precise registration with the environment, with dedicated visual feedback for both successful and failed calibrations. During task evaluation, adaptive logic such as affected side selection, 50% slower decomposition teaching and 3-second keyframe freeze, trajectory deviation alarms and circuit breakers based on physiological signals such as heart rate, and automatic simplification of actions after three failures or allowing the voice command "Next" to skip the task, are all designed to minimize operational difficulty and cognitive load. Upon task completion, the system presents a visual summary report and can output PDF reports easily understood by family members and raw data packages for professional analysis. These designs collectively ensure that patients, even those with cognitive impairment, can use the system relatively independently or with minimal assistance.

[0214] Throughout the assessment and rehabilitation process, the system records various user data through multi-dimensional data acquisition, processing, transmission, and export modules. Specifically, the mixed reality glasses are responsible for real-time acquisition of various sensor data, including optional eye-tracking data, joint data output from the hand posture recognition module, and the glasses' own spatial positioning data. The raw data acquired undergoes a series of preprocessing steps on the glasses, including NTP-based timestamp synchronization, targeted data compression, AES-256 end-to-end encryption, and SHA-256 integrity verification. The preprocessed data is then transmitted to the computer via an optimized wireless transmission protocol. Upon receiving the data, the computer performs decryption and decompression, data stream merging, timeline realignment, and other operations before persistently storing the data. This data can be exported for further analysis, and the computer can also selectively perform real-time visualization to assist in monitoring or adjustments.

[0215] Finally, through the automated assessment and analysis module, the system comprehensively analyzes the collected data to quantify the user's motor function level. For example, it calculates joint range of motion, compares the difference between the actual movement trajectory and the standard trajectory, and evaluates the coordination and efficiency of the movement, ultimately generating an assessment report that includes various indicators and a comprehensive score.

[0216] This embodiment achieves automated and gamified assessment and assisted rehabilitation of the fine motor abilities of the upper limbs of patients with cognitive impairment in a mixed reality environment through the coordinated work of the above modules. It has the advantages of reducing cognitive load and improving the objectivity and fun of the assessment.

[0217] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A fine motor assessment and rehabilitation system based on mixed reality technology, suitable for patients with cognitive impairment, characterized in that, The application relates to a mixed reality system, comprising: a mixed reality glasses module for data collection and analysis of all collected data, and providing interaction and feedback functions; an arm motion capture module for collecting arm motion data and transmitting the data to the mixed reality glasses module; the mixed reality glasses module is internally provided with a depth camera, and further comprises: a real scene data collection module for collecting real scene three-dimensional point cloud data; a virtual-real registration module for identifying the plane and object boundary in the real scene three-dimensional point cloud data, and refining the relative position between the real scene and the virtual scene; a hand data collection module for collecting hand three-dimensional point cloud data; a hand posture recognition module for processing the hand three-dimensional point cloud data, and using a deep learning model to segment and recognize the hand three-dimensional point cloud data, and further obtaining hand skeleton key node three-dimensional coordinates; a real-time tracking module for calculating the arm skeleton key node three-dimensional coordinates based on the arm acceleration and angular velocity data collected by the arm motion capture module, combining the hand skeleton key node three-dimensional coordinates, and obtaining upper limb kinematics information according to an inverse kinematics algorithm, and combining a preset upper limb model to complete the construction of a virtual upper limb; a key node detection module for triggering the next node or providing error correction feedback by judging the motion achievement according to the threshold of the preset upper limb model in real time according to task analysis of preset evaluation or rehabilitation motions, and providing instant feedback according to the detection result; an evaluation and analysis module for comprehensively analyzing the collected arm motion data and hand three-dimensional point cloud data by means of a machine learning model, realizing accurate rehabilitation evaluation, motion parameter analysis and health progress record functions; a display and interaction module for displaying and providing virtual interaction, voice guidance, adaptive difficulty adjustment and gaze confirmation functions; a demonstration module for demonstrating correct motions by means of the virtual upper limb, and assisting with voice encouragement and progress display; a multi-modal feedback module for presenting the results of the key node detection module in combination with the feedback modes of touch, vision and hearing.

2. The mixed-reality technology based fine motor assessment and rehabilitation system according to claim 1, characterized in that, The application further comprises a terminal module, the terminal module comprising a mobile terminal and a fixed terminal, and being used for realizing data receiving, data storage and visualization, wherein in the process of visualization, the terminal module provides the exported data in a standard format to researchers or clinicians, and is used for subsequent quantitative analysis, model training or generation of more detailed clinical reports.

3. The mixed-reality technology based fine motor assessment and rehabilitation system according to claim 1, wherein, The arm motion capture module comprises: an arm data collection module comprising one or more inertial measurement units, each of which is internally provided with a three-axis accelerometer and a three-axis gyroscope, and is used for collecting arm acceleration and angular velocity data; the arm data collection module is further internally provided with a magnetometer and an infrared sensor to calibrate the data; an arm data transmission module for transmitting the data collected by the arm data collection module to the mixed reality glasses module.

4. The mixed-reality technology based fine motor assessment and rehabilitation system according to claim 1, wherein, The information collected by the depth camera comprises depth image and color image information; the virtual-real registration module adopts a random sample consensus algorithm for plane detection, adopts a Euclidean clustering segmentation algorithm for object boundary identification, and adopts an iterative closest point algorithm for pose refinement when the threshold is not met. The hand posture recognition module adopts a deep learning model combining a convolutional neural network and a long short-term memory network.

5. The mixed-reality technology based fine motor assessment and rehabilitation system according to claim 1, wherein, The real-time tracking module fuses the three-dimensional coordinates of the hand skeleton key nodes with the three-dimensional coordinates of the arm skeleton key nodes, converts the fused data to obtain the three-dimensional space position and posture information of the upper limb key nodes in a preset coordinate system using an inverse kinematics algorithm, reconstructs the motion trajectory of the upper limb segments to obtain upper limb kinematics information, and constructs a virtual upper limb in combination with a preset upper limb model.

6. The mixed-reality technology based fine motor assessment and rehabilitation system according to claim 1, characterized in that, The key node detection module can perform real-time feature extraction on the data collected by the arm motion capture module and the hand data acquisition module. The real-time extracted features include spatial position features and joint angle features, the spatial position features include an envelope area formed by the actual motion trajectory of the user, and the residence time of the user's limbs in the target area; The joint angle features include the flexion angle of the elbow joint or the abduction angle of the shoulder joint; 7. The mixed-reality technology based fine motor assessment and rehabilitation system according to claim 1, wherein, The error correction feedback of the key node detection module includes visual prompts for local joint angles that do not meet the standard.

8. The mixed-reality technology based fine motor assessment and rehabilitation system of claim 1, wherein, The machine learning model in the evaluation and analysis module includes a random forest regression model or a support vector machine classification model.

9. The mixed-reality technology based fine motor assessment and rehabilitation system of claim 1, wherein, The motion parameter analysis performed in the evaluation and analysis module includes muscle rehabilitation condition analysis, neural rehabilitation condition analysis, fine motor ability analysis, and gross motor ability analysis.

10. A fine motor assessment and rehabilitation method based on the fine motor assessment and rehabilitation system according to any one of claims 1 to 9, characterized in that, In the display and interaction module, the adaptive difficulty adjustment function refers to automatically simplifying the action difficulty after a certain task node fails for a set number of times, or skipping the current task or node by setting instructions. The gaze confirmation function refers to an interactive way of confirming options based on the user's gaze duration. An augmented reality auxiliary guidance function is also provided to guide the user to correctly wear the mixed reality glasses module or related sensors. A physiological signal monitoring function is also provided to automatically pause the task when an abnormal physiological indicator is detected. The visual feedback of the multi-modal feedback module includes key object highlighting and progress bar prompts. The auditory feedback includes sound prompts. The tactile feedback includes touch object feedback. The method comprises the following steps: S1. Wear the mixed reality glasses module and the arm motion capture module; S2. Enter the virtual-real registration module with the help of the display and interaction module; S3. The virtual-real registration module processes the obtained real scene three-dimensional point cloud data to achieve accurate registration of the real scene and the virtual scene; S4. The demonstration module starts demonstration, including virtual hand demonstration, voice encouragement, and progress display; S5. The real scene data acquisition module and the hand data acquisition module respectively acquire real scene three-dimensional point cloud data and hand three-dimensional point cloud data, and the arm data acquisition module acquires arm acceleration and angular velocity information; S6. The hand posture recognition module processes the obtained hand three-dimensional point cloud data to obtain the three-dimensional coordinates of the hand skeleton key nodes; S7. The real-time tracking module uses the extracted arm acceleration and angular velocity information to calculate the three-dimensional coordinates of the key nodes of the arm skeleton, and combines the three-dimensional coordinates of the key nodes of the hand skeleton to obtain upper limb kinematics information for evaluation using an inverse kinematics algorithm, and can also construct a virtual upper limb based on a pre-set upper limb model for recording of demonstration animation; S8. The key node detection module determines whether the motion reaches each key node in the pre-set motion sequence based on the obtained upper limb kinematics information, and provides immediate feedback based on the detection result; S9. The multi-modal feedback module provides multi-modal feedback based on the detection result of the key node detection module, including key object highlighting, progress bar prompting, sound prompting, and touch object feedback; S10. The evaluation and analysis module comprehensively quantitatively evaluates based on the collected upper limb kinematics information and the returned result of the key node detection module, and completes motion parameter analysis and health progress recording.

Images