Apparatus and method for enabling community-customized post-stroke rehabilitation

Abstract

The invention discloses a post-stroke rehabilitation system for providing community customization. The post-stroke rehabilitation system comprises a user side module, a cloud platform module and a therapist side module. The user-side module is configured to provide schedule information having instructions for at least one particular exercise and to display at least one image or video, where the user-side module provides a view page of the camera to record a target image or video and to record performance metrics of a target user. The cloud platform module is configured to receive and store performance indicators of the target user from the client module. The therapist end module is allowed to log in the cloud platform module and receives performance metrics of the target user from the cloud platform module, and the therapist end module is further configured to visualize the performance metrics of the target user, thereby displaying motion waveforms including quantitative data for qualitative analysis.

Description

Technical Field

[0001] The present invention generally relates to digital validation of human activity assessment. More specifically, the present invention relates to a method and apparatus for validating upper limb exercises based on human activity assessment facilitated by consumer-grade digital cameras, and also for developing a therapist-guided automated telerehabilitation framework and a stroke rehabilitation platform. Background Art

[0002] Stroke is a global health problem that can cause a large amount of disability and death. As the prevalence of stroke increases, innovative rehabilitation solutions are urgently needed to meet the growing demand and mitigate the impact on public health systems. Among them, there are three main challenges from the following aspects.

[0003] I: Global burden of stroke and need for rehabilitation services:

[0004] Stroke is the leading cause of disability and death worldwide. Timely and adequate rehabilitation is essential to promote stroke recovery and ultimately affect people's quality of life. With an aging population and rising prevalence of non-communicable diseases, including stroke, the global demand for rehabilitation services continues to increase. During the COVID-19 pandemic in 2019, face-to-face rehabilitation services were severely affected globally, further limiting the rehabilitation potential of stroke survivors. At that time, the World Health Organization also declared that strengthening rehabilitation services was a top priority; therefore, innovative solutions were urgently needed to improve the accessibility of stroke rehabilitation services and reduce the burden on public healthcare.

[0005] II: Current strategies to promote motor rehabilitation in community settings:

[0006] In addition to traditional face-to-face rehabilitation services, the COVID-19 pandemic has also demonstrated a significant increase in the actual adoption rate of telemedicine and telerehabilitation around the world, which is specifically reflected in various professional fields such as cardiology, respiratory medicine, neurology and orthopedics. These platforms can overcome physical, functional and economic barriers to the continuation of rehabilitation services. Barriers include high costs, difficulty in traveling to face-to-face service centers, additional pressure on caregivers, and lack of incentives or motivation.

[0007] Current telerehabilitation solutions rely heavily on subjective reporting via video conferencing or telephone interviews, which can be labor-intensive and susceptible to recall bias. Other technology solutions utilize sensor-based (e.g., Xsens and smartwatches) and camera-based (e.g., Vicon and Kinect) devices to obtain objective measurements and have demonstrated good prediction and measurement of motor performance, such as range of motion, including home motor rehabilitation. These systems do show benefits in improving functional outcomes for stroke patients in addition to traditional treatments. However, these system approaches still have shortcomings in practical home telerehabilitation applications due to their high cost per unit and large footprint.

[0008] III: Advanced technologies to facilitate home exercise rehabilitation

[0009] Based on the recent developments in artificial intelligence, sensor technology and hardware specifications, this has enabled the potential of consumer-grade products (i.e. smartphones and tablet devices) to be tapped as an affordable and accessible alternative. Human pose estimation is a state-of-the-art computer vision method, such as Figure 1 As shown, it can be used to segment and obtain key information about human joints in space, including shoulders, elbows, knees, etc.

[0010] Figure 1 The plot highlights the key joints of the body frame predicted using computer vision algorithms. The target key joints of interest include the head, shoulder, elbow, wrist, hip, knee, and ankle joints. The figure shows a sample frontal view of the motion waveforms predicted by different posture estimation models when a participant performs left shoulder abduction on the right. This view is plotted with all 13 key joints together.

[0011] According to the diagram, this involves the process of identifying and predicting the position and orientation of human body parts using images or videos captured by sensors (e.g., red, green, and blue (RGB) digital cameras, depth cameras, infrared sensors). Human pose estimation models initially relied on high-computing-power devices to run complex algorithms in order to achieve optimal performance. Currently, more complex and compressed models exist and are able to achieve similar performance on a variety of consumer devices.

[0012] Several devices have implemented camera-based motion-related rehabilitation. However, these devices require a console to run their pose estimation models on the environment captured by the built-in infrared camera. In addition, a relatively large spatial area is required to adequately capture the target user and the surrounding environment. These combined factors make this approach difficult to scale up and use for home exercise, especially in low-and middle-income countries (LMICs). Summary of the invention

[0013] The purpose of the present invention is to provide a method and device for a smartphone and paired with artificial intelligence to promote community-customized post-stroke rehabilitation. Timely and adequate rehabilitation activities are very important for promoting recovery after stroke. However, the organization and implementation of rehabilitation require a lot of resources, and only about 25% of stroke patients in low- and middle-income countries can receive rehabilitation services. Therefore, there is an urgent need for innovative solutions to improve the accessibility of stroke rehabilitation services. Telerehabilitation can transfer care to home and community settings and has become a promising solution. However, current methods using video tutorials, teleconferencing or other dedicated equipment face inherent defects that limit their adoption.

[0014] This disclosure proposes and validates the use of open source, markerless motion capture models and consumer-grade devices to overcome these challenges. The proposed solution is able to reliably measure the end range of motion during upper limb exercises, and has near-perfect waveform similarity and intra-class correlation with the gold standard Kinect method. In this disclosure, an automated telerehabilitation framework that combines proven markerless technology is developed to facilitate a seamless telerehabilitation process. The proposed technical solution uses objective quantitative and qualitative features to implement customized rehabilitation plans and provides real-time feedback and personal progress reports to improve patient monitoring and management, as well as the acceptance and compliance of home rehabilitation services. Further combined with training modes (such as cognitive and voice) and other Internet of Things (IoT) devices (such as biometric sensors for monitoring vital signs), the overall telerehabilitation process can also be facilitated. This disclosure serves as a proof of concept to prepare for future development of detailed care models, as well as feasibility, usability and cost-effectiveness studies of automated telerehabilitation platforms and frameworks in improving post-stroke rehabilitation and functional outcomes.

[0015] To overcome the above challenges, the present disclosure proposes a remote rehabilitation platform that can be accessed via a smartphone or tablet. This cutting-edge solution enables users to participate in personalized rehabilitation programs at home, providing customized guidance, real-time automatic feedback, and progress reports to effectively improve patient treatment outcomes.

[0016] Figure 2The development of information and communication systems using systems such as 5G IoT and cloud services is depicted. With these systems, "mobile health" or "mHealth" has rapidly developed into a field involving the use of mobile devices such as smartphones, tablets, and wearable devices in the provision and management of healthcare. To address hardware limitations and allow accessibility and scalability to promote community rehabilitation, the RGB camera built into smartphone devices can be utilized. Computer vision models are able to predict the locations of key points in the human body and space (e.g., the x and y coordinates of the elbow joint in space at time t), which is valuable for evaluating a person's rehabilitation performance. The body frame key points are then used to infer joint angles, velocities, motion smoothness, and other properties to objectively and automatically evaluate performance. By automating the evaluation process and incorporating gamification elements into the feedback system, the provided technology solution can act like a customized "smart therapist" to promote home rehabilitation.

[0017] By developing a smartphone model, the aim is to minimise the physical barriers to traditional face-to-face services provided by rehabilitation day centres. Smartphone-based rehabilitation assessment allows stroke patients to start their training sessions anytime and anywhere, thus matching rehabilitation sessions to personal schedules with great flexibility. Therefore, a platform called SmartRehab system was designed that acts as an AI therapist to analyse and assess an individual’s performance by leveraging computer vision models, including image processing, object detection and posture estimation, to facilitate high-quality home rehabilitation exercises. The model employed captures images of the patient, identifies key points of the patient’s body frame and analyses performance using biometric data and joint angles. During the exercise, the SmartRehab system provides real-time visual and audio feedback to correct any inaccuracies in the exercise, simulating the therapist providing commentary during face-to-face rehabilitation sessions, using metrics and rules developed by experts from the Hong Kong Society for Rehabilitation (HKSR) and the University of Hong Kong Stroke Centre. In addition, connecting sensor-based headphones during rehabilitation allows for direct auditory guidance and feedback. Potential uses include respiratory rate and pulse oximeter functionality for stroke, heart disease and respiratory patient populations for remote rehabilitation.

[0018] A cloud-based approach can be adopted to upload and store rehabilitation data. The responsible therapist can then remotely access and view the patient's progress, and even prescribe and modify rehabilitation exercise plans to promote customized treatment. Cloud-based systems and automatic assessments using artificial intelligence models allow remote and telerehabilitation in the community, greatly reducing the pressure on therapists and reducing waiting times for stroke patients.

[0019] Therefore, the SmartRehab system can be used and expanded by the community as an application on smartphones and smart device platforms. The proposed technology provides a unique and innovative solution for stroke patients who need rehabilitation, especially in the COVID-19 era, when rehabilitation services have been severely affected worldwide.

[0020] According to a first aspect of the present invention, a system for providing community-customized post-stroke rehabilitation is disclosed. The system includes a user-end module, a cloud platform module, and a therapist-end module. The user-end module is configured to provide schedule information, which has instructions for at least one specific exercise and displays at least one image or video, wherein the user-end module provides a view page of a camera to record a target image or video and record the performance indicators of the target user. The user-end module includes a feed module, a human activity recognition module, and a human activity evaluation module. The feed module is configured to obtain images or videos from a consumer-grade smartphone or tablet device. The human activity recognition module is configured to receive images or videos from the feed module and analyze input data using a human posture estimation model. The human activity evaluation module is configured to receive results transmitted from the human activity recognition module and evaluate performance based on performance indicator information, wherein the performance indicator information is formulated based on rules or templates. The cloud platform module is configured to receive and store the performance indicators of the target user from the user-end module. The therapist-end module communicates with the user-end module through the cloud platform module. The therapist-side module is allowed to log into the cloud platform module and receive the performance indicators of the target user from the cloud platform module, and the therapist-side module is also configured to visualize the performance indicators of the target user, thereby displaying motion waveforms including quantitative data for qualitative analysis.

[0021] According to a second aspect of the present invention, a method for providing community-customized post-stroke rehabilitation is disclosed. The method comprises the following steps: providing schedule information having at least one instruction for a specific exercise through a user-end module; displaying at least one image or video through the user-end module, wherein the user-end module provides a view page of a camera to record a target image or video and to record performance indicators of a target user. The user-end module comprises: a feed module configured to acquire the image or video from a consumer-grade smartphone or tablet device; a human activity recognition module configured to receive the image or video from the feed module and analyze input data using a human posture estimation model; and a human activity evaluation module configured to receive a result transmitted from the human activity recognition module and evaluate performance based on performance indicator information, wherein the performance indicator information is rule-based or template-based. The method also includes the following steps: receiving and storing the performance indicators of the target user from the user-side module through the cloud platform module; using the therapist-side module to record the performance indicators of the target user into the cloud platform module to receive the performance indicators of the target user from the cloud platform module; and using the therapist-side module to visualize the performance indicators of the target user to display motion waveforms containing quantitative data for qualitative analysis.

[0022] Through the above configuration, the present disclosure proposes a more accessible and affordable solution, which uses an improved posture estimation model that relies only on consumer-grade equipment combined with a human activity assessment method to automate the assessment of large-scale motor exercises after stroke. The proposed solution has been validated against the existing and widely accepted Microsoft Kinect posture estimation model, and a remote rehabilitation care model has been developed using a validated automated assessment method, laying the foundation for future trials to determine the effectiveness of such a platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings, in which:

[0024] Figure 1 Prediction maps of key joints of the body frame using computer vision algorithms are presented;

[0025] Figure 2 The information and communication system diagram for system development using 5G IoT and cloud services was presented;

[0026] Figure 3 A schematic diagram of a data collection method in some embodiments of the present invention is shown;

[0027] Figure 4A schematic diagram of a system framework designed to facilitate telerehabilitation in some embodiments of the present invention is shown;

[0028] Figure 5 According to some embodiments of the present invention, Figure 4 A diagram of the user interface of the mobile application of the system;

[0029] Figure 6 According to some embodiments of the present invention, Figure 4 A schematic diagram of the therapist portal system of the SmartRehab system;

[0030] Fig. 7A and Figure 7B Presents visualizations of motion waveforms captured during shoulder abduction and elbow extension; and

[0031] Figure 8 A schematic diagram of a system architecture for community-customized post-stroke rehabilitation in one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] In the following description, systems and methods for implementing community-based customized post-stroke rehabilitation, etc. will be described in the form of preferred examples. Those skilled in the art will appreciate that modifications, including additions and / or substitutions, may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the present disclosure is written to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0033] First, the preparatory work and related work of the present invention are described.

[0034] (1) Data Collection

[0035] Figure 3 A schematic diagram depicting a data collection method in some embodiments of the invention is shown. Participants were video recorded performing the thirteen listed rehabilitation exercises shown in Table 1 below using a consumer-grade smartphone and an infrared camera device in a well-lit environment. Figure 3 Shown are the front view (left) and side view (right) of the recording setup, which consists of a consumer smartphone device stacked above an infrared camera and raised to a height of 1 meter above the ground using a tripod.

[0036] The stacked devices were aligned and calibrated using the built-in leveling tool, and the consumer smartphone was inverted to minimize the distance between the RGB digital camera and the infrared camera. The devices were supported by a tripod 1 meter above the ground to minimize tilt and video distortion. Each recording session involved the participant performing ten repetitions of all listed rehabilitation exercises to the best of their ability. The consumer smartphone device and the infrared camera recorded video at 60 frames per second (fps). Table 1. Checklist of major motor exercises after stroke # Included in this invention and verified

[0037] 2) Mobile healthcare

[0038] In this disclosure, "mobile health" or "mHealth" refers to the use of mobile devices (such as smartphones, tablets, and wearable devices) to provide medical services and information. The development of mHealth has been driven by technological advances, the increase in the use of mobile devices, and the growing demand for remote and customized medical care.

[0039] In the early stages of mHealth, the focus was mainly on text messages and voice calls to provide health information and reminders to patients. Today, mHealth is used in a variety of healthcare settings, including disease prevention, diagnosis, treatment and management. It has shown outstanding use in improving healthcare services in remote and underserved areas and providing more customized care for patients.

[0040] Some of the key benefits of mHealth include increased patient engagement, improved health outcomes, and reduced healthcare costs. However, there are challenges with mHealth, such as ensuring data privacy and security, addressing disparities in the use of mobile technology, and integrating mHealth into existing healthcare systems. Overall, the development of mobile health is a major advancement in healthcare.

[0041] 3) Pose estimation and data preprocessing

[0042] In some embodiments, pose estimation and data preprocessing are performed using software, such as MATLAB and Python. Two-dimensional (2D) video recordings captured using gold standard and consumer devices are used, which can be input into at least one pre-trained model, such as the Kinect pre-trained pose estimation model and the open source pre-trained MediaPipe pose estimation model. The x and y coordinates of the thirteen key joints are obtained and converted into a time series data format to synchronize the two sets of data, thereby generating two 2*13 matrix blocks based on the original spatial position of the joints relative to the capture device resolution (refer to Figure 1 , which has 26 degrees of freedom). The time series data of the x and y coordinates of the key joints were passed through a high pass filter using the MATLAB Signal Processing Toolbox to remove random spikes in the key point coordinates from the data frame that could not be represented. It is worth noting that no smoothing techniques were applied to the time series data, and the following data was processed in raw format to represent the real-time and practical use of the posture estimation model in evaluating rehabilitation performance. After filtering and synchronizing the Kinect and MediaPipe skeleton data outputs, the video recordings can be reviewed and specific recording time periods can be manually marked and segmented into corresponding exercise segments to facilitate the validation analysis process, which is shown in Table 1 above.

[0043] Furthermore, in the present disclosure, 2D pose estimation refers to the process of identifying the position and orientation of a human body in a 2D image or video. The development of 2D pose estimation technology has been greatly improved by the advancement of computer vision and machine learning algorithms, as well as the improvement of hardware performance.

[0044] For algorithm development, in the early stages of 2D pose estimation, simple algorithms were used that detected body parts based on color and intensity. Over time, more sophisticated techniques were developed, including model-based methods that use statistical models of human body shape and appearance to estimate pose. In some embodiments, deep learning methods (such as convolutional neural networks (CNNs)) can be used for 2D pose estimation. These methods have achieved state-of-the-art performance on challenging datasets and can estimate poses in real time on a variety of devices.

[0045] 4) Validate the accuracy of consumer-grade computer vision models against the Kinect gold standard

[0046] In the present disclosure, the performance of key joints predicted by the open source MediaPipe pose estimation model is compared with the performance of the Kinect gold standard pose estimation model. The waveform similarity of the segmented time series of each movement in the pose estimation model is checked using Pearson's correlation coefficient. In order to evaluate the ability of the MediaPipe pose estimation model in estimating and predicting the range of motion (ROM) endpoints, three representative estimated ROM endpoints were manually selected from ten repetitions of each movement. In some embodiments, the estimated ROM endpoints can be calculated using trigonometric properties (such as Formula 1), while combining key joints into a vector (for example, linking the wrist and elbow key joints into a forearm vector a, and linking the elbow and shoulder key points into a proximal arm vector b to calculate the elbow angle for elbow extension). Next, the root mean squared error (RMSE) is used, which analyzes the error between the ROM endpoint prediction performance of different pose estimation models.

[0047] Among them, θ = the angle of ∠ABC; a = the vector from joint A to B; b = the vector from joint B to C.

[0048] In some embodiments, intraclass correlation (ICC) was performed using mean ratings (k=2), absolute agreement, and a two-way mixed effects model to analyze the agreement between posture estimation models in predicting ROM endpoints. Values ​​less than 0.5 indicate poor reliability, values ​​between 0.5 and 0.75 indicate moderate reliability, values ​​between 0.75 and 0.9 indicate good reliability, and values ​​greater than 0.90 indicate excellent reliability. All statistical analyses were performed using MATLAB, and data were reported with 95% confidence intervals. For this, a p-value < 0.05 was statistically significant.

[0049] 5) Sensor

[0050] In this disclosure, a sensor device refers to a device that is capable of detecting and measuring a physical or chemical phenomenon, such as temperature, pressure, light, sound, or motion. The development of sensor devices has been enhanced by advances in materials science, microelectronics, and wireless communications.

[0051] Simple sensors can be used to measure physical quantities, such as thermometers and pressure gauges. Over time, more complex sensors have been developed, including accelerometers, gyroscopes, and magnetometers, which are commonly used in smartphones and wearable devices. The miniaturization of sensors has also enabled the development of smart sensors, which can process data and make decisions based on the measured data. In addition, wireless communication technology has enabled the deployment of sensor networks, which consist of multiple sensors that can communicate with each other and with a central server or cloud-based platform. Sensor networks can be used in a variety of applications, including smart homes, smart cities, and industrial automation.

[0052] In recent years, the development of sensors has been driven by the rise of the Internet of Things (IoT), which refers to the network of physical devices, vehicles, buildings and other objects embedded with sensors, software and connectivity. The IoT enables the integration of sensor data with other data sources, such as social media, weather data and traffic data, creating new insights and opportunities for new developments.

[0053] 6) Development of remote rehabilitation nursing model

[0054] Given the urgent need for innovative solutions to advance rehabilitation services after stroke, in this disclosure, a novel solution is provided to identify the limitations faced by current approaches that may be addressed by validated pose estimation models. Therefore, a framework and example solution are developed that implement camera-based pose estimation techniques and human activity assessment methods for automated home-based stroke rehabilitation.

[0055] Next, the architecture design of the present invention is provided as follows.

[0056] Figure 4 A schematic diagram of a framework for facilitating telerehabilitation according to some embodiments of the present invention is shown. In this regard, core technologies include human activity recognition (HAR) and human activity evaluation (HAE) on mobile devices, statistical methods, and network and database design and deployment.

[0057] Architecture Design Overview: A system called "SmartRehab System" is presented that aims to address the limitations faced by current post-stroke rehabilitation methods and the available literature and products mentioned above. The presented system combines interdisciplinary technologies and expert knowledge from the Internet of Things (IOT), computer vision (CV), physical therapy, and neurology. Figure 4 There are three parts in the diagram. The system can be accessed through the cloud (see Figure 4Part (b)) connects to the user end (see Figure 4 part (a)) and the therapist side (see Figure 4 part (c)) of the study, thereby promoting customized and high-quality post-stroke rehabilitation programs.

[0058] User-side mobile applications in architecture design:

[0059] Figure 5 According to some embodiments of the present invention, it is shown Figure 4 The user interface of the mobile application of the system. Figure 5 There are four parts in the diagram. Figure 5 The illustrated portion (a) is the home page, which displays the time schedule for the day. The schedule page lists the active exercise plan prescribed by the therapist, such as various large-scale exercises. Figure 5 The illustrated portion (b) is a preview page that displays a video and written instructions for a specific exercise. The instructions page includes a video demonstration provided by a physical therapist and written instructions on how to perform the exercise. Figure 5 The diagram part (c) is the camera view page, which interacts with HAR and HAE to evaluate the patient's rehabilitation quality. The camera page can use AI and deep learning technology to track the user's movements and provide real-time posture estimation, range of motion estimation, exercise evaluation and feedback functions for repetition counting. Figure 5 The illustrated portion (d) is the Statistics page, which displays statistics and the progress of the patient's performance indicators. The Results page provides a performance summary of the completed exercise session and records the results and sends them back to the therapist.

[0060] More specifically, for Figure 5 In the user end shown in parts (a) and (b) of the diagram, the patient can enter a rehabilitation exercise course as prescribed by the physical therapist and set specific goals (i.e., ten left shoulder abductions per day). Using the provided human posture estimation (HPE) and activity assessment modules, the user end can automatically evaluate a person's rehabilitation performance. Figure 5 As shown in the diagram part (c), the HPE model can extract relevant information and predict the key points of the human body from each camera input frame. The provided HPE process can be completed without the need for additional markers or trackers. The predicted key points can then be used to infer human skeleton, biometrics, and other motion data.

[0061] In some embodiments, the predicted human key points can be used in the HPE process to infer relevant biometric data for analysis and evaluation based on the gold standard determined by experts from the Hong Kong Society for Rehabilitation (HKSR) and the Stroke Centre of the University of Hong Kong.

[0062] For example, information such as joint angles, velocity, acceleration, motion smoothness, distance to key points, etc. can be recorded. By using the inferred body data, the evaluation model can accurately compare the time series data with the gold standard template, or check the joint angles against the gold standard rules. This data is then passed to the scoring, feedback, and repetition counter functions. This summary data can be used to provide real-time visual or audio feedback to the user to correct and reinforce high-quality rehabilitation performance, such as Figure 5 In one embodiment, gamification elements are also incorporated into the user interface to make the use process more enjoyable and exciting, motivating and promoting home rehabilitation exercises.

[0063] Therapist-side web portal system in architecture design:

[0064] Figure 6 According to some embodiments of the present invention, Figure 4 Schematic diagram of the therapist portal system of the SmartRehab system in FIG. 1 , which is used to objectively review the patient's motor performance. The portal system includes "patient information", "activity map", "rehabilitation plan", "progress record" and "statistics" functions. These functions can be viewed as a web page or through an application (e.g., smartphone operation).

[0065] Patient information is used to provide a brief summary of the patient's condition and rehabilitation goals.

[0066] The activity map is used to visualize daily adherence and completion of prescribed exercise using a traffic light system; for example, it can default to: red = not completed, yellow = partially completed, green = completed).

[0067] The Rehabilitation Plan is a calendar format that is used to display proactively prescribed exercise plans and execution prescriptions for new plans.

[0068] Progress logs are used to record the therapist's actions on the patient's data, including editing patient information or prescriptions, or modifying exercise plans.

[0069] Statistics are used to review exercise performance measured and estimated by the posture estimation model: Range of Motion: average angle or distance of interest achieved during home exercises (e.g., angles between the shoulder and elbow and vertical during shoulder abduction); Quality of Performance: detection of skillful movements (e.g., recognized movements proposed by the Hong Kong Society for Rehabilitation (HKSR)); Counts: number of repetitions; Motion Trajectory: for visualization and qualitative analysis of key exercise waveforms (e.g., angles between the shoulder and elbow and vertical during shoulder abduction).

[0070] Furthermore, the therapist-side web portal system in the architecture design also includes:

[0071] After completing the rehabilitation course, users can upload their rehabilitation performance data to the cloud service through the cloud path, such as Figure 4 As shown in the diagram part (b).

[0072] Therapists and healthcare professionals can remotely access patient performance data by logging into a password-protected and encrypted portal, such as Figure 4 The diagram part (c) and Figure 6 The web portal system is designed to allow therapists to remotely navigate and monitor the patient's performance and progress. The portal system includes a summary of the patient's detailed information (e.g., patient details, active disease, rehabilitation goals, and relevant past medical history).

[0073] In various embodiments, the web portal system may have an activity map that employs a traffic light reporting system to visualize the patient's activity on their assigned or prescribed exercises; for example: "green light" for completion of all prescribed exercises; "yellow light" for partial completion; and "red light" for incomplete. The activity map is used to monitor patient compliance and identify the potential for patients who may be experiencing difficulty using the platform. The progress log records the therapist's actions on the patient's profile, including editing patient information or prescriptions or modifying exercise plans, etc.

[0074] In various embodiments, the web portal system may include a statistics dashboard that displays three main data domains of importance in facilitating telerehabilitation (e.g., domains determined by experts from the Hong Kong Society of Rehabilitation (HKSR)). These areas include "range of motion" of the joints of interest for a particular exercise, "exercise quality" compared to the experts' standards and metrics, and "repetition count". Therapists can regularly monitor and follow up on the patient's performance and compliance through the web portal, and further adjust the difficulty and intensity of their exercises to facilitate a seamless telerehabilitation process. At the same time, when the scoring function shows that the patient has achieved sufficient performance, the therapist can continue to modify the plan and increase the difficulty to further provide a higher level of challenge to the user. Through the portal system, therapists can easily modify editable content, including exercise type, target number of repetitions, exercise duration or frequency, and target range of motion, which can all be customized according to individual needs. These modification functions provide great flexibility to develop personalized rehabilitation plans based on the needs of patients.

[0075] Therefore, the SmartRehab system includes two unique interfaces connected through a cloud server, one of which is a mobile application for stroke patients who need rehabilitation services (i.e. the user side); and the other is a web-based portal system for therapists and healthcare professionals to manage patients (i.e. the therapist side).

[0076] In response to the system provided above, the following results and evaluations can be obtained.

[0077] The SmartRehab system can incorporate ten large-scale motion exercises, six of which involve the upper extremities, as shown in Table 2 below. Table 2: List of currently available exercises within the SmartRehab system *Indicates upper limb exercise

[0078] These exercises are recommended and designed by HKSR's therapists and are specifically tailored for stroke patients to improve upper limb function, weight transfer, and balance. Compared with traditional telerehabilitation platforms, such as Kinect, the SmartRehab system can utilize the built-in RGB camera of tablet or smartphone devices. Accordingly, in the technical solution of this article, a computer vision-based pose estimation algorithm is developed and provided, which uses a deep learning model to predict accurate human key points and segment key joint positions.

[0079] Fig. 7A and Figure 7B A visualization of the captured motion waveforms during shoulder abduction and elbow extension is shown. The waveform data can be extracted to calculate angles (x vs. y vs. z in space), angular velocity (θ vs. time), and spasticity (smoothness of the curve). The system provided allows SmartRehab to calculate motion-related features, including motion trajectory (reference Fig. 7A ), changes in joint angles, speed, and occurrence of spastic movements (refer to Figure 7B ), all of which can be used to assess the quality of the patient's movements and provide immediate feedback.

[0080] Furthermore, as mentioned in the introduction, Kinect is a markerless motion capture device that is widely used in validated rehabilitation centers. However, its cost, computational requirements, and need for a specific setup limit accessibility and widespread dissemination in home-based telerehabilitation. Therefore, for better evaluation, the posture estimation model of the SmartRehab system was validated against Kinect. These data points are within the range of impaired upper limb dysfunction (Medical Research Council (MRC) strength score is less than 5 for the proximal upper limb of 44% of patients, and Modified Ashworth Scale (MAS) score is less than 0 for the proximal upper limb of 44% of patients). The results show that the movement trajectory and range of motion between the SmartRehab system and Kinect are similar (refer to Fig. 7A). In addition, the intra-class correlations between the predicted range of motion of the upper limb (abduction, shoulder flexion, and elbow extension) detected using the SmartRehab system and Kinect were all close to 1, as shown in Table 3 below. Such excellent results further demonstrate the promise of automated telerehabilitation and prescription-based platforms. Table 3: Root mean square error (RMSE) and intraclass correlation coefficient analysis of active motion range prediction between Kinect and SmartRehab systems Training Exercise Root mean square error (angle) Intraclass Correlation Shoulder abduction 9.65 0.977(0.965-0.985) Shoulder flexion 9.268 0.971(0.956-0.981) Elbow extension 2.986 0.915(0.863-0.948)

[0081] More results are provided below. During the sensor validation study, several data points were collected, as shown in Table 4 below. Table 4: Demographics and characteristics of stroke patients in the validation study

[0082] Among them, for the nine data points (i.e., nine participants) for data preprocessing and validation analysis, seven were male. The overall median age of the participants was 60 years (IQR: 56-67 years), height was 169 cm (IQR: 165-174 cm), and weight was 67 kg (IQR: 63-70 kg). The average number of days after admission was 97 days (IQR: 65-240) when the assessment and video were taken. Six participants had hemorrhagic stroke and three participants had ischemic stroke. None had experienced a stroke. One participant had paranoid schizophrenia, but with minimal motor manifestations. Four participants required a walker, three of whom required crutches and one required a wheelchair. Regarding motor performance at the time of assessment and video shooting, five participants obtained a complete Medical Research Council (MRC) score for the affected proximal upper limb, two participants had an MRC score of 3, and two participants had an MRC score of 2. The results of muscle tone assessment using the modified Ashworth Scale (MAS) showed that five participants had a MAS of 0, three participants had a MAS of 1, and one participant had a MAS of 3. The global functional disability assessment using the modified Rankin Scale (mRS) showed that six participants had no symptoms to mild disability after stroke (mRS < 3), and three participants had moderate to severe disability (mRS ≥ 3).

[0083] About validation analysis:

[0084] When comparing the pose estimation performance of MediaPipe to Kinect, a perfect correlation was found between the waveform motions for all upper limb movements across all participants, as shown in Table 5 below. Table 5: Pearson correlation between the motion trajectories of the pose estimation model MediaPipe and Kinect during upper limb movement.

[0085] Notably, noticeably smoother motion trajectory and range of motion waveforms can be observed in the MediaPipe pose estimation output without applying any smoothing filters or processing to the data, whereas the Kinect pose estimation output appears more “jagged” (e.g. Figure 1 , Fig. 7A and Figure 7B ). The "jagged" appearance may be due to signal noise captured by the infrared camera.

[0086] The RMSE analysis of the difference in ROM endpoints between MediaPipe and Kinect outputs in shoulder abduction, shoulder flexion, and elbow extension was 9.65, 9.27, and 2.99 degrees, respectively, as shown in Table 6 below. Compared with the Kinect pose estimation model, the ICC analysis showed that the MediaPipe pose estimation model had significant and excellent reliability in predicting ROM endpoints in all exercises. The ICC results for the above exercises were 0.98 (95% CI: 0.96-0.99), 0.97 (0.80-0.99), and 0.92 (0.86-0.95), respectively. Table 6: Range of motion (ROM) between MediaPipe and Kinect pose estimation models during upper limb exercises, root mean square error (RMSE) and intraclass correlation (ICC) analysis of their predictions.

[0087] Figure 8A schematic architecture diagram of a system 100 according to an embodiment of the present invention is shown, which uses SmartRehab to perform a community-customized post-stroke rehabilitation program. The system 100 includes a user-end module 110, a cloud platform module 120, and a therapist-end module 130, which can communicate with the user-end module 110 through the cloud platform module 120. The user-end module 110 is configured to provide schedule information, which has instructions for at least one specific exercise, and displays at least one image or video, wherein the user-end module 110 can provide a view page of a camera to record a target image or video through a camera, and record the performance indicators of the target user. The cloud platform module 120 is used to receive and store the target image or video from the user-end module 110, and store the performance indicators of the target user. The therapist-end module 130 may be allowed to log in to the cloud platform module 120, and receive the target image or video and the performance indicators of the target user from the cloud platform module 120, wherein the therapist-end module 130 is also used to visualize the performance indicators of the target user to display a motion waveform including quantitative data for qualitative analysis. In one embodiment, the performance indicators of the target user include joint angles, speed, acceleration, motion smoothness, key point distances, and combinations thereof.

[0088] More specific details are provided below.

[0089] The client module 110 includes a feed module 112, a human activity recognition module 114, a human activity assessment module 116, and an information receiver 118. The client module 110 can be installed on a smart phone.

[0090] The feed module 112 is configured to obtain photos or videos (e.g., input from an RGB digital camera) from a consumer smartphone or tablet device, which can serve as the primary data source. Once the photos or videos are obtained, they are input into the human activity recognition module 114. The human activity recognition module 114 analyzes the input data using a human posture estimation model implemented using a convolutional neural network (CNN). The human posture estimation model can identify and predict the position and orientation of each body joint, providing detailed information about the participant's movement.

[0091] For example, a human pose estimation model can extract relevant information and predict key points of the human body from each camera input frame. This process can be completed without additional markers or tracers. The predicted key points can then be used to infer the human skeleton, biometrics, and other motion data. In some embodiments, the human pose estimation process uses the predicted human key points to infer relevant biometric data for analysis and evaluation. For example, the information used by the human pose estimation model includes joint angles, velocities, accelerations, motion smoothness, and key point distances.

[0092] In multiple embodiments, the human pose estimation model includes a computer vision model that can predict the position of the human body and key points in space (e.g., the x and y coordinates of the elbow joint in space at time t). The body frame key points are then used to infer joint angles, speeds, motion smoothness, and other properties, and performance is objectively and automatically evaluated. The human pose estimation process can use a deep learning model to predict accurate human key points and segment key joint locations.

[0093] After the analysis, the results are transmitted to the human activity assessment module 116. The human activity assessment module 116 assesses the performance based on the performance indicator information, which can be rule-based or template-based. The assessment results from the human activity assessment module 116 can be transmitted to the therapist end module 130 via the cloud platform module 120.

[0094] In this regard, various rehabilitation exercises require attention to specific assessment angles. By focusing on the angles formed by the three key points associated with each exercise and referring to the thresholds set by the Hong Kong Society for Rehabilitation (HKSR), patients' rehabilitation exercises can be effectively evaluated. For example, in the case of shoulder abduction, two angles are concerned: left elbow-left shoulder-left hip and right elbow-right shoulder-right hip.

[0095] Assume that the coordinates of the right elbow point are (x1, y1); the coordinates of the right shoulder point are (x2, y2); and the coordinates of the right hip point are (x3, y3). Then, two vectors can be determined: V1=((x1-x2),(y1-y2)) V2=((x3-x2),(y3-y2))

[0096] Next, the key point angles can be calculated as follows: θ=cos -1 [(V1·V2) / (|V1||V2|)]

[0097] The average angle and score are calculated as the final evaluation result for each batch of exercises, where each batch can consist of n repetitions. With this evaluation method, the provided SmartRehab system is able to transmit the patient's evaluation data from their portable device to the rehabilitation expert's system. Therefore, the therapist can easily monitor the patient's rehabilitation progress and make real-time adjustments as needed.

[0098] In actual application environments, patients may make inaccurate movements due to the lack of direct supervision from therapists. These incorrect movements may lead to inefficient rehabilitation exercises and even have a negative impact on the rehabilitation process. To address this challenge, the SmartRehab system uses reference angles for routine evaluation of rehabilitation movements and considers potential errors by providing text and visual guidance. For example, in shoulder abduction exercises, the standard evaluation angle is the left elbow-left shoulder-left hip angle in the XY plane. In contrast, the evaluation of incorrect movements is based on the left elbow-left shoulder-right shoulder angle in the XZ plane. This approach helps to identify and correct any inaccuracies during exercise.

[0099] The therapist-side module 130 includes a web-based portal system 132 and a feedback module 134. The therapist-side module 130 is allowed to log in to the cloud platform module 120 and send the training course description and store it in the cloud platform module 120. The training course description is further sent to the user-side module 110, and can be allowed to modify the schedule information and video and written instructions of the user-side module 110. The web-based portal system 132 allows healthcare professionals to remotely manage the patient's personal profile and ensure that the therapist instruction information is updated as needed. Based on the updated therapist instruction information stored in the web-based portal system 132, the feedback module 134 can send a signal to the information receiver 118 of the user-side module 110, allowing the user to proceed to the next evaluation cycle.

[0100] In summary, the user-side feeds input from the RGB digital camera of a consumer smartphone or tablet device into two modules, namely human activity recognition and human pose estimation, followed by a human activity assessment module. Here, the MediaPipe human pose estimation model validated above is used, and the key joint information is fed into a rule-based human activity assessment model, where the performance of the listed exercises can be evaluated according to predetermined criteria and indicators developed by experienced therapists at HKSR. The exercises can be qualitatively analyzed using kinematic features inferred from the waveform motion of the key joints, including the position of the joints (exhibiting baseline posture, such as flexor muscle tension), the distance, velocity and acceleration of the movement, the presence of spastic jerking movements, and the final range of motion. Quantitative analysis can be performed by detecting the number of repetitions and the number of training sessions completed. Real-time feedback can be provided using key joint data to inform the user of their correctness, and reminders can be provided through predetermined and manually developed rules. In addition, the detection, assessment and feedback process is automated using objective measures (i.e. key points and inferred qualitative and quantitative data), which can reproduce the process of therapists monitoring patient performance, and is estimated to reduce the amount of work required compared to face-to-face services.

[0101] The web-based portal system on the therapist side then allows healthcare professionals to remotely manage the patient's profile. Qualitative and quantitative objective measurements are collected from the automatic human activity recognition and human activity assessment modules on the user-side mobile application and uploaded to the cloud server. Therapists can access and give new exercise plans, review progress and performance, and modify current exercise plans. Therapists have high flexibility in modifying and customizing exercise plans and can personalize plan attributes including number of sessions, number of repetitions per day, and exercise difficulty according to the patient's needs. The completion of the prescribed exercise plan can be easily quantified and tracked using traffic light reports to monitor adherence and compliance. Overall, the proposed telerehabilitation framework enables remote distribution and dissemination of services to areas with adequate internet services through more easily available consumer-grade smartphone or tablet devices without the need for additional dedicated sensors or equipment.

[0102] As mentioned above, the rule-based human activity assessment module adopted in SmartRehab can achieve objective measurement through predicted and inferred motion data. Currently available telemedicine and telerehabilitation solutions mainly rely on individual subjective recollection, which leads to response bias. By using accurate and reliable key joint markers, automatic continuous assessment and monitoring of rehabilitation exercise performance can be achieved, which, on the contrary, may be labor-intensive and almost impossible to achieve in face-to-face services. Important movement information, including body posture, velocity, acceleration, and movement spasm, can be easily assessed without additional work. However, the rule-based assessment module requires expert opinion and hand-crafted features, which may be difficult or almost impossible in relatively complex and complicated exercises. In addition, the accuracy, reliability, and effectiveness of the rule-based assessment algorithm depend on the quality and comprehensiveness of the predetermined criteria (e.g., the expected ROM endpoint of healthy subjects, any possible skill movements similar to the exercise, any specific instructions or warnings). Future directions may combine deeper classification models and further automate the assessment process by utilizing template-based or other assessment methods with higher dimensions.

[0103] SmartRehab uses artificial intelligence and computer vision to perform automatic posture analysis and performance assessment during home stroke rehabilitation exercises. The application can act as a "digital coach" to enable patients to continue rehabilitation remotely in the community. Therapists can prescribe personalized exercise plans and monitor progress through the SmartRehab portal. Physiological indicators collected during exercises provide valuable data for treatment. The platform proposed in the present disclosure may reduce the need for face-to-face meetings with therapists and travel time to rehabilitation centers, and is particularly valuable in situations such as COVID-19, where social distancing needs to be maintained in rehabilitation centers to minimize cross-infection.

[0104] The functional units and modules of the apparatus and methods according to the embodiments disclosed herein can be implemented using computing devices, computer processors or electronic circuits, including but not limited to application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), microcontrollers and other programmable logic devices configured or programmed according to the teachings of the present disclosure. A person skilled in the art of software or electronics can easily prepare computer instructions or software codes executed in a computing device, a computer processor or a programmable logic device based on the teachings of the present disclosure.

[0105] All or part of the method according to the embodiment may be executed in one or more computing devices, including server computers, personal computers, laptop computers, mobile computing devices (such as smart phones and tablet computers).

[0106] Embodiments may include computer storage media, transient and non-transitory memory devices having stored therein computer instructions or software codes, which may be used to program or configure a computing device, a computer processor, or an electronic circuit to perform any of the processes of the present invention. Storage media, transient and non-transitory memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray disks, DVDs, CD-ROMs, magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of medium or device suitable for storing instructions, codes, and / or data.

[0107] Each functional unit and module according to various embodiments may also be implemented in a distributed computing environment and / or a cloud computing environment, where all or part of a machine instruction is executed in a distributed manner by one or more processing devices that are interconnected through a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0108] The above description of the present invention is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present invention to the precise form disclosed. For those skilled in the art, many modifications and variations will be apparent.

[0109] The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A system for providing community-customized post-stroke rehabilitation, characterized in that: include: A client module configured to provide schedule information having instructions for at least one specific exercise and display at least one image or video, wherein the client module provides a view page of a camera to record a target image or video and record performance indicators of a target user, the client module comprising: A feeding module configured to obtain the image or video from a consumer-grade smartphone or tablet device; a human activity recognition module configured to receive the image or video from the feed module and analyze input data using a human posture estimation model; and a human activity evaluation module configured to receive the result transmitted from the human activity recognition module and evaluate the performance based on performance indicator information, wherein the performance indicator information is formulated based on a rule or based on a template; a cloud platform module, configured to receive and store the performance indicator of the target user from the user terminal module; and A therapist-side module communicates with the user-side module through the cloud platform module, wherein the therapist-side module is allowed to log into the cloud platform module and receive the performance indicators of the target user from the cloud platform module, and the therapist-side module is also configured to visualize the performance indicators of the target user, thereby displaying motion waveforms including quantitative data for qualitative analysis.

2. The system according to claim 1, characterized in that The cloud platform module is also configured to store the target image or video from the user-side module, and the therapist-side module is allowed to receive and store the target image or video from the cloud platform module.

3. The system according to claim 1, characterized in that The performance indicators of the target user include joint angle, speed, acceleration, motion smoothness, key point distance and a combination thereof.

4. The system according to claim 1, characterized in that The human posture estimation model of the human activity recognition module is implemented using a convolutional neural network (CNN) to analyze the input data, and the human posture estimation model is configured to identify and predict the position and direction of each body joint based on the input data, thereby providing detailed information about the user's movement.

5. The system according to claim 4, characterized in that The human pose estimation model is configured to extract relevant information from each input frame of the target image or video and predict key points of the body appearing in the target image or video.

6. The system according to claim 5, characterized in that The human pose estimation model is configured to use the predicted key points to infer skeletal, biometric, and motion data.

7. The system according to claim 6, characterized in that The human pose estimation model is configured to perform a human pose estimation process, which utilizes the predicted key points to infer relevant biometric data for analysis and evaluation, and the information applied by the human pose estimation model to the human pose estimation process includes joint angles, velocities, accelerations, motion smoothness, and key point distances.

8. The system according to claim 7, characterized in that The human pose estimation model includes a computer vision model for predicting the position of the body and the key points in space as two-dimensional coordinates at a given time.

9. The system according to claim 8, characterized in that The human activity assessment module is further configured to transmit the assessment result to the therapist end module through the cloud platform module, wherein the therapist end module includes: A web-based portal system is configured to allow external visitors to remotely manage the user's personal profile based on the evaluation results to ensure that therapist guidance information is updated for the user-side module.

10. The system according to claim 9, characterized in that The therapist-side module also includes: A feedback module is in communication with the web-based portal system, wherein the feedback module is configured to send a course description to the client module and allow modification of the schedule information and instructions for the specific exercise of the client module.

11. A method for providing community-based customized post-stroke rehabilitation, characterized in that: include: providing, by the client module, schedule information having instructions for at least one specific exercise; At least one image or video is displayed through the user terminal module, wherein the user terminal module provides a view page of the camera to record the target image or video and record the performance index of the target user, wherein the user terminal module includes: A feeding module configured to obtain the image or video from a consumer-grade smartphone or tablet device; a human activity recognition module configured to receive the image or video from the feed module and analyze input data using a human posture estimation model; and a human activity evaluation module configured to receive the result transmitted from the human activity recognition module and evaluate the performance based on performance indicator information, wherein the performance indicator information is formulated based on a rule or based on a template; Receiving and storing the performance indicator of the target user from the user terminal module through the cloud platform module; Using a therapist-side module, recording the performance indicator of the target user into the cloud platform module to receive the performance indicator of the target user from the cloud platform module; and The performance indicators of the target user are visualized using a therapist-side module to display motion waveforms containing quantitative data for qualitative analysis.

12. The method according to claim 11, characterized in that Also includes: The target image or video from the user-side module is stored by the cloud platform module, wherein the therapist-side module is allowed to receive and store the target image or video from the cloud platform module.

13. The method according to claim 11, characterized in that The performance indicators of the target user include joint angle, speed, acceleration, motion smoothness, key point distance and a combination thereof.

14. The method according to claim 11, characterized in that The human posture estimation model of the human activity recognition module is implemented using a convolutional neural network (CNN) to analyze the input data, and the human posture estimation model is configured to identify and predict the position and direction of each body joint based on the input data, thereby providing detailed information about the user's movement.

15. The method according to claim 14, characterized in that Also includes: extracting relevant information from each input frame of the target image or video through the human pose estimation model; as well as The human body posture estimation model is used to predict key points of the body appearing in the target image or video.

16. The method according to claim 15, characterized in that The human pose estimation model is configured to use the predicted key points to infer skeletal, biometric, and motion data.

17. The method according to claim 16, characterized in that Also includes: A human posture estimation process is performed through the human posture estimation model, which uses the predicted key points to infer relevant biometric data for analysis and evaluation, and the information applied by the human posture estimation model to the human posture estimation process includes joint angles, speeds, accelerations, motion smoothness and key point distances.

18. The method according to claim 17, characterized in that The human pose estimation model includes a computer vision model for predicting the position of the body and the key points in space as two-dimensional coordinates at a given time.

19. The method according to claim 18, characterized in that Also includes: The evaluation result is transmitted to the therapist end module through the human activity evaluation module and the cloud platform module; as well as By using the web-based portal system of the therapist-side module, external visitors are allowed to remotely manage the user's personal profile based on the evaluation results to ensure that the therapist guidance information is updated for the user-side module.

20. The method according to claim 19, characterized in that Also includes: Using the feedback module of the therapist-side module, the course instructions are sent to the user-side module to modify the schedule information and the instructions of the specific exercise of the user-side module.

Images