Smart wearable system with fast trajectory planning and mixed-information posture reconstruction

Abstract

The present invention discloses an intelligent wearable system with fast trajectory planning and hybrid posture reconstruction, including a robot body module, a visual posture monitoring module, a hybrid posture reconstruction module, a wearing trajectory planning module and an underlying safety control module; the robot body module assists people with upper limb dysfunction in performing exoskeleton wearing tasks; the visual posture monitoring module collects and obtains the patient's posture data; the hybrid posture reconstruction module estimates the unreliable part of the visual information in the process of reconstructing the exoskeleton wearing task to ensure improved accuracy in a hybrid environment; the wearing trajectory planning module is used to plan the real-time trajectory in the exoskeleton wearing task based on the reconstructed posture information. The underlying safety control module is used to ensure safety protection and flexibility in the process of the exoskeleton performing the wearing task based on the interactive force information. The present invention can observe, estimate and generate a wearing posture that matches the patient, thereby quickly and intelligently assisting the therapist in completing the wearing task.

Description

Technical Field

[0001] The present invention relates to the field of upper limb rehabilitation, and in particular to an intelligent wearable system with fast trajectory planning and mixed information posture reconstruction. Background Art

[0002] Faced with growing medical needs, rehabilitation exoskeletons offer a potential solution, potentially offsetting rising care costs and a shortage of caregivers. Of all daily activities, dressing places the greatest burden on caregivers, while assistive technology is least utilized. Fitting patients with exoskeletons places a greater burden on caregivers than dressing them. Currently, most exoskeleton developments focus on increasing patient engagement and motivation during rehabilitation training, but the donning phase is often overlooked. Most exoskeletons still require manual adjustments by therapists, which not only increases the burden on caregivers but also causes discomfort for patients. Therefore, it is crucial to enable collaboration between patients, therapists, and exoskeletons to jointly complete the donning process. During the donning phase, the exoskeleton's involvement should be maximized, with the therapist determining the overall direction while the exoskeleton handles the details of the movements.

[0003] Sensors used to collect human posture signals can be divided into three categories: wearable sensors, sensors that combine vision and wearable technology, and pure vision sensors. Regarding wearable sensors, the paper "Drift-Free and Self-Aligned IMU-Based Human Gait Tracking System With Augmented Precision and Robustness" (IEEE Robotics and Automation Letters) proposes an IMU-based online human gait estimation framework. Regarding sensors combining vision and wearables, Chen, Y proposed a visual-inertial skeleton tracking (VIST) framework (Chen, Y. ,Fu, C. ,Leung, S. ,&Shi, L. . (2020). Drift-free and self-aligned imu-based human gait tracking system with augmented precisionand robustness. IEEE Robotics and Automation Letters, PP(99), 1-1.), and G Nagymáté proposed a posture recognition method based on augmented reality (AR) markers (G Nagymáté,Kiss, RM,&Masani, K. . (2019). Affordable gait analysis using augmented realitymarkers. PLoS ONE, 14(2).). Regarding pure visual sensors, Wu, S, et al. used multiple perspectives to locate the center of the 3D human body and then performed 3D human pose estimation (Wu, S. ,Jin, S. ,Liu, W. ,Bai, L. ,Qian, C. ,&Liu, D. , et al. (2021). Graph-Based 3D Multi-Person Pose Estimation UsingMulti-View Images.).

[0004] However, although wearable sensors such as IMU are one of the solutions for posture recognition, they have anti-occlusion capabilities and excellent measurement accuracy. However, they cannot locate the target and require troublesome wearing. Therefore, a wearable combined with vision solution has been proposed. Although the wearing steps are greatly reduced, it still makes the patient feel unnatural and even interferes with the movement of the exoskeleton. Based on this, a pure vision solution is the optimal solution for this task. However, in the task of wearing an exoskeleton, due to the limitations of various complex medical venues, it is impossible to fully deploy multiple cameras, which will result in the anti-occlusion advantage of multiple cameras not being well reflected. Therefore, in this case, it is necessary to estimate the required depth information and three-dimensional posture from more reliable visual information. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the present invention provides an intelligent wearable system with fast trajectory planning and hybrid information posture reconstruction, which realizes real-time visual posture monitoring, hybrid posture reconstruction, and wearing trajectory tracking of therapist and patient data in exoskeleton wearing tasks for upper limb rehabilitation training. It can observe, estimate and generate wearing postures that match the patient, and thus quickly and intelligently assist therapists in completing wearing tasks.

[0006] In order to achieve the purpose of the present invention, the present invention discloses an intelligent wearable system with fast trajectory planning and hybrid information posture reconstruction, including a robot body module, a visual posture monitoring module, a hybrid posture reconstruction module, a wearable trajectory planning module and an underlying safety control module.

[0007] The robot body module serves as an actuator to assist people with upper limb dysfunction in wearing the exoskeleton.

[0008] The visual posture monitoring module includes a visual posture information acquisition system, which is used to obtain real-time signals from visual sensors and feedback posture data generated by the patient and therapist's movements during the exoskeleton wearing task. The posture information includes frame images of the patient and therapist, and the three-dimensional key point position information of the target posture is obtained through openpose and depth information, where the key point information includes: three-dimensional information of the shoulder joint, elbow joint, and wrist joint.

[0009] The hybrid posture reconstruction module is used to divide the posture data into reliable information and unreliable information, and then estimate and reconstruct the unreliable information using the reliable information to ensure improved accuracy and reliability in the hybrid environment, and obtain a more reliable target posture;

[0010] The wearing trajectory planning module is used to plan the real-time trajectory in the exoskeleton wearing task according to the reconstructed posture information;

[0011] The underlying safety control module is used to ensure safety and flexibility during the exoskeleton's wearing task based on the interaction force and target trajectory between the patient and the exoskeleton, and to drive the exoskeleton to assist the patient and therapist in completing the wearing task safely.

[0012] Furthermore, the robot body module is used to assist patients and therapists in completing the task of wearing an upper limb rehabilitation exoskeleton, and drives the exoskeleton according to the target trajectory output by the wearing trajectory planning module to achieve the wearing task trajectory, and corrects the actual target position of the exoskeleton according to the interaction force based on the underlying safety control module to achieve a safe exoskeleton wearing task.

[0013] Furthermore, the hybrid posture reconstruction module divides the obtained posture information into reliable information and unreliable information, and the reliability evaluation model is established based on two indicators, which are shown in (1).

[0014]

[0015] in, is the 2D planar distance between the patient's elbow and wrist joints in the 2D image. It is the standard length of the user's forearm and can be measured with a ruler or other reliable means. yes and It is an evaluation indicator of the difference between the two dimensions, which is aimed at the two-dimensional plane. The smaller it is, the more reliable the information is. is the 3D planar distance between the patient's elbow and wrist joints in the 2D image. yes and It is an evaluation index of the difference between the two, which is aimed at the three-dimensional plane. The smaller is, the more reliable the information is. The reliability evaluation model is shown in formula (2). Greater than 1, it is considered unreliable data. Data less than 1 are considered reliable.

[0016]

[0017] Furthermore, the unreliable information is estimated and reconstructed through the reliable information to ensure improved accuracy and reliability in mixed environments, and ultimately obtain a more reliable target pose.

[0018] Furthermore, the steps of performing posture reconstruction by the hybrid posture reconstruction module include:

[0019] The parametric equation of a straight line in a three-dimensional coordinate system is expressed by formula (3), where , , , represent the slope and intercept of the straight line respectively;

[0020]

[0021] The user's forearm and upper arm are discretized into n key points and concentrated to form a three-dimensional point set. The iterative least squares method is used to fit the posture of the upper arm and forearm. The target loss function is designed as shown in formulas (4)-(5):

[0022]

[0023]

[0024] in, , , P is a set of 3 rows and n columns of points, t is the number of iterations, T is the transpose, is the self-adjusting weight;

[0025] Among them, the initial value , represents the initial slope and intercept of the line, Represents the initial weight, as shown in equations (6)-(7):

[0026]

[0027]

[0028]

[0029] when Close to zero, and updated is obtained, similarly, the updated , Be obtained, and then, get the reconstructed posture.

[0030] Among them, the user's forearm and upper arm are discretized into n key points and concentrated to form a three-dimensional point set, and the locally occluded points and visual error points are ignored as outliers.

[0031] Furthermore, the wear trajectory planning module adopts a local trajectory planning method based on the reconstructed target posture to generate an adaptive virtual guide pipeline potential field to solve the problem of local minimum, and finally plans the real-time trajectory in the exoskeleton wearing task according to the reconstructed posture information.

[0032] Furthermore, in the wear trajectory planning module, the attraction potential field and repulsive potential field The expressions are

[0033] (9)

[0034] (10)

[0035] in, represents the attraction coefficient, represents the target position, Represents the actual location, represents the repulsion coefficient, The preset safe distance between the path point and the obstacle, Represents the actual distance between the waypoint and the obstacle, is an adjustable parameter for the performance of the repulsive potential field.

[0036] Furthermore, the underlying safety control module designs an underlying compliance control algorithm based on the target trajectory and the interaction force between the patient and the exoskeleton, and ultimately drives the exoskeleton to assist the patient and therapist in completing the safe wearing task.

[0037] The underlying compliance control algorithm in the underlying safety control module is:

[0038] (11)

[0039] When the motor position When the above control rules are met, the compliant control stiffness of the system is , by adjusting K P The value of can adjust the system stiffness, where K p Adjustable parameters for smooth control performance, K s is the system's own stiffness.

[0040] Compared with the prior art, the present invention has the following advantages and technical effects:

[0041] The present invention proposes an intelligent wearable system with fast trajectory planning and mixed information posture reconstruction, aiming to solve the problem that the existing upper limb exoskeleton wearing task requires the cooperation of multiple therapists to complete the exoskeleton wearing task. The robot body module assists people with upper limb dysfunction in performing exoskeleton wearing tasks, the visual posture monitoring module collects and obtains the patient's posture data, and the mixed posture estimation reconstructs the unreliable part of the visual information in the process of exoskeleton wearing tasks to ensure improved accuracy in mixed environments. The wearing trajectory planning module is used to plan the real-time trajectory in the exoskeleton wearing task based on the reconstructed posture information. The bottom layer safety control module is used to ensure safety protection and flexibility in the process of the exoskeleton performing the wearing task based on the interactive force information. An intelligent wearable system with fast trajectory planning and mixed information posture reconstruction realizes the posture monitoring and estimation reconstruction of patients and therapists in the wearing task of rehabilitation training, which is conducive to therapists and patients to complete the exoskeleton wearing task more easily, and then smoothly carry out exoskeleton rehabilitation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of an intelligent wearable system with fast trajectory planning and mixed information posture reconstruction provided by an embodiment of the present invention.

[0043] As shown in the figure: 1-robot body module, 2-visual posture monitoring module, 3-hybrid posture reconstruction module, 4-wearable trajectory planning module, 5-bottom-level safety control module.

[0044] Figure 2 Schematic diagram of artificial potential field. The left picture is the attractive potential field, and the right picture is the repulsive potential field.

[0045] Figure 3 Schematic diagram of the mathematical model of the compliant control algorithm. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to specific embodiments, but is not limited thereto.

[0047] like Figure 1 As shown, the present invention provides an intelligent wearable system with fast trajectory planning and hybrid information posture reconstruction, including a robot body module 1, a visual posture monitoring module 2, a hybrid posture reconstruction module 3, a wear trajectory planning module 4 and an underlying safety control module 5;

[0048] The robot body module 1 acts as an actuator to assist people with upper limb dysfunction in wearing the exoskeleton.

[0049] The visual posture monitoring module 2 includes a visual posture information acquisition system, which is used to obtain real-time signals from visual sensors and feedback posture data generated by the patient and therapist's movements during the exoskeleton wearing task, including the patient's posture data and the therapist's posture data; the posture data includes frame images of the patient and the therapist, and the three-dimensional key point position information of the target posture is obtained through openpose and depth information, where the key point information includes: three-dimensional information of the shoulder joint, elbow joint, and wrist joint.

[0050] The hybrid posture reconstruction module divides the obtained posture data into reliable information and unreliable information, and then estimates and reconstructs the unreliable information based on the reliable information to ensure improved accuracy and reliability in the hybrid environment, and ultimately obtain a more reliable target posture.

[0051] The wear trajectory planning module adopts the local trajectory planning method and generates an adaptive virtual guide pipeline potential field to solve the problem of local minimum, and finally plans the real-time trajectory in the exoskeleton wearing task based on the reconstructed posture information.

[0052] The underlying safety control module designs an underlying compliance control algorithm based on the interaction force between the patient and the exoskeleton, and ultimately drives the exoskeleton to assist the patient and therapist in completing safe wearing tasks.

[0053] The robot body module is used to assist patients and therapists in completing the task of wearing an upper limb rehabilitation exoskeleton, and drives the exoskeleton according to the target trajectory output by the wearing trajectory planning module to achieve the wearing task trajectory, and corrects the actual target position of the exoskeleton according to the interaction force based on the underlying safety control module to achieve a safe exoskeleton wearing task.

[0054] In some embodiments of the present invention, the hybrid posture reconstruction module divides the obtained posture information into reliable information and unreliable information, and the reliability evaluation model is established based on two indicators, which are shown in formula (1).

[0055]

[0056] in, is the 2D planar distance between the patient's elbow and wrist joints in the 2D image. It is the standard length of the user's forearm and can be measured with a ruler or other reliable means. yes and It is an evaluation indicator of the difference between the two dimensions, which is aimed at the two-dimensional plane. The smaller it is, the more reliable the information is. is the 3D planar distance between the patient's elbow and wrist joints in the 2D image. yes and It is an evaluation index of the difference between the two, which is aimed at the three-dimensional plane. The smaller is, the more reliable the information is. The reliability evaluation model is shown in formula (2). If it is greater than 1, it is considered as unreliable information. If it is less than 1, it is considered to be reliable information.

[0057]

[0058] Represents the reliability evaluation index, when If it is greater than 1, it is considered as unreliable information. If it is less than 1, it is considered to be reliable information.

[0059] Then, the unreliable information is estimated and reconstructed through reliable information to ensure improved accuracy and reliability in mixed environments, and ultimately obtain a more reliable target posture.

[0060] The parametric equation of a straight line in a three-dimensional coordinate system can be expressed by formula (3), where , , , represent the slope and intercept of the line respectively.

[0061]

[0062] The user's forearm and upper arm are discretized into n key points and concentrated to form a three-dimensional point set. The locally occluded points and visual error points are ignored as outliers. Therefore, the iterative least squares method is used to fit the posture of the upper arm and forearm to reduce the error caused by the visual signal. The target loss function is designed as shown in equations (4) and (5). ; ; P is a set of 3 rows and n columns of points; t is the number of iterations; T is the transpose. In addition, is the self-adjusting weight, initial value , represents the initial slope and intercept of the line, represents the initial weight, as shown in equations (6)-(7).

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] when Close to zero, and updated is obtained. Similarly, the updated , Then, the reconstructed posture is obtained.

[0069] The wearable trajectory planning module adopts the local trajectory planning method and generates an adaptive virtual guide pipe potential field to solve the problem of local minima. Finally, it plans the real-time trajectory in the exoskeleton wearing task based on the reconstructed posture information.

[0070] In some embodiments of the present invention, the wear trajectory planning module adopts a local trajectory planning method based on artificial potential field based on the reconstructed target posture to plan the real-time trajectory in the exoskeleton wearing task according to the reconstructed posture information. Expressed as formula (9), the repulsive force potential field It is expressed as formula (10):

[0071] (9)

[0072] (10)

[0073] in, represents the attraction coefficient, represents the target position, Represents the actual location, represents the repulsion coefficient, The preset safe distance between the path point and the obstacle, Represents the actual distance between the waypoint and the obstacle, is an adjustable parameter for the performance of the repulsive potential field.

[0074] The underlying safety control module designs the underlying compliance control algorithm based on the target trajectory and the interaction force between the patient and the exoskeleton, and ultimately drives the exoskeleton to assist the patient and therapist in completing the safe wearing task. The control rate of the motor, i.e., the underlying compliance control algorithm, is (11):

[0075] (11)

[0076] When the motor position When the above control rules are met, the compliant control stiffness of the system is , by adjusting K P The value of can adjust the system stiffness. K p Adjustable parameters for smooth control performance, Ks is the system's own stiffness, D For damping, m L For system quality, f L For external force, X m is the motor position. is the displacement of the compliant control.

[0077] The present invention proposes an intelligent wearable system with fast trajectory planning and mixed information posture reconstruction, which aims to solve the problem that the existing upper limb exoskeleton wearing task requires the cooperation of multiple therapists to complete the exoskeleton wearing task. The robot body module assists people with upper limb dysfunction in performing exoskeleton wearing tasks, the visual posture monitoring module collects and obtains the patient's posture data, and the mixed posture estimation reconstructs the unreliable part of the visual information in the process of exoskeleton wearing tasks to ensure improved accuracy in mixed environments. The wearing trajectory planning module is used to plan the real-time trajectory in the exoskeleton wearing task based on the reconstructed posture information. The bottom layer safety control module is used to make the exoskeleton have safety protection and flexibility in the process of performing the wearing task based on the interactive force information. The present invention realizes the posture monitoring and estimation reconstruction of patients and therapists in the wearing task of rehabilitation training, which is conducive to therapists and patients to complete the exoskeleton wearing task more easily, and then smoothly carry out exoskeleton rehabilitation training.

[0078] The usage process of the wearable system provided in the above embodiment is as follows:

[0079] In some embodiments of the present invention, the robot body module 1 is positioned behind the patient and therapist, assisting them in performing the exoskeleton donning task. While the robot body module 1 is performing the exoskeleton-assisted donning task, the visual posture monitoring module 2 captures frame images of the patient and therapist, deriving the target posture using open pose and depth information. The monitored image data is then transmitted as input parameters to the hybrid posture reconstruction module 3. The hybrid posture reconstruction module 3 estimates and reconstructs unreliable information using reliable information to improve accuracy and reliability in a hybrid environment, ultimately obtaining a more reliable target posture. The reconstructed posture is then transmitted to the donning trajectory planning module 4. Based on the reconstructed target posture, the donning trajectory planning module 4 employs a local trajectory planning method to generate an adaptive virtual guide channel potential field to resolve local minima. Ultimately, the real-time trajectory for the exoskeleton donning task is planned based on the reconstructed posture information. The actual trajectory is then transmitted to the underlying safety control module 5. Based on the target trajectory and the interaction forces between the patient and the exoskeleton, the underlying safety control module 5 designs an underlying compliant control algorithm, ultimately driving the exoskeleton to assist the patient and therapist in completing the donning task safely.

[0080] This invention provides an intelligent wearable system with rapid trajectory planning and hybrid information posture reconstruction. The robot module assists people with upper limb dysfunction in wearing an exoskeleton. The visual posture monitoring module collects and obtains the patient's posture data. Hybrid posture estimation reconstructs unreliable visual information during the exoskeleton wearing task to ensure improved accuracy in hybrid environments. The wearing trajectory planning module plans the real-time trajectory during the exoskeleton wearing task based on the reconstructed posture information. The underlying safety control module uses interaction force information to ensure safety protection and flexibility during the exoskeleton wearing task.

[0081] According to the disclosure and teachings of the above description, those skilled in the art to which the present invention belongs may also make changes and modifications to the above-mentioned embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the scope of protection of the claims of the present invention. According to the disclosure and teachings of the above description, those skilled in the art to which the present invention belongs may also make changes and modifications to the above-mentioned embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the scope of protection of the claims of the present invention.

Claims

1. An intelligent wearable system with fast trajectory planning and mixed information posture reconstruction, characterized in that: It includes a robot body module (1), a visual posture monitoring module (2), a hybrid posture reconstruction module (3), a wearable trajectory planning module (4) and an underlying safety control module (5); The robot body module (1) is used to assist people with upper limb dysfunction in wearing an exoskeleton; The visual posture monitoring module (2) is used to obtain real-time signals from the visual sensor and to feed back posture data generated by the patient and the therapist's movements during the exoskeleton wearing task, including the patient's posture data and the therapist's posture data; The hybrid posture reconstruction module (3) is used to divide the posture data into reliable information and unreliable information, and then estimate and reconstruct the unreliable information through the reliable information to obtain a more reliable target posture; The wearing trajectory planning module (4) is used to plan the real-time trajectory in the exoskeleton wearing task according to the reconstructed posture information; The bottom safety control module (5) is used to enable the exoskeleton to perform the wearing task with safety protection and flexibility according to the interaction force information and the target trajectory, and to drive the exoskeleton to assist the patient and the therapist in completing the wearing task safely; The hybrid posture reconstruction module divides posture data into reliable information and unreliable information according to the reliability evaluation model. The reliability evaluation model is: Represents the reliability evaluation index, when If it is greater than 1, it is considered as unreliable information. Less than 1 is considered reliable information, where for and The difference evaluation index, is the 2D plane distance between the patient's elbow and wrist joints in the 2D image, is the standard length of the user's forearm, yes and The difference evaluation index, is the 3D planar distance between the patient's elbow and wrist joints in the 2D image.

2. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 1, characterized in that: The visual posture monitoring module (2) includes a visual posture information acquisition system; the posture information includes frame images of the patient and the therapist, and the three-dimensional key point position information of the target posture is obtained through openpose and depth information.

3. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 2, characterized in that: The three-dimensional key point position information includes three-dimensional information of the shoulder joint, elbow joint and wrist joint.

4. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 1, characterized in that: In the reliability assessment model, and The expressions are: .

5. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 1, characterized in that: The steps of posture reconstruction in the hybrid posture reconstruction module include: The parametric equation of a straight line in a three-dimensional coordinate system is: in , , , represent the slope and intercept of the straight line respectively; The user's forearm and upper arm are discretized into n key points and concentrated to form a three-dimensional point set. The iterative least squares method is used to fit the position of the upper arm and forearm. The target loss function is: in, , , P is a set of 3 rows and n columns of points, t is the number of iterations, T is the transpose, is the self-adjusting weight; Among them, the initial value , represents the initial slope and intercept of the line, Represents the initial weight, the expression is: ； when Close to zero, and updated , is obtained, similarly, the updated , Be obtained, and then, get the reconstructed posture.

6. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 5, characterized in that: The user's forearm and upper arm are discretized into n key points and concentrated to form a three-dimensional point set, and the locally occluded points and visual error points are ignored as outliers.

7. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 1, characterized in that: The wearing trajectory planning module (4) adopts a local trajectory planning method based on artificial potential field to plan the real-time trajectory in the exoskeleton wearing task according to the reconstructed posture information.

8. The intelligent wearable system with fast trajectory planning and mixed information posture reconstruction according to claim 1, characterized in that: In the wear trajectory planning module (4), the attractive potential field and repulsive potential field The expressions are: ； in, represents the attraction coefficient, represents the target position, Represents the actual location, represents the repulsion coefficient, The preset safe distance between the path point and the obstacle, Represents the actual distance between the waypoint and the obstacle, is an adjustable parameter for the performance of the repulsive potential field.

9. A smart wearable system with fast trajectory planning and mixed information posture reconstruction according to any one of claims 1 to 8, characterized in that: The underlying compliance control algorithm in the underlying safety control module is: When the motor position When the underlying compliant control algorithm is satisfied, the compliant control stiffness of the system is , by adjusting K P The value of can adjust the system stiffness, where K p Adjustable parameters for smooth control performance, K s is the system's own stiffness, is the displacement of the compliant control.

Images