A lower limb exoskeleton control device and method based on IMU and video stream

Abstract

The present invention discloses a lower limb exoskeleton control device and method based on an IMU and video stream. The control device comprises an IMU (1), a hip joint servo motor (2), a knee joint servo motor (4), a central processing unit (5), an embedded controller (6), a power module (7) and a camera (8). The IMU (1) is placed at four joint rings of the device, the hip joint servo motor (2) and the knee joint servo motor (4) are both placed on the isometric side wall of the device, the central processing unit (5), the embedded controller (6) and the power module (7) are all placed on the back panel of the device, and the camera (8) is independent of the device and faces the side axis of the device. The exoskeleton control method proposed by the present invention has accurate recognition and high prediction accuracy, can provide users with real-time, high-intensity auxiliary torque, and can help users restore muscle function to achieve the purpose of rehabilitation training.

Description

Technical Field

[0001] The present invention belongs to the field of exoskeleton robots, and in particular relates to a lower limb exoskeleton control device and method based on IMU and video stream. Background Art

[0002] Exoskeleton robots have broad application prospects in the military, medical, and industrial fields. In the field of medical rehabilitation, rehabilitation exoskeletons can assist patients with spinal cord injuries, stroke injuries, and brain trauma with rehabilitation training, helping them regain their mobility. For rehabilitation exoskeletons, obtaining accurate gait information from the human body and converting it into exoskeleton control signals is a very important technology. Current research mainly uses sensors such as surface electromyographs, plantar pressure sensors, and infrared sensors to obtain human motion information. However, these methods of obtaining human motion status have large errors and cannot accurately identify human gait information. This results in inaccurate gait control solutions and fails to achieve ideal human-computer interaction effects. In addition, current research also applies electrical stimulation to the user's corresponding muscles when performing assistance, which makes it difficult to implement and may cause harm to the user. This leads to poor accuracy and reduced robustness of the exoskeleton control algorithm, which in turn makes rehabilitation exoskeleton unable to provide accurate rehabilitation training for patients. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention provides a lower limb exoskeleton control device and method based on IMU and video stream, which can accurately obtain human gait information. At the same time, the present invention uses non-invasive means to obtain motion data, which can avoid harm to the human body and reduce the difficulty of use, thereby providing users with precise rehabilitation training.

[0004] In order to achieve the above-mentioned purpose of the invention, the technical solution adopted by the present invention is: a lower limb exoskeleton control device based on IMU and video stream includes IMU, hip joint servo motor, knee joint servo motor, central processing unit, embedded controller, power module and camera, the IMU is placed at the four joint rings of the device, the hip joint servo motor and knee joint servo motor are both placed on the isometric side wall of the device, the central processing unit, embedded controller and power module are all placed on the back panel of the device, the camera is independent of the device and faces the side axis of the device, the central processing unit is respectively communicated with the IMU, camera and embedded controller, the embedded controller is simultaneously communicated with the hip joint servo motor and knee joint servo motor, and the power module is respectively electrically connected with the IMU, hip joint servo motor, knee joint servo motor and central processing unit, embedded controller and camera.

[0005] The beneficial effect of the above solution is that through the above technical solution, the patient's movement information can be obtained in real time and accurately, good human-computer interaction can be achieved, and accurate auxiliary torque can be provided to the user to complete rehabilitation training.

[0006] Furthermore, the hip joint servo motor is connected to the two upper hip joint rings, and the knee joint servo motor is connected to the two lower knee joint rings.

[0007] The beneficial effect of the above further solution is: in the above technical solution, the hip joint servo motor controls the hip joint ring to train the user's hip joint, and the knee joint servo motor controls the knee joint ring to train the user's knee joint.

[0008] Furthermore, the IMU is placed outside the joint ring, and the joint ring is connected by metal and flexible straps.

[0009] The beneficial effect of the above further solution is that the joint ring in the above technical solution is composed of metal and flexible straps, which can provide good support and binding for the user, so that the exoskeleton fits the human body better and ensures the accuracy of applying auxiliary torque.

[0010] Furthermore, the hip joint ring and knee joint ring are both placed on the inner side of the vertical axis of the device, the hip joint servo motor and knee joint servo motor are placed on the outer side of the vertical axis of the device, and the hip joint ring, knee joint ring and foot pedal placed on the inner side of the two vertical axes are all located in the same vertical plane.

[0011] The beneficial effect of the above further solution is that the above technical solution can ensure the flexibility of the user when wearing the exoskeleton for exercise, and the movement will not be hindered by the structure of the exoskeleton. The structural design of the device also provides a better experience for the user.

[0012] Furthermore, the control device also includes a height adjustment device, and the two vertical axes in the control device can be freely adjusted in height using the height adjustment device.

[0013] The beneficial effect of the above further solution is that the above technical solution can flexibly adjust the height of the vertical axis of the device, and adjust different heights according to the user's height and other personal conditions, providing users with more comfortable and convenient training services. At the same time, the flexible adjustment also expands the user range.

[0014] In addition, the present invention also adopts a technical solution: a lower limb exoskeleton control method based on IMU and video stream, characterized in that the method includes the following steps:

[0015] S1: Use IMU and video streaming to collect user motion information;

[0016] S2: The central processing unit integrates the collected motion information according to the deep learning algorithm to generate the input signal of the embedded control system;

[0017] S3: Transmitting the signal processed by the central processing unit to the embedded controller for signal conversion;

[0018] S4: Using the embedded controller to control the joint servo motor according to the converted signal, the motor outputs auxiliary torque and rotation angle;

[0019] S5: Use the angle sensors and torque sensors in the joint servo motors to monitor whether the lower limb exoskeleton has reached the target position and feed back the execution status to the central processor. If the expected target value is not reached, the central processor will calculate the error and make timely adjustments to the predicted values ​​of the next joint angle and joint torque.

[0020] The beneficial effects of the above scheme are: the above technical scheme uses IMU and video streaming to accurately obtain the athlete's gait information, which is easier to implement and can reduce harm to the human body. At the same time, the joint sensors and torque sensors will monitor the current movement status of the lower limb exoskeleton and promptly feedback to the central processor for analysis and processing, reducing the use error of the device and providing users with more accurate training.

[0021] Furthermore, the deep learning algorithm in S2 includes the following sub-steps:

[0022] S2-1: Input the video stream data collected by the camera into the stacked hourglass network for human posture estimation to obtain the coordinates of the key points of the human lower limbs and calculate the joint angles;

[0023] S2-2: The data collected by the IMU and the joint angles calculated based on the video stream data are fused according to certain weights and then input into the CNN-LSTM deep neural network;

[0024] S2-3: Filter and extract features from the fused IMU data and the joint angles calculated based on the video stream data, and use the CNN-LSTM deep neural network to determine the current motion state and gait phase;

[0025] S2-4: Use the CNN-LSTM deep neural network algorithm to predict the joint torque and joint angle at time T+1ms based on the previous 50ms data including the current time T;

[0026] S2-5: Generate input signals for the embedded control system.

[0027] The beneficial effect of the above further scheme is: the above technical scheme uses a deep learning algorithm to analyze and process the acquired human gait information, and then predict the joint torque and joint angle at the next moment, and at the same time convert the prediction results into signal inputs for the embedded control system.

[0028] Furthermore, the embedded controller in S3 adopts a cascade PID closed-loop control algorithm, takes the predicted joint angle and predicted joint torque as inputs of the embedded controller, and uses the position controller and torque controller to control the motor operation.

[0029] The beneficial effect of the above further solution is that the above solution converts the control signal into an auxiliary torque and rotation angle output by the motor, so as to achieve good human-computer interaction and good rehabilitation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 An isometric view of a lower limb exoskeleton control device based on an IMU and video stream.

[0031] Figure 2 Back view of the exoskeleton, which is a lower limb exoskeleton control device based on IMU and video stream.

[0032] Figure 3 This is a schematic diagram of the lower limb exoskeleton control method based on IMU and video stream.

[0033] Figure 4 The flowchart of the lower limb exoskeleton control method based on IMU and video stream.

[0034] Figure 5 This is the flow chart of the cascade PID control algorithm.

[0035] Among them: 1. Inertial sensing unit (IMU); 2. Hip joint servo motor; 3. Height adjustment device; 4. Knee joint servo motor; 5. Central processing unit; 6. Embedded controller; 7. Power module; 8. Camera. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1, as Figure 1 and Figure 2As shown, a lower limb exoskeleton control device based on IMU and video stream includes IMU1, hip joint servo motor 2, knee joint servo motor 4, central processing unit 5, embedded controller 6, power module 7 and camera 8, wherein the IMU1 is placed at each joint ring of the device, the hip joint servo motor 2 and the knee joint servo motor 4 are both placed on the isometric side wall of the device, the central processing unit 5, the embedded controller 6 and the power module 7 are all placed on the back panel of the device, the camera 8 is independent of the device and faces the lateral axis of the device, the central processing unit 5 is communicated with the IMU1, the camera 8 and the embedded controller 6 respectively, the embedded controller 6 is communicated with the hip joint servo motor 2 and the knee joint servo motor 4 at the same time, and the power module 7 is electrically connected to the IMU1, the hip joint servo motor 2, the knee joint servo motor 4 and the central processing unit 5, the embedded controller 6 and the camera 8 respectively.

[0038] Among them, the hip joint servo motor 2 is connected to the two hip joint rings above, the knee joint servo motor 4 is connected to the two knee joint rings below, the IMU1 is placed on the outside of the joint ring, the joint ring is connected by metal and flexible straps, the hip joint ring and the knee joint ring are both placed on the inside of the vertical axis of the device, the hip joint servo motor 2 and the knee joint servo motor 4 are placed on the outside of the vertical axis of the device, and the hip joint ring, knee joint ring and foot pedal placed on the inside of the two vertical axes are all in the same vertical plane. The control device also includes a height adjustment device 3, and the two vertical axes in the control device can be freely adjusted in height using the height adjustment device 3.

[0039] Example 2, as Figure 3 and Figure 4 As shown, a lower limb exoskeleton control method based on IMU and video stream is characterized in that the method includes the following steps:

[0040] S1: Use IMU and video streaming to collect user motion information;

[0041] S2: The central processing unit integrates the collected motion information according to the deep learning algorithm to generate the input signal of the embedded control system;

[0042] S3: Transmitting the signal processed by the central processing unit to the embedded controller for signal conversion;

[0043] S4: Using the embedded controller to control the joint servo motor according to the converted signal, the motor outputs auxiliary torque and rotation angle;

[0044] S5: Use the angle sensors and torque sensors in the joint servo motors to monitor whether the lower limb exoskeleton has reached the target position and feed back the execution status to the central processor. If the expected target value is not reached, the central processor will calculate the error and make timely adjustments to the predicted values ​​of the next joint angle and joint torque.

[0045] The deep learning algorithm in S2 includes the following steps:

[0046] S2-1: Input the video stream data collected by the camera into the stacked hourglass network for human posture estimation to obtain the coordinates of the key points of the human lower limbs and calculate the joint angles;

[0047] S2-2: The data collected by the IMU and the joint angles calculated based on the video stream data are fused according to certain weights and then input into the CNN-LSTM deep neural network;

[0048] S2-3: Filter and extract features from the fused IMU data and the joint angles calculated based on the video stream data, and use the CNN-LSTM deep neural network to determine the current motion state and gait phase;

[0049] S2-4: Use the CNN-LSTM deep neural network algorithm to predict the joint torque and joint angle at time T+1ms based on the previous 50ms data including the current time T;

[0050] S2-5: Generate input signals for the embedded control system.

[0051] The embedded controller in S3 adopts a cascade PID closed-loop control algorithm, takes the predicted joint angle and predicted joint torque as the input of the embedded controller, and uses the position controller and torque controller to control the motor operation.

[0052] The cascade PID control algorithm is divided into three loops: the position controller is the first loop, the speed controller is the second loop, and the torque controller is the third loop. The specific implementation process is as follows: Figure 5 As shown: the expected joint angle and expected joint torque obtained based on the deep learning algorithm are used as inputs to the PID control algorithm. The position controller controls the rotation angle of the motor according to the expected angle θ, and uses the actual rotation angle θ1 monitored by the angle sensor as feedback for real-time adjustment to eliminate the cumulative error of the motor rotation angle. The speed loop controller calculates the expected speed ω based on the position increment. The torque controller controls the output torque of the motor according to the expected output torque M, and uses the actual output torque M1 monitored by the torque sensor as feedback for real-time adjustment to eliminate the cumulative error of the output torque.

[0053] In one embodiment of the present invention, the user's motion information is collected through video streaming and an IMU. A deep learning algorithm is used to achieve real-time posture recognition of the user based on the fusion data of the video stream and the IMU. The joint angle and joint torque at the next moment are predicted to generate the control input of the embedded system. The embedded controller uses a cascade PID closed-loop control algorithm to convert the control signal into the auxiliary torque, rotation angle and angular velocity output by the motor to achieve good human-computer interaction and good rehabilitation effect. The exoskeleton motor is controlled to output the corresponding auxiliary torque and move to the expected position. The torque sensor and angle sensor are used to monitor the output torque and rotation angle of the motor, and the execution status is fed back to the central processing unit. If the expected target value is not reached, the central processing unit calculates the error and promptly adjusts the predicted values ​​of the next joint angle and joint torque to achieve closed-loop control of the exoskeleton. This solution can avoid the cumulative error that may occur after the system has been running for a period of time, while improving the stability and execution accuracy of the system.

[0054] The deep learning algorithm used in the present invention integrates two deep neural networks: a stacked hourglass network for human posture estimation and a CNN-LSTM deep neural network for predicting joint angles and joint torques. Its specific implementation process is as follows: a video of a person wearing an exoskeleton walking, captured by a camera, is broken down into frames and input into the stacked hourglass network for human posture estimation. The network outputs a two-dimensional human posture result, namely the key point position. The key point position coordinates of each lower limb joint are used to calculate the hip and knee joint angles, which are then input into the next CNN-LSTM deep neural network for the next step of prediction. While the joint angles are obtained using video stream data and input into the CNN-LSTM deep neural network, the three-axis angular velocity and three-axis acceleration collected by the IMU are simultaneously input into the CNN-LSTM deep neural network. All data are filtered and feature value extracted, and then used to generate predicted joint angles and joint torques. The trained model is lightweight and then deployed on the central processing unit. The correspondence between joint angles, joint torques and gait phases is established during the early model training. As long as the user wears the exoskeleton and starts walking, the central processing unit can determine the user's motion state at the current time T based on the deep learning algorithm, and predict the joint angles and joint torques at the time T+1ms based on the previous 50ms data including the current time T to generate the motor control signal, thereby realizing real-time control of the exoskeleton.

[0055] Before the deep neural network algorithm is trained using the data set, the present invention determines the data processing calculation formula:

[0056] The formula for calculating the joint angle is as follows:

[0057]

[0058]

[0059] Among them, the vector and They are (x1-x2, y1-y2), (x2-x3, y2-y3) and (x3-x4, y3-y4) respectively, (x2, y2), (x3, y3) and (x4, y4) are all joint positions;

[0060] The formula for calculating joint torque is as follows:

[0061]

[0062] Where J is the moment of inertia of the thigh or calf about the center of mass, is the angular acceleration of the thigh or calf, M1 is the first joint torque, M2 is the second joint torque, F 1x and F 1y are the horizontal and vertical forces acting on the distal end of the thigh or calf, respectively, 2x and F 2y are the horizontal and vertical forces acting on the proximal end of the thigh or calf, L 1x , L 2x , L 1y and L 2y are the moment arms of the relevant forces about the center of mass.

[0063] Through the technical solutions of the above-mentioned embodiments 1 and 2, the user's motion information can be accurately obtained, the human body's motion parameters can be tracked in real time, human-computer interaction performance can be achieved, and more accurate rehabilitation training can be provided for the user.

[0064] The lower limb exoskeleton control device and method proposed in the present invention utilizes non-invasive means such as video streaming and IMU to obtain accurate human motion data, combines deep learning algorithms to predict joint torque and joint angle, and then obtains the target values ​​of the auxiliary torque and joint angle output by the motor at different times in the gait cycle. At the same time, the torque sensor and angle sensor monitor the execution status of the exoskeleton to achieve closed-loop control of the exoskeleton. In this way, the exoskeleton can follow the human body's movement in real time, obtain good human-computer interaction performance, and output personalized auxiliary torque for each patient, which can better assist the patient in rehabilitation. The technical solution proposed in the present invention changes the control method of the previous exoskeleton fixation program, obtains good human-computer interaction, has high prediction accuracy, can provide users with real-time, high-intensity auxiliary torque, and at the same time helps users restore muscle function and provide patients with better auxiliary effects.

[0065] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the invention.

Claims

1. A lower limb exoskeleton control method based on IMU and video stream, which is realized by a lower limb exoskeleton control device based on IMU and video stream, wherein the control device comprises an IMU (1), a hip joint servo motor (2), a knee joint servo motor (4), a central processing unit (5), an embedded controller (6), a power module (7) and a camera (8), wherein the IMU (1) is placed at four joint rings of the device, the hip joint servo motor (2) and the knee joint servo motor (4) are both placed on the isometric side wall of the device, the central processing unit (5), the embedded controller (6) and the power module (7) are all placed on the back panel of the device, the camera (8) is independent of the device and faces the side axis of the device, the central processing unit (5) is respectively connected to the IMU (1), the camera (8) and the embedded controller (6), the embedded controller (6) is simultaneously connected to the hip joint servo motor (2) and the knee joint servo motor (4), and the power module (7) is respectively connected to the IMU (1), the camera (8) and the embedded controller (6). The hip joint servo motor (2), the knee joint servo motor (4), the central processing unit (5), the embedded controller (6), and the camera (8) are electrically connected; It is characterized by: The lower limb exoskeleton control method comprises the following steps: S1: Use IMU and video streaming to collect user motion information; S2: The central processing unit integrates the collected motion information according to the deep learning algorithm to generate the input signal of the embedded control system; The deep learning algorithm in S2 includes the following steps: S2-1: Input the video stream data collected by the camera into the stacked hourglass network for human posture estimation to obtain the coordinates of the key points of the human lower limbs and calculate the joint angles; S2-2: The data collected by the IMU and the joint angles calculated based on the video stream data are fused according to certain weights and then input into the CNN-LSTM deep neural network; S2-3: Filter and extract features from the fused IMU data and the joint angles calculated based on the video stream data, and use the CNN-LSTM deep neural network to determine the current motion state and gait phase; S2-4: Use the CNN-LSTM deep neural network algorithm to predict the joint torque and joint angle at time T+1ms based on the previous 50ms data including the current time T; S2-5: Generate input signals for embedded control systems; The formula for calculating the joint angle is as follows: Among them, the vector 、 and They are 、 and , 、 and All are joint point positions; The formula for calculating joint torque is as follows: in, is the moment of inertia of the thigh or calf about the center of mass, is the angular acceleration of the thigh or calf, is the first joint torque, is the second joint torque, and are the horizontal and vertical forces acting on the distal end of the thigh or calf, and are the horizontal and vertical forces acting on the proximal end of the thigh or calf, 、 、 and are the moment arms of the relevant forces about the center of mass; S3: Transmitting the signal processed by the central processing unit to the embedded controller for signal conversion; The embedded controller in S3 adopts a cascade PID closed-loop control algorithm, uses the joint angle and joint torque predicted by the deep learning algorithm as inputs of the embedded controller, and uses the position controller and torque controller to control the motor operation; The cascade PID closed-loop control algorithm is divided into three loops: the position controller is the first loop, the speed controller is the second loop, and the torque controller is the third loop. The specific implementation process is as follows: the desired joint angle and desired joint torque obtained based on the deep learning algorithm are used as inputs to the PID control algorithm. The position controller controls the rotation angle of the motor according to the desired angle θ, and uses the actual rotation angle θ1 monitored by the angle sensor as feedback for real-time adjustment to eliminate the cumulative error of the motor rotation angle. The speed loop controller calculates the desired speed ω based on the position increment. The torque controller controls the output torque of the motor according to the desired output torque M, and uses the actual output torque M1 monitored by the torque sensor as feedback for real-time adjustment to eliminate the cumulative error of the output torque. S4: Using the embedded controller to control the joint servo motor according to the converted signal, the motor outputs auxiliary torque and rotation angle; S5: Use the angle sensors and torque sensors in the joint servo motors to monitor whether the lower limb exoskeleton has reached the target position and feed back the execution status to the central processor. If the expected target value is not reached, the central processor will calculate the error and make timely adjustments to the predicted values ​​of the next joint angle and joint torque.

2. The lower limb exoskeleton control method based on IMU and video stream according to claim 1, characterized in that: The hip joint servo motor (2) is connected to the two upper hip joint rings, and the knee joint servo motor (4) is connected to the two lower knee joint rings.

3. The lower limb exoskeleton control method based on IMU and video stream according to claim 1, characterized in that: The IMU (1) is placed outside the joint ring, and the joint ring is formed by connecting metal and flexible straps.

4. The lower limb exoskeleton control method based on IMU and video stream according to claim 2, characterized in that: The hip joint ring and the knee joint ring are both placed inside the vertical axis of the device, the hip joint servo motor (2) and the knee joint servo motor (4) are placed outside the vertical axis of the device, and the hip joint ring, the knee joint ring and the foot pedal placed inside the two vertical axes are all located in the same vertical plane.

5. The lower limb exoskeleton control method based on IMU and video stream according to claim 1, characterized in that: The control device further comprises a height adjustment device (3), and the heights of the two vertical axes in the control device can be freely adjusted by the height adjustment device (3).

Images