Intelligent lower limb prosthesis control mode early switching method

Abstract

The application discloses a kind of intelligent lower limb prosthesis control mode leading switching method, it is related to rehabilitation robot control technical field, including the following steps: target terrain detection based on monocular vision, and the orientation description of the target terrain is carried out;Shift centroid in horizontal plane pose estimation;Combining the rule determination of motion mode switching with the orientation description and pose estimation.The present application can realize the leading prediction of prosthesis motion mode switching by constructing the effective perception channel of prosthesis to walking movement environment and the motion intention of prosthesis wearer, thereby reducing the delay required for human-machine system interaction communication, which helps to realize the natural switching of different motion modes of prosthesis wearer, and improves the reliability of prosthesis wearer walking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rehabilitation robot control, and particularly relates to a control mode advanced switching method for intelligent lower limb prosthesis. BACKGROUND

[0002] Due to traffic accidents, accidental injuries, joint infections, bone tumors and complications of vascular diseases, hundreds of thousands of people around the world undergo lower limb amputation every year, and there are currently about 360 million lower limb amputees in the world. Amputation has a great impact on their daily activities and psychology, and wearing a lower limb prosthesis can effectively offset these adverse effects.

[0003] At present, there are lower limb prostheses on the market that can perform multiple gait task control systems, but these prostheses often need to introduce relatively complex pattern recognition algorithms to identify different gait tasks, and pattern recognition usually occurs after the start of prosthesis movement, so there is a large time delay in switching between different gait tasks. The human-computer interaction lag problem reduces the wearing experience of the lower limb prosthesis, especially greatly limits the application scenarios that require rapid movement of amputees, thereby causing a high abandonment rate of prostheses.

[0004] Therefore, the skilled in the art is committed to developing a prosthesis control mode switching method to achieve fast and reliable advanced prediction of prosthesis movement mode switching. SUMMARY

[0005] In view of the above defects of the prior art, the present application proposes a gait mode switching prediction method that combines mobile scene understanding and human motion intention recognition to solve the interaction lag problem of intelligent lower limb prosthesis when performing mode switching between different gait tasks, with the purpose of providing an advanced gait mode switching prediction method to reduce the response delay of the prosthesis controller.

[0006] To achieve the above purpose, the present application provides a control mode advanced switching method for intelligent lower limb prosthesis, which comprises the following steps:

[0007] S1: target terrain detection based on monocular vision, and orientation description of the target terrain;

[0008] S2: transfer of center of mass pose estimation in the horizontal plane;

[0009] S3: rule determination of motion mode switching combined with the orientation description and pose estimation.

[0010] Preferably, the monocular vision-based target terrain detection in the step S1 comprises: realizing the identification of the target terrain based on an SSD target detection network to obtain image coordinate information of the target terrain frame in the field of view, and then calculating the orientation information of the target terrain and the prosthetic wearer in the horizontal plane at the current time through an orientation calculation link.

[0011] Preferably, the model deployment of the SSD target detection network comprises an offline training stage and an online inference stage.

[0012] Preferably, the offline training stage is performed by first performing data enhancement on images with target terrain annotation information, then inputting the images into the SSD target detection network for domain task transfer learning, outputting the final target detection frame through non-maximum suppression, stopping training when the detection accuracy of the SSD target detection network reaches a preset index, and then exporting the trained network model for model deployment.

[0013] Preferably, the orientation description comprises: the straight-line distance of the prosthetic wearer and the target terrain in the horizontal plane; and the orientation angle of the prosthetic wearer and the target terrain in the horizontal plane.

[0014] Preferably, the pose estimation of the transfer center of mass in the horizontal plane in the step S2 is obtained by directly integrating the horizontal plane projection vector after the body coordinate system is converted to the global coordinate system through the six-axis IMU processing data.

[0015] Preferably, the pose estimation comprises: zero drift correction of the IMU sensor data, walking speed estimation and turning angle estimation based on the zero drift correction data, and inferring the transfer trajectory of the transfer center of mass relative to the target obstacle in the horizontal plane.

[0016] Preferably, the walking speed estimation calculates the transfer speed of the transfer center of mass according to the observation of the accelerometer at time t, and the formula is:

[0017]

[0018] wherein v g (0) is the initial speed of the transfer center of mass, g g is the gravity acceleration vector in the global coordinate system, and a g is the acceleration of the transfer center of mass in the global coordinate system.

[0019] Preferably, the method for rule determination in the step S3 is: combining the orientation description of the target terrain, excluding the turning and stopping motion of the human-machine system through the walking speed and the turning angle, determining that the human-machine system will switch the motion mode, and providing predictive control instruction input for the prosthetic controller when the human-machine system is away from the target terrain by a predetermined value.

[0020] The application also provides an electronic device, comprising: at least one processor; and

[0021] A memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the foregoing method.

[0022] The beneficial effects of the application are: by constructing an effective sensing channel for the walking moving environment and the motion intention of the prosthesis wearer, and by fusing the information interpretation of the two, the advanced prediction of the prosthesis motion mode switching can be realized, thereby reducing the delay required for the human-machine system interaction communication, helping to realize the natural switching of different motion modes of the prosthesis wearer, and improving the reliability of the prosthesis wearer walking.

[0023] The concept, specific structure and technical effects of the application will be further described below in combination with the drawings, so as to fully understand the purposes, features and effects of the application. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A schematic diagram of the control mode advanced switching method module for the intelligent lower limb prosthesis facing different moving tasks in the embodiments of the application;

[0025] Figure 2 A schematic diagram of the local semantic map creation process in the moving scene in the embodiments of the application;

[0026] Figure 3 A principle schematic diagram of measuring the target terrain distance based on the visual method in the embodiments of the application;

[0027] Figure 4 A process schematic diagram of the center of mass transfer pose estimation of the amputee in the embodiments of the application;

[0028] Figure 5 A flowchart of the prosthesis moving mode switching prediction in the embodiments of the application. DETAILED DESCRIPTION

[0029] The following reference description of the drawings introduces a plurality of preferred embodiments of the application, so that the technical content thereof is more clear and convenient to understand. The application can be embodied in many different forms of embodiments, and the protection scope of the application is not limited to the embodiments mentioned herein.

[0030] In the embodiment of the present application, the target terrain in the advancing direction is recognized based on visual information, and the straight-line distance and the angle of orientation of the target terrain and the amputee in the horizontal plane are estimated by using the monocular ranging principle. Then, the case where the amputee turns and stops moving is excluded, so that it can be determined that the amputee will switch the motion mode, and further, the control instruction input of the prophase is provided for the artificial limb controller when the distance between the amputee and the target terrain reaches a predetermined distance.

[0031] The intelligent lower-limb artificial limb control mode prophase switching method provided by the embodiment of the present application is as shown in the figure, which comprises the following steps: Figure 1

[0032] S1: terrain detection and orientation description based on monocular vision;

[0033] S2: estimation of the position and posture of the shifted center of mass in the horizontal plane;

[0034] S3: rule determination of motion mode switching.

[0035] The step S1 realizes the recognition of the target terrain based on monocular vision, and the target terrain refers to obstacles such as roadblocks, stairs, slopes, etc., and the image vertex coordinate information of the target terrain frame in the field of view is obtained. As shown in the figure, Figure 2 In a preferred embodiment of the present application, the terrain detection and orientation description based on monocular vision comprises: realizing the recognition of the target terrain based on the SSD target detection network 10, obtaining the image coordinate information of the target terrain frame 11 in the field of view, and then calculating the orientation information of the target terrain and the amputee in the horizontal plane at the current time through the orientation calculation link 12.

[0036] The backbone network of the SSD target detection network 10 is preferably MobileNet V2, so as to deploy the model to an embedded device.

[0037] The network model deployment comprises an offline training phase and an online inference phase.

[0038] In the offline training phase, the images with target terrain annotation information are first subjected to data enhancement, and then input to the SSD pre-training network for transfer learning of the domain task. The network model outputs the final target detection frame through non-maximum suppression. When the detection accuracy of the network model reaches the preset index, the training is stopped, and then the network model is exported for model deployment.

[0039] In the online inference phase, the real-time acquired key frames of the camera are directly input to the network model to obtain the real-time monitored target detection frame.

[0040] The position of the target detection frame is defined by the coordinates of four vertices in the image.

[0041] ​The camera keeps the camera imaging plane vertical to the ground through the gimbal device.

[0042] The phase key frame of the input network model is a single sampling at a relatively stable body posture period when the amputee enters a double-foot support period, instead of continuous sampling, so as to save computing power and reduce device power consumption.

[0043] In the embodiment of the application, the camera used adopts a standard lens, and there is no need to consider the correction of image distortion.

[0044] Further, for the currently detected target terrain, by comparing with the updated orientation information of the last saved map, only the new map needs to be stored, and the method can handle the case that multiple terrain target to be transferred exist in the field of view at the same time.

[0045] The orientation information includes a straight-line distance of the amputee from the target terrain in a horizontal plane, and an orientation angle of the amputee from the target terrain in the horizontal plane.

[0046] The principle of calculating the straight-line distance information is shown in Figure 3 The distance D calculation formula can be obtained by using the principle of similar triangles:

[0047]

[0048] Wherein, P bottom , P center respectively represent the coordinates of the target detection frame bottom center point on the imaging plane and the coordinates of the camera imaging plane, H represents the height of the camera from the ground plane when the camera is in a vertical position, and f represents the focal length of the camera, which can be directly obtained through camera calibration.

[0049] On the basis of distance calculation, the calculation formula of the orientation θ is:

[0050]

[0051] Wherein, X represents the horizontal distance of the coordinates of the target detection frame bottom center point on the imaging plane and the coordinates of the camera imaging plane. The value range of the orientation information θ is related to the viewing angle of the selected camera.

[0052] In this embodiment, the monocular vision detects that there is an obstacle in its forward direction through step 1, stores the target terrain type as an obstacle, and stores the orientation information including the straight-line distance of the obstacle from the amputee and the deflection angle of the obstacle in the forward direction of the amputee.

[0053] The implementation method of step 2 is shown in Figure 4The method comprises: zero drift correction 20 of IMU sensor data, walking speed estimation 21 and turning angle estimation 22 based on the corrected data, and inference of a transfer trajectory of the center of mass in a horizontal plane relative to a target obstacle.

[0054] The IMU is installed at a body surface position defined by the transfer of the center of mass of the amputee, and the x-axis and y-axis of the IMU are defined to be aligned with the coordinate axes of a horizontal plane in the anatomical plane.

[0055] Since the direct measurement by the IMU is a description quantity in the body coordinate system of the human-machine system, i.e., the system composed of the amputee and the prosthesis, and the aforementioned environmental orientation information is a description quantity in the global coordinate system, in order to realize the transformation between the two, a rotation matrix needs to be solved first, as follows:

[0056] The angular velocity ω directly measured by the three-axis gyroscope of the IMU b (t) = [ω bx (t), ω by (t), ω bz (t)] T Construct Ω(t):

[0057]

[0058] The rotation matrix representing the pose is solved in real time by the rectangular numerical integration method where C(0) represents the initial pose.

[0059] The transfer center of mass obtained by the body coordinate system p b is converted to the coordinate p g in the global coordinate system, which can be represented as:

[0060] p g = Cp b

[0061] On the basis of the pose estimation, the implementation method of the walking speed estimation 21 in the embodiment of the application is as follows:

[0062] According to the observation a b (t) = [a bx (t), a by (t), a bz (t)] T of the accelerometer at time t, the transfer speed of the transfer center of mass in the global coordinate system at this moment is calculated, i.e., the walking speed of the human body is reflected.

[0063] First, the acceleration of the transfer center of mass in the body coordinate system is converted to the global coordinate system, which can be represented as:

[0064] a g = Cab

[0065] Then, the gravity is subtracted from the acceleration, and the remaining acceleration is subjected to rectangular numerical integration to obtain the center-of-mass transfer velocity in the global coordinate system, which can be expressed by the following formula:

[0066]

[0067] where v g (0) is the initial velocity of the device, g g is the gravity acceleration vector in the global coordinate system.

[0068] On the basis of the pose estimation, the turning angle estimation 22 is implemented in the embodiment of the present application as follows:

[0069] The initial moment transfer center-of-mass coordinate p g (0) = [1, 0, 0] T Then, the subsequent center-of-mass coordinate p g (t) after the pose transformation can be continuously solved according to the above transformation, and then the deflection angle θ relative to the initial moment can be estimated by using the horizontal plane projection angle of the coordinate point.

[0070]

[0071] In addition, since the pose estimation all needs to use the rotation matrix calculated based on the gyroscope, and the MEMS gyroscope has white noise and bias noise, although the process from detecting the target terrain to switching the movement mode is not a long-term process, when the IMU positioning is used alone, the error will have a significant impact on the calculation accuracy, and therefore the complementary filtering based on the acceleration and angular velocity output is introduced in the module 20 for zero bias error correction.

[0072] In the embodiment, the center-of-mass continuous transfer state of the amputee in the flat ground walking mode can be solved by the IMU six-axis filtered data in step 2: the walking speed and the turning angle reflected by the center-of-mass transfer speed in the horizontal plane.

[0073] The rule constituting elements of step 3 are shown in Figure 1 , and the determination logic flow is shown in Figure 5 .

[0074] The rule elements include: orientation estimation 31, turning determination 32, and zero speed determination 33.

[0075] The determination logic flow is: on the basis of the orientation estimation 31 of the target terrain relative to the man-machine system in the movement scene, the turning and stopping motion are excluded by the walking speed and the turning angle, and when the target is in the forward direction and reaches a predetermined distance, the movement mode switching is started.

[0076] In this embodiment, orientation estimation 31 is completed in conjunction with step S1.

[0077] In this embodiment, the turning determination 32 is based on the turning angle estimation 22. By measuring the turning angle θ in the horizontal plane, when the cumulative offset between the turning angle and the terrain of the target to be moved exceeds a preset threshold, it is determined to be a turning mode. The threshold can be set according to the specific application scenario.

[0078] In this embodiment, the zero-speed determination is based on the walking speed estimation 21. By continuously detecting the walking speed over a period of time, when the output of each sampled speed in the detection window is continuously zero, it is determined to be a zero-speed mode. At this time, it can be considered that the prosthesis wearer has stopped moving.

[0079] In this embodiment, as Figure 5 As shown, step 3 determines the occurrence of movement mode switching: starting from the moment the target terrain obstacle that can be moved is first detected, until the estimated distance the prosthetic wearer reaches in the forward direction, if the cumulative calculated turning angle θ in the horizontal plane and the cumulative offset of the obstacle target exceed the preset threshold θ0 during this period, the prosthetic wearer is determined to be in a turning state; if the walking speed is continuously detected to be zero within the preset detection window during this period, the prosthetic wearer is determined to be in a stopped state; if neither turning nor stopping occurs during this period, when the target is in the forward direction and reaches the predetermined distance (preferably set to half a step length of the user), the next switching state of the prosthetic wearer can be determined to be obstacle-crossing gait mode, and the mode switching can be initiated when the prosthesis takes a step, and the retained transferable terrain type and corresponding orientation information are deleted.

[0080] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for intelligent lower limb prosthesis control mode ahead switching, characterized in that, A six-axis IMU is installed at a body surface position defined by the transfer of the center of mass of a prosthesis wearer, and the x-axis, y-axis of the IMU are aligned with the coordinate axes of a horizontal plane in the anatomical plane; the method comprises the following steps: S1: monocular vision-based target terrain detection, and orientation description of the target terrain; Wherein the monocular vision-based target terrain detection includes: realizing the identification of the target terrain based on the SSD target detection network, the target terrain refers to obstacles; obtaining the image coordinate information of the target terrain frame in the field of view, and then calculating the orientation information of the target terrain and the prosthesis wearer in the horizontal plane at the current time through the orientation calculation link, the orientation information includes: the straight-line distance of the obstacle from the prosthesis wearer in the horizontal plane and the deflection angle of the obstacle in the prosthesis wearer's forward direction in the horizontal plane calculated at the current time; the model deployment of the SSD target detection network includes an offline training stage and an online inference stage; the offline training stage first performs data enhancement on images with target terrain annotation information, and then inputs the images into the SSD target detection network for domain task transfer learning, and outputs the final target detection frame through non-maximum suppression, and stops training when the detection accuracy of the SSD target detection network reaches the preset index, and then exports the trained network model for model deployment; S2: transfer of the center of mass in the horizontal plane; the pose estimation includes: zero drift correction of IMU data, walking speed estimation and turning angle estimation based on the zero drift corrected data, and inference of the transfer trajectory of the transfer center of mass in the horizontal plane relative to the target obstacle; the walking speed estimation calculates the transfer speed of the transfer center of mass according to the observation of the accelerometer at time t, and the formula is: ； wherein is the initial velocity of the transfer mass center, is the gravity acceleration vector in the global coordinate system, is the acceleration of the transfer mass center in the global coordinate system; S3: rule determination of motion mode switching combined with the orientation description and pose estimation: combined with the orientation description of the target terrain, the turning and stopping motion cases are excluded through the walking speed and the turning angle, then when the obstacle is in the forward direction and reaches a predetermined distance, it is determined that the motion mode of the prosthesis wearer is switched to the obstacle crossing gait mode.

2. The method of claim 1, wherein the control mode of the intelligent lower-limb prosthesis is switched in advance. The pose estimation of the transfer center of mass in the horizontal plane in S2 is obtained by directly integrating the horizontal plane projection vector of the six-axis IMU data converted from the body coordinate system to the global coordinate system.

3. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-2.

Images