A lower limb exoskeleton motion mode prediction method and system based on a VIO system

Abstract

The present application relates to a kind of lower extremity exoskeleton motion mode prediction method and system based on VIO system, prediction method includes according to the input depth camera field of view environmental information and motion information, user state information is estimated, and constructs three-dimensional point cloud model diagram, constructs topographic information by extracting plane information from the depth map under the current perspective of depth camera and mapping to three-dimensional point cloud model diagram, and user current topography is judged in combination with user state information, accurately predict the motion mode of lower extremity exoskeleton and send switching instruction.System includes visual integrated wearing module, lower extremity sensing module and VIO system operation module, respectively for collecting visual motion data, gait cycle phase and the step of executing exoskeleton motion mode prediction method.The present application can effectively solve the problem of insufficient terrain recognition accuracy, user positioning, motion state estimation is not accurate, to realize the accurate prediction of the switching state of lower extremity exoskeleton motion mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lower extremity exoskeleton movement, and in particular to a lower extremity exoskeleton movement mode prediction method and system based on a VIO system. BACKGROUND

[0002] In order to facilitate daily travel, there are a large number of flat ground, stairs, slopes and other terrains in living places. Through different force and balance control methods, healthy people can freely and smoothly move between these terrains. However, lower extremity dysfunction such as amputation, nerve injury and muscle atrophy makes patients lose the ability to walk, causing great inconvenience in life. Inspired by the way people walk, lower extremity prostheses, exoskeletons and other devices are widely used to assist lower extremity patients in daily walking. In order to adapt to various terrains in daily life, the above-mentioned devices usually save many preset movement modes. Therefore, how to detect the user state and perform coordinated movement has become the main development direction of lower extremity exoskeletons.

[0003] At present, lower extremity exoskeletons and other devices mainly use two types of sensors to obtain the user state, user state detection sensors and environment information detection sensors. The user state detection sensors mainly include inertial measurement units (IMU), electromyography (EMG) and electroencephalogram (EEG), etc. The environment detection sensors mainly include laser radars, depth cameras, etc. The user state detection sensors can directly obtain the user state, but lack prediction function; the environment information detection sensors have prediction potential, but can only roughly estimate or use the user state detection sensors to detect the state. The current research and engineering development direction mainly focuses on using the above-mentioned sensors to classify the terrain gait, or using some preset parameters, such as fixed environment collection angle, to predict the gait, so as to realize the guidance of lower extremity exoskeleton movement mode switching.

[0004] However, the above-mentioned method still has deviations when used for predicting the precise switching moment of the lower extremity exoskeleton movement mode, such as detection delay, state estimation error, etc. Therefore, the present application accurately estimates the user state based on a visual-inertial odometry (VIO), and accurately predicts the lower extremity exoskeleton movement mode by constructing the environment terrain information. SUMMARY

[0005] The purpose of the present application is to provide a lower extremity exoskeleton movement mode prediction method and system based on a VIO system, so as to solve the problem of low estimation accuracy of user movement state, insufficient correlation between user movement and terrain information and ultimately low prediction accuracy of lower extremity exoskeleton movement mode in the prior art.

[0006] To achieve the above object, the present application adopts the following technical solutions:

[0007] The present application provides a lower limb exoskeleton motion mode prediction method based on a VIO system, comprising:

[0008] Obtaining environment information and motion information within the field of view of the depth camera, estimating user state information based on the VIO system input environment information and motion information, and constructing a three-dimensional point cloud model graph of the scene around the depth camera;

[0009] Extracting plane information from the environment information under the current perspective of the depth camera, reconstructing the terrain by mapping the plane information onto the three-dimensional point cloud model graph, and projecting the user's current foothold on the three-dimensional point cloud model graph in combination with the user state information to determine the terrain where the user is currently located;

[0010] In combination with the user state information and the terrain where the user is currently located, the user's foothold in the future period of time is determined, and the motion mode of the lower limb exoskeleton is predicted.

[0011] Further, it further comprises a three-dimensional point cloud model graph construction method: the VIO system initializes the world coordinate and the user's pose according to the input environment information and motion information, and calculates the key frames on the time sequence of the depth camera, uses a plurality of key frames to form point cloud data, and constructs and updates the three-dimensional point cloud model graph of the scene around the depth camera through the point cloud data.

[0012] Further, it further comprises an environment information and motion information acquisition method: the environment information acquisition comprises obtaining a certain number of camera color graphs and depth graphs through the depth camera, and the motion information acquisition comprises obtaining user motion data through the fixed inertial measurement unit on the depth camera, wherein the motion data comprises acceleration and angular velocity of the depth camera motion.

[0013] Further, the user state information comprises the user's pose, speed, and the motion phase of the user in the walking cycle obtained by measuring the pressure from the user's foot.

[0014] Further, it further comprises a terrain information construction method: extracting and updating plane information from the depth graph under the current perspective of the depth camera, mapping the extracted plane information onto the three-dimensional point cloud model graph to match the coincidence degree with the existing plane information, and reconstructing and updating the terrain according to the matched plane information.

[0015] Further, it further comprises a plane information extraction method: sequentially performing plane segmentation and filtering on the depth graph under the current perspective of the depth camera to obtain a complete plane, extracting plane information from the complete plane and storing it, wherein the plane information comprises plane area, plane center point coordinates, and plane normal vector value.

[0016] A system for predicting the motion mode of lower extremity exoskeleton, comprising a visual integrated wearing module, a lower extremity sensing module and a VIO system operation module;

[0017] The visual integrated wearing module comprises a depth camera and an inertial measurement unit fixed on the depth camera, the depth camera is used to collect a certain number of camera color graphs and depth graphs, and the inertial measurement unit is used to detect the motion data of the user when the user moves, the motion data comprising acceleration and angular velocity of the depth camera motion;

[0018] The lower extremity sensing module is a pressure insole, which is used to obtain the motion stage of the user in the walking cycle by measuring the pressure from the foot of the user;

[0019] The signal output ends of the visual integrated wearing module and the lower extremity sensing module are connected with the signal input end of the VIO system operation module, and are configured to execute the steps of the lower extremity exoskeleton motion mode prediction method.

[0020] Further, the VIO system operation module comprises a sensing and mapping unit, a plane extraction and terrain reconstruction unit and a lower extremity exoskeleton motion mode prediction unit;

[0021] The sensing and mapping unit is used to obtain the environment information and motion information in the field of view of the depth camera, and estimate the user state information through the input environment information and motion information, and construct a three-dimensional point cloud model graph for the scene around the depth camera;

[0022] The plane extraction and terrain reconstruction unit is used to extract plane information from the environment information under the current view angle of the depth camera, reconstruct the terrain by mapping the plane information to the three-dimensional point cloud model graph, and project the current foot point of the user on the three-dimensional point cloud model graph in combination with the user state information to judge the terrain where the user is currently located;

[0023] The lower extremity exoskeleton motion mode prediction unit is used to judge the foot point of the user in a future period of time in combination with the user state information and the terrain where the user is currently located, and predict the motion mode of the lower extremity exoskeleton.

[0024] The present application has the following beneficial effects due to the above technical solutions:

[0025] The VIO system inputs the environment information and motion information in the field of view of the depth camera, completes VIO system initialization, mapping, terrain reconstruction and other work, extracts plane information from the depth map under the current view angle of the depth camera, maps the plane information to a three-dimensional point cloud model graph to construct terrain information, judges the current terrain of the user, and accurately predicts the motion mode of the lower limb exoskeleton and sends a switching instruction in combination with the estimated user state information. The application can solve the problems of insufficient terrain recognition accuracy, inaccurate user motion state estimation, inaccurate motion mode switching timing judgment and the like, and has strong compatibility for different wearers, different wearing positions, different sensing hardware and different assistance hardware. BRIEF DESCRIPTION OF DRAWINGS

[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application. Throughout the drawings, like reference numerals will be used to designate like components. In the drawings:

[0027] Figure 1 is a flow diagram of a lower limb exoskeleton motion mode prediction method provided by an embodiment of the application;

[0028] Figure 2 is a schematic block diagram of the overall structure of a lower limb exoskeleton motion mode prediction system provided by an embodiment of the application;

[0029] Figure 3 is a schematic block diagram of the operation structure of a VIO system operation module of a lower limb exoskeleton motion mode prediction system provided by an embodiment of the application.

[0030] The various signs in the drawings represent the following:

[0031] 1, visual integrated wearing module; 11, depth camera; 12, inertial measurement unit; 2, lower limb sensing module; 3, VIO system operation module; 31, sensing and mapping unit; 32, plane extraction and terrain reconstruction unit; 33, lower limb exoskeleton motion mode prediction unit. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be accurately conveyed to those skilled in the art.

[0033] Since the traditional lower limb exoskeleton and other devices are used, the prediction of the switching time of the user lower limb exoskeleton motion mode is easy to deviate. The application provides a lower limb exoskeleton motion mode prediction method and system based on a VIO system, based on the VIO system, the user state information is estimated according to the input depth camera field of view environmental information and motion information, and the three-dimensional point cloud model graph of the scene around the depth camera is constructed, the terrain is reconstructed by extracting the plane information under the current view angle of the depth camera, the motion posture of the user in a period of time in the future is judged combined with the user state information, and the motion mode of the lower limb exoskeleton is predicted. It effectively solves the problems of insufficient terrain recognition accuracy, inaccurate user positioning and motion state estimation, inaccurate motion mode switching time judgment and the like.

[0034] The scheme of the application will be described in detail through examples.

[0035] Embodiments

[0036] As shown in Figure 1 The application provides a lower limb exoskeleton motion mode prediction method based on a VIO system, comprising:

[0037] The environmental information and motion information in the field of view of the depth camera 11 are obtained, the user state information is estimated based on the VIO system input environmental information and motion information, and the three-dimensional point cloud model graph of the scene around the depth camera 11 is constructed. Among them, the user state information includes the pose, speed of the user and the motion stage of the user in the walking cycle by measuring the pressure from the user's foot, the comprehensive posture information including the user's landing time point, motion direction, step length and the like can be obtained by using the obtained user pose, speed information and pressure measurement. Among them, the pose of the user includes the position and attitude of the user.

[0038] The plane information is extracted from the environmental information under the current view angle of the depth camera 11, the terrain is reconstructed by mapping the plane information to the three-dimensional point cloud model graph, and the current landing point of the user is projected on the three-dimensional point cloud model graph combined with the position information contained in the user state information, the terrain where the user is currently located is judged;

[0039] Combined with the pose, speed information contained in the user state information and the terrain where the user is currently located, the landing point of the user in a period of time in the future is judged, and the motion mode of the lower limb exoskeleton is predicted.

[0040] As described above, the VIO system is an algorithm that fuses camera and IMU data to realize simultaneous localization and mapping. Based on the VIO system, further, the VIO system can initialize the world coordinate and the pose of the user according to the input environment information and motion information, and calculate the key frames in the time sequence of the depth camera, use a plurality of key frames to form point cloud data, and construct and update a three-dimensional point cloud model graph of the scene around the depth camera 11 through the point cloud data. Among them, the generation and update of the three-dimensional point cloud model graph also includes updating and weighted voting of point cloud data at different times according to the weight to smooth the time sequence information.

[0041] Further, the method for obtaining environment information and motion information is: the obtaining of environment information includes obtaining a certain number of camera color graphs and depth graphs through the depth camera 11; the obtaining of motion information includes obtaining the motion data of the user by using the fixed inertial measurement unit 12 on the depth camera 11. Among them, the motion data includes the acceleration and angular velocity of the depth camera 11 motion.

[0042] Further, the lower extremity exoskeleton motion mode prediction method of the present application further includes a method for constructing terrain information: extracting and updating plane information from the depth graph under the current view angle of the depth camera 11, mapping the extracted plane information to the three-dimensional point cloud model graph, and performing coincidence matching with the existing plane information (the plane mapped before in the time sequence of the depth camera), and reconstructing and updating the terrain according to the matched plane information (updating the matched plane information). Such mapping is only an attribute of the three-dimensional point cloud model graph and can be operated by dictionary (hash), which can speed up the running speed.

[0043] Further, the method for extracting plane information includes sequentially performing plane segmentation and filtering on the depth graph under the current view angle of the depth camera 11 to obtain a complete plane, extracting plane information from the complete plane and storing it, wherein the plane information includes plane area, plane center point coordinates and plane normal vector value. Preferably, the extracted plane filtering can use various methods, such as plane overall mean square error, area, completeness, etc. Conventional methods for filtering, and the coincidence matching can also take various methods, such as intersection-over-union, time sequence voting method, etc.

[0044] As described above, due to hardware and user state estimation reasons, the same plane will have a certain deviation on different frame depth graphs, so the system will update the plane information in real time according to the user's position and the newly acquired sensing information. And the construction and update of the terrain can strengthen the geometric features of the terrain and further exclude the interference of non-terrain plane information. Among them, the three-dimensional geometric features of the specific terrain include terrain area, terrain and horizontal plane angle, and spatial relationship with the surrounding plane.

[0045] The specific prediction steps are as follows:

[0046] S1, synchronously acquire a certain number of camera color images, depth images (20-30 Hz), and IMU data (>100 Hz) fixed with the depth camera 11 external parameters;

[0047] S2, the VIO system is initialized, and initial pose data of the world coordinates and the user is acquired;

[0048] S3, if the VIO system initialization is successful, the next step is entered, and if the initialization fails, the step S1 is returned;

[0049] S4, based on the sensing information acquired in step S1, the key frames on the time sequence of the depth camera 11 are calculated through the camera color images, depth images and other information, and the user pose information is updated, and the speed information is calculated;

[0050] S5, a three-dimensional point cloud model graph is constructed by using the key frames and the pose data of the key frames, the three-dimensional point cloud model graph and the corresponding point cloud are weighted and voted, and the three-dimensional point cloud model graph is updated in real time;

[0051] S6, the depth image under the current view angle of the depth camera 11 is subjected to plane information extraction, and is matched with the existing plane on the three-dimensional point cloud model graph in time sequence, and new plane addition and plane update are performed;

[0052] S7, the terrain is constructed and the terrain information is updated by using the updated plane information;

[0053] S8, the current foot point of the user is projected by using the VIO system, and the terrain where the user is located is judged through the terrain information, the next foot point in time sequence is predicted in combination with the user state information and the terrain where the user is located, and then the time of switching the lower limb exoskeleton mode is predicted;

[0054] S9, if the VIO system is not ended by the user, the step S4 is returned to continue.

[0055] The present application inputs the environment information and motion information around the depth camera 11 based on the VIO system, completes the VIO system initialization, mapping, terrain reconstruction and other work, constructs the terrain information by mapping the plane information extracted from the depth image under the current view angle of the depth camera 11 to the three-dimensional point cloud model graph, judges the terrain where the user is currently located, and accurately predicts the motion mode of the lower limb exoskeleton and sends the switching instruction in combination with the estimated user state information. The present application can solve the problems of insufficient terrain recognition accuracy, inaccurate user motion state estimation, inaccurate motion mode switching time judgment and the like, and has strong compatibility for different wearers, different wearing positions, different sensing hardware and different assistive hardware.

[0056] The application further provides a system for predicting a lower extremity exoskeleton movement mode, comprising a visual integrated wearing module 1, a lower extremity sensing module 2, and a VIO system operation module 3.

[0057] The visual integrated wearing module 1 comprises a depth camera 11 and an inertial measurement unit 12 fixed on the depth camera 11. The depth camera 11 is used to collect a certain number of camera color graphs and depth graphs, and the inertial measurement unit 12 is used to detect motion data of the user when the user moves, wherein the motion data comprises acceleration and angular velocity of the depth camera 11.

[0058] The lower extremity sensing module 2 is preferably a pressure insole, which is used to obtain the motion stage of the user in a walking cycle by means of pressure measurement from the foot of the user. The pressure detection method is not limited to the pressure insole structure, and any method that can obtain user motion stage and attitude data is applicable.

[0059] The signal output ends of the visual integrated wearing module 1 and the lower extremity sensing module 2 are connected to the signal input ends of the VIO system operation module 3, and are configured to perform the steps of the lower extremity exoskeleton movement mode prediction method of the application.

[0060] Further, the VIO system operation module 3 comprises a sensing and mapping unit 31, a plane extraction and terrain reconstruction unit 32, and a lower extremity exoskeleton movement mode prediction unit 33.

[0061] The sensing and mapping unit 31 is used to obtain environmental information and motion information in the field of view of the depth camera 11, and to estimate user state information through the input environmental information and motion information, and to construct a three-dimensional point cloud model graph of the scene around the depth camera 11.

[0062] The plane extraction and terrain reconstruction unit 32 is used to extract plane information from the environmental information under the current view angle of the depth camera 11, to reconstruct the terrain by mapping the plane information to the three-dimensional point cloud model graph, and to project the current foot point of the user on the three-dimensional point cloud model graph in combination with the user state information, so as to determine the terrain where the user is currently located.

[0063] The lower extremity exoskeleton movement mode prediction unit 33 is used to determine the foot point of the user in a future period of time in combination with the user state information and the terrain where the user is currently located, and to predict the movement mode of the lower extremity exoskeleton.

[0064] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A lower extremity exoskeleton motion pattern prediction method based on a VIO system, characterized in that, the lower extremity exoskeleton motion pattern prediction method comprises: obtaining environmental information and motion information in the field of view of a depth camera, inputting the environmental information and motion information into a VIO system to estimate user state information, and constructing a three-dimensional point cloud model graph of the scene around the depth camera; extracting plane information from the environmental information under the current perspective of the depth camera, reconstructing the terrain by mapping the plane information onto the three-dimensional point cloud model graph, and projecting the user's current foot point on the three-dimensional point cloud model graph in combination with the user state information to determine the terrain where the user is currently located; judging the user's foot point in a future period of time and predicting the motion pattern of the lower extremity exoskeleton according to the user state information and the terrain where the user is currently located; and further comprising a method for constructing terrain information: extracting and updating plane information from the depth map under the current perspective of the depth camera, mapping the extracted plane information onto the three-dimensional point cloud model graph to match the degree of coincidence with the existing plane information, and reconstructing and updating the terrain according to the matched plane information. 2.The lower extremity exoskeleton motion pattern prediction method based on the VIO system according to claim 1, characterized in that, further comprising a method for constructing a three-dimensional point cloud model graph: the VIO system initializes the world coordinates and the user's pose according to the input environmental information and motion information, calculates the key frames in the time sequence of the depth camera, uses a plurality of key frames to form point cloud data, and constructs and updates the three-dimensional point cloud model graph of the scene around the depth camera through the point cloud data. 3.The lower extremity exoskeleton motion pattern prediction method based on the VIO system according to claim 2, characterized in that, further comprising a method for obtaining environmental information and motion information: the acquisition of environmental information includes obtaining a certain number of camera color maps and depth maps through the depth camera, and the acquisition of motion information includes obtaining user motion data by using an inertial measurement unit fixed on the depth camera, wherein the motion data includes the acceleration and angular velocity of the depth camera motion. 4.The lower extremity exoskeleton motion pattern prediction method based on the VIO system according to claim 3, characterized in that: the user state information includes the user's pose, speed, and the motion phase of the user in the walking cycle obtained by measuring the pressure from the user's foot. 5.The lower extremity exoskeleton motion pattern prediction method based on the VIO system according to claim 1, characterized in that, further comprising a method for extracting plane information: sequentially performing plane segmentation and filtering on the depth map under the current perspective of the depth camera to obtain a complete plane, extracting plane information from the complete plane and storing it, wherein the plane information includes the plane area, the plane center point coordinates, and the plane normal vector value. 6.A lower extremity exoskeleton motion pattern prediction system, characterized in that: the system comprises a visual integrated wearing module, a lower extremity sensing module, and a VIO system operation module. The visual integrated wearable module comprises a depth camera for collecting a certain number of camera color images and depth images, and an inertial measurement unit fixed on the depth camera for detecting motion data of the user when the user moves, the motion data including acceleration and angular velocity of the depth camera motion; The lower limb sensing module is a pressure insole, which is configured to obtain the motion phase of the user in a walking cycle by measuring the pressure from the foot of the user; The signal output ends of the visual integrated wearable module and the lower limb sensing module are connected to the signal input ends of the VIO system operation module, and are configured to perform the steps of the lower limb exoskeleton motion mode prediction method according to any one of claims 1-5.

7. The lower limb exoskeleton motion mode prediction system according to claim 6, wherein The VIO system operation module comprises a sensing and mapping unit, a plane extraction and terrain reconstruction unit, and a lower limb exoskeleton motion mode prediction unit; The sensing and mapping unit is configured to obtain the environment information and motion information within the field of view of the depth camera, and to estimate the user state information based on the input environment information and motion information, and to construct a three-dimensional point cloud model graph of the scene around the depth camera; The plane extraction and terrain reconstruction unit is configured to extract plane information from the environment information under the current view angle of the depth camera, to map the plane information to the three-dimensional point cloud model graph to reconstruct the terrain, and to project the current foot point of the user on the three-dimensional point cloud model graph in combination with the user state information to determine the terrain where the user is currently located; The lower limb exoskeleton motion mode prediction unit is configured to determine the foot point of the user in a future period of time in combination with the user state information and the terrain where the user is currently located, and to predict the motion mode of the lower limb exoskeleton.

Images