An exoskeleton robot gait autonomous decision-making method based on environment perception
By using terrain parameter recognition based on point cloud convolution and deep learning algorithms, the exoskeleton robot has achieved autonomous decision-making to switch gait according to the current terrain, solving the human-machine interaction problem in complex terrain and improving the comfort and safety of patients.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-07-06
- Publication Date
- 2026-05-29
AI Technical Summary
Existing exoskeleton robots lack the ability to interact with the environment in everyday life scenarios, and cannot autonomously decide on a gait suitable for the current terrain. In particular, there are safety hazards when switching gaits in complex terrain.
A terrain parameter recognition method based on point cloud convolution is adopted, which combines visual sensors and deep learning algorithms to detect ground type and terrain size parameters in real time. The robot's motion state is obtained through an inverted pendulum model and foot pressure information, and a gait trajectory suitable for the wearer and the environment is planned to achieve autonomous gait decision-making.
It improves the exoskeleton robot's ability to perceive and interact with the environment, enabling it to switch gait without external operation, enhancing the comfort and safety of patients wearing it, and allowing it to more smoothly complete global path walking in complex terrain.
Smart Images
Figure CN116901036B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of automatic control, specifically an automatic control method for robotic exoskeletons. Background Technology
[0002] In recent years, many lower limb exoskeleton robots have been designed for medical rehabilitation and daily mobility assistance. Currently, when applied to everyday life scenarios, exoskeleton robots often employ a fixed gait and rely on gesture recognition, voice recognition, and user-controlled button interfaces for human-machine information and movement transmission. This lack of interaction between the human-machine system and the surrounding environment fails to meet the daily needs of patients.
[0003] In practical applications, existing exoskeleton robots can assist patients in rehabilitation training within fixed gait patterns. Common typical terrains include walking, climbing stairs, and ascending / descending slopes. For a person to complete a global path, it is usually composed of a combination of different individual gait patterns. For example, walking to climbing stairs requires combining flat-ground gait with stair-climbing gait. This is not a simple task and requires consideration of multiple factors. For instance, switching from flat-ground gait to stair-climbing gait requires considering factors such as the current distance to the stairs and the height of the stairs to ensure a proper gait transition at the appropriate location and safe stepping onto the stairs. When paraplegic patients face complex terrain, they often need to switch gait patterns to adapt to changes in the external environment. Environmental perception in complex terrain is the foundation for exoskeleton robots to make autonomous gait switching decisions. The ability to autonomously decide on a gait suitable for the current terrain is an urgent problem to be solved. Summary of the Invention
[0004] This invention solves the problem of environmental information interaction between human-machine systems and the environment. Terrain parameter recognition based on point cloud convolution can provide accurate information for exoskeleton robots to adjust their gait in real time. Real-time, efficient and reliable environmental perception makes the robot's communication with the environment more natural and barrier-free, improving the intelligence and humanization of the exoskeleton robot system.
[0005] The technical solution of this invention is an autonomous gait decision-making method for exoskeleton robots based on environmental perception. The method includes three modules: a data acquisition module, an environmental perception module, and an autonomous gait decision-making module.
[0006] The data acquisition module is used to collect depth data and RGB data of images during robot walking and transmit the data to the environment perception module.
[0007] The minimum range of the visual sensor in the data acquisition module is set to within 3.5m directly in front of the exoskeleton, ensuring that the visual sensor can see the information in front of the current position in real time.
[0008] The environmental perception module includes a ground type detection module and a terrain size parameter detection module. The terrain type detection module obtains terrain parameters based on the RGB data from the data acquisition module, and the terrain size parameter detection module uses point cloud data to perform staircase size parameter regression to obtain terrain parameters.
[0009] The specific calculation method in the ground type detection module is as follows:
[0010] After image preprocessing, the RGB images acquired by the data acquisition module are entered into the semantic segmentation network model and classified into four categories based on the region detection results: stairs, slopes, flat ground, and obstacles.
[0011] Step A1: Image alignment;
[0012] Align the RGB image with the depth image so that the pixels of the depth image correspond to those of the RGB image;
[0013] Step A2: Use a semantic segmentation network model for region detection;
[0014] The semantic segmentation network model downsamples the original image by 8 times to obtain a feature map, inputs the feature map into the Pyramid Pooling Module, and adds the output of the Pyramid Pooling Module to the feature map. Then, it is upsampled by convolution and 8 times bilinear interpolation to obtain the result.
[0015] Step A3: Region detection;
[0016] The semantic segmentation results are used to determine the scene based on priority, which is: obstacle > stairs > ramp > flat ground. When there is an area in the perception area that the robot cannot walk on at present, it is treated as an obstacle scene, and the unwalkable information is retained as additional decision information. When the scene category is set as stairs and flat ground, specific parameter information is further detected.
[0017] The calculation method of the terrain size parameter detection module is as follows:
[0018] Step B1: Point cloud processing;
[0019] Step B11: Coordinate transformation;
[0020] Obtain color and distance information for each pixel, and transform all point cloud data into point cloud data in the same three-dimensional coordinate system;
[0021] Step B12: Point cloud denoising;
[0022] First, voxel downsampling is performed on the point cloud, and then outliers are removed using a statistical outlier removal method.
[0023] Then, a random sampling method is used to randomly extract the same number of points from each depth image as new point cloud data;
[0024] Step B2: The processed point cloud data is processed through two sets of feature extraction parts to extract global and local features of the point cloud and combine them. Then, it is passed through a PointNet layer to further extract the features obtained above. Finally, after passing through a fully connected layer, the size parameters of the current terrain are output.
[0025] The gait autonomous decision-making module;
[0026] The robot's motion state is obtained by combining an inverted pendulum model with foot pressure information. Then, based on the obtained human-machine motion state, environmental perception information is integrated to parametrically describe the gait of the transitional terrain. In order to enable the human-machine system to assist human movement, and according to the human's movement intention, the trajectory of the swing phase end is modulated through dynamic motion primitives to plan a gait trajectory suitable for the wearer and the characteristics of the environment, and to make single-step dynamic adjustments based on environmental information. Finally, the hip and knee joint angles of the swinging leg can be obtained through inverse kinematics, and the generated gait trajectory is ultimately triggered to meet the gait requirements of human-machine-environment transitional terrain.
[0027] Step C1: Gait parameterization;
[0028] With the wearer's safety as the key factor, constraints are set, and the defined safety constraint parameters are:
[0029] , The minimum safe distance and minimum safe height for a robot to climb stairs are the forward distance between the ankle joint and the stairs.
[0030] The starting point for dynamic adjustments during the robot's gait transition phase, and ;
[0031] The safe walking distance between the robot and the stairs. , here This indicates the stride length of the swinging leg, ensuring that the forward distance between the landing point and the stairs is the minimum safe distance for the robot to climb the stairs;
[0032] Set the transition point between the terrain transition phase and the transition phase, that is, the position of the supporting leg in the last step of the flat gait;
[0033] , : The maximum stride length and maximum height that a robot can walk on in typical flat terrain;
[0034] Based on the model of the terrain environment, the terrain parameters in the transformation stage are described as follows: , respectively representing the starting point of the transition phase, the landing point of the oscillation phase in the sagittal plane when entering the transition phase, and the different step lengths of the intermediate movement;
[0035] Similarly, the parameters of the transition phase are described as follows: These represent the starting and ending points of the transitional terrain, respectively. Here, the ending point indicates the landing point of the swinging leg on the first step. These represent the height and width of the corresponding staircases, respectively.
[0036] When ascending stairs and during the transition phase, consider the forward steps. Therefore, it is described as follows , representing the support point of the current step and the landing point of the next step, respectively;
[0037] To enable the exoskeleton robot to smoothly and naturally ascend stairs, it is necessary to first determine the forward distance from the foot to the stairs based on the height of the stairs. In the formula Indicates the incline of the stairs;
[0038]
[0039] Step C2: Gait parameter representation during the transition phase;
[0040] During the gait transition phase when approaching stairs, the exoskeleton robot's gait is parameterized; during walking on flat ground, the single-step step length is adjusted as if the center of mass only changes in the sagittal X-axis direction and not in the Z-axis direction; the coordinates of the swing phase start point are determined. Maximum safe feasible distance and single-step step length sequence The following formula describes the process;
[0041]
[0042] Combining the relationship between stride length and step length, we can obtain the relationship between the two:
[0043]
[0044] in, Indicates stride length when walking. Indicates the step length of the previous movement;
[0045] Considering the variable stride length walking of a normal person approaching an object, a weighting coefficient is added to the single-step stride length. Its value Combined with the forward distance from the stairs Safe walking distance between the robot and the stairs The gait parameters are described in the following formula;
[0046]
[0047] Step C3: Representation of gait parameters during the transition phase;
[0048] For scene modeling of the transition phase from flat ground to stairs, the parameters of the transition phase are described, including the points of the terrain interface. The foothold of the first step and the height and width of the first step The exoskeleton robot is guaranteed to transition safely when the following constraint condition is met.
[0049]
[0050] Therefore, the step size and step height for the transition phase are:
[0051]
[0052] The foot closest to the safe distance from the forward staircase becomes the supporting leg, while the swinging leg on the other side first reaches the tread of the first step from the transition point and becomes the supporting leg; at this time, the supporting leg on the other side becomes the swinging leg and reaches the tread of the next step from the point closest to the safe distance from the staircase, completing the transition from the flat ground to the staircase.
[0053] Step C4: Gait parameter representation during the stair-climbing phase;
[0054] The modeling of the stair-climbing scene is based on the fact that normal people walk in a crisscross pattern on stairs, combined with the parameter description of the stair-climbing terrain, including the starting point being the current step's foot position. The point where the foot lands on the forward steps serves as the endpoint. and the height and width of the forward steps Therefore, the step length and step height of the staircase terrain are defined as follows:
[0055] ;
[0056] Step C5: Autonomous gait decision-making based on environmental parameters;
[0057] When an exoskeleton robot detects staircase terrain while walking forward, the first key information for decision-making is the size and distance of the staircase to determine whether the exoskeleton robot needs to change its stride length to approach the staircase and continue walking; based on gait parameters and terrain parameters, when there are no stairs in the walking environment or there are stairs but the forward distance is limited, the robot will adjust its stride accordingly. At this time, the human-machine system will maintain its original gait trajectory.
[0058] When the forward distance of the stairs meets the requirement When this occurs, it indicates that the exoskeleton robot has entered the transition phase, requiring dynamic adjustment of the single-step length; otherwise, the forward distance of the stairs meets the requirements. If the dimensions of the stairs do not exceed the robot's range of motion, it indicates that the exoskeleton robot has entered the transition phase. It needs to recalculate the parameters of the stairs and the starting position of the swing leg to take the first step in the transition phase. Otherwise, it cannot enter the transition phase and cannot enter the new terrain.
[0059] when When the forward distance is too small, the exoskeleton robot cannot walk normally.
[0060] This invention not only enables the lower limb exoskeleton robot to perceive and interact with the environment without requiring external operation when switching gait, which is beneficial to the user, but also improves the comfort of patients wearing the lower limb exoskeleton robot in their daily lives, and allows them to complete a global path more smoothly. Attached Figure Description
[0061] Figure 1 This is a schematic diagram of the process of the present invention;
[0062] Figure 2 A schematic diagram showing the exoskeleton robot model and sensor installation locations;
[0063] Figure 3 A schematic diagram of region detection based on semantic segmentation.
[0064] Figure 4 This is a schematic diagram of parameter detection based on point cloud convolution.
[0065] Figure 5 The diagrams are schematic diagrams of the point cloud processing flow. (a) is a schematic diagram of the original point cloud before processing; (b) is a schematic diagram of the point cloud after downsampling; (c) is a schematic diagram of the point cloud after removing discrete points; (d) is a schematic diagram of the point cloud without random sampling, and a schematic diagram of the point cloud with the required number of points taken in sequence; (e) is a schematic diagram of the point cloud with the required number of points randomly selected.
[0066] Figure 6 Diagram of gait control structure;
[0067] Figure 7 A schematic diagram illustrating the gait phase division for terrain transitions;
[0068] Figure 8 A schematic diagram of scene modeling for the transition phase;
[0069] Figure 9 A schematic diagram of scene modeling for the transition phase;
[0070] Figure 10A schematic diagram of a scene where someone is going up the stairs;
[0071] Figure 11 A schematic diagram of an exoskeleton robot walking;
[0072] Figure 12 Flowchart for autonomous decision-making during transitional gait in a lower limb exoskeleton robot;
[0073] Figure 13 This is a schematic diagram of trajectory planning based on dynamic motion primitives;
[0074] Figure 14 This is a system verification diagram;
[0075] Figure 15 This is a diagram illustrating the addition of a loss during the training of the pyramid module. Implementation
[0076] 1) Data Acquisition Module
[0077] The device used for data acquisition in this invention is an Intel RealSense D435i, which is used to collect depth data and RGB data of images of the robot walking, and then transmit the data to the environmental perception module.
[0078] 2) Environmental perception module
[0079] The environmental perception module of this invention includes a ground type detection module and a terrain size parameter detection module. The terrain type detection module uses RGB data from the data acquisition module. This invention employs the semantic segmentation network PSPNet, which has shown good segmentation performance on other datasets, and uses deep learning methods to perform end-to-end identification of the stair area, obtaining category information in the environmental scene, such as flat ground, slope, stairs, and obstacles. The robot can then respond with gait switching based on this scene type information. To further improve the gait progression, a terrain size parameter detection module is needed. This invention uses point clouds to regress stair size parameters. Based on PointNet and PointNet++, a deep learning framework is designed. This algorithm can directly input unordered point clouds and output the width and height of the stairs to obtain the terrain parameters.
[0080] 3) Gait Autonomous Decision-Making Module
[0081] This module introduces an autonomous gait parameter decision-making method for lower limb exoskeleton robots facing transitional terrain environments from a gait decision-making perspective. The robot's motion state is acquired by combining an inverted pendulum model with foot pressure information. Then, based on the acquired human-machine motion state, environmental perception information is integrated to provide a parameterized description of the gait in transitional terrain. To enable the human-machine system to assist human movement, and based on the user's movement intention, dynamic motion primitives modulate the trajectory of the swing phase's end point, planning a gait trajectory suitable for the wearer and environment characteristics, and making dynamic adjustments for each step based on environmental information. Finally, inverse kinematics is used to obtain the hip and knee joint angles of the swinging leg, ultimately triggering the generated gait trajectory to meet the gait requirements of the human-machine-environment transitional terrain.
[0082] In addition, the human, as the primary operator, guides the robot's walking direction. When the human is walking normally, they can control the walking direction of the human-machine system based on the current terrain and gait.
[0083] 1) Sensor Installation
[0084] In this invention, the sensor used in the data acquisition module can be installed at the waist, and the minimum range of the visual sensor is set to within 3.5m directly in front of the exoskeleton, ensuring that the camera can see the information in front of the current position in real time. For example... Figure 2 As shown;
[0085] 2) Region detection methods based on semantic segmentation
[0086] The acquired RGB images undergo image preprocessing before entering the semantic segmentation network model. The regions are then classified into four categories based on detection results: stairs, slopes, flat ground, and obstacles. A flowchart is shown below. Figure 3 As shown:
[0087] (2.1) Image alignment. Before processing, due to the positional difference between the two cameras that acquired the data, a certain area in the image is located differently in the unaligned color image and depth image, resulting in a large deviation. Therefore, it is necessary to first align the RGB image with the depth image so that the pixels of the depth image correspond to those of the RGB image.
[0088] (2.2) Semantic Segmentation Network Model. To distinguish scene regions, the semantic segmentation network PSPNet, which has a good segmentation effect, was selected. Its network architecture incorporates more contextual and multi-scale information. The network architecture downsamples the original image by 8 times to form a feature map. The feature map is input into the Pyramid Pooling Module (PPM) and its output is added. Finally, the result is obtained by convolution and upsampling by 8 times bilinear interpolation.
[0089] PSPNet employs a network structure to more effectively acquire and combine prior information at different scales. This module is a hierarchical global prior module containing feature information at different scales, called the pyramid pooling module. The pyramid pooling module fuses features at different scales; the coarsest feature level is global pooling, generating global features. The remaining pyramid levels divide the feature map into different sub-regions and obtain input features at different scales. By changing the number of pyramid levels and the size of the pooling kernels at different levels, features from different stages can be obtained. To keep the weights of the global features constant, if the number of pyramid levels is N, a 1×1 convolutional layer is used after each pyramid level to reduce the dimension of the context representation to 1 / N of the original context representation. Then, bilinear interpolation is used to directly upsample the low-dimensional feature map to obtain features of the same size as the original feature map. Finally, the context features are concatenated to form the final pyramid pooled global features. The pyramid structure extracts features in different regions at different scales using pooling kernels of different sizes; therefore, when setting the size of multi-level kernels, a reasonable interval should be maintained in the representation. The pyramid pooling module in this network is a four-level module, with pooling layer sizes of 1×1, 2×2, 3×3 and 6×6.
[0090] Since this algorithm uses ResNet to obtain feature maps before the pyramid module, increasing the network depth may make network optimization more difficult. Therefore, a loss function is added during the training phase to help the network learn, as follows: Figure 15 As shown;
[0091] Therefore, the optimization problem of deep networks is decomposed into two, each more easily reaching its optimum. In addition to using softmax loss to train the main branch of the final classifier, another softmax loss is applied after the fourth stage of network propagation to aid learning. The softmax loss is shown in Equation (1-1). This algorithm passes both losses through all previous layers, rather than just through a few shallow layers of the network as in relay backpropagation.
[0092] (1-1)
[0093] In the formula, y(i) is the true value corresponding to the i-th sample; Pi,y(i) is the predicted probability of the i-th sample in its class; and n is the batch size.
[0094] To balance the impact of the loss on network learning, weights are added to both losses to balance the auxiliary loss and help the network optimize the learning process. The main branch loss has a larger weight than the overall loss. The formula for the network loss is shown in Equation (1-2). During the testing phase, only the well-optimized main branch is used for the final prediction, without adding auxiliary branches. This deep supervised training strategy based on ResNet's FCN can be applied under different experimental conditions, and when combined with a pre-trained ResNet model, it can help the model converge more quickly.
[0095] (1-2)
[0096] In the formula, loss is the final total loss of the network, main_loss is the loss of the main branch (loss1), and aux_loss is the auxiliary loss (loss2).
[0097] During the training of this model, a pre-trained ResNet is first used to extract feature maps. To expand the model's receptive field and better obtain multi-scale contextual information, a dilated convolution strategy is also employed in this part. The feature maps are then input into a pyramid module, where the pooling kernels of the four-level pyramid cover the entire, half, and a small portion of the feature image, respectively, and these are fused into global prior information. Finally, a prediction map is generated through convolutional layers. This algorithm yields the semantic segmentation results of the region color image.
[0098] (2.3) Area detection.
[0099] The semantic segmentation results are used to determine the scene based on priority, which is: obstacle > stairs > ramp > flat ground. When there is an area within the perception area that the robot cannot currently walk on, it is treated as an obstacle scene, and the unwalkable information is retained as additional decision information; when the scene category is set to stairs, further detection of specific parameter information is required (3); the same applies to flat ground.
[0100] 3) Point cloud-based parameter detection method:
[0101] (3.1) Point cloud processing.
[0102] (3.1.1) Coordinate transformation. After (3.1), the color information and distance information of each pixel can be obtained. However, the pixel data is different from the data in the world coordinate system. Therefore, it is transformed into point cloud data in the three-dimensional coordinate system.
[0103] (3.1.2) Point Cloud Denoising. After the depth map acquired by the depth camera is converted into a point cloud, a large amount of noise is generated due to equipment accuracy errors and segmentation errors in the stairwell area. A large number of discrete points need to be removed, and the point cloud data density also needs smoothing due to equipment errors. Furthermore, since there is a one-to-one correspondence between pixels in the depth map and points in space, an excessive number of points after the depth map is converted to a point cloud will affect the speed of subsequent processing. To improve the efficiency of network training, point cloud filtering and sampling are required on the generated raw point cloud data.
[0104] This invention uses a point cloud processing filtering method. First, the point cloud is voxel downsampled, and then a statistical outlier removal method is used.
[0105] Then, a random sampling method is used. Since the number of point cloud points obtainable from each depth map is not the same, even if the same voxels are selected, the number of remaining points after downsampling will also differ. The number of remaining points after removing discrete points will also differ, so a random sampling method is needed to obtain the required number of points from the point cloud. The point cloud processing flow is as follows: Figure 5 As shown.
[0106] (3.2) PointNet++ network model. The structure of this part is to input the processed point cloud data into the network. The point cloud data goes through two sets of feature extraction parts to extract the global and local features of the point cloud and combine them. Then it goes through a PointNet layer to further extract the features obtained above. Finally, after passing through a fully connected layer, the current terrain size parameters are output, such as the height and width of the stairs.
[0107] 4) Gait Autonomous Decision-Making Module
[0108] (4.1) Gait parameterization
[0109] This paper sets constraints with the wearer's safety as the key element, defining the following safety constraint parameters:
[0110] 1. , The minimum safe distance and minimum safe height for a robot to climb stairs can be understood as the forward distance between the ankle joint and the stairs.
[0111] 2. The starting point for dynamic adjustments during the robot's gait transition phase, and .
[0112] 3. The safe walking distance between the robot and the stairs ( ), here This indicates the stride length of the swinging leg, ensuring that the forward distance between the landing point and the stairs is the minimum safe distance for the robot to climb the stairs.
[0113] 4. : Set the transition point between the terrain transition phase and the transition phase, that is, the position of the supporting leg in the last step of the flat gait.
[0114] 5. , : The maximum stride length and maximum height that the robot can walk on in normal flat terrain.
[0115] Based on environmental perception, to accurately and stably obtain terrain dimension information in the flat-to-staircase transition environment, we obtain the current environmental parameter information during the double-support stage, because at this stage, the single supporting leg is relatively stable and the human body is in an upright position. For example... Figure 7 As shown, the blue area represents the transition phase from flat ground to stairs, and the green area represents the transition phase. The transition phase involves dynamic gait planning, primarily dynamic step length adjustment, gradually moving towards the stairs to adapt to the optimal position and reach the transition phase. The dual-support state during the transition phase is a consideration in transition gait planning. As for entering the staircase terrain scenario after the transition phase, it can be considered as planning the end trajectory of the gait under the staircase terrain between the endpoints of the specified walking mode.
[0116] Based on the model of the terrain environment, the terrain parameters in the transformation stage can be described as follows: , respectively, represent the starting point of the transition phase, the landing point of the oscillating phase in the sagittal plane when entering the transition phase, and the different step lengths of the intermediate movement.
[0117] Similarly, the parameters of the transition phase are described as follows: These represent the starting and ending points of the transitional terrain, respectively. Here, the ending point indicates the landing point of the swinging leg on the first step. These represent the height and width of the corresponding staircase, respectively.
[0118] The process of going up the stairs is similar to the transition phase, mainly considering the forward steps. Therefore, it can be simplified as follows: , representing the support point of the current step and the landing point of the step in front, respectively.
[0119] To enable the exoskeleton robot to smoothly and naturally ascend stairs, it is necessary to first determine the forward distance from the foot to the stairs based on the height of the stairs. In the formula Indicates the inclination of the staircase.
[0120]
[0121] (4.2) Representation of gait parameters during the transition phase
[0122] The gait transition phase of an exoskeleton robot when approaching stairs, such as Figure 8 As shown, the gait of the exoskeleton robot can be parameterized. During flat ground walking, it can be considered as a single-step step length adjustment where the center of mass changes only in the sagittal X-axis direction and remains unchanged in the Z-axis direction. Combined with the parameter description of the transition phase, this mainly includes the coordinates of the starting point of the swing phase. Maximum safe feasible distance and single-step step length sequence Obviously, the parameters are described as shown in the following formula.
[0123]
[0124] Combining the relationship between stride length and step length, we can obtain the relationship between the two:
[0125]
[0126] Since we consider the variable stride length walking of a normal person approaching an object, we add a weighting coefficient to the single-step stride length. Its value The forward distance from the stairs needs to be considered. Safe walking distance between the robot and the stairs The gait parameters are described in the following formula.
[0127]
[0128] (4.3) Representation of gait parameters during the transition phase.
[0129] For scene modeling of the transition phase from flat ground to stairs, the parameters of the transition phase are described, including the points of the terrain interface. The foothold of the first step and the height and width of the first step The exoskeleton robot is guaranteed to transition safely when the following constraint condition is met.
[0130]
[0131] Therefore, the step size and step height for the transition phase can be defined as shown in the following formula.
[0132]
[0133] As can be seen from the above formula, the step length and step height of the transition phase depend on the height and width of the first step, the minimum safe distance and minimum safe height for the robot to climb the stairs, and the distance from the robot to the stairs.
[0134] The foot closest to the safe distance from the staircase becomes the supporting leg. The other swing leg, starting from the transition point, first reaches the tread of the first step and becomes the supporting leg. Then, the other supporting leg becomes the swing leg, moving from the point closest to the safe distance from the staircase to the tread of the next step, completing the transition from flat ground to the stairs. Detailed transition stages can be illustrated as follows: Figure 9 The scene modeling description is shown.
[0135] (4.4) Gait parameters during the stair climbing phase.
[0136] Modeling the upstairs scene, such as Figure 10 As shown. Based on the fact that normal people walk in a crisscross pattern on stairs, and combined with the description of the staircase terrain parameters, including the starting point being the current step's foot position. The point where the foot lands on the forward steps serves as the endpoint. and the height and width of the forward steps Therefore, the step length and step height of the staircase terrain can be defined as shown in the following formula.
[0137]
[0138] As can be seen from the above formula, the step length and step height when going up stairs depend entirely on the size information of the steps.
[0139] (4.5) Autonomous gait decision-making based on environmental parameters
[0140] Based on the exoskeleton robot's perception of environmental information, the size information of the stairs on the ground is obtained. Combined with the gait parameter description in the previous section, this section mainly discusses the autonomous gait decision-making of the exoskeleton robot. This method takes into account that normal people will choose appropriate locations to change their gait during walking according to the terrain environment; especially in mixed terrain scenarios, advance planning is required after entering the mixed terrain. Compared with previous walking modes, this method can avoid collisions between the robot and the wearer during the transition phase when climbing stairs, fundamentally ensuring the safety of the human-machine system's movement. This paper only considers mixed scenarios of flat ground and stairs. A schematic diagram of the exoskeleton robot's walking / climbing stairs is shown below. Figure 11 As shown.
[0141] When the perception system detects the staircase terrain and its parameters in front of the exoskeleton robot during walking on flat ground, the adaptive transition gait autonomous decision-making flowchart is as follows: Figure 12 As shown.
[0142] When an exoskeleton robot detects staircase terrain while walking forward, the first key information for decision-making is the size and distance of the staircase to determine whether the robot needs to change its stride length to approach the staircase and continue walking. Based on the gait and terrain parameters described in the previous section, when there are no stairs in the walking environment or there are stairs but the forward distance is limited... At this time, the human-machine system will maintain its original gait trajectory.
[0143] When the forward distance of the stairs meets the requirement When this occurs, it indicates that the exoskeleton robot has entered the transition phase, requiring dynamic adjustment of the single-step length. Otherwise, the forward distance of the stairs must meet the requirements. If the dimensions of the stairs do not exceed the robot's range of motion, it indicates that the exoskeleton robot has entered the transition phase. The parameters of the stairs and the starting position of the swing leg need to be considered to take the first step in the transition phase. Otherwise, it cannot enter the transition phase and cannot enter the new terrain.
[0144] when When the forward distance is too small, the exoskeleton robot cannot walk normally.
[0145] (5) Gait planning and system verification
[0146] Simultaneously, combining the above method (3) with the detected terrain parameter information, new gait parameters, such as stride length and stride height, can be specified before each step, and the gait trajectory is as follows: Figure 13 As shown, gait trajectories with stride lengths ranging from 70 to 120 cm are planned for walking on flat ground, while gait trajectories adapting to different stair sizes are planned for walking up stairs. This ensures that the exoskeleton robot can fine-tune its gait when the terrain remains unchanged, improving the robot's gait generalization ability. The effectiveness of this method is verified in a real-world scenario, such as... Figure 14 As shown.
[0147] The above methods not only enable the lower limb exoskeleton robot to perceive and interact with the environment, but also eliminate the need for external operation when switching gait, which is beneficial to the user. Furthermore, they improve the comfort of patients wearing the lower limb exoskeleton robot for daily life and allow them to complete a global path more smoothly.
Claims
1. A gait autonomous decision-making method for exoskeleton robots based on environmental perception, comprising three modules: a data acquisition module, an environmental perception module, and a gait autonomous decision-making module; The data acquisition module is used to collect depth data and RGB data of images during robot walking and transmit the data to the environment perception module. The minimum range of the visual sensor in the data acquisition module is set to within 3.5m directly in front of the exoskeleton, ensuring that the visual sensor can see the information in front of the current position in real time. The environmental perception module includes a ground type detection module and a terrain size parameter detection module. The terrain type detection module obtains terrain parameters based on the RGB data from the data acquisition module, and the terrain size parameter detection module uses point cloud data to perform staircase size parameter regression to obtain terrain parameters. The specific calculation method in the ground type detection module is as follows: After image preprocessing, the RGB images acquired by the data acquisition module are entered into the semantic segmentation network model and classified into four categories based on the region detection results: stairs, slopes, flat ground, and obstacles. Step A1: Image alignment; Align the RGB image with the depth image so that the pixels of the depth image correspond to those of the RGB image; Step A2: Use a semantic segmentation network model for region detection; The semantic segmentation network model downsamples the original image by 8 times to obtain a feature map, inputs the feature map into the Pyramid Pooling Module, and adds the output of the Pyramid Pooling Module to the feature map. Then, it is upsampled by convolution and 8 times bilinear interpolation to obtain the result. Step A3: Region detection; The semantic segmentation results are used to determine the scene based on priority, which is: obstacle > stairs > ramp > flat ground. When there is an area in the perception area that the robot cannot walk on at present, it is treated as an obstacle scene, and the unwalkable information is retained as additional decision information. When the scene category is set as stairs and flat ground, specific parameter information is further detected. The calculation method of the terrain size parameter detection module is as follows: Step B1: Point cloud processing; Step B11: Coordinate transformation; Obtain color and distance information for each pixel, and transform all point cloud data into point cloud data in the same three-dimensional coordinate system; Step B12: Point cloud denoising; First, voxel downsampling is performed on the point cloud, and then outliers are removed using a statistical outlier removal method. Then, a random sampling method is used to randomly extract the same number of points from each depth image as new point cloud data; Step B2: The processed point cloud data is processed through two sets of feature extraction parts to extract global and local features of the point cloud and combine them. Then, it is passed through a PointNet layer to further extract the features obtained above. Finally, after passing through a fully connected layer, the size parameters of the current terrain are output. The gait autonomous decision-making module; The robot's motion state is obtained by combining an inverted pendulum model with foot pressure information. Then, based on the obtained human-machine motion state, environmental perception information is integrated to parametrically describe the gait of the transitional terrain. In order to enable the human-machine system to assist human movement, and according to the human's movement intention, the trajectory of the swing phase end is modulated through dynamic motion primitives to plan a gait trajectory suitable for the wearer and the characteristics of the environment, and to make single-step dynamic adjustments based on environmental information. Finally, the hip and knee joint angles of the swinging leg can be obtained through inverse kinematics, and the generated gait trajectory is ultimately triggered to meet the gait requirements of human-machine-environment transitional terrain. Step C1: Gait parameterization; With the wearer's safety as the key factor, constraints are set, and the defined safety constraint parameters are: , The minimum safe distance and minimum safe height for a robot to climb stairs are the forward distance between the ankle joint and the stairs. The starting point for dynamic adjustments during the robot's gait transition phase, and ; The safe walking distance between the robot and the stairs. , here This indicates the stride length of the swinging leg, ensuring that the forward distance between the landing point and the stairs is the minimum safe distance for the robot to climb the stairs; Set the transition point between the terrain transition phase and the transition phase, that is, the position of the supporting leg in the last step of the flat gait; , : The maximum stride length and maximum height that a robot can walk on in typical flat terrain; Based on the model of the terrain environment, the terrain parameters in the transformation stage are described as follows: , respectively representing the starting point of the transition phase, the maximum safe feasible distance, and the single-step step size sequence; Similarly, the parameters of the transition phase are described as follows: These represent the starting and ending points of the transitional terrain, respectively. Here, the ending point indicates the landing point of the swinging leg on the first step. These represent the height and width of the corresponding staircases, respectively. When ascending stairs and during the transition phase, consider the forward steps. Therefore, it is described as follows , representing the support point of the current step and the landing point of the next step, respectively; To enable exoskeleton robots to smoothly and naturally ascend stairs, it is necessary to first determine the minimum forward distance threshold from the foot to the stairs based on the stair height. In the formula Indicates the incline of the stairs; ； Step C2: Gait parameter representation during the transition phase; During the gait transition phase when approaching stairs, the exoskeleton robot's gait is parameterized; during walking on flat ground, the single-step step length is adjusted as if the center of mass only changes in the sagittal X-axis direction and not in the Z-axis direction; the coordinates of the swing phase start point are determined. Maximum safe feasible distance and single-step step length sequence The following formula describes the process; ； Combining the relationship between stride length and step length, we can obtain the relationship between the two: ； in, Indicates stride length when walking. Indicates the step length of the previous movement; Considering the variable stride length walking of a normal person approaching an object, a weighting coefficient is added to the single-step stride length. Its value Combined with the forward distance from the stairs Safe walking distance between the robot and the stairs The gait parameters are described in the following formula; ； Step C3: Representation of gait parameters during the transition phase; For scene modeling of the transition phase from flat ground to stairs, the parameters of the transition phase are described, including the points of the terrain interface. The foothold of the first step and the height and width of the first step The exoskeleton robot is guaranteed to transition safely when the following constraint condition is met. ； Therefore, the step size and step height for the transition phase are: ； The foot closest to the safe distance from the forward staircase becomes the supporting leg, while the swinging leg on the other side first reaches the tread of the first step from the transition point and becomes the supporting leg; at this time, the supporting leg on the other side becomes the swinging leg and reaches the tread of the next step from the point closest to the safe distance from the staircase, completing the transition from the flat ground to the staircase. Step C4: Gait parameter representation during the stair-climbing phase; The modeling of the stair-climbing scene is based on the fact that normal people walk in a crisscross pattern on stairs, combined with the parameter description of the stair-climbing terrain, including the starting point being the current step's foot position. The point where the foot lands on the forward steps serves as the endpoint. and the height and width of the forward steps Therefore, the step length and step height of the staircase terrain are defined as follows: ； Step C5: Autonomous gait decision-making based on environmental parameters; When an exoskeleton robot detects staircase terrain while walking forward, the first key information for decision-making is the size and distance of the staircase to determine whether the exoskeleton robot needs to change its stride length to approach the staircase and continue walking; based on gait parameters and terrain parameters, when there are no stairs in the walking environment or there are stairs but the forward distance is limited, the robot will adjust its stride accordingly. At this time, the human-machine system will maintain its original gait trajectory. When the forward distance of the stairs meets the requirement When this occurs, it indicates that the exoskeleton robot has entered the transition phase, requiring dynamic adjustment of the single-step length; otherwise, the forward distance of the stairs meets the requirements. If the dimensions of the stairs do not exceed the robot's range of motion, it indicates that the exoskeleton robot has entered the transition phase. It needs to recalculate the parameters of the stairs and the starting position of the swing leg to take the first step in the transition phase. Otherwise, it cannot enter the transition phase and cannot enter the new terrain. when When the forward distance is too small, the exoskeleton robot cannot walk normally.