Exoskeleton gait intelligent regulation method based on binocular depth information perception

Abstract

The application provides a kind of exoskeleton gait intelligent regulation method based on binocular depth information perception, realizes high-precision forward-looking terrain identification by binocular RGB-D fusion perception, combines multi-source information fusion to construct a drift-free gait tracker, realizes predictive control mode switching and parameterized fine assistance, realizes predictive adaptive control, solves the problem of traditional control lag. The assistance torque is dynamically associated with the measured step height, slope angle and other terrain parameters, and the adaptive scaling of the assistance curve is realized through the quantitative formula. Compared with the fixed assistance curve scheme, the problem of insufficient or excessive assistance is completely avoided, and the assistance efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of exoskeleton technology, and in particular to an intelligent gait control system for exoskeleton based on binocular depth information perception, as well as an intelligent gait control method, electronic device, and computer-readable storage medium for exoskeleton based on binocular depth information perception. Background Technology

[0002] Lower limb exoskeleton robots have broad application prospects in rehabilitation medicine, elderly and disabled assistance, and industrial assistance, but their adaptability in complex terrain has always been a core pain point in the industry. Existing technologies have the following unavoidable core defects:

[0003] 1. Inherent limitations of monocular vision perception: Existing monocular vision solutions can only achieve terrain semantic classification and cannot obtain accurate 3D depth information. The estimation of terrain parameters relies entirely on single-point scanning of auxiliary lasers. It has insufficient contour coverage for complex terrains such as continuous stairs and irregular slopes, and the estimation error of key parameters such as step height and slope angle is large. At the same time, monocular vision is susceptible to environmental interference such as lighting, texture loss, and reflection. The misclassification rate is high in complex scenes, and the robustness is extremely poor.

[0004] 2. Reactive control has inherent lag: Traditional exoskeletons rely solely on body sensors such as IMU and encoders. They can only identify the terrain and adjust the control strategy after the user steps onto stairs or ramps, which is a reactive control that is "adjusted only after stepping on". The assistance lag is 300-500ms, resulting in unnatural gait, insufficient leg lift assistance, and safety hazards such as tripping and falling.

[0005] 3. Gait phase estimation has accumulated errors: Existing IMU-based gait phase estimation schemes rely on integral operations to achieve continuous phase advancement, which is prone to accumulated drift over long-term use; relying solely on plantar pressure switches and single laser signals for ground contact correction lacks multi-dimensional verification, resulting in a high misjudgment rate of ground contact events in complex terrain, and failing to achieve drift-free phase estimation, leading to misalignment of assist timing.

[0006] 4. Insufficient precision in assist control: Existing solutions mostly use assist curves with fixed parameters, which cannot dynamically adjust the assist magnitude according to actual terrain parameters such as step height and slope angle. This easily leads to problems such as excessive assist for low steps and insufficient assist for high steps, making it unsuitable for complex and ever-changing real-world scenarios, resulting in extremely poor user experience and assist efficiency.

[0007] 5. Cost and performance cannot be balanced: Although the solution using 3D LiDAR and high-end stereo cameras can achieve high-precision perception, the hardware cost is high, the computing power requirement is large, and the power consumption is high, which makes it impossible to commercialize in consumer-grade and rehabilitation-grade exoskeleton products; low-cost solutions have problems with insufficient perception accuracy and poor robustness, which cannot meet the actual use needs. Summary of the Invention

[0008] To address the technical problems existing in the prior art, the present invention provides the following technical solution:

[0009] On the one hand, an intelligent gait control system for exoskeleton based on binocular depth information perception is provided, including a multimodal sensing subsystem, a binocular depth perception and terrain recognition module, a multi-source fusion gait tracking module, a forward-looking adaptive control module, and an embedded execution and drive module, wherein:

[0010] The multimodal sensing subsystem is used to synchronously acquire multi-source sensing data, including binocular vision data, laser ranging data, IMU inertial data and joint encoder data, and to complete data preprocessing and spatiotemporal synchronization.

[0011] The binocular depth perception and terrain recognition module is used to generate a dense depth map based on binocular vision data, fuse RGB-D information to complete terrain semantic classification, accurately estimate and quantize terrain parameters based on three-dimensional point cloud data, and output terrain type and corresponding parameters.

[0012] The multi-source fusion gait tracking module is used to integrate the body kinematics observation, the absolute reference signal of heel contact detected by laser ranging, and the binocular terrain context information, with the extended Kalman filter (EKF) as the core, to achieve drift-free continuous gait phase estimation and gait event recognition.

[0013] The forward-looking adaptive control module is used to switch between predictive control modes based on forward-looking terrain information and real-time gait phase, and generate a parameterized target assist curve by combining terrain parameters, and achieve precise assist output through underlying impedance control.

[0014] The embedded execution and drive module is used to receive control commands, drive the exoskeleton joint motors to perform corresponding assistive actions, and provide feedback on the real-time operating status.

[0015] Preferably, the multimodal sensing subsystem includes a binocular depth camera, a one-dimensional laser rangefinder, multiple IMU inertial measurement units, and a joint absolute encoder, wherein:

[0016] The binocular depth camera is installed in front of the waist of the exoskeleton, with a baseline distance of 50-80mm, and is used to simultaneously acquire RGB images of the left and right eyes, with a field of view covering a range of 0.5-5m in the direction of travel;

[0017] The one-dimensional laser rangefinder is installed on the front of the lower leg of the exoskeleton, with the beam pointing towards the ground, and is used to collect distance data between the foot and the ground at high frequency to detect heel contact events.

[0018] The IMU (Inertial Measurement Unit) is deployed on the exoskeleton, thigh, and calf to collect acceleration, angular velocity, and attitude angle data of various parts of the human body.

[0019] The joint absolute encoder is deployed at the hip, knee, and ankle joints of the exoskeleton to collect real-time angle and angular velocity data of the joints;

[0020] The multimodal sensing subsystem achieves time synchronization of all sensors through hardware triggering.

[0021] Preferably, the binocular depth perception and terrain recognition module includes a binocular depth map generation unit, a terrain semantic segmentation and feature extraction unit, and a terrain parameter accurate estimation unit, wherein:

[0022] The binocular depth map generation unit is used to perform stereo correction and stereo matching on the synchronized left and right eye RGB images, generate a disparity map, and calculate a dense depth map based on the disparity-depth mapping formula, converting it into 3D point cloud data in the world coordinate system; the disparity-depth mapping formula is:

[0023] ,in:

[0024] This represents the target depth value corresponding to pixel (x, y). For the equivalent focal length of a binocular camera, The baseline distance of the binocular camera. This is the disparity value of that pixel;

[0025] The terrain semantic segmentation and feature extraction unit is used to fuse RGB images and depth maps into 4-channel RGB-D input data, input to a lightweight semantic segmentation network that has been pruned and quantized, and output terrain classification results, region of interest (ROI) masks and classification confidence scores. The terrain categories include flat land, up stairs, down stairs, up slope, down slope and uneven ground.

[0026] The terrain parameter precise estimation unit is used to perform planar fitting and feature extraction on the 3D point cloud within the ROI region, and calculate quantified terrain parameters, including step height. Step depth Slope angle Flatness with the ground; including the slope angle The calculation formula is:

[0027] In the formula, A, B, and C are the coefficients of the plane equation fitted to the slope.

[0028] Preferably, the multi-source fusion gait tracking module includes a body observation preprocessing unit, a ground contact event absolute reference detection unit, and an EKF multi-source fusion gait estimation unit, wherein:

[0029] The proprioception preprocessing unit is used to calculate the lower limb joint pose and foot kinematic parameters based on IMU and encoder data through the DH positive kinematics model, and output high-frequency proprioception gait observation values.

[0030] The ground contact event absolute reference detection unit is used to fuse laser ranging data, foot IMU acceleration data and binocular terrain ground height information, detect heel contact HS events through a multi-condition fusion judgment algorithm, and output the absolute reference signal of the gait cycle start point;

[0031] The EKF multi-source fusion gait estimation unit is used to construct an EKF model with continuous gait phase as the core state variable, fuse three types of observation information: ontological kinematics observation, HS event absolute benchmark observation, and binocular terrain context observation, perform prediction and update steps, and output continuous and smooth gait phase and gait stage division results.

[0032] The state vector of the EKF model is: ,in Let k be the continuous gait phase at time k, ranging from [0,1]. The gait phase change rate, The phase change rate is zero bias; the state transition matrix is , This represents the sampling time interval.

[0033] Preferably, in the EKF multi-source fusion gait estimation unit, when the HS event is triggered, an absolute benchmark observation update is performed, and the observed values... Its observation noise covariance is taken as 1e-6, forcibly adjusting the gait phase. Reset to 0 to eliminate the accumulated error caused by IMU integration; at the same time, based on the binocular terrain classification results, adaptively adjust the process noise covariance and observation noise covariance of EKF to optimize the estimation accuracy under complex terrain.

[0034] Preferably, the forward-looking adaptive control module includes:

[0035] The control mode prediction and decision unit is used to enter the mode preparation state and preload the corresponding control mode parameters when the binocular vision recognizes an effective shape change at a distance of 2-5 meters; when the gait phase enters the initial stage of the swing phase and the predicted foot landing point is located in a new terrain area, the control mode is smoothly switched within the swing phase.

[0036] The parameterized assist curve generation unit is used to dynamically scale the basic assist curve according to the control mode, real-time terrain parameters, and gait phase to generate target assist torque curves for the hip, knee, and ankle joints; wherein, in the stair-climbing mode, the formula for the target assist torque of the knee joint is:

[0037] In the formula: A stair-assisted curve for knee joint support. To measure the actual step height, The standard step height is defined by a scaling factor with upper and lower limits of 0.5-2.0.

[0038] The underlying impedance servo control unit is used to employ a position-based impedance control algorithm, combined with a dynamically adjusted stiffness coefficient. With damping coefficient To achieve precise and smooth output of the assist torque, the impedance control formula is as follows:

[0039] ,

[0040] In the formula: To output control torque, , For the target and measured joint angles, , For the target and measured joint angular velocities, This is the feedforward compensation torque.

[0041] On the other hand, a method for intelligent gait control of an exoskeleton based on binocular depth information perception is provided, which is implemented based on the above system and includes the following steps:

[0042] System initialization and calibration: Complete hardware self-test, sensor calibration, system parameter initialization and multi-sensor spatiotemporal synchronization, and establish a unified world coordinate system and time reference;

[0043] Multimodal sensing data acquisition and preprocessing: Simultaneously acquire binocular vision, laser ranging, IMU and encoder data, complete noise reduction, correction, filtering preprocessing, and add a unified timestamp;

[0044] Binocular Depth Perception and Terrain Recognition Parameter Estimation: Dense depth maps and 3D point clouds are generated based on binocular images. RGB-D information is fused to complete terrain semantic classification, accurately estimate and output terrain type and quantified terrain parameters;

[0045] Multi-source fusion gait phase estimation and event recognition: Solve the body kinematic parameters, detect the absolute reference event of heel strike, achieve drift-free continuous gait phase estimation through EKF multi-source fusion, and divide gait stages;

[0046] Proactive adaptive control strategy generation and execution: Based on forward-looking terrain information, predictive control mode switching is completed, and parameterized assist curves are generated by combining terrain parameters. The exoskeleton is driven to perform precise assistance through underlying impedance control.

[0047] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0048] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0049] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0050] This invention employs a binocular RGB-D fusion perception scheme, which not only achieves terrain semantic classification but also outputs dense depth maps and 3D point clouds, completely solving the core pain points of missing depth information in monocular vision, inaccurate terrain parameter estimation, and poor environmental robustness. The terrain classification accuracy is ≥95%, the step height estimation error is ≤1cm, and the slope angle error is ≤1°, achieving high-precision forward-looking perception with long-distance and full terrain coverage.

[0051] This invention uses EKF as its core, integrating high-frequency observation of body kinematics, absolute reference of laser ground contact events, and binocular terrain context information. By forcibly resetting the phase through HS events, it completely eliminates the cumulative drift of IMU integration. Even in complex terrain, it achieves drift-free, high-precision phase estimation, providing a reliable timing reference for precise assistance. By recognizing terrain changes 2-5 meters in advance with binocular vision, it enters a preparatory state ahead of time and smoothly switches control modes within the swing phase. Compared to traditional reactive control, the response is 300-500ms earlier, completely solving the industry pain point of control lag. Gait transitions are smooth and imperceptible, without impact, significantly improving the exoskeleton's passability and safety in complex terrain. The assist torque is dynamically correlated with measured terrain parameters such as step height and slope angle. An adaptive scaling of the assist curve is achieved through a quantification formula. Compared to fixed assist curve solutions, it completely avoids the problems of insufficient or excessive assistance, improving assistance efficiency by over 30% and reducing user metabolic consumption by over 25%. It is adaptable to steps of different heights and ramps of different slopes, exhibiting extremely strong generalization ability.

[0052] This invention adopts a low-cost hardware solution of binocular camera + one-dimensional laser rangefinder, with a total cost far lower than 3D LiDAR and high-end stereo cameras. At the same time, all algorithms have been lightweighted and quantized, and can run in real time on embedded MCU + AI coprocessor without the need for high-performance GPU. It has low power consumption and small size, making it perfectly suitable for the commercial application of consumer-grade and rehabilitation-grade exoskeletons. The multiple sensors are redundant, and the failure of a single sensor can degrade the operation, making it extremely robust. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the principle of this method;

[0055] Figure 2 This is a schematic diagram of a system architecture provided in an embodiment of the present invention;

[0056] Figure 3 This is a functional structure block diagram of the binocular depth perception and terrain recognition module of the present invention;

[0057] Figure 4 This is a schematic diagram illustrating the working mechanism of the multi-source fusion gait tracking module of the present invention;

[0058] Figure 5 This is a schematic diagram of the working mechanism of the forward-looking adaptive control module of the present invention;

[0059] Figure 6 This is a schematic diagram of the control system composition structure of the embedded execution and drive module of the present invention;

[0060] Figure 7 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0061] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0062] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0063] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0064] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0065] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0066] This invention proposes an intelligent gait control scheme for exoskeletons based on binocular depth information perception. It achieves high-precision forward-looking terrain recognition through binocular RGB-D fusion perception, and constructs a drift-free gait tracker by combining multi-source information fusion. This enables predictive control mode switching and parameterized fine-tuning assistance, completely solving the core pain points of existing technologies.

[0067] This invention addresses the shortcomings of monocular vision in feature extraction (mere image recognition) by fusing laser ranging, binocular vision (in this solution, the binocular camera not only acquires the terrain environment of the RGB image but also captures depth information such as target environment feature values), IMU, and encoder multimodal information to construct a gait tracker using an extended Kalman filter; it uses laser ranging to detect heel touch events as an absolute benchmark, and combines the depth information (distance, number, slope) detected by binocular vision to upgrade the exoskeleton's driving strategy (the amount of assistance is dynamically correlated with the measured step height, slope, and other parameters, achieving refined and quantitative assistance, avoiding insufficient or excessive assistance, and thus increasing efficiency); and it intelligently switches the exoskeleton's strategy based on the depth information, realizing depth recognition and intelligent switching of the exoskeleton gait.

[0068] like Figure 1 As shown, the exoskeleton gait intelligent control method based on binocular depth information perception proposed in this application adopts the core control logic of "look-ahead prediction - real-time adaptation - closed-loop feedback", and mainly includes the following implementation steps:

[0069] 1. Forward-looking perception stage: Binocular vision identifies terrain changes 2-5 meters in advance, completes terrain classification and parameter estimation, and reserves sufficient decision-making and preparation time for the system;

[0070] 2. Preliminary decision-making phase: The system enters the mode preparation state, loads the corresponding control parameters in advance, and continuously tracks the gait phase through EKF, waiting for the best switching opportunity;

[0071] 3. Mode switching phase: When the gait enters the initial swing phase and the predicted foot landing point is located in the new terrain, the control mode is smoothly switched to avoid sudden torque changes;

[0072] 4. Assisted execution phase: Based on real-time terrain parameters and gait phase, a target assist curve is dynamically generated, and the motor is driven to output precise assist through an impedance control algorithm;

[0073] 5. Closed-loop optimization: Collect joint feedback data in real time, compare the deviation between the target value and the actual value, adjust the control parameters online, and continuously monitor changes in terrain and gait, and repeat the above process in a loop.

[0074] The following section will describe in detail the implementation process of the method described in this application, in conjunction with the system introduction.

[0075] Since exoskeleton technology and its components, as well as its hardware devices, are all mature technologies, this application does not limit the hardware structure and model of the exoskeleton, and can refer to existing commercially available or publicly available exoskeleton devices.

[0076] The application of the exoskeleton and the intelligent gait control algorithm of this application will be described in detail below.

[0077] like Figure 2 As shown, the gait intelligent control system adopted in this application employs a hierarchical architecture of "perception-cognition-decision-execution," which mainly includes the following functional subsystems:

[0078] (a) Multimodal sensing subsystem

[0079] This module is the sensing foundation of the system, responsible for the synchronous acquisition, preprocessing and spatiotemporal unification of multi-source data, providing high-quality input data for subsequent modules. All sensors achieve time synchronization through hardware triggering, with a timestamp error of ≤1ms, ensuring the spatiotemporal consistency of multi-source data.

[0080] 1. Multimodal sensing hardware configuration

[0081] Binocular depth camera The exoskeleton is positioned directly in front of the waist, horizontally forward. Baseline distance 60mm, resolution 640×480, frame rate 30fps, field of view 90° horizontal / 60° vertical, measurement range 0.5-5m. Simultaneously acquire RGB images from both eyes, output dense depth maps and 3D point clouds, providing terrain semantic information and accurate depth information. One-dimensional laser rangefinder The beam of light is directed perpendicularly to the ground from the front of the exoskeleton's lower leg. Model VL53L1X, frame rate 100Hz, measurement range 2cm-4m, ranging accuracy ±1mm High-frequency data acquisition of the real-time distance between the foot and the ground, detection of heel strike events, and provision of an absolute reference for the start of the gait cycle. IMU (Inertial Measurement Unit) Deployed on the exoskeleton's torso, thigh, and lower leg respectively. Model MPU9250, 9-axis, 200Hz frame rate, acceleration range ±16g, angular velocity range ±2000° / s Acceleration, angular velocity, and attitude angle data of various parts of the human body are collected for lower limb kinematics calculation and attitude estimation. Absolute encoder Deployed at the hip, knee, and ankle joints of the exoskeleton 16-bit resolution, 1000Hz frame rate, angle accuracy ±0.05° Real-time joint angle and angular velocity data are collected for lower limb forward kinematics calculation and low-level servo control.

[0082] 2. Data Preprocessing Unit

[0083] This unit performs noise reduction, correction, and filtering on the raw sensor data to eliminate systematic errors and random noise. For binocular images, it performs distortion correction, white balance correction, and Gaussian filtering; for laser ranging data, it performs moving average filtering to eliminate impulse noise; for IMU data, it performs zero-bias compensation and low-pass filtering (cutoff frequency 50Hz); and for encoder data, it performs nonlinear correction and digital filtering.

[0084] In this scheme, the preprocessed data collected by the multimodal sensing subsystem is uploaded to the binocular depth perception module and the multi-source fusion gait tracking module via SPI / I2C / CAN bus, respectively; the terrain information and gait information output by the two modules are uploaded to the forward-looking adaptive control module via high-speed serial port; the motor status and sensor feedback data of the execution module are uploaded to the main MCU in real time to form a closed-loop feedback.

[0085] The motor drive commands generated by the forward-looking adaptive control module are sent to the motor drive unit via the CAN bus; system parameter configuration commands are sent from the main MCU to each sensor and coprocessor; mode switching commands and parameter update commands are synchronized to the corresponding modules in real time.

[0086] For a detailed understanding of the specific data communication and control, please refer to the module descriptions below.

[0087] (II) Binocular Depth Perception and Terrain Recognition Module

[0088] This module is the core innovative unit of this invention. Unlike monocular vision, which can only perform semantic classification, it achieves long-distance, high-precision terrain recognition and quantization parameter estimation by fusing RGB-D information through binocular vision.

[0089] like Figure 3 As shown, this module is divided into 3 core sub-units:

[0090] 1. Binocular Depth Map Generation Unit

[0091] This unit, based on the principle of binocular parallax, generates a dense depth map aligned with the RGB image and converts it into 3D point cloud data in the world coordinate system. Specifically:

[0092] First, intrinsic parameters (equivalent focal length f, principal point coordinates cx / cy) and extrinsic parameters (rotation matrices R and translation vector T for the left and right cameras, baseline distance b = ||T||) are obtained through binocular camera calibration. Stereo correction is then performed on the preprocessed left and right camera images to eliminate distortion and achieve line alignment. A lightweight improved SGBM stereo matching algorithm is used to calculate the pixel-level disparity map d(x,y). Finally, the depth value of each pixel is calculated using the disparity-depth mapping formula to generate a dense depth map. The specific calculation method is as follows: In the formula: The target depth value (in mm) corresponding to pixel (x, y); Equivalent focal length for a binocular camera (unit: pixels); Baseline distance of the binocular camera (unit: mm); This is the disparity value of that pixel (unit: pixels).

[0093] Coordinate Transformation: Using the pose data output in real time by the torso IMU, a transformation matrix is ​​constructed from the camera coordinate system to the world coordinate system (with the user's standing point as the origin, horizontal forward as the X-axis, vertical upward as the Y-axis, and perpendicular to the ground as the Z-axis). This transforms the depth map into a 3D point cloud in the world coordinate system. The transformation relationship is as follows: In the formula:

[0094] These are the point cloud coordinates in the camera coordinate system; The coordinates of the three-dimensional point cloud in the world coordinate system; This is the rotation matrix from the camera to the world coordinate system; This is the translation vector from the camera to the world coordinate system (obtained from the camera's mounting position).

[0095] 2. Terrain semantic segmentation and feature extraction unit

[0096] This unit integrates RGB images and depth information to achieve high-precision terrain semantic classification and locate regions of interest (ROIs). The unit employs a lightweight semantic segmentation network (MobileNetV2 + DeepLabV3Lite) with pruned and INT8 quantization, fusing RGB images and depth maps into 4-channel RGB-D input data. Compared to monocular RGB input, this adds depth dimension features, significantly improving classification accuracy in low-light and texture-deficient scenes. The network outputs the terrain category probability distribution for each pixel, including: flat ground, uphill stairs, downhill stairs, uphill slope, downhill slope, and uneven ground. It also outputs the ROI mask and classification confidence for stair edges and slope areas.

[0097] Specifically, the application process of the lightweight semantic segmentation network (MobileNetV2 + DeepLabV3Lite) is as follows: 1) Pruning operation: Based on the channel importance evaluation algorithm, redundant convolutional channels in MobileNetV2 are pruned. First, the weight contribution of each channel is calculated through training data, and a threshold is set to remove channels with low contribution, compressing the number of network parameters to 60% of the original, while ensuring that the accuracy loss is controlled within 2%. 2) INT8 quantization: A post-training quantization scheme is adopted. First, RGB-D data in typical scenarios are collected as a calibration dataset. Then, the weights and activation values ​​in the network are converted to 8-bit integers, converting floating-point operations to integer operations, which improves the model inference speed by 3 times and reduces memory usage by 75%. MobileNetV2, as the backbone network for feature extraction, uses its core inverse residual structure to retain rich semantic information while reducing computational load through 1×1 convolution for dimensionality increase, 3×3 depthwise separable convolution for feature extraction, and 1×1 convolution for dimensionality reduction. DeepLabV3Lite utilizes atrous convolution to expand the receptive field without reducing resolution, combined with Spatial Pyramid Pooling (ASPP) to capture terrain features at multiple scales, and finally outputs the terrain category probability distribution for each pixel through a fully connected layer. Examples are as follows: 1) Low-light scenarios: In a dimly lit underground parking garage, monocular RGB images struggle to distinguish changes in ground slope. After fusing depth maps, the network can accurately identify uphill (deep values ​​continuously decreasing) and downhill (deep values ​​continuously increasing) terrain using depth information, improving classification accuracy. 2) Scenarios lacking texture: Faced with smooth marble surfaces and reflective tile slopes, RGB images have similar texture features and are easily confused. Depth maps, however, can clearly distinguish them through height differences, reducing the classification error rate to 5%. 3) ROI mask application: When the robot detects a ROI mask at the edge of a staircase, it automatically adjusts its motion planning, slowing down in advance and switching to stair climbing mode; ROI masks in slope areas help the robot identify passable areas and avoid steep slopes. 4) Confidence reference: When the confidence of the terrain classification is below 80%, the robot will activate the lidar for secondary detection and combine the results of multi-sensor data fusion to confirm the terrain type and avoid movement risks caused by misjudgment.

[0098] 3. Accurate Estimation Unit for Terrain Parameters

[0099] This unit, based on 3D point clouds and ROI regions, accurately calculates and quantifies terrain parameters, providing a quantitative basis for control. The specific implementation process is as follows: ① Step parameter estimation (ascending / descending stairs): Within the stair ROI region, the RANSAC algorithm is used to separate the planar point clouds of adjacent steps and extract step edge features; in the world coordinate system, the difference in Y-axis coordinates between two adjacent steps is calculated to obtain the step height. The step depth is obtained from the difference in X-axis coordinates. The parameters are optimized by using a moving average filter based on 5 consecutive frames of data to eliminate noise. The filter formula is as follows: In the formula: This is the estimated step height for the current frame; This is the estimated value from the previous frame; The filter coefficient is set to 0.4 to balance response speed and smoothness. ② Slope parameter estimation (uphill / downhill): Within the slope ROI region, the least squares method is used to perform plane fitting on the 3D point cloud to obtain the plane equation. The plane normal vector is The ground normal vector in the world coordinate system is Calculate the slope angle using the angle between the normal vectors. For example: In the formula: The slope angle is expressed in degrees, with positive for uphill and negative for downhill. If the sum of squared residuals (SSE) of the plane fitting exceeds the threshold, the terrain is considered uneven.

[0100] Result confirmation mechanism: When the classification confidence is ≥90% and the parameter fitting residual is ≤threshold, the effective shape type and parameters are output; otherwise, it is marked as uncertain, and the effective result of the previous frame is used to avoid misjudgment.

[0101] (III) Multi-source fusion gait tracking module

[0102] This module is the temporal core of the system. Using an extended Kalman filter (EKF) as its core, it fuses multi-source observation information to achieve drift-free, high-precision continuous gait phase estimation and gait event recognition. By tracking body kinematics observations, HS event signals, and binocular terrain observations, it fuses and outputs a continuous and smooth gait phase. Gait phase segmentation results, gait parameters such as gait frequency / stride length, etc.

[0103] like Figure 4 As shown, the module mainly includes the following functions:

[0104] 1. Body observation preprocessing unit

[0105] This unit calculates lower limb kinematic parameters based on IMU and encoder data, providing high-frequency gait state observations. A lower limb linkage model of the exoskeleton is established using the DH parameter method, calibrating the DH parameters (link length, torsion angle, joint angle, offset) of the hip, knee, and ankle joints. The spatial pose of each link is calculated using the DH transformation matrix, ultimately obtaining the foot's position in the world coordinate system. ,speed acceleration It outputs high-frequency body gait observations.

[0106] Specifically:

[0107] 1) DH parameter calibration

[0108] A 3D motion capture system (such as Vicon) combined with manual measuring tools was used to accurately obtain the physical dimensions of each link in the lower limb of the exoskeleton. First, the actual lengths of the links in the thigh and lower leg were measured using vernier calipers. Then, the spatial coordinates of the joints in different postures were collected using the motion capture system, and the torsion angle, initial joint angle values, and offset parameters were calculated. Finally, the DH parameters were optimized and calibrated using the least squares method to ensure that the model error was controlled within 1 mm.

[0109] 2) Kinematic solution

[0110] Based on the calibrated DH parameters, the transformation matrix for each link is constructed sequentially. The transformation matrix consists of a rotation matrix and a translation vector. The rotation matrix is ​​calculated using the link's torsion angle and joint angle, while the translation vector is determined by the link length and offset parameters. By multiplying the transformation matrices of each link sequentially using matrix multiplication, a homogeneous transformation matrix from the base to the foot is obtained. Finally, the position, velocity, and acceleration of the foot in the world coordinate system are calculated.

[0111] 3) Data fusion processing

[0112] A hardware synchronization triggering mechanism is used to achieve time alignment between IMU and encoder data, with the time synchronization error controlled within 1ms. An extended Kalman filter algorithm is used for data fusion, with acceleration and angular velocity measured by the IMU as observation values ​​and joint angles measured by the encoder as state update values. Through a prediction-update iterative process, high-precision joint angles and foot motion parameters are optimized, and high-frequency body gait observation values ​​with a frequency of up to 100Hz are output.

[0113] High-frequency gait observations can accurately analyze indicators such as stride symmetry and gait cycle stability, providing data support for program adjustments. In assistive exoskeleton control, real-time foot motion parameters can be used to predict the user's walking intentions and adjust the exoskeleton joint torque output in advance, achieving natural walking in human-machine collaboration.

[0114] 2. Ground contact event absolute reference detection unit

[0115] This unit uses multi-condition fusion to detect heel strike (HS) events, serving as an absolute benchmark for the start of the gait cycle and eliminating phase accumulation errors. By fusing laser ranging data, foot IMU acceleration data, and binocular terrain elevation information, a multi-condition judgment rule is constructed. Simultaneously, the judgment threshold is adaptively adjusted based on the terrain type identified by the binocular system to avoid misjudgments in complex terrain. The HS event triggering conditions must simultaneously meet: ① Laser ranging value... ① Reduce the ground height to below the terrain-matching ground height threshold (≤5cm on flat ground, ≤ step height + 2cm when climbing stairs); ② First-order difference laser ranging A negative peak appears (distance decreases rapidly); ③ Second-order difference laser ranging ④ A positive peak appears (deceleration upon ground contact); ⑤ A positive impact peak appears in the vertical acceleration of the foot IMU (ground contact impact); ⑥ The error between the ground height of the foot landing point detected by the binocular depth map and the height measured by the laser is ≤2cm.

[0116] 3. EKF Multi-Source Fusion Gait Estimation Unit

[0117] This unit constructs an EKF model based on multi-source observation fusion, outputting continuous, smooth gait phase and gait phase segmentation results without cumulative error. The process includes the following steps: ① State vector definition: Using continuous gait phase as the core state variable, a state vector is constructed: In the formula: The continuous gait phase at time k, ranging from [0,1] (corresponding to 0%-100% gait cycle); Gait phase change rate (unit: ); The zero bias of the phase change rate is used to compensate for IMU drift error. ② State prediction equation: Based on the uniform motion model, the discretized prediction formula is as follows: Where: State transition matrix ,

[0118] The sampling time interval; The noise is a process noise that follows a Gaussian distribution. ; The prior covariance matrix; This is the process noise covariance matrix. ③ Multi-source observation model: Construct three types of complementary observations to achieve accurate phase estimation and error correction:

[0119] Observation 1 (High-Frequency Ontology Observation): Based on the kinematic parameters of the IMU and encoder, a lightweight MLP model was pre-trained (the specific pre-training process is as follows: 1. Data Preparation: Raw data from IMU sensors (accelerometer, gyroscope) and joint encoders were collected. Multi-source data were aligned using a time synchronization algorithm, and data augmentation was performed using a sliding window method to generate a training dataset containing 100,000+ samples; 2. Model Construction: A 3-layer lightweight MLP network was designed. The input layer contained 12 neurons (corresponding to 6-axis IMU data + 6 joint angles), the hidden layer used the ReLU activation function, and the output layer output 8 gait phase categories using the Softmax function; 3. Training Process: The Adam optimizer was used, with an initial learning rate of 0.001, a batch size of 64, and 100 training iterations. An early stopping mechanism was introduced to prevent overfitting; 4. Performance Evaluation: Gait phase classification accuracy exceeding 98% was achieved on the test set, and the inference speed reached 1000Hz, meeting real-time requirements. If the evaluation was unsatisfactory, the training data was readjusted and retraining was performed), outputting gait phase observations. Observation equation: , , To observe noise, its covariance is adaptively adjusted according to the gait phase;

[0120] Observation 2 (Absolute baseline observation): Observation value when the HS event is triggered. (Gait cycle start point), observation equation: , Observation noise covariance This forces the phase to be reset to 0, completely eliminating the accumulated error.

[0121] Observation 3 (Terrain Context Observation): Based on the binocular terrain classification results and foothold prediction, output phase observation values. Observation equation:

[0122] , , Adaptive adjustment based on terrain classification confidence level. ④ State update step: For valid observations, calculate the Kalman gain and update the state and covariance:

[0123] Gait phase segmentation: based on continuous phase The gait cycle is divided into support phases ( ), oscillating phase ( It is further divided into seven sub-stages: pre-oscillation, initial oscillation, middle oscillation, final oscillation, initial support, middle support, and final support.

[0124] (iv) Forward-looking adaptive control module

[0125] This module is the core of the system's decision-making and execution. Based on forward-looking terrain information and real-time gait phase, it enables predictive mode switching and parameterized refinement assistance, and is divided into the following sub-units:

[0126] 1. Control Mode Prediction and Decision-Making Unit

[0127] This unit can identify terrain changes in advance and achieve seamless switching of control modes, solving the lag problem of traditional reactive control.

[0128] When the binocular vision detects terrain changes at a distance of 2-5 meters and the classification confidence level is ≥90%, the system enters the mode preparation state, loads the parameter library of the corresponding control mode in advance, and does not immediately switch the current control mode.

[0129] When the gait phase enters the early swing phase, and the next foot landing point predicted based on the foot movement trajectory is located in the new terrain area, the control mode is smoothly switched within the swing phase to avoid sudden torque changes caused by the support phase switching.

[0130] Control modes include: flat ground mode, up stairs mode, down stairs mode, uphill mode, downhill mode, and uneven ground mode.

[0131] 2. Parametric Assist Curve Generation Unit

[0132] This unit dynamically correlates the assist torque with measured terrain parameters to achieve refined and quantitative assist control, avoiding under- or over-assistance. Each control mode corresponds to a baseline assist curve calibrated through clinical gait analysis. , is the gait phase The function corresponds to the basic assist torque of the three joints of hip, knee, and ankle; based on the real-time estimated terrain parameters, the basic curve is dynamically scaled to generate the target assist torque. The scaling factor is set to an upper and lower limit of 0.5-2.5 to avoid security risks caused by outliers.

[0133] In this unit, the control modes are introduced as follows: ① Stair climbing mode: focuses on maximizing hip flexion and knee flexion assistance, peak torque and step height. Positive correlation, the formula is as follows: In the formula: Target torque for knee joint (unit: N·m); The standard staircase support curve for knee joints; The actual measured step height (unit: cm); The standard step height is taken as 15cm, the standard value for civil buildings. The hip joint assistance formula follows the same logic as above, with the scaling factor adapted to the joint assistance characteristics. ② Uphill mode: Focuses on maximizing hip extension and ankle plantar flexion assistance, peak torque and slope angle. Positive correlation, the formula is as follows: In the formula: Target torque for hip joint; The standard uphill basic assist curve for the hip joint; The measured slope angle (unit: °); The standard slope angle is 5°. The scaling factor is set to 1.0. ③ Smooth mode transition: When switching control modes, linear interpolation over two gait cycles is used to achieve a smooth torque transition, avoiding abrupt changes. The formula is: In the formula: The transition coefficient increases linearly from 0 to 1, with a transition period of 2 gait cycles.

[0134] 3. Underlying impedance servo control unit

[0135] This unit achieves precise and smooth output of assist torque, ensuring the smoothness and safety of human-machine interaction. It employs a position-based impedance control algorithm to dynamically adjust the stiffness coefficient according to terrain type and gait phase. Damping coefficient (Oscillating phase decrease) Increase flexibility and support phase (To improve stability), the algorithm is as follows: In the formula: The final control torque output to the motor (unit: N·m); , For the target and measured joint angles; , The target and the measured joint angular velocity; The feedforward compensation torque is based on terrain parameters and gait phase feedforward compensation to improve the system response speed.

[0136] like Figure 5 As shown, this module performs adaptive control analysis on the input target assist torque curve, dynamic impedance parameters, and measured joint angle / angular velocity, and finally outputs motor drive commands.

[0137] (v) Embedded Execution and Driver Module

[0138] This module serves as the system's hardware execution platform. It employs a heterogeneous computing architecture, balancing real-time performance with AI inference capabilities, and is suitable for lightweight embedded deployments. Figure 6 As shown, the module includes the following facilities:

[0139] 1. Core Processor: The main MCU uses an STM32H743 (Cortex-M7 core, 400MHz) to handle sensor data acquisition, EKF fusion, real-time control, and motor drive, with a control closed-loop frequency ≥1kHz; the AI ​​coprocessor uses a K210 to handle binocular depth calculation and lightweight inference for semantic segmentation networks, with an inference frame rate ≥15fps, eliminating the need for a high-performance GPU and meeting the requirements of embedded deployment.

[0140] 2. Motor drive unit: The DRV8301 three-phase gate driver is used to drive the brushless DC motors of the hip, knee and ankle joints. The current sampling accuracy is 1mA and it has overcurrent, overvoltage and overheat protection functions.

[0141] 3. Supporting Units: The power management unit uses a 24V / 10Ah lithium battery, providing a battery life of ≥4 hours; the data storage unit uses a MicroSD card to store operating data and logs in real time, supporting offline analysis; the safety protection unit monitors the system status in real time and triggers emergency stop and safety degradation modes in case of abnormalities.

[0142] Regarding the above system application, the implementation process of the exoskeleton gait intelligent control method based on binocular depth information perception will be described in detail below.

[0143] like Figure 7 As shown, the method includes the following steps.

[0144] Step 1: System Initialization and Calibration

[0145] The hardware performs a power-on self-test, completing the initialization and communication status detection of all sensors, processors, and drive units, and triggering an alarm when an abnormality occurs.

[0146] Sensor calibration: Complete single-target and dual-target calibration of the binocular camera, acquire intrinsic and extrinsic parameters and baseline distance, and correct image distortion; complete zero-bias calibration of the IMU and calibration of the accelerometer / gyroscope; complete zero-position calibration of the joint encoder and distance calibration of the laser rangefinder.

[0147] System parameter initialization: Load user-personalized parameters (height, weight, leg length), basic assist curve library, EKF initial state and noise matrix, control mode default parameters, and lightweight AI model weights; 1.4 Spatiotemporal synchronization: Achieve global time synchronization of all sensors through hardware trigger signals, establish a unified world coordinate system, and complete coordinate transformation matrix calibration.

[0148] Step 2: Multimodal sensor data acquisition and preprocessing

[0149] The binocular cameras simultaneously acquire RGB images from both eyes at 30fps, transmitting them to the AI ​​coprocessor for distortion correction, white balance adjustment, and Gaussian filtering preprocessing. A laser rangefinder acquires distance data between the feet and the ground at 100Hz, performing moving average filtering to eliminate impulse noise. The trunk, thigh, and calf IMUs acquire acceleration, angular velocity, and attitude data at 200Hz, performing zero-bias compensation and low-pass filtering. The joint encoder acquires joint angle and angular velocity data at 1000Hz, performing nonlinear correction and digital filtering. All preprocessed data is stamped with a unified global timestamp and synchronized to the corresponding processing modules. This preprocessing enables simultaneous acquisition and standardized preprocessing of multi-source data, eliminating noise and interference, and providing high-quality, real-time input data for subsequent modules.

[0150] Step 3: Parameter Estimation for Binocular Depth Sensing and Terrain Recognition

[0151] Binocular depth map generation: Stereo matching is performed on the corrected left and right eye images to calculate the disparity map. Dense depth maps are generated using the disparity-depth mapping formula and converted into 3D point clouds in the world coordinate system by combining IMU pose data. Terrain semantic segmentation: RGB images and depth maps are fused into 4-channel RGB-D data, which are input into a lightweight semantic segmentation network to output terrain classification results, ROI masks and classification confidence.

[0152] Terrain parameter estimation: Plane fitting and feature extraction are performed on the 3D point cloud within the ROI region to calculate quantitative parameters such as step height, step depth, slope angle, and ground flatness. The parameters are optimized by multi-frame moving average filtering.

[0153] Results confirmed that when the classification confidence is ≥90% and the fitting residual is ≤threshold, the effective shape type and parameters are output; otherwise, the effective results of the previous frame are used.

[0154] This system achieves long-distance, high-precision terrain recognition through binocular RGB-D fusion perception. Compared with monocular vision, the terrain classification accuracy is ≥95%, an improvement of more than 10%. The step height estimation error is ≤1cm, and the slope angle estimation error is ≤1°, providing accurate quantitative basis for predictive control.

[0155] Step 4: Multi-source fusion gait phase estimation and event recognition

[0156] To completely eliminate the accumulated error caused by IMU integration, this application controls the gait phase estimation error to ≤2%, achieving a ground contact event detection accuracy of ≥99%. Even in complex terrain, it achieves high-precision, drift-free gait phase estimation, providing a reliable timing reference for precise assisted control. The specific process is as follows:

[0157] Propriometry kinematics calculation: Based on IMU and encoder data, the lower limb joint pose, foot movement trajectory and kinematic parameters are calculated through the DH forward kinematics model, and high-frequency proprioceptive gait observations are output.

[0158] Ground contact event detection: By fusing laser ranging, foot IMU, and binocular terrain height information, ground contact events of the HS are detected through multi-condition judgment rules, and the absolute reference signal of the gait cycle start point is output;

[0159] EKF multi-source fusion estimation: Input ontological observations, HS absolute benchmark observations, and binocular terrain context observations into the EKF model, perform prediction and update steps, and output a continuous and smooth gait phase; when an HS event is triggered, the phase is forcibly reset to 0 to eliminate accumulated errors;

[0160] Gait phase segmentation: Based on continuous phase, the gait cycle is divided into support phase, swing phase and its sub-phases, and parameters such as stride frequency, stride length and gait symmetry are calculated.

[0161] Step 5: Generation and Execution of Proactive Adaptive Control Strategy

[0162] When binocular vision detects effective shape changes at a distance of 2-5 meters, the system enters a mode preparation state and preloads the corresponding control parameters. When the gait enters the initial swing phase and the predicted foot placement is located on the new terrain, a smooth switch of control mode is completed within the swing phase. Based on the control mode, real-time terrain parameters, and gait phase, the basic assist curve is dynamically scaled to generate a target assist torque curve, while the stiffness and damping coefficient of the impedance control are dynamically adjusted. During mode switching, a smooth torque transition is achieved through linear interpolation over two gait cycles to avoid sudden shocks. Based on the target assist curve and impedance parameters, the motor control torque is calculated using an impedance control algorithm, and drive commands are output to drive the exoskeleton joints to perform assist actions. Encoder and motor current feedback data are collected in real time, and the deviation between the target value and the actual value is compared to fine-tune the control parameters online, achieving closed-loop precise control.

[0163] Here, the adaptive control strategy can solve the lag problem of traditional reactive control, with control response 300-500ms ahead and smooth, imperceptible mode switching; through the dynamic correlation between terrain parameters and assist torque, refined and quantitative assist is achieved, improving assist efficiency and reducing the user's metabolic consumption during walking (by up to 25%-30%).

[0164] During operation, the system continuously monitors the user's gait symmetry and walking status, and uses a lightweight optimization algorithm to fine-tune the assist scaling factor and impedance parameters online to adapt to the user's personalized walking habits.

[0165] Simultaneously, it performs anomaly monitoring and safety protection: real-time monitoring of the operating status of sensors, motors, and batteries; triggering audible and visual alarms when an anomaly occurs; and simultaneously entering a safety degradation mode, switching to flat ground assist mode to ensure user safety; real-time storage of key operating data, sensor data, and control logs, supporting offline export for user gait analysis, system optimization, and troubleshooting.

[0166] Therefore, this solution enables online personalized adaptation of the system, continuously optimizing the assistance effect and user experience; it has a comprehensive security protection mechanism, which greatly improves the security and reliability of system operation and reduces the risk of failure.

[0167] The actual application process and effects of this solution will be described below with reference to the embodiments.

[0168] Example 1

[0169] This embodiment is applied to a lower limb assistive exoskeleton in a rehabilitation center in Hangzhou, which is suitable for patients with lower limb muscle strength grade 4 after stroke (who can walk on flat ground but have difficulty going up and down stairs). The exoskeleton gait intelligent control system based on binocular depth information perception of the present invention is used to complete hardware modification and algorithm deployment, and the core solution is to address the pain points of insufficient assistance in going up and down stairs and unnatural gait.

[0170] The configuration of the example is as follows:

[0171] 1. Sensor configuration: A binocular depth camera (baseline distance 60mm, 30fps) is mounted on the front of the waist; a VL53L1X laser rangefinder is mounted on the front of the lower leg; three MPU9250IMUs are deployed on the torso, thigh, and lower leg respectively; and a 16-bit absolute encoder is deployed on the hip, knee, and ankle joints.

[0172] 2. Processor configuration: The main MCU is STM32H743, which is responsible for real-time control and EKF fusion; the AI ​​coprocessor is K210, which is responsible for binocular depth calculation and semantic segmentation inference. The model is quantized by INT8 and the inference frame rate is 15fps.

[0173] 3. Algorithm parameters: Binocular depth measurement range 0.5-5m, depth error ≤2%@2m; terrain classification accuracy ≥95%; step height estimation error ≤1cm; gait phase estimation error ≤2%; control closed loop frequency 1kHz; knee joint assist scaling factor 1.2 for stair climbing mode, standard step height 15cm.

[0174] 4. Functional Configuration: Enables predictive mode switching, parameterized adaptive assistance, HS event absolute phase reset, and full-featured anomaly safety protection.

[0175] To compare the technical advantages of the method in this application, the following comparative examples have been set up:

[0176] Comparative Example 1: Traditional reactive control exoskeleton, equipped only with IMU and encoder body sensors, without environmental perception unit, switches control mode only after the user steps on the terrain, and adopts fixed assist curve.

[0177] Comparative Example 2: Monocular vision + single laser ranging exoskeleton (current mainstream solution), monocular camera realizes terrain classification, single laser realizes parameter estimation, no dense depth information, fixed threshold ground contact correction and fixed assist curve are used.

[0178] The comparative analysis is as follows:

[0179] Terrain recognition capability Recognition from 2-5 meters ahead, RGB-D fusion classification, accuracy ≥95%, strong robustness in low-light / weak texture scenes. It lacks forward-looking terrain recognition capabilities and can only perform post-event identification. Recognition at a distance of 2-3 meters, monocular RGB classification, accuracy ≤85%, prone to misjudgment in low-light / weak texture scenes. Accuracy of terrain parameter estimation Step height error ≤1cm, slope angle error ≤1°, continuous staircase outline fully covered. No ability to estimate terrain parameters Step height error ≥3cm, slope angle error ≥3°, single-point scanning is prone to missed detection Gait phase estimation accuracy Error ≤2%, absolute reset of HS events, no cumulative error, stable under complex terrain. Error ≥8%, severe IMU integration drift, and the error continues to increase with long-term use. Error ≥5%, single laser correction, high ground contact misjudgment rate in complex terrain, and drift exists. Control response characteristics Predictive switching, responding 300-500ms in advance, smooth switching within the swing phase, with no lag. Reactive control only responds after the terrain is stepped on, with a lag of ≥400ms, which is prone to assist misalignment. Semi-predictive handover requires a 100-200ms lead time; inaccurate parameters can easily lead to incorrect handover timing. Assisted control precision Parametric dynamic boosting, strongly correlated with measured terrain, with boosting error ≤5%, and no under- or over-boosting. A fixed assist curve cannot adapt to changes in terrain, and is prone to insufficient or excessive assist. A fixed assist curve cannot adapt to different step heights / slopes, resulting in poor generalization ability. Walking metabolic consumption 28% reduction compared to walking without assistance 12% reduction compared to walking without assistance 18% reduction compared to walking without assistance Safety in complex terrain The ground contact misjudgment rate is ≤0.5%, the mode switching is seamless, and the risk of tripping is extremely low. The ground contact misjudgment rate is ≥6%, indicating delayed assistance and a high risk of tripping. A ground contact misjudgment rate of ≥5% indicates that terrain misjudgment can easily lead to assist errors, posing a safety hazard. Hardware costs Low to medium cost, binocular + single laser solution, total cost ≤ 800 yuan Low cost, only the sensor itself, total cost ≤ 300 yuan Low to medium cost, monocular + single laser solution, total cost ≤ 600 yuan Embedded Adaptability The algorithm is lightweight, can run on an MCU and an AI coprocessor, requires no GPU, and has low power consumption. The algorithm is simple, can be run on an MCU, and has extremely low power consumption. The algorithm is lightweight, can be run on an MCU, and has low power consumption.

[0180] In this embodiment, the patient, using the exoskeleton of this invention, can independently walk up and down a 10-story residential building without the assistance of a caregiver. The gait is natural and smooth, without any lag or hesitation, and the leg lift provides ample assistance, resulting in no fatigue. Compared to Comparative Example 1, the patient's gait symmetry is improved by 40%, the time spent climbing stairs is reduced by 25%, and metabolic consumption during walking is reduced by 28%. Compared to Comparative Example 2, the terrain recognition misjudgment rate is reduced from 8% to 1%, the step height estimation error is reduced from 3cm to 0.8cm, and the ground contact event misjudgment rate is reduced from 5% to 0.5%, significantly improving stability and safety in complex terrain.

[0181] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0182] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0183] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0184] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0185] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0186] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0187] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0188] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0190] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intelligent gait control system for exoskeleton based on binocular depth information perception, characterized in that, It includes a multimodal sensing subsystem, a binocular depth perception and terrain recognition module, a multi-source fusion gait tracking module, a forward-looking adaptive control module, and an embedded execution and drive module, among which: The multimodal sensing subsystem is used to synchronously acquire multi-source sensing data, including binocular vision data, laser ranging data, IMU inertial data and joint encoder data, and to complete data preprocessing and spatiotemporal synchronization. The binocular depth perception and terrain recognition module is used to generate a dense depth map based on binocular vision data, fuse RGB-D information to complete terrain semantic classification, accurately estimate and quantize terrain parameters based on three-dimensional point cloud data, and output terrain type and corresponding parameters. The multi-source fusion gait tracking module is used to integrate the body kinematics observation, the absolute reference signal of heel contact detected by laser ranging, and the binocular terrain context information, with the extended Kalman filter (EKF) as the core, to achieve drift-free continuous gait phase estimation and gait event recognition. The forward-looking adaptive control module is used to switch between predictive control modes based on forward-looking terrain information and real-time gait phase, and generate a parameterized target assist curve by combining terrain parameters, and achieve precise assist output through underlying impedance control. The embedded execution and drive module is used to receive control commands, drive the exoskeleton joint motors to perform corresponding assistive actions, and provide feedback on the real-time operating status.

2. The system according to claim 1, characterized in that, The multimodal sensing subsystem includes a binocular depth camera, a one-dimensional laser rangefinder, multiple IMU inertial measurement units, and a joint absolute encoder, wherein: The binocular depth camera is installed in front of the waist of the exoskeleton, with a baseline distance of 50-80mm, and is used to simultaneously acquire RGB images of the left and right eyes, with a field of view covering a range of 0.5-5m in the direction of travel; The one-dimensional laser rangefinder is installed on the front of the lower leg of the exoskeleton, with the beam pointing towards the ground, and is used to collect distance data between the foot and the ground at high frequency to detect heel contact events. The IMU (Inertial Measurement Unit) is deployed on the exoskeleton, thigh, and calf to collect acceleration, angular velocity, and attitude angle data of various parts of the human body. The joint absolute encoder is deployed at the hip, knee, and ankle joints of the exoskeleton to collect real-time angle and angular velocity data of the joints; The multimodal sensing subsystem achieves time synchronization of all sensors through hardware triggering.

3. The system according to claim 1, characterized in that, The binocular depth perception and terrain recognition module includes a binocular depth map generation unit, a terrain semantic segmentation and feature extraction unit, and a terrain parameter accurate estimation unit, wherein: The binocular depth map generation unit is used to perform stereo correction and stereo matching on the synchronized left and right eye RGB images, generate a disparity map, and calculate a dense depth map based on the disparity-depth mapping formula, converting it into 3D point cloud data in the world coordinate system; the disparity-depth mapping formula is: ,in: This represents the target depth value corresponding to pixel (x, y). For the equivalent focal length of a binocular camera, The baseline distance of the binocular camera. This is the disparity value of that pixel; The terrain semantic segmentation and feature extraction unit is used to fuse RGB images and depth maps into 4-channel RGB-D input data, input to a lightweight semantic segmentation network that has been pruned and quantized, and output terrain classification results, region of interest (ROI) masks and classification confidence scores. The terrain categories include flat land, up stairs, down stairs, up slope, down slope and uneven ground. The terrain parameter precise estimation unit is used to perform planar fitting and feature extraction on the 3D point cloud within the ROI region, and calculate quantified terrain parameters, including step height. Step depth Slope angle Flatness with the ground; including the slope angle The calculation formula is: In the formula, A, B, and C are the coefficients of the plane equation fitted to the slope.

4. The system according to claim 1, characterized in that, The multi-source fusion gait tracking module includes a body observation preprocessing unit, a ground contact event absolute benchmark detection unit, and an EKF multi-source fusion gait estimation unit, wherein: The proprioception preprocessing unit is used to calculate the lower limb joint pose and foot kinematic parameters based on IMU and encoder data through the DH positive kinematics model, and output high-frequency proprioception gait observation values. The ground contact event absolute reference detection unit is used to fuse laser ranging data, foot IMU acceleration data and binocular terrain ground height information, detect heel contact HS events through a multi-condition fusion judgment algorithm, and output the absolute reference signal of the gait cycle start point; The EKF multi-source fusion gait estimation unit is used to construct an EKF model with continuous gait phase as the core state variable, fuse three types of observation information: ontological kinematics observation, HS event absolute benchmark observation, and binocular terrain context observation, perform prediction and update steps, and output continuous and smooth gait phase and gait stage division results. The state vector of the EKF model is: ,in Let k be the continuous gait phase at time k, ranging from [0,1]. The gait phase change rate, The phase change rate is zero bias; the state transition matrix is , This represents the sampling time interval.

5. The system according to claim 4, characterized in that, In the EKF multi-source fusion gait estimation unit, when the HS event is triggered, an absolute baseline observation update is performed, and the observed values ​​are updated accordingly. Its observation noise covariance is taken as 1e-6, forcibly adjusting the gait phase. Reset to 0 to eliminate the accumulated error caused by IMU integration; Simultaneously, based on the binocular terrain classification results, the process noise covariance and observation noise covariance of EKF are adaptively adjusted to optimize the estimation accuracy under complex terrain.

6. The system according to claim 1, characterized in that, The forward-looking adaptive control module includes: The control mode prediction and decision unit is used to enter the mode preparation state and preload the corresponding control mode parameters when the binocular vision recognizes an effective shape change at a distance of 2-5 meters; when the gait phase enters the initial stage of the swing phase and the predicted foot landing point is located in a new terrain area, the control mode is smoothly switched within the swing phase. The parameterized assist curve generation unit is used to dynamically scale the basic assist curve according to the control mode, real-time terrain parameters, and gait phase to generate target assist torque curves for the hip, knee, and ankle joints; wherein, in the stair-climbing mode, the formula for the target assist torque of the knee joint is: In the formula: A stair-assisted curve for knee joint support. To measure the actual step height, The standard step height is defined by a scaling factor with upper and lower limits of 0.5-2.

0. The underlying impedance servo control unit is used to employ a position-based impedance control algorithm, combined with a dynamically adjusted stiffness coefficient. With damping coefficient To achieve precise and smooth output of the assist torque, the impedance control formula is as follows: ， In the formula: To output control torque, , For the target and measured joint angles, , For the target and measured joint angular velocities, This is the feedforward compensation torque.

7. The system according to claim 1, characterized in that, The embedded execution and drive module adopts a heterogeneous computing architecture, including a main MCU and an AI coprocessor. The main MCU is responsible for sensor data acquisition, EKF fusion, real-time control, and motor drive. The AI ​​coprocessor is responsible for binocular depth calculation and lightweight inference of semantic segmentation networks. The module also includes a motor drive unit, a power management unit, and a data storage unit to realize the exoskeleton's drive execution, power management, and data recording.

8. A method for intelligent gait control of exoskeleton based on binocular depth information perception, characterized in that, The system implementation based on any one of claims 1-7 includes the following steps: System initialization and calibration: Complete hardware self-test, sensor calibration, system parameter initialization and multi-sensor spatiotemporal synchronization, and establish a unified world coordinate system and time reference; Multimodal sensing data acquisition and preprocessing: Simultaneously acquire binocular vision, laser ranging, IMU and encoder data, complete noise reduction, correction, filtering preprocessing, and add a unified timestamp; Binocular Depth Perception and Terrain Recognition Parameter Estimation: Dense depth maps and 3D point clouds are generated based on binocular images. RGB-D information is fused to complete terrain semantic classification, accurately estimate and output terrain type and quantified terrain parameters; Multi-source fusion gait phase estimation and event recognition: Solve the body kinematic parameters, detect the absolute reference event of heel strike, achieve drift-free continuous gait phase estimation through EKF multi-source fusion, and divide gait stages; Proactive adaptive control strategy generation and execution: Based on forward-looking terrain information, predictive control mode switching is completed, and parameterized assist curves are generated by combining terrain parameters. The exoskeleton is driven to perform precise assistance through underlying impedance control.

Images