A lower limb wearable terrain recognition system and method based on RGB-IMU-single point ToF fusion

Abstract

The application discloses a lower limb wearable terrain recognition system and method based on RGB-IMU-single-point ToF fusion, and belongs to the terrain recognition field.Under the lightweight hardware constraint of single IMU and single-point ToF, multi-source signal depth fusion of RGB semantics, IMU kinematic characteristics and central alignment single-point ToF distance is achieved, and high-precision terrain recognition in the lower limb wearable scene is realized.With the help of the flat ground calibration mechanism of the ToF phase residual error calibration module, the influence of the wearer's height, sensor installation deviation and gait phase on the original distance value is effectively eliminated, and individual rapid adaptation is realized.A posteriori gating fusion module dynamically adjusts the fusion weight of the vision and distance categories according to the ToF effectiveness and residual error amplitude, and solves the problems of environmental reflection interference and single-mode confidence decline.Meanwhile, MCU unified time synchronization ensures multi-sensor data frame-level alignment, and reduces system time delay.The whole does not need RGB-D or laser radar, has low power consumption and computing burden, is easy to embed into lower limb exoskeletons or artificial limb equipment, and can stably recognize terrain.

Description

Technical Field

[0001] This application relates to the field of terrain recognition technology, and in particular, to a lower limb wearable terrain recognition system and method based on RGB-IMU-single-point ToF fusion. Background Technology

[0002] Powered prostheses, lower limb exoskeletons, and soft walking aids need to accurately identify terrain such as flat ground, going upstairs, going downstairs, uphill, and downhill during walking in order to adjust the assistance mode and gait switching strategy in a timely manner.

[0003] In existing solutions, RGB cameras can provide the appearance of the scene in front, but it is difficult to directly obtain geometric information such as slope, distance and height difference, and cannot directly distinguish between flat ground and slope; IMU (Inertial Measurement Unit) can reflect the movement state of human limbs, but it is not an external terrain sensor and cannot perceive the type of terrain in front in advance; RGB-D cameras, binocular vision or LiDAR can provide relatively complete depth maps, point clouds or spatial structure information, but they are large in size, power consumption, installation calibration and computational burden, and are difficult to integrate into lower limb wearable devices.

[0004] Single-point Time-of-Flight (ToF) ranging sensors offer advantages such as small size, low power consumption, and easy embedding. They output only a single-point scalar distance along the ranging direction, without forming a depth map or point cloud, thus providing minimal external geometric cues. However, their raw distance values ​​are affected by the wearer's height, sensor installation, and gait phase, and are susceptible to interference from ambient glare. Furthermore, their recognition performance relies on manually set thresholds. Therefore, a terrain recognition system and method are needed that integrates RGB semantics, IMU motion context, and center-aligned single-point ToF distance information within the constraints of a single, lightweight IMU hardware, and can rapidly calibrate and adapt to different wearers through short-term flat-surface calibration. Summary of the Invention

[0005] To overcome the shortcomings of related technologies, this application provides a lower limb wearable terrain recognition system and method based on RGB-IMU-single-point ToF fusion, in order to solve the technical problems in existing lower limb wearable terrain recognition schemes, such as the lack of direct geometric information in RGB vision, the large size and computational burden of dense depth sensing devices, the influence of individual differences in wearers, installation position and posture phase on the original distance of single-point ToF, and the difficulty of fixed weight fusion in adapting to single-modal failure.

[0006] The technical solution adopted by this application to solve its technical problem is: On the one hand, a lower limb wearable terrain recognition system based on RGB-IMU-single-point ToF fusion is provided, including: Wearable sensing components, MCU data processing module, ToF phase residual calibration module, RGB semantic recognition module, ToF-IMU distance recognition module, posterior gated fusion module, and terrain output module; The wearable sensing component includes an RGB camera mounted on the lower limb, a single-point ToF ranging sensor, and a single IMU; the ranging direction of the single-point ToF ranging sensor is aligned with the central line of sight of the RGB camera, and it only outputs a single-point scalar distance value along the ranging direction. The MCU data processing module is used to maintain a unified system time, and to perform time synchronization and preprocessing on the data from the RGB camera, the single-point ToF ranging sensor and the single IMU to form a synchronization frame indexed by a unified timestamp. The ToF phase residual calibration module is used to establish a flat ground ToF distance reference table corresponding to the posture phase interval based on the calibration data of the target wearer walking on flat ground, and convert the currently collected single-point ToF distance value into a phase residual value relative to the flat ground reference. The RGB semantic recognition module is used to output the posterior probability of the visual category based on the RGB image of the current synchronization frame; The ToF-IMU distance identification module is used to output a posterior probability of the distance category based on distance geometry information, according to the phase residual value and the kinematic features of the single IMU. The posterior gating fusion module is used to dynamically calculate the fusion weight based on the visual category posterior probability, the distance category posterior probability and the preset gating factor, and to perform weighted fusion of the visual category posterior and the distance category posterior to obtain the final terrain category probability. The terrain output module is used to output the terrain recognition result at the current moment based on the final terrain category probability.

[0007] Furthermore, the single-point ToF ranging sensor and the RGB camera are fixed on the same bracket, and the central line of sight of the single-point ToF ranging sensor is parallel or coincident with the optical axis of the RGB camera, so that the single-point ToF ranging sensor can measure the single-point distance corresponding to the center area of ​​the field of view of the RGB camera.

[0008] Furthermore, the MCU data processing module is used for: According to the unified system time, each sensor is asynchronously triggered to collect data, and timestamps are written for the received RGB images, single-point ToF distance values ​​and IMU data respectively; During data synchronization, the RGB image with the smallest time difference, the single-point ToF distance value, and the IMU data are selected as an index using the timestamp of the target time as an index to form a synchronization frame. In addition, the validity of the single-point ToF distance value is checked, and data points with no return value or exceeding the preset valid distance measurement range are removed, and a ToF validity flag is generated.

[0009] Furthermore, the ToF phase residual calibration module is specifically used for: After the target wearer uses the device for the first time or after the sensor is reinstalled, collect flat walking data for a preset duration as a calibration dataset. Based on the thigh pitch angle or gravity reference pitch angle measured by the single IMU, the gait cycle is divided into multiple posture phase intervals; For each attitude phase interval, the median or mean of the ToF distance values ​​of all single points falling into the attitude phase interval in the calibration dataset is calculated and used as the flat ground ToF distance reference value corresponding to the attitude phase interval. In the current real-time recognition phase, the corresponding flat ground ToF distance reference value is found based on the thigh pose phase index of the current frame, and the original single-point ToF distance value of the current frame is subtracted from the flat ground ToF distance reference value to obtain the phase residual value.

[0010] Furthermore, the posterior gating fusion module is specifically used for: Obtain the visual category posterior probability vector output by the RGB semantic recognition module and the distance category posterior probability vector output by the ToF-IMU distance recognition module; Obtain the ToF validity flag and phase residual amplitude of the current synchronization frame; Based on the maximum confidence of the visual category posterior probability, the maximum confidence of the distance category posterior probability, the ToF validity flag, and the phase residual amplitude, the visual fusion weight and the distance fusion weight are calculated using a preset gating network or gating rules. Based on the visual fusion weight and the distance fusion weight, the visual category posterior and the distance category posterior are fused using a posterior probability layer to obtain the final terrain category probability.

[0011] On the other hand, a lower limb wearable terrain recognition method based on RGB-IMU-single-point ToF fusion is provided, which is applied to the aforementioned lower limb wearable terrain recognition system based on multi-source signal fusion. The method includes the following steps: Acquire RGB images, single-point ToF scalar distance values, and single thigh IMU inertial data to form a synchronization frame with a unified timestamp; Based on the flat ground walking calibration data of the target wearer, a reference relationship between the attitude phase interval and the flat ground ToF distance is established, and the original single-point ToF distance value of the current synchronization frame is converted into the phase residual value. The RGB image of the current synchronization frame is input into the first lightweight classifier, which outputs the posterior probability of the visual category; the phase residual value is combined with the kinematic features of the single IMU and input into the second lightweight classifier, which outputs the posterior probability of the distance category. Based on the confidence levels of the visual category posterior probability and the distance category posterior probability, the validity of the single-point ToF data, and the phase residual amplitude, the fusion weight is dynamically calculated, and the visual category posterior and the distance category posterior are fused at the posterior probability layer to obtain the final terrain category probability. The terrain recognition result at the current moment is output based on the final terrain category probability. The terrain recognition result includes one of the following: flat ground, upstairs, downstairs, uphill, and downhill.

[0012] Furthermore, the step of establishing a reference relationship between the attitude phase interval and the ToF distance on flat ground based on the target wearer's flat ground walking calibration data, and converting the original single-point ToF distance value of the current synchronization frame into a phase residual value, includes: Collect data on flat ground as the target wearer walks continuously, and record the thigh pose phase and single-point ToF distance value for each frame; According to the preset phase interval division rules, the attitude phase interval b is obtained, where, Where bin is the mapping function, The thigh pose phase in the i-th frame; Establish a flat-ground ToF distance reference for phase interval b: ,in, Let be the standard ToF ranging reference value for the target wearer s while walking on flat ground within the posture phase interval b; median is a median function. The ToF distance value of a single point in the j-th frame. For a dataset of flat ground where the target wearer s walks continuously, Let LG represent the ground truth label for frame j; Calculate the phase residual: ,in, Let be the phase residual of the i-th frame. This represents the ToF distance value of a single point in the i-th frame. This is the standard ToF ranging reference value for walking on flat ground in the i-th frame.

[0013] Further, the step of fusing the visual category posterior and the distance category posterior through a posterior probability layer to obtain the final terrain category probability includes: ; in, This represents the probability distribution of all terrain categories after fusion. For normalization function, and These are the fusion weights for the visual category and the distance category at frame i, respectively. For the visual class posterior probabilities mapped to the class set C, This represents the posterior probability of the distance category. To prevent a small constant with a logarithm of zero, the final terrain category probability is the maximum value among the probability distributions of all terrain categories.

[0014] Further, the step of dynamically calculating the fusion weights based on the confidence levels of the visual category posterior probability and the distance category posterior probability, the validity of the single-point ToF data, and the phase residual amplitude includes: Constructing gated feature vectors: ,in, Let be the gated feature vector of the i-th frame. For the visual class posterior probabilities mapped to the class set C, This represents the posterior probability of the distance category. This serves as a validity identifier for the single-point ToF data in the i-th frame. Let be the phase residual of the i-th frame. The gated feature vector is input into a preset gated network or gated function, and after processing by a normalized exponential function, visual fusion weights and distance fusion weights are output.

[0015] Further, the step of outputting the terrain recognition result at the current moment based on the final terrain category probability includes: Output the terrain category and confidence level at the current moment, where the confidence level is the probability of the final terrain category.

[0016] Beneficial effects: This application provides a lower limb wearable terrain recognition system and method based on RGB-IMU-single-point ToF fusion. Through multi-source signal fusion and modular design, it achieves lightweight and robust terrain recognition on lower limb wearable devices. The sensing components consist of an RGB camera, a single-point ToF ranging sensor, and a single IMU. The single-point ToF outputs only the scalar distance in the central line-of-sight direction, providing crucial external geometric cues while avoiding the size, power consumption, and computational burden of RGB-D or LiDAR, making it easy to integrate. The MCU data processing module unifies time synchronization, ensuring frame-level alignment of multi-sensor data and reducing processing latency. The ToF phase residual calibration module establishes a posture phase correlation reference table using flat-ground calibration, effectively eliminating the influence of wearer height, installation deviation, and gait cycle on the original distance value, enabling rapid individual adaptation. RGB semantic recognition and ToF-IMU distance recognition extract terrain information from visual appearance and distance geometry dimensions, respectively. The posterior-gated fusion module dynamically adjusts the fusion weights based on data validity and confidence, eliminating the need for manually setting fixed thresholds and adaptively handling environmental glare interference or single-mode failure. The terrain output module directly provides five types of terrain results. The overall architecture ensures recognition accuracy and stability under low power consumption and low computing power constraints, providing a reliable perception basis for real-time assistive control of exoskeletons or prostheses. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural block diagram of a wearable terrain recognition system for lower limbs provided in an embodiment of this application; Figure 2 This is a schematic diagram of a wearable sensor installation provided in an embodiment of this application; Figure 3 This is a flowchart of a ToF phase asymmetry calibration provided in an embodiment of this application; Figure 4 This is a flowchart of a post-approval gating fusion process provided in an embodiment of this application; Figure 5 This is a flowchart of a terrain recognition method provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] While theoretically, RGB-D cameras, binocular vision, or LiDAR in laboratories can acquire accurate geometric information, in practical applications, space and cost constraints make it impossible for lower limb wearable devices like RGB-D cameras to be industrially viable. Although single-point ToF ranging sensors meet the cost and space requirements, the directly measured data cannot satisfy actual needs.

[0021] Therefore, in order to solve the above-mentioned technical problems, referring to... Figure 1 This application provides a lower limb wearable terrain recognition system based on RGB-IMU-single-point ToF fusion, comprising: Wearable sensing component 101, MCU data processing module (i.e. Figure 1 The module includes an MCU102, a ToF phase residual calibration module103, an RGB semantic recognition module104, a ToF-IMU distance recognition module105, a posterior gated fusion module106, and a terrain output module107. The wearable sensing component includes an RGB camera, a single-point ToF ranging sensor, and an IMU, preferably a single IMU, all mounted on the lower limbs. The ranging direction of the single-point ToF ranging sensor is aligned with the central line of sight of the RGB camera, and it only outputs a single-point scalar distance value along this ranging direction. For example, such as Figure 2 As shown, the wearable sensing component 101 is installed on one lower limb of the user. An RGB camera 101-1 and a single-point ToF ranging sensor 101-2 are positioned on the thigh near the knee, facing the ground or the area where the feet will rest. The ranging direction of the single-point ToF ranging sensor 101-2 is aligned with the central line of sight of the RGB camera 101-1, and it only outputs a single-point scalar distance in that ranging direction. A single IMU 101-3 is positioned on the thigh to collect thigh segment movement and posture data. These sensors are connected to the MCU module 102 via wired or wireless means. The MCU is connected to or is part of the computing unit, used for unified system time, acquisition triggering, data reception, and timestamp recording.

[0022] The MCU data processing module is used to maintain a unified system time and perform time synchronization and preprocessing on the data from the RGB camera, the single-point ToF ranging sensor, and the single IMU to form a synchronization frame indexed by a unified timestamp. Specifically, the MCU maintains a unified system time and writes timestamps for each sensor data according to the arrival time of the data. When forming the i-th synchronization frame, the RGB image, ToF distance, and IMU data with the smallest time difference are selected using the unified timestamp as an index. The ToF distance value can be recorded in millimeters or meters, and outliers with no return value or exceeding the valid range are removed to form a ToF validity flag. The IMU data includes three-axis acceleration, three-axis angular velocity, and three-axis Euler angles, which are filtered to form the kinematic features of a single IMU. The RGB image is cropped, scaled, and normalized before being input into the visual recognition network.

[0023] The ToF phase residual calibration module is used to establish a flat ground ToF distance reference table corresponding to the posture phase interval based on the calibration data of the target wearer walking on flat ground, and convert the currently collected single-point ToF distance value into a phase residual value relative to the flat ground reference. For example, such as Figure 3 As shown, the ToF phase residual calibration module collects a preset duration of flat-ground walking data as calibration data after the target wearer's first use or after the sensor is reinstalled, and divides the phase intervals according to the thigh pitch angle, gravity reference pitch angle, or gait phase index. For each phase interval, the ToF distance reference value for the target wearer in flat-ground conditions is calculated. During recognition, the current ToF distance is subtracted from the flat-ground reference value of the corresponding phase interval to obtain the ToF phase residual.

[0024] The RGB semantic recognition module is used to output the posterior probability of the visual category based on the RGB image of the current synchronization frame; For example, the RGB semantic recognition module uses a lightweight convolutional neural network or other image classification model to output a visual category posterior for the current RGB image. The visual category posterior can directly correspond to flat ground, upstairs, and downstairs, and then be converted into five visual posteriors corresponding to flat ground, upstairs, downstairs, uphill, and downhill through a compatible mapping. The visual evidence corresponding to the appearance of flat ground allows the ToF-IMU distance branch to further distinguish geometric differences such as uphill, downhill, and flat ground.

[0025] The ToF-IMU distance identification module is used to output a posterior probability of the distance category based on distance geometry information, according to the phase residual value and the kinematic features of the single IMU. For example, the ToF-IMU distance recognition module combines the ToF phase residual with the features of a single thigh IMU to form a distance feature vector, and inputs it into a random forest, support vector machine, shallow neural network or other lightweight classifier, and outputs a terrain category distance posterior based on the distance residual and thigh motion context.

[0026] The posterior gating fusion module is used to dynamically calculate the fusion weight based on the visual category posterior probability, the distance category posterior probability and the preset gating factor, and to perform weighted fusion of the visual category posterior and the distance category posterior to obtain the final terrain category probability. For example, such as Figure 4 As shown, the posterior-gated fusion module receives five types of visual posteriors and distance posteriors, and calculates fusion weights based on the maximum confidence of the two posteriors, the class distribution, the Time-of-Flight (ToF) validity flag, the phase residual amplitude, or a preset gating network. The fusion weights are used to adjust the contributions of visual and distance evidence to obtain the final class probabilities.

[0027] The terrain output module is used to output the terrain recognition result at the current moment based on the final terrain category probability.

[0028] The output categories include flat ground, upstairs, downstairs, uphill, and downhill; when the dataset or application scenario only contains some categories, a subset of the above categories can also be output. The output results include terrain categories and timestamps, and can be sent to the lower limb assistive device controller via serial port, bus, wireless communication, or software interface.

[0029] The implementation process is as follows: After the user puts on the lower limb assistive device, the RGB camera and single-point ToF ranging sensor are fixed to the thigh near the knee, facing forward towards the foot placement area, and the single-point ToF ranging direction is aligned with the center line of sight of the RGB camera; a single IMU is fixed to the thigh. After the system starts, the MCU establishes a unified system time and triggers the acquisition and timestamp recording of RGB images, ToF distance, and IMU data, so that the sensor data form a synchronized frame under the same system time.

[0030] After the target wearer uses or reinstalls the sensor for the first time, approximately 60 seconds of flat walking data is collected. The system establishes a flat ground ToF distance reference table for the target wearer based on the thigh posture phase. In subsequent recognition processes, the current ToF distance is converted into a residual value through the phase reference table, and then used together with the individual thigh IMU features for distance recognition.

[0031] When the RGB image has a high confidence level for the stairs, the posterior gated fusion module increases the visual posterior weight; when the RGB image does not clearly distinguish between flat ground and ramp appearance, but the ToF phase residual and single IMU features show obvious height or slope changes, the posterior gated fusion module increases the distance posterior weight, thereby obtaining the final terrain category and timestamp.

[0032] Understandably, without changing the concepts of lightweight deployment of a single IMU, phase residual calibration, and posterior gating fusion, the single IMU can be a six-axis or nine-axis inertial measurement unit; the attitude phase can be determined by the thigh pitch angle, gravity reference pitch angle, or gait phase percentage; the ToF ground reference can be estimated using the median, mean, quantile, or regression model; and the posterior gating can be implemented using regular weights, linear models, or small neural networks.

[0033] The main improvements in this application are as follows: First, the single-point ToF is set to be aligned with the center line of sight of the RGB camera, so that it can provide the minimum geometric cues of the terrain ahead without forming a depth map or point cloud; Second, a flat ground distance reference related to the attitude phase is established using the target wearer's approximately 60s flat ground walking data, and the original ToF distance is converted into a phase residual; Third, the RGB visual posterior and the ToF-IMU distance posterior are gated and fused at the posterior layer, so that visual appearance evidence and distance geometric evidence are complementary.

[0034] Based on the above improvements, this application has the following effects: 1. Lightweight sensor configuration: It adopts an RGB camera, a center-aligned single-point ToF sensor, and a single thigh IMU, all of which are consumer-grade small-volume sensors. They do not rely on RGB-D cameras, LiDAR, or dense point cloud processing, making them easy to integrate into lower limb wearable devices.

[0035] 2. Single-point geometrically caliable: Through phase residual calibration, the original distance, which is affected by individual differences, is converted into a geometric residual relative to the target wearer's flat ground reference.

[0036] 3. Clear fusion relationship: The RGB branch provides scene semantics, the ToF-IMU distance branch provides distance geometry and thigh motion context, and the gated fusion module dynamically adjusts the two types of evidence based on confidence, ToF validity and residual amplitude.

[0037] 4. Lightweight deployment with a single IMU: The system uses a single large IMU as the main implementation method. The classifier type, gating method and phase division method can be replaced according to the hardware and application scenario without affecting the core phase residual calibration concept.

[0038] 5. Clear output results: The system outputs terrain category and timestamp, which can be used as mode switching input for lower limb exoskeletons, powered prostheses, or soft walking aids.

[0039] Based on the same inventive concept, this application provides a lower limb wearable terrain recognition method based on RGB-IMU-single-point ToF fusion, which is applied to the aforementioned multi-source signal fusion lower limb wearable terrain recognition system, such as... Figure 5 As shown, the method includes the following steps: S1: Acquire RGB images, single-point ToF scalar distance values, and single thigh IMU inertial data to form a synchronization frame with a unified timestamp; As a preferred implementation of this application, the terrain category set is defined as follows: , representing flat ground, going upstairs, going downstairs, going uphill, and going downhill, respectively. The input of the i-th synchronization frame is defined as ,in For an RGB image synchronized with the i-th frame, The ToF scalar distance at a single point aligned with the line of sight of the RGB camera center. The thigh pitch angle or the attitude phase variable obtained from a single IMU. Kinematic characteristics of a single thigh IMU. To unify system timestamps.

[0040] S2: Based on the target wearer's flat ground walking calibration data, establish a reference relationship between the attitude phase interval and the flat ground ToF distance, and convert the original single-point ToF distance value of the current synchronization frame into a phase residual value; Specifically, data on flat ground during continuous walking by the target wearer is collected, and the thigh posture phase and single-point ToF distance value are recorded for each frame. According to the preset phase interval division rules, the posture phase interval is obtained, a flat ground ToF distance reference for the phase interval is established, and the phase residual is calculated.

[0041] For example, the attitude phase interval is defined as . For the target wearer s, data on walking on flat ground for approximately 60 seconds. Establish a flat ground ToF distance reference within phase interval b: ,in, Let be the standard ToF ranging reference value for the target wearer s while walking on flat ground within the posture phase interval b; median is a median function. The ToF distance value of a single point in the j-th frame. For a dataset of flat ground where the target wearer s walks continuously, Let LG represent the ground truth label for frame j; The ToF phase residual of the current synchronization frame is defined as .in, Let be the phase residual of the i-th frame. This represents the ToF distance value of a single point in the i-th frame. This is the standard ToF ranging reference value for walking on flat ground in the i-th frame.

[0042] When there are insufficient flat ground samples in a certain phase interval, backtracking can be performed using interpolation between adjacent phase intervals, the global flat ground median of the target wearer, or a pre-stored universal flat ground reference. The Time-of-Flight (ToF) validity indicator is denoted as... This is used to indicate whether the current ToF distance has no return, exceeds the effective ranging range, or has abnormal reflection.

[0043] S3: Input the RGB image of the current synchronization frame into the first lightweight classifier and output the posterior probability of the visual category; combine the phase residual value with the kinematic features of the single IMU and input it into the second lightweight classifier to output the posterior probability of the distance category; The first lightweight classifier can be a lightweight convolutional neural network, decision tree, or random forest, while the second lightweight classifier can be a random forest, support vector machine, or shallow neural network. Since these solutions are all conventional techniques in the field, and this application does not improve upon them, they will not be described in detail here.

[0044] For example, the ToF-IMU distance branch input is defined as The posterior distance is ,in, Let represent the mapping function of the first lightweight classifier. The RGB semantic posterior is... , This represents the mapping function of the second lightweight classifier; when the RGB semantic recognition module first outputs three visual posteriors (flat ground, upstairs, and downstairs), it maps them to the five visual posteriors corresponding to the category set C through the compatible mapping M, denoted as . .

[0045] S4: Based on the confidence levels of the visual category posterior probability and the distance category posterior probability, the validity of the single-point ToF data, and the phase residual amplitude, dynamically calculate the fusion weights, perform posterior probability layer fusion on the visual category posterior and the distance category posterior, and obtain the final terrain category probability; In posterior gating fusion, the gating input is defined as: Gating weight is defined as .

[0046] The final posterior optimization adopts log-posterior fusion: ,in, This represents the probability distribution of all terrain categories after fusion. For normalization function, and These are the fusion weights for the visual category and the distance category at frame i, respectively.

[0047] The final category is Terrain output module output This includes terrain type and timestamp; it can also output fused posterior when needed. As a confidence level.

[0048] S5: Output the terrain recognition result at the current moment according to the final terrain category probability. The terrain recognition result includes one of flat ground, upstairs, downstairs, uphill, and downhill.

[0049] The lower limb wearable terrain recognition method based on RGB-IMU-single-point ToF fusion provided in this application integrates RGB vision, single-point ToF ranging, and IMU kinematic information through a standardized process, achieving high-precision terrain recognition under lightweight hardware constraints. Step S2 establishes a ToF distance reference related to the posture phase interval using short-time flat ground walking calibration, converting the original distance into phase residuals, effectively eliminating individual differences caused by wearer height, sensor installation deviation, and gait cycle, achieving rapid adaptation without parameter tuning. Step S4 constructs a gated feature vector, dynamically calculates fusion weights based on the confidence of the visual and distance branches, ToF effectiveness, and residual amplitude, and adopts a logarithmic posterior fusion strategy to avoid the performance degradation problem of fixed weights or manual thresholds under environmental interference such as reflection and occlusion, adaptively enhancing the contribution of the reliable modality. The overall method relies solely on single-point ToF instead of RGB-D or LiDAR, resulting in low computational burden and good real-time performance. It can stably output five types of terrain: flat ground, stairs, slopes, and sloping ground, providing reliable and robust perception data for gait switching and assistive control of lower limb exoskeletons or prostheses.

[0050] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0051] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0052] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0053] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0054] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0055] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0056] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0058] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A lower limb wearable terrain recognition system based on RGB-IMU-single-point ToF fusion, characterized in that, include: Wearable sensing components, MCU data processing module, ToF phase residual calibration module, RGB semantic recognition module, ToF-IMU distance recognition module, posterior gated fusion module, and terrain output module; The wearable sensing component includes an RGB camera, a single-point ToF ranging sensor, and an IMU, all mounted on the lower limbs. The ranging direction of the single-point ToF ranging sensor is aligned with the central line of sight of the RGB camera, and it only outputs a single-point scalar distance value along this ranging direction. The MCU data processing module is used to maintain a unified system time, and to perform time synchronization and preprocessing on the data from the RGB camera, the single-point ToF ranging sensor and the single IMU to form a synchronization frame indexed by a unified timestamp. The ToF phase residual calibration module is used to establish a flat ground ToF distance reference table corresponding to the posture phase interval based on the calibration data of the target wearer walking on flat ground, and convert the currently collected single-point ToF distance value into a phase residual value relative to the flat ground reference. The RGB semantic recognition module is used to output the posterior probability of the visual category based on the RGB image of the current synchronization frame; The ToF-IMU distance identification module is used to output a posterior probability of the distance category based on distance geometry information, according to the phase residual value and the kinematic features of the single IMU. The posterior gating fusion module is used to dynamically calculate the fusion weight based on the visual category posterior probability, the distance category posterior probability and the preset gating factor, and to perform weighted fusion of the visual category posterior and the distance category posterior to obtain the final terrain category probability. The terrain output module is used to output the terrain recognition result at the current moment based on the final terrain category probability.

2. The system according to claim 1, characterized in that, The single-point ToF ranging sensor is fixed on the same bracket as the RGB camera, and the central line of sight of the single-point ToF ranging sensor is parallel or coincident with the optical axis of the RGB camera, so that the single-point ToF ranging sensor can measure the single-point distance corresponding to the center area of ​​the field of view of the RGB camera.

3. The system according to claim 1, characterized in that, The MCU data processing module is used for: According to the unified system time, each sensor is asynchronously triggered to collect data, and timestamps are written for the received RGB images, single-point ToF distance values ​​and IMU data respectively; During data synchronization, the RGB image with the smallest time difference, the single-point ToF distance value, and the IMU data are selected as an index using the timestamp of the target time as an index to form a synchronization frame. In addition, the validity of the single-point ToF distance value is checked, and data points with no return value or exceeding the preset valid distance measurement range are removed, and a ToF validity flag is generated.

4. The system according to claim 1, characterized in that, The ToF phase residual calibration module is specifically used for: After the target wearer uses the device for the first time or after the sensor is reinstalled, collect flat walking data for a preset duration as a calibration dataset. Based on the thigh pitch angle or gravity reference pitch angle measured by the single IMU, the gait cycle is divided into multiple posture phase intervals; For each attitude phase interval, the median or mean of the ToF distance values ​​of all single points falling into the attitude phase interval in the calibration dataset is calculated and used as the flat ground ToF distance reference value corresponding to the attitude phase interval. In the current real-time recognition phase, the corresponding flat ground ToF distance reference value is found based on the thigh pose phase index of the current frame, and the original single-point ToF distance value of the current frame is subtracted from the flat ground ToF distance reference value to obtain the phase residual value.

5. The system according to claim 1, characterized in that, The posterior gating fusion module is specifically used for: Obtain the visual category posterior probability vector output by the RGB semantic recognition module and the distance category posterior probability vector output by the ToF-IMU distance recognition module; Obtain the ToF validity flag and phase residual amplitude of the current synchronization frame; Based on the maximum confidence of the visual category posterior probability, the maximum confidence of the distance category posterior probability, the ToF validity flag, and the phase residual amplitude, the visual fusion weight and the distance fusion weight are calculated using a preset gating network or gating rules. Based on the visual fusion weight and the distance fusion weight, the visual category posterior and the distance category posterior are fused using a posterior probability layer to obtain the final terrain category probability.

6. A method for wearable terrain recognition of the lower limbs based on RGB-IMU-single-point ToF fusion, characterized in that, Applied to the system according to any one of claims 1-5, the method comprises the following steps: Acquire RGB images, single-point ToF scalar distance values, and single thigh IMU inertial data to form a synchronization frame with a unified timestamp; Based on the flat ground walking calibration data of the target wearer, a reference relationship between the attitude phase interval and the flat ground ToF distance is established, and the original single-point ToF distance value of the current synchronization frame is converted into the phase residual value. The RGB image of the current synchronization frame is input into the first lightweight classifier, which outputs the posterior probability of the visual category; the phase residual value is combined with the kinematic features of the single IMU and input into the second lightweight classifier, which outputs the posterior probability of the distance category. Based on the confidence levels of the visual category posterior probability and the distance category posterior probability, the validity of the single-point ToF data, and the phase residual amplitude, the fusion weight is dynamically calculated, and the visual category posterior and the distance category posterior are fused at the posterior probability layer to obtain the final terrain category probability. The terrain recognition result at the current moment is output based on the final terrain category probability. The terrain recognition result includes one of the following: flat ground, upstairs, downstairs, uphill, and downhill.

7. The method according to claim 6, characterized in that, The process involves establishing a reference relationship between the attitude phase interval and the ToF distance on flat ground based on the target wearer's flat-ground walking calibration data, and converting the original single-point ToF distance value of the current synchronization frame into a phase residual value, including: Collect data on flat ground as the target wearer walks continuously, and record the thigh pose phase and single-point ToF distance value for each frame; According to the preset phase interval division rules, the attitude phase interval b is obtained, where, Where bin is the mapping function, The thigh pose phase in the i-th frame; Establish a flat-ground ToF distance reference for phase interval b: ,in, Let be the standard ToF ranging reference value for the target wearer s while walking on flat ground within the posture phase interval b; median is a median function. The ToF distance value of a single point in the j-th frame. For a dataset of flat ground where the target wearer s walks continuously, Let LG represent the ground truth label for frame j; Calculate the phase residual: ,in, Let be the phase residual of the i-th frame. This represents the ToF distance value of a single point in the i-th frame. This is the standard ToF ranging reference value for walking on flat ground in the i-th frame.

8. The method according to claim 6, characterized in that, The step of fusing the visual category posterior and the distance category posterior with a posterior probability layer to obtain the final terrain category probability includes: ； in, This represents the probability distribution of all terrain categories after fusion. For normalization function, and These are the fusion weights for the visual category and the distance category at frame i, respectively. For the visual class posterior probabilities mapped to the class set C, This represents the posterior probability of the distance category. To prevent a small constant with a logarithm of zero, the final terrain category probability is the maximum value among the probability distributions of all terrain categories.

9. The method according to claim 8, characterized in that, The dynamic calculation of fusion weights based on the confidence levels of the visual category posterior probability and the distance category posterior probability, the validity of the single-point ToF data, and the phase residual amplitude includes: Constructing gated feature vectors: ,in, Let be the gated feature vector of the i-th frame. For the visual class posterior probabilities mapped to the class set C, This represents the posterior probability of the distance category. This serves as a validity identifier for the single-point ToF data in the i-th frame. Let be the phase residual of the i-th frame. The gated feature vector is input into a preset gated network or gated function, and after processing by a normalized exponential function, visual fusion weights and distance fusion weights are output.

10. The method according to claim 6, characterized in that, The step of outputting the terrain recognition result at the current moment based on the final terrain category probability includes: Output the terrain category and confidence level at the current moment, where the confidence level is the probability of the final terrain category.

Images