Lower extremity exoskeleton control method based on deep learning
By detecting and recognizing the movement status and environmental context of the elderly through image data, a precise lower limb exoskeleton control mode is generated, which solves the comfort and stability problems of existing devices in complex scenarios and achieves automatic environmental adaptation and safety redundancy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN MILEBOT ROBOTICS CO LTD
- Filing Date
- 2026-01-28
- Publication Date
- 2026-06-12
AI Technical Summary
Existing lower limb exoskeleton devices are difficult to match the actual movement intentions of the elderly, especially in complex scenarios where they are prone to misjudgment. They also lack environmental perception and automatic control mode switching, resulting in insufficient comfort, stability and safety.
By acquiring image data of the lower limbs and the environment in the direction of travel of users wearing lower limb exoskeletons, leg and environmental target detection is performed, motion state and environmental context labels are fused, and precise control modes and parameters are generated to output walking assistance.
It improves the comfort, stability and safety of the lower limb exoskeleton device in complex scenarios, reduces misjudgments, and enhances the applicability and safety redundancy of the system.
Smart Images
Figure CN122185157A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotics technology, and in particular to a deep learning-based method for controlling a lower limb exoskeleton. Background Technology
[0002] Existing lower limb exoskeletons or assistive devices for elderly mobility mostly rely on preset gait parameters, fixed assist curves, or triggering strategies based on simple thresholds to provide auxiliary output to flexion and extension joints. However, the gait of the elderly has significant individual differences and time-varying characteristics, and is prone to shortening of stride, unstable swing phase, and gait rhythm drift under the influence of factors such as fatigue, pain, and changes in ground surface. As a result, existing devices are difficult to continuously match the user's true movement intention in actual use, and are prone to the phenomenon of "not keeping up" or "over-keeping up," affecting comfort and stability.
[0003] Some existing control schemes primarily rely on single or limited sensor information, such as inertial measurement units (IMUs), plantar pressure, joint angles, or electromyography, for gait phase recognition and control decisions. Due to the small range of motion, irregular movement patterns, and the prevalence of signal noise and wear misalignment among the elderly, these sensor signals are prone to misjudgment or inaccurate phase switching in complex movement scenarios such as going uphill / downhill, turning, crossing obstacles, and standing / sitting, leading to timing deviations in the auxiliary torque output and increasing safety risks such as tripping and imbalance. Furthermore, in real-world home and community environments, factors such as ground material, steps, ramps, thresholds, narrow passages, and dynamic pedestrians significantly alter the required gait assistance strategies. Additionally, existing deep learning-based lower limb exoskeleton control systems generally lack effective perception and semantic understanding of the surrounding environment, making it difficult to automatically switch control modes between scenarios such as "flat ground—ramp—steps—obstacles." Manual switching is often required, relying on user experience, or the system can only operate under limited conditions, thus reducing its applicability and safety redundancy. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, this invention provides a deep learning-based lower limb exoskeleton control method, which solves the problems of existing lower limb exoskeletons or assistive devices relying on preset parameters and single sensor information, making it difficult to match the user's real movement intentions, prone to misjudgment in complex scenarios, and lacking environmental perception and automatic control mode switching, thereby improving comfort, stability, applicability and safety redundancy.
[0005] The first aspect of this application provides a deep learning-based method for controlling a lower limb exoskeleton, the method comprising: Acquire image data of the lower limbs and the environment in the direction of travel of a user wearing a lower limb exoskeleton; Perform target detection on each frame of the image data and output the leg target detection result and the environment target detection result; The control mode and control parameters of the lower limb exoskeleton are determined based on the leg target detection results and the environmental target detection results. The lower limb exoskeleton control parameters are generated according to the control mode and the control parameters, and the lower limb exoskeleton control parameters are output to the lower limb exoskeleton actuator to realize the user's walking assistance.
[0006] In an optional implementation, determining the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results includes: The leg target detection results are processed by leg inference, and motion state labels are output. The environmental target detection results are processed by environmental reasoning to output environmental context labels; The motion state label is fused with the environmental context label, and the control mode and the corresponding control parameters are determined based on the fusion result.
[0007] In an optional implementation, the leg inference processing performed on the leg target detection results to output motion state labels includes: The leg target detection results are processed for temporal consistency. If the leg target in the leg target detection result after time-series consistency processing exhibits the detection characteristics of "moving up and moving forward" in the first consecutive preset frames, then the leg target is identified as a swing cycle and a swing cycle motion state label is output. If the leg target is stable in the first consecutive preset frames, the leg target is identified as a support cycle and a support cycle motion state label is output. If the leg target is not detected in the second consecutive preset frame, the leg target is identified as a stopped or abnormal state and a stopped or abnormal state motion status label is output.
[0008] In an optional implementation, the temporal consistency processing of the leg target detection results includes: The cross-union ratio of detection boxes of adjacent similar leg targets in the detection results is calculated by detection box matching; When the intersection-union ratio is greater than a preset threshold, the adjacent leg targets of the same type are determined to be continuous observations of the same target; When multiple candidates of the same type of target are detected in any leg target, the candidate with the largest intersection-union ratio with the leg target in the previous frame and whose detection confidence meets the preset threshold is selected as the continuation target of the leg target in the current frame. The confidence level and frame position of the continued target are smoothed using a time window. If no leg target is detected in the second consecutive preset frame or the detection confidence is lower than the preset threshold, the "leg target exists" status will continue to be output within the second consecutive preset frame using the previous stable observation value.
[0009] In an optional implementation, the environmental inference processing of the environmental target detection results, and the output of environmental context labels, includes: Obtain the detection confidence score for each environmental category within N consecutive frames of the environmental target detection results; When a certain environment category is detected within N consecutive frames and the detection confidence is greater than a preset confidence threshold, it is determined that the current actual environment has entered the specific environmental scenario corresponding to the certain environment category. Output the environmental scenario label corresponding to the specific environmental scenario.
[0010] In an optional implementation, the step of performing target detection inference on each frame of the image data and outputting leg target detection results and environment target detection results includes: The image data is input into a preset leg target detection model so that the leg target detection model outputs the leg target detection result; The image data is input into a preset environmental target detection model so that the environmental target detection model outputs the environmental target detection result.
[0011] In an optional implementation, before performing target detection on each frame of the image data, the method further includes: The image data is preprocessed, including any one or a combination of distortion correction, scaling to the model input size, brightness normalization, and denoising.
[0012] A second aspect of this application provides a deep learning-based lower limb exoskeleton control device, the device comprising: The data acquisition module is used to acquire image data of the lower limbs of the user wearing the lower limb exoskeleton and the environment in the direction of travel; The target detection and inference module is used to perform target detection on each frame of the image data and output the leg target detection result and the environmental target detection result. The fusion decision module is used to determine the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results; The control output module is used to generate lower limb exoskeleton control quantities according to the control mode and the control parameters, and output the lower limb exoskeleton control quantities to the lower limb exoskeleton actuator to realize the user's walking assistance.
[0013] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the deep learning-based lower limb exoskeleton control method.
[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described deep learning-based lower limb exoskeleton control method.
[0015] In summary, the deep learning-based lower limb exoskeleton control method provided in this application has at least one of the following beneficial effects: 1. Acquire image data of the user's lower limbs and the surrounding environment in the direction of movement while wearing the lower limb exoskeleton. Perform target detection on each frame of the image and output the leg target detection results and the environmental target detection results. By acquiring comprehensive image information and performing target detection, the actual movement state of the user's lower limbs and the surrounding environment can be captured more accurately. Compared with methods that rely on single or limited sensor information, it can more comprehensively and realistically reflect the user's movement intentions, thereby avoiding the phenomenon of "not keeping up / not keeping up" due to insufficient or inaccurate information, and improving comfort and stability. 2. Utilizing image data, which contains rich information and is not limited by factors such as the small range of motion, irregular movement patterns, signal noise, and wearing misalignment in elderly individuals. By performing target detection on the image data, gait phase can be identified more accurately and control decisions can be made, reducing misjudgments or inaccurate phase switching in complex movement scenarios such as going uphill, downhill, and turning. This avoids timing deviations in the auxiliary torque output and reduces safety risks such as tripping and imbalance. 3. Acquire image data of the environment in the direction of travel and perform environmental target detection, enabling the perception of various surrounding environmental scenarios, such as ground texture, steps, ramps, and obstacles. Based on the environmental target detection results, determine the control mode and control parameters of the lower limb exoskeleton, achieving automatic switching of control modes under different scenarios without manual switching or reliance on user experience, thus improving the system's applicability in different environments and increasing safety redundancy. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating a deep learning-based lower limb exoskeleton control method according to an embodiment of this application; Figure 2 This is a schematic diagram of an image data acquisition scene during the movement of a lower limb exoskeleton, as shown in an embodiment of this application. Figure 3 This is a schematic diagram illustrating the training process of an object detection model according to an embodiment of this application; Figure 4 This is a functional block diagram of a deep learning-based lower limb exoskeleton control device shown in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation
[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0018] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.
[0019] The following describes a deep learning-based lower limb exoskeleton control method using an electronic device as the execution subject. The electronic device can be integrated into the lower limb exoskeleton or a mobile terminal that communicates with the lower limb exoskeleton.
[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a deep learning-based lower limb exoskeleton control method according to an embodiment of this application. The deep learning-based lower limb exoskeleton control method includes the following steps.
[0021] S11, acquire image data of the user's lower limbs and the environment in the direction of travel for the user wearing the lower limb exoskeleton.
[0022] Refer to together Figure 2 In some embodiments, a visual acquisition unit acquires image data containing the user's lower limbs and the surrounding environment in the direction of travel, along with the user's lower limb exoskeleton. The visual acquisition unit can be a monocular wide-angle camera, mounted on the chest support of the lower limb exoskeleton, so that the field of view simultaneously covers the user's legs and the road surface area in front (also called the detection area). After the user puts on the lower limb exoskeleton and starts the system, the visual acquisition unit continuously acquires image or video frames containing information about the user's lower limbs and the road surface in the direction of travel at a preset frame rate, obtaining image data. A timestamp is added to each frame of the image data.
[0023] In some embodiments, to ensure the stability of subsequent object detection model inference, the electronic device needs to perform uniform preprocessing on each input frame of image. Specifically, using pre-calibrated camera intrinsic parameters and distortion parameters, the cv2.undistort() function from the OpenCV library is used to perform distortion correction on the input frame. For example, for input frames where the image edges are curved and deformed due to camera lens distortion, this function can restore the geometric shapes such as straight lines in the image to their normal state, eliminating the impact of lens distortion on subsequent object detection. Next, according to the input image size required by the subsequent object detection model, such as 640×480 pixels, the cv2.resize() function is used to scale the distortion-corrected image. During the scaling process, a bilinear interpolation algorithm is used to ensure that the image still maintains good visual quality and detail information after scaling, so that the image size meets the model input requirements. Then, to eliminate the influence of different lighting conditions on image brightness, brightness normalization processing is performed on the scaled image. Specifically, the global average brightness value of the image is calculated. Then, based on a preset target brightness value (e.g., 128), the brightness value of each pixel in the image is adjusted through a linear transformation to achieve a relatively uniform overall brightness. For example, if the average brightness of the image is lower than the target brightness value, the brightness value of each pixel is increased proportionally; conversely, it is decreased. Finally, a median filtering algorithm is used to denoise the brightness-normalized image. The `cv2.medianBlur()` function is used, and an appropriate filter kernel size (e.g., 3×3 or 5×5) is selected based on the noise level of the image. Median filtering effectively removes isolated noise points such as salt-and-pepper noise in the image while preserving edge information well, further improving image quality. Through a series of preprocessing operations, the input images received by the subsequent object detection model have consistent distribution characteristics under different lighting conditions (e.g., strong light, weak light, cloudy days) and different scenes (e.g., indoor, outdoor, complex backgrounds).
[0024] A single wide-angle camera can simultaneously perceive the user's lower limb movement and the surrounding environment, avoiding the system complexity, calibration costs, and reliability issues associated with multiple sensors such as inertial sensors, force sensors, and plantar pressure sensors. This significantly reduces hardware costs and system integration difficulty, and improves the stability and maintainability of the lower limb exoskeleton system in real-world applications.
[0025] S12, perform target detection on each frame of the image data, and output the leg target detection result and the environment target detection result.
[0026] After image data preprocessing is complete, the electronic device can use the object detection model to perform object detection inference on each frame of the image, outputting leg object detection results and environment object detection results. The leg object detection results include leg detection categories, such as "thigh, calf, foot". The environment object detection results include environment detection categories, such as "slope area, step edge, obstacle, threshold, pothole / uneven area, pedestrian / dynamic target", etc.
[0027] In an optional implementation, the step of performing target detection inference on each frame of the image data and outputting leg target detection results and environment target detection results includes: The image data is input into a preset leg target detection model so that the leg target detection model outputs the leg target detection result; The image data is input into a preset environmental target detection model so that the environmental target detection model outputs the environmental target detection result.
[0028] In some embodiments, the preprocessed image data is input into preset target detection models, namely a leg target detection model and an environment target detection model, so that the leg target detection model and the environment target detection model perform forward inference once to obtain a set of detection results for each frame of the image, namely the leg target detection result and the environment target detection result. The set of detection results consists of several detection instances, and each detection instance contains a target category. Bounding box and confidence level .in This represents the coordinates of the top-left corner of the bounding box in the horizontal direction (usually the x-axis) of the image. This represents the coordinates of the top-left corner of the bounding box in the vertical direction (usually the y-axis) of the image. It is the width of the bounding box, that is, the length of the bounding box in the horizontal direction; This represents the height of the bounding box, which is its length in the vertical direction. Target category. That is, the leg inspection category and the environmental inspection category.
[0029] In some embodiments, the object detection model is acquired and trained in advance before the system is deployed and run. See also... Figure 3First, a visual acquisition unit is used to collect video image data containing the user's lower limbs and the walking environment in real or simulated walking scenarios. The collected data is meticulously screened to remove blurry, duplicate, and other invalid data. Then, the screened data is labeled, covering leg-related targets, including the thigh, calf, and leg bounding boxes, as well as environmental targets such as ramps, step edges, and obstacles. Furthermore, to enhance data diversity and richness and improve the model's generalization ability, data augmentation processing, such as rotation, flipping, and brightness adjustment, is performed on the labeled data to form a training dataset. Next, based on the training dataset, the object detection network is trained offline or fine-tuned to obtain two independent detection models: a leg target detection model and an environmental target detection model. The leg target detection model focuses on outputting leg target detection results, accurately identifying the user's movement posture, such as leg swing and support states; the environmental target detection model mainly outputs environmental target detection results, effectively identifying surrounding context information, such as the presence of ramps, steps, and obstacles. Regarding model selection, mature object detection network architectures such as YOLO or DETR can be chosen based on actual needs and performance considerations. Furthermore, after the leg object detection model and the environment object detection model are trained and optimized, the parameters of the two detection models are deployed to the computing unit (development board) of the deep learning-based lower limb exoskeleton control system. During subsequent system operation, the same image is used as input and processed by both detection models. The leg object detection model outputs the user's leg object detection results (i.e., motion posture information), while the environment object detection model outputs the environment object detection results (i.e., surrounding scene recognition results). In this way, the leg posture and surrounding scene recognition results are fused, providing a comprehensive and accurate basis for subsequent intent recognition and lower limb exoskeleton control decisions.
[0030] It should be noted that the leg target detection model and the environment target detection model are trained independently, aiming to focus on the detection tasks of leg posture and surrounding scene respectively, in order to achieve more accurate recognition results.
[0031] S13, determine the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results.
[0032] After acquiring the target detection results of consecutive frames (including leg target detection results and environmental target detection results), the electronic device needs to perform time sequence consistency processing on the target detection results in order to obtain a stable recognition sequence that can be used for online control.
[0033] In an optional implementation, determining the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results includes: The leg target detection results are processed by leg inference, and motion state labels are output. The environmental target detection results are processed by environmental reasoning to output environmental context labels; The motion state label is fused with the environmental context label, and the control mode and the corresponding control parameters are determined based on the fusion result.
[0034] In some embodiments, for leg target detection results, the intersection-union ratio (IUR) is calculated for targets of the same category in two adjacent frames. As a match rate, the interaction ratio is defined as: ; Set a specific threshold condition (e.g., 45%), when the calculated value is... When the value exceeds a preset threshold, targets in two adjacent frames are considered as continuous observations of the same target. If multiple candidate detection results exist for the same type of target in a frame, the system will select the candidate target with the largest IoU and a detection confidence that meets a preset threshold as the continuing target for that target in the current frame.
[0035] To suppress potential jitter during target detection, a time-window smoothing process is applied to the target confidence score and the detection box position. Let the detection confidence score of a key target in the current frame be... The smoothness confidence of the previous frame is The smoothing calculation formula is: In order to achieve stability and response sensitivity in model inference, A value of 0.2 to 0.6 is acceptable.
[0036] When the target is lost in a short period of time (for example, no target is detected for several consecutive frames (i.e., the second consecutive preset frame)) or the detection confidence is lower than the preset threshold, the system does not immediately determine that the target has disappeared. Instead, it allows the system to continue to use the previous stable observation value to output the "target exists" status for a limited period of time. This effectively avoids the problem of frequent switching of control modes caused by the target being temporarily occluded.
[0037] After obtaining a stable recognition sequence through temporal consistency processing, the electronic device performs state recognition according to the following rules and outputs motion state labels: If, in multiple consecutive frames (i.e., the first consecutive preset frame), the leg target exhibits the detection feature of "moving upward and forward," meaning that the position of the leg target in the image changes positively in both the vertical and horizontal directions, then the system identifies it as a swing cycle and outputs a swing cycle motion state label.
[0038] If the position of the leg target remains basically stable in multiple consecutive frames (i.e., the first consecutive preset frame) and the position change is within a very small range, the system will identify it as a support cycle and output a support cycle motion state label.
[0039] If the leg target suddenly disappears (not detected after the limited holding time, i.e., no leg target is detected within the second consecutive preset frame) or the detection confidence drops rapidly (e.g., drops sharply from a high value to below the preset threshold in a short period of time), the system identifies it as a stopped or abnormal state and outputs a stopped or abnormal state motion status label.
[0040] At the same time, corresponding confidence thresholds are set for different environmental categories, such as ramps and steps. ,in This is based on pre-set confidence thresholds, such as those for setting ramp environment categories, after extensive experimentation and analysis. =0.7, the confidence interval threshold for the step environment category =0.65, etc. After obtaining the environmental target detection results, extract the detection confidence score for each environmental category (ramp, steps, etc.) within N consecutive frames (e.g., N=5). and the detection confidence level Compared with the preset confidence threshold Compare, when determined > If so, it is determined that the current actual environment has entered the specific environmental scenario corresponding to the ramp. For example, taking the ramp environment category as an example, the detection confidence of each frame within 5 consecutive frames is checked. Are all greater than the preset reliability threshold? =0.7. If this condition is met, it is determined that the current actual environment has entered the specific environmental scenario corresponding to the ramp. Similarly, for the step environment category, if its detection confidence score for 5 consecutive frames is greater than 0.7, then... If the value is 0.65, then the specific environmental scenario corresponding to the steps is determined. Once a specific environmental scenario corresponding to a certain environmental category is determined, the system immediately outputs the environmental scenario label corresponding to that specific environmental scenario. If it is determined that a ramp environmental scenario has been entered, the label "Ramp Environment" is output; if it is determined that a step environmental scenario has been entered, the label "Step Environment" is output.
[0041] After entering a specific environmental scenario, the system continuously monitors the environmental detection results. When the detection confidence level for that environmental category falls below the corresponding lower exit threshold, the system determines to exit the environmental scenario. To suppress jitter, a lower exit threshold can be set, such as 0.5 for a ramp environment and 0.45 for a step environment. After entering a ramp environment scenario, if the subsequent detection confidence level for the ramp environment falls below 0.5, the system determines to exit the ramp environment scenario; similarly, in a step environment scenario, if the detection confidence level for the step environment falls below 0.45, the system determines to exit the step environment scenario.
[0042] Once motion state labels (covering leg movement behavior labels such as swing cycle, support cycle, pause, or abnormal states) and environmental context labels (including environmental context-related labels such as flat ground, approaching step edge, stable environment, and approaching obstacle) are obtained through target detection and inference, the electronic device can fuse the motion state labels and environmental context labels to infer the user's movement intention and determine the control mode and corresponding control parameters of the lower limb exoskeleton. Specifically, when the motion state label is determined to be "swinging / walking behavior" and the environmental context label is "flat ground," the user is determined to be in a flat ground walking state, and the control mode is determined to be the flat ground assist mode. Based on the gait characteristics and biomechanical requirements of the human body when walking on flat ground, control parameters such as the amount and timing of assistance are set. The amount of assistance can be adjusted according to factors such as the user's weight and walking speed, while the timing of assistance is precisely controlled according to the leg movement phase. When the motion status label is determined to be "swinging / walking behavior" and the environmental context label is "approaching the edge of a step," it is determined that the user is about to perform an up / down step action. The control mode is set to step mode, and corresponding control parameters are configured according to the motion characteristics of going up / down steps. For example, when going up a step, greater assistance is provided to help the user lift their legs; when going down a step, appropriate damping is added to prevent the user from falling quickly and ensure safety. When the motion status label is determined to be "support / standing behavior" and the environmental context label indicates that the environment is stable, it is determined that the user is in a standing or stopped state. The control mode is set to low assistance or hold mode, and control parameters are set according to the standing / stopping state. For example, in low assistance mode, less assistance is provided to maintain the user's standing posture; in hold mode, the current state of the lower limb exoskeleton is maintained, and no additional assistance or damping changes are provided. When the environmental context label indicates that the environment has detected an obstacle that is close to the foot target in the image travel area, a collision risk is determined, the control mode is set to protection mode, and the corresponding control parameters are determined based on obstacle avoidance or risk suppression requirements. For example, the damping may be rapidly increased to slow down the foot movement speed, or the angle of the lower limb exoskeleton joint may be adjusted to allow the user to change the direction of travel to avoid the obstacle.
[0043] By directly identifying leg targets and environmental targets through a deep learning target detection model, and completing gait phase determination, motion intention recognition and environmental scenario classification based on the change characteristics of target detection results in the time dimension, the system avoids the dependence on joint angles, angular velocities or human kinematics models in traditional methods. This makes the system more robust to different heights, gait habits and wearing position errors, and improves the versatility and adaptability of the algorithm.
[0044] S14, generate lower limb exoskeleton control quantities according to the control mode and the control parameters, and output the lower limb exoskeleton control quantities to the lower limb exoskeleton actuator to realize the user's walking assistance.
[0045] Once the control mode and corresponding control parameters are acquired, the electronic device can generate deep learning-based control quantities for the lower limb exoskeleton, including target torque, target joint angular trajectory, target impedance / admittance parameters, or combinations thereof, and output the deep learning-based lower limb exoskeleton control quantities to the lower limb exoskeleton actuator to drive the hip, knee, and ankle joints to provide assistive output, thereby enabling the user to walk.
[0046] In one optional implementation, the electronic device implements a safety constraint and degradation strategy based on the reliability of deep learning recognition results throughout the operation of the lower limb exoskeleton. This strategy aims to suppress the amplification effect of misjudgments on the lower limb exoskeleton output when recognition instability is caused by occlusion, backlighting, motion blur, or complex backgrounds. For key targets (including at least foot targets and key environmental targets used for scene determination), smoothing confidence is continuously calculated, and exponential smoothing is used to suppress single-frame noise. When the confidence level falls below a preset safety threshold and continues to reach a set number of frames, the system determines that the current recognition reliability is insufficient and triggers degradation control, causing the assist output to gradually decrease according to a preset attenuation strategy rather than abruptly changing. Simultaneously, the system also monitors recognition stability based on the consistency of consecutive frames: when the same key target cannot maintain continuous observation through detection box matching between adjacent frames (e.g., the target frequently disappears / reappears), or when the behavior label derived from the recognition result frequently reverses within a short time window (e.g., high-frequency switching of walking / standing labels), the system considers this recognition jitter and increases the safety constraint level, limiting the mode switching frequency and forcing entry into a "conservative output" state to avoid control mode jumps caused by recognition jitter.
[0047] Compared to existing technologies, this application uses only visually acquired image data as external sensory input. Through deep learning, it simultaneously identifies leg movement posture and surrounding environment, thereby determining the required assistance mode and corresponding assistance torque, and generating a deep learning-based lower limb exoskeleton control output to achieve this.
[0048] Reference Figure 4The diagram shown is a functional block diagram of a deep learning-based lower limb exoskeleton control device according to an embodiment of this application.
[0049] In some embodiments, the deep learning-based lower limb exoskeleton control device 40 may include multiple functional modules composed of computer program segments. The computer programs for each program segment of the deep learning-based lower limb exoskeleton control device 40 may be stored in the memory of an electronic device and executed by at least one processor to perform (see details). Figure 1 (Description) Functions of a deep learning-based lower limb exoskeleton control system. Based on its functions, it can be divided into multiple functional modules. These modules may include: a data acquisition module 401, a target monitoring and inference module 402, a fusion decision module 403, and a control output module 404. The term "module" in this application refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.
[0050] The data acquisition module 401 is used to acquire image data of the lower limbs of the user wearing the lower limb exoskeleton and the environment in the direction of travel.
[0051] The target detection inference module 402 is used to perform target detection on each frame of the image data and output the leg target detection result and the environmental target detection result.
[0052] The fusion decision module 403 is used to determine the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results.
[0053] The control output module 404 is used to generate lower limb exoskeleton control quantities according to the control mode and the control parameters, and output the lower limb exoskeleton control quantities to the lower limb exoskeleton actuator to realize the user's walking assistance.
[0054] It should be understood that the various variations and specific embodiments of the deep learning-based lower limb exoskeleton control method provided in the above embodiments are also applicable to the deep learning-based lower limb exoskeleton control device of this embodiment. Through the foregoing detailed description of the deep learning-based lower limb exoskeleton control method, those skilled in the art can clearly understand the implementation method of the deep learning-based lower limb exoskeleton control device of this embodiment. For the sake of brevity, it will not be described in detail here.
[0055] See Figure 5 The diagram shown is a schematic representation of the structure of an electronic device according to an embodiment of this application. In a preferred embodiment of this application, the electronic device 5 includes a memory 51, at least one processor 52, and at least one communication bus 53.
[0056] Those skilled in the art should understand that Figure 5 The structure of the electronic device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The electronic device 5 may also include more or fewer other hardware or software than shown, or different component arrangements.
[0057] In some embodiments, the electronic device 5 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 5 may also include user equipment, which includes, but is not limited to, any electronic product capable of human-computer interaction with a user via a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smartphone, or digital camera.
[0058] In the embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple components or modules may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices, components, or modules may be electrical, mechanical, or other forms.
[0059] The components described as separate parts may or may not be physically separate. The components shown as components may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the components can be selected to achieve the purpose of this embodiment according to actual needs.
[0060] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each component can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0061] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, portable hard drive, read-only memory (ROM). Various media that can store program code, such as only memory, random access memory (RAM), magnetic disks or optical disks.
[0062] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.
[0063] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0064] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A deep learning-based lower limb exoskeleton control method, characterized in that, The method includes: Acquire image data of the lower limbs and the environment in the direction of travel of a user wearing a lower limb exoskeleton; Perform target detection on each frame of the image data and output the leg target detection result and the environment target detection result; The control mode and control parameters of the lower limb exoskeleton are determined based on the leg target detection results and the environmental target detection results. The lower limb exoskeleton control parameters are generated according to the control mode and the control parameters, and the lower limb exoskeleton control parameters are output to the lower limb exoskeleton actuator to realize the user's walking assistance.
2. The deep learning-based lower limb exoskeleton control method according to claim 1, characterized in that, The process of determining the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results includes: The leg target detection results are processed by leg inference, and motion state labels are output. The environmental target detection results are processed by environmental reasoning to output environmental context labels; The motion state label is fused with the environmental context label, and the control mode and the corresponding control parameters are determined based on the fusion result.
3. The deep learning-based lower limb exoskeleton control method according to claim 2, characterized in that, The leg target detection result is processed by leg inference, and the output motion state label includes: The leg target detection results are processed for temporal consistency. If the leg target in the leg target detection result after time-series consistency processing exhibits the detection characteristics of "moving up and moving forward" in the first consecutive preset frames, then the leg target is identified as a swing cycle and a swing cycle motion state label is output. If the leg target is stable in the first consecutive preset frames, the leg target is identified as a support cycle and a support cycle motion state label is output. If the leg target is not detected in the second consecutive preset frame, the leg target is identified as a stopped or abnormal state and a stopped or abnormal state motion status label is output.
4. The deep learning-based lower limb exoskeleton control method according to claim 3, characterized in that, The time-series consistency processing of the leg target detection results includes: The cross-union ratio of detection boxes of adjacent similar leg targets in the detection results is calculated by detection box matching; When the intersection-union ratio is greater than a preset threshold, the adjacent leg targets of the same type are determined to be continuous observations of the same target; When multiple candidates of the same type of target are detected in any leg target, the candidate with the largest intersection-union ratio with the leg target in the previous frame and whose detection confidence meets the preset threshold is selected as the continuation target of the leg target in the current frame. The confidence level and frame position of the continued target are smoothed using a time window. If no leg target is detected in the second consecutive preset frame or the detection confidence is lower than the preset threshold, the "leg target exists" status will continue to be output within the second consecutive preset frame using the previous stable observation value.
5. The deep learning-based lower limb exoskeleton control method according to claim 2, characterized in that, The environmental inference processing performed on the environmental target detection results, and the output environmental context labels, include: Obtain the detection confidence score for each environmental category within N consecutive frames of the environmental target detection results; When a certain environment category is detected within N consecutive frames and the detection confidence is greater than a preset confidence threshold, it is determined that the current actual environment has entered the specific environmental scenario corresponding to the certain environment category. Output the environmental scenario label corresponding to the specific environmental scenario.
6. The deep learning-based lower limb exoskeleton control method according to claim 1, characterized in that, The step of performing target detection inference on each frame of the image data and outputting leg target detection results and environment target detection results includes: The image data is input into a preset leg target detection model so that the leg target detection model outputs the leg target detection result; The image data is input into a preset environmental target detection model so that the environmental target detection model outputs the environmental target detection result.
7. The deep learning-based lower limb exoskeleton control method according to claim 1, characterized in that, Before performing target detection on each frame of the image data, the method further includes: The image data is preprocessed, including any one or a combination of distortion correction, scaling to the model input size, brightness normalization, and denoising.
8. A deep learning-based lower limb exoskeleton control device, characterized in that, The device includes: The data acquisition module is used to acquire image data of the lower limbs of the user wearing the lower limb exoskeleton and the environment in the direction of travel; The target detection and inference module is used to perform target detection on each frame of the image data and output the leg target detection result and the environmental target detection result. The fusion decision module is used to determine the control mode and control parameters of the lower limb exoskeleton based on the leg target detection results and the environmental target detection results; The control output module is used to generate lower limb exoskeleton control quantities according to the control mode and the control parameters, and output the lower limb exoskeleton control quantities to the lower limb exoskeleton actuator to realize the user's walking assistance.
9. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the deep learning-based lower limb exoskeleton control method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the deep learning-based lower limb exoskeleton control method as described in any one of claims 1 to 7.