User-Active Vision-Based Adaptive Assistive Exoskeleton System and Its Control Method
By integrating wearable visual sensors and microphones into the hip exoskeleton system, environmental and user commands are acquired. Combined with joint encoder signals, predictive assistive control is achieved, solving the problem that existing exoskeleton systems cannot predict terrain changes and understand user intentions, thus improving safety and convenience.
Patent Information
- Application Number
- CN202611601563.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing hip exoskeleton systems lack the ability to anticipate environmental changes and understand users' natural language, resulting in assistive strategies that cannot predict terrain changes, leading to response delays and safety hazards. Furthermore, the human-computer interaction is inconvenient and fails to understand users' ambiguous intentions.
By introducing wearable visual sensors and microphones into the exoskeleton system, structured environmental information and user natural language commands are acquired. Combined with the joint encoder signals of the hip exoskeleton robot, the gait phase is calculated in real time, predictive assist control decisions are generated, and the joint motors are driven to output target torque.
This system enables predictive environmental assistance from the exoskeleton, improving walking safety and smoothness, simplifying human-computer interaction, enhancing personalized adaptability, and reducing system complexity and cost.
Smart Images

Figure CN123103603A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent wearable robot technology, specifically to an adaptive assistive exoskeleton system based on user active vision and its control method, particularly a hip adaptive assistive exoskeleton system based on user active vision and its control method. Background Technology
[0002] Hip-assisted exoskeletons are wearable human-machine collaborative devices that use motors to provide additional torque to assist the human hip joint in flexion and extension. They aim to enhance the wearer's walking, running, weight-bearing, and climbing abilities, showing broad application prospects in industrial logistics, military load-bearing, and lower limb rehabilitation. Their core control logic relies on built-in encoders, inertial measurement units (IMUs), or torque sensors to sense the wearer's gait phase and movement intention in real time, and then drive the motors to output a preset assistance curve. This "human-in-the-loop" control mode is the technological foundation of current mainstream exoskeleton products.
[0003] However, existing hip exoskeleton control methods generally suffer from a fundamental limitation: the system's perception of the environment and user intent is entirely passive and delayed. Because decisions are made solely based on proprioceptive signals generated by the wearer's own movements, the system cannot anticipate upcoming terrain changes, such as suddenly transitioning from flat ground to stairs or a slope. This lack of "foresight" means that the assist strategy can only be switched after the wearer has already interacted with the terrain and their gait has begun to change. This mechanical lag can cause a momentary mismatch between the assist and the wearer's actual needs in complex terrain, not only disrupting the natural flow of walking but also creating additional safety hazards due to untimely output of assist torque when the wearer's center of gravity shifts abruptly.
[0004] Furthermore, the existing human-computer interaction methods of exoskeletons severely limit their adaptability and convenience. Currently, the commonly used physical buttons, knobs, or mobile applications require the wearer to operate them with their hands during movement. This not only disrupts the continuity of work but is also virtually unusable in scenarios where both hands are occupied, such as carrying heavy objects or holding tools. More importantly, this discrete and patterned interaction method cannot understand the complex and ambiguous intentions in human language, such as "walk slower," "go up the step with more force," or "it feels a bit heavy." The system lacks an intelligent channel to translate these qualitative and ambiguous user sensations into quantitative control parameter adjustments, causing the exoskeleton to always respond to ever-changing usage needs and user states with a pre-set, single logic, resulting in a severe lack of personalization and adaptability.
[0005] In summary, traditional exoskeletons, limited by a single proprioception architecture and rigid human-computer interaction interfaces, have formed a "passive response" technological paradigm, making it difficult to achieve truly intuitive and predictive human-computer collaboration. Therefore, there is an urgent need for a new technological approach that endows exoskeletons with forward-looking environmental perception capabilities and the ability to understand user natural language, deeply integrating this prior information with real-time proprioception status. This would drive control strategies to shift from "passive response" to "active prediction," fundamentally improving the intelligence, safety, and smoothness of human-computer collaboration. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide an adaptive assistive exoskeleton system based on user active vision and its control method.
[0007] A control method for an adaptive assistive exoskeleton system based on user active vision, provided by the present invention, includes the following steps: Acquire structured environmental information representing changes in the terrain ahead, collected by wearable vision sensors; Acquire user natural language commands collected by the microphone that qualitatively describe the assist effect, and parse them into quantitative correction values for control parameters; The phase of the current gait is calculated in real time based on the angle signals output by the encoders built into the joints on both sides of the exoskeleton robot. By integrating the structured environmental information, the quantitative correction value, and the phase of the current gait, a control decision command is generated; wherein, when the structured environmental information indicates a change in the terrain ahead, the control decision command is configured to adjust the assist strategy in advance before the exoskeleton robot reaches the change in terrain. Based on the control decision commands and real-time gait phase, the joint motors of the exoskeleton robot are driven to output target torque to assist the user's movement.
[0008] Preferably, obtaining the structured environment information specifically involves: processing the collected images of the foreground environment using a visual deep learning model deployed on a smart glasses terminal to output the structured environment information containing terrain category semantic labels and their spatial geometric attributes.
[0009] Preferably, the visual deep learning model is a large visual model; The processing of the foreground environment image specifically involves semantic segmentation and depth estimation of the foreground environment image.
[0010] Preferably, the phase of the current gait is calculated in real time by: calculating the angle difference signal of the encoder built into the bilateral joints of the exoskeleton robot, and inputting the angle difference signal into the adaptive oscillator for calculation.
[0011] Preferably, the advance adjustment of the assist strategy is as follows: when it is detected that there is an upward staircase or slope in front, the assist torque amplitude in the direction of hip flexion is increased in advance at the end of the support phase before the user's lower limbs contact the terrain.
[0012] Preferably, parsing the user's natural language instructions specifically involves: performing semantic parsing through a natural language processing model, and mapping instructions containing qualitative descriptions of the assistance effect to quantitative correction values for at least one control parameter in the assistance algorithm.
[0013] Preferably, the target torque output of the drive joint motor is specifically achieved by: coordinating and optimizing the generation of the target torque based on the physical characteristic parameters of the joint motor.
[0014] The present invention also provides an adaptive assistive exoskeleton system based on user active vision, comprising: Wearable visual sensors and microphones; Hip exoskeleton actuator, which includes bilateral joint motors with built-in encoders; A controller configured to execute the control method described above for an adaptive assistive exoskeleton system based on user active vision.
[0015] Preferably, the wearable visual sensor and microphone are integrated into a smart glasses terminal, which has a built-in edge computing unit for running visual deep learning models and natural language processing models.
[0016] Preferably, the bilateral joint motors of the hip exoskeleton actuator are high torque density frameless torque motors.
[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention introduces wearable visual sensors as the front end of environmental perception, giving the exoskeleton system the ability to "know in advance". Based on structured environmental information, the predictive assistance can actively adjust the torque output before the user comes into contact with changes in terrain, fundamentally eliminating the assistance response delay of traditional solutions in complex terrain, and significantly improving the safety and smoothness of walking.
[0018] 2. This invention utilizes natural language processing technology to transform users' qualitative and ambiguous voice commands into quantitative correction values, achieving intuitive and convenient "hands-free" human-computer interaction. This not only lowers the barrier to entry for the system but also enables the exoskeleton to adapt to users' personalized needs and subjective feelings in real time, greatly improving the tacit understanding of human-computer collaboration.
[0019] 3. This invention uses an adaptive oscillator to process the angle difference signal of the bilateral joint encoder to calculate the phase of the real-time gait. This solution does not require additional inertial measurement units or plantar pressure sensors, which are easily affected by wearable devices. This simplifies the system hardware architecture, reduces costs, and enhances the robustness and convenience of the system in practical applications.
[0020] 4. The core of this invention lies in enabling users to actively perceive environmental conditions, including terrain features, by wearing wearable visual sensors (smart glasses). By utilizing the user's active participation intention and visual analysis features, the exoskeleton's assistance method is adaptively adjusted to achieve active and efficient human-machine collaboration. Attached Figure Description
[0021] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the adaptive assistive exoskeleton system based on user active vision according to the present invention; Figure 2 This is a schematic diagram of the terrain recognition process in the control method of the adaptive assistive exoskeleton system based on user active vision provided by the present invention. Figure 3 This is a schematic diagram of the assist control process in the control method of the adaptive assist exoskeleton system based on user active vision provided by the present invention. Figure 4 This is a schematic diagram of the wearable adaptive assistive exoskeleton system based on user active vision according to the present invention.
[0022] The diagram shows: 10 - Smart glasses terminal; 20 - Hip exoskeleton actuator. Detailed Implementation
[0023] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0024] Example 1 like Figure 1 and Figure 4 As shown, this embodiment provides an adaptive assistive exoskeleton system based on user active vision. Figure 1 As shown, the system mainly includes a smart glasses terminal 10 worn on the user's head and a hip exoskeleton actuator 20 worn on the user's waist and thighs. Figure 1As shown, from a hardware structure perspective, the system consists of three core components: a smart glasses terminal 10, a hip exoskeleton actuator 20, and a controller that serves as the system's computational core. These three components are connected via wired or wireless high-speed communication links, forming a complete human-machine interaction and control closed loop. This design decouples environmental perception, user intent understanding, and physical assistive execution functions, and achieves efficient coordination through the controller, providing a hardware foundation for realizing predictive adaptive assistive control.
[0025] In this implementation, the smart glasses terminal 10 serves as the system's prior perception entry point, its core function being to acquire environmental information in front of the user and the user's natural language commands. The smart glasses terminal 10 integrates a wearable visual sensor and a microphone. The wearable visual sensor is preferably an RGB-D camera, capable of simultaneously capturing color images and depth information of the scene in front. The microphone is used to pick up the user's voice. Furthermore, to achieve efficient local processing, the smart glasses terminal 10 incorporates an edge computing unit. This unit is specifically designed for running complex artificial intelligence algorithms on the device side, and it deploys a visual deep learning model and a natural language processing model. By completing most of the computation on the terminal side, the reliance on cloud communication and the computational burden on the controller are greatly reduced, ensuring the real-time response of the system, which is crucial for assisted control requiring instantaneous reactions.
[0026] The hip exoskeleton actuator 20 is the physical carrier that directly applies auxiliary torque to the user. Its main structure includes a load-bearing structure surrounding the user's waist and binding structures fixed to both thighs. At the rotation axis of each hip joint, a dual-joint motor is installed. In this embodiment, to achieve high dynamic and high torque assist output, the joint motor is a high torque density frameless torque motor. This motor undergoes specialized electromagnetic and structural optimization, enabling it to provide powerful torque output within a compact volume, and features low cogging torque and a high torque response bandwidth. Each joint motor integrates an encoder for accurately measuring the joint's rotation angle. This design allows the system to acquire all the proprioceptive kinematic information required for gait research using only the built-in encoder, eliminating the need for additional sensors, thus simplifying the system hardware and improving wearability.
[0027] The controller is the brain of the entire exoskeleton robot system, responsible for executing the adaptive assist control method proposed in this invention. The controller is connected to both the smart glasses terminal 10 and the hip exoskeleton actuator 20 via communication interfaces. Its internal processor is configured to run a series of algorithm modules, including but not limited to information fusion, gait phase calculation, decision generation, and torque control. The controller receives structured environmental information and quantitative correction values from the smart glasses terminal 10, and simultaneously receives real-time angle signals from the encoder of the hip exoskeleton actuator 20. After a series of complex calculations and decisions, it ultimately sends precise drive commands to the joint motors to achieve an assist effect that adapts to the user's movement intentions and the external environment.
[0028] The control method for an adaptive assistive exoskeleton system based on user active vision provided in this embodiment is detailed as follows: Figure 2 and Figure 3 As shown in the figure. The core of this method lies in fusing multimodal information to achieve a shift in control paradigm from passive response to active prediction. The entire process can be summarized into four main stages: multimodal information acquisition, gait phase recognition, decision fusion, and assist torque calculation.
[0029] During the multimodal information acquisition phase, the system collects information in parallel from two different sources. On one hand, the system acquires structured environmental information representing changes in the terrain ahead, collected by wearable visual sensors. Specifically, a visual deep learning model deployed on the smart glasses terminal 10 processes the images of the foreground environment collected by the wearable visual sensors in real time. This model is a visual deep learning model trained on a large amount of terrain data, capable of pixel-level semantic segmentation and accurate depth estimation of the images. For example, when a staircase appears 3.2 meters in front of the user, the structured environmental information output by the model might be a data packet containing the following information: {Terrain type: Upward staircase, Distance: 3.2 meters, Estimated step height: 15 centimeters, Number of steps: 12}. This structured data provides accurate quantitative input for subsequent predictive decision-making.
[0030] On the other hand, during the multimodal information acquisition phase, the system also acquires user natural language commands collected by the microphone that qualitatively describe the assistance effect, and parses them into quantitative correction values. When the user issues commands such as "use more force when going up the stairs" or "feels a bit heavy," the natural language processing model built into the smart glasses terminal 10 performs semantic analysis. This model can identify the key intent (such as "increase force") and the application scenario (such as "going up the stairs") in the command, and map them into specific adjustments to one or more control parameters in the underlying assistance algorithm. For example, "use more force" is parsed as a quantitative correction value of "increase the peak torque gain coefficient of the assistance by 15%." Thus, the system can understand and execute the user's fuzzy, qualitative commands, achieving truly intuitive human-computer interaction.
[0031] In the gait phase recognition stage, the core task of the system is to calculate the phase of the current gait in real time. This embodiment adopts an innovative scheme that relies solely on joint encoders. First, the controller reads the angle signals output by the encoders built into the joints of both sides of the exoskeleton robot in real time. Then, the difference between these two angle signals is calculated to obtain an angle difference signal that reflects the alternating swing state of the legs. This signal exhibits quasi-periodicity during walking. In order to extract stable phase information from this potentially noisy signal, this embodiment inputs the angle difference signal into an adaptive oscillator for calculation. The adaptive oscillator is a dynamic system that can track the frequency and phase of the input signal in real time through a phase-locked loop mechanism. Even when the user's walking speed changes (i.e., the gait frequency changes), the oscillator can quickly converge and remain synchronized with it. The output of the adaptive oscillator is a normalized variable that changes continuously and monotonically in the interval [0, 2π], namely the phase of the gait. This phase accurately marks the user's position in the current gait cycle; for example, a phase of 0 can be defined as the heel striking one side, and a phase of π can be defined as the heel striking the opposite side. In this way, the system obtains the time reference necessary for precise timing control, and its robustness and ease of use are greatly enhanced because it does not rely on an additional IMU or plantar pressure sensor.
[0032] During the decision fusion phase, the controller integrates all the aforementioned information to generate a comprehensive control decision command. A "state discriminator" module within the controller is the core of this phase. Within each control cycle, this module simultaneously receives and evaluates three dimensions of input: the phase of the real-time gait output from the adaptive oscillator, structured environmental information provided by the smart glasses terminal 10, and quantitative correction values parsed from user commands, also from the smart glasses terminal 10. Based on a set of preset rules and state machine logic, the state discriminator determines the assistance mode that the user is most likely to need in the current and near future. The key to this fusion decision process is that when structured environmental information indicates a change in terrain ahead, the control decision command is configured to adjust the assistance strategy in advance before the exoskeleton robot reaches that change. For example, when environmental information indicates an upward staircase 0.8 meters ahead, the state discriminator predicts that the user will begin lifting their leg to climb the staircase in the next gait cycle. At this point, it generates a special control decision command that not only includes the basic assistance mode but also adds a predictive assistance trigger flag and related parameters. At the same time, if there is a quantitative correction value input by the user (such as "increase the peak torque gain coefficient by 15%), the correction value will also be integrated into the instruction to adjust the target torque to be generated.
[0033] In the assist torque calculation stage, the system drives the joint motors of the exoskeleton robot to output a target torque to assist the user's movement, based on the control decision command and the phase of the real-time gait. The "adaptive assist algorithm" module within the controller receives the control decision command from the decision fusion stage. This algorithm calculates the target torque value to be applied to the hip joint at the current moment by looking up a table or performing function calculations, based on the assist mode defined in the command (e.g., flat ground mode, stair climbing mode), torque parameters, and the phase of the real-time gait. To make the output torque more accurate and efficient, this step further includes: co-optimizing the generation of the target torque based on the physical characteristic parameters of the joint motors. This means that the parameters of the assist algorithm are not set in isolation, but are jointly calibrated and optimized with the physical characteristic parameters of the high torque density frameless torque motor, such as torque density, torque response bandwidth, and cogging torque characteristics. For example, the algorithm ensures that the dynamic range and rate of change of the generated target torque curve are within the performance envelope of the motor, thereby giving full play to the high responsiveness of the motor and achieving a steep yet smooth torque output. This avoids control delays and distortions caused by hardware performance limitations and ensures that the torque ultimately applied to the human body is highly consistent with the expected value.
[0034] To more clearly illustrate the implementation process of predictive assistance in this embodiment, a typical scenario of a user walking from flat ground to an upward staircase is used as an example. In this scenario, when the user is walking on flat ground, the system executes a conventional flat-ground assistance strategy, mainly providing a small flexion assistance at the beginning of the swing phase of the gait cycle to help the user lift their leg. As the user gradually approaches the staircase, the vision system of the smart glasses terminal 10 recognizes the terrain ahead and continuously outputs structured environmental information containing distance information. The controller monitors that this distance value gradually decreases as the user moves forward. When the controller determines that the distance between the user and the staircase is less than a preset trigger threshold (e.g., 0.8 meters, which usually means that the user will step onto the next step), the decision fusion module immediately triggers the predictive assistance strategy. The core of this strategy is to adjust the assistance strategy in advance, that is, to increase the amplitude of the auxiliary torque in the hip flexion direction in advance at the end of the support phase before the user's lower limbs contact the terrain (staircase). This pre-applied, larger-amplitude flexion assistance can actively help the user overcome gravity and lift their leg to the height required for the first step with less metabolic energy consumption. This "predictive" assistance method effectively avoids the "tripping sensation" or insufficient assistance caused by the delayed response when encountering steps in traditional exoskeletons, greatly improving the user's walking smoothness and safety when changing terrain.
[0035] Example 2 Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.
[0036] This embodiment provides a control method for an adaptive assistive exoskeleton system based on user active vision, including: Step S1: Receive prior perception information from the smart glasses terminal, the prior perception information including visual recognition results of the environmental terrain and assistance adjustment intention instructions parsed from the user's continuous voice stream; Step S2: Synchronously collect the real-time status information of the exoskeleton actuator, which includes at least the hip joint angle, angular velocity, and human-machine interaction torque; Step S3: Perform time-series alignment and multimodal fusion of the prior perception information and the real-time status information of the body to generate control decision instructions that characterize the wearer's comprehensive assistance needs at the current moment; Step S4: Based on the control decision command, the target torque is calculated and output in real time using an adaptive assist algorithm, and the exoskeleton joint motor is driven to actively execute the target torque during the gait cycle.
[0037] Furthermore, the visual recognition result of the environmental terrain is obtained by performing semantic segmentation and depth estimation processing on the acquired terrain image through a large visual model deployed on the smart glasses terminal, and includes at least terrain category semantic labels and their spatial geometric attributes.
[0038] Furthermore, the multimodal fusion of the prior perceived information and the real-time state information of the ontology includes: Based on the semantic tags and spatial geometric attributes of the terrain categories; Predicting upcoming topographic change events; The predictive control decision instructions are generated in advance of one or more gait cycles.
[0039] Furthermore, the predictive control decision instruction includes, when recognizing an upward staircase or slope ahead, increasing the amplitude of the auxiliary torque in the hip flexion direction at the end of the support phase before the lower limbs of the human body contact the terrain.
[0040] Furthermore, the assist adjustment intention command is semantically parsed by a natural language processing model deployed on the smart glasses terminal, which analyzes the natural language command issued by the user that contains a qualitative description of the assist intensity or response speed, and maps it into a quantitative correction value for at least one control parameter in the adaptive assist algorithm.
[0041] Furthermore, the adaptive assist algorithm is co-optimized based on the physical characteristic parameters of the exoskeleton joint motor. The physical characteristic parameters include at least the torque density, torque response bandwidth, and cogging torque characteristics of the self-developed motor, so that the output of the target torque matches the performance envelope of the motor.
[0042] This embodiment also provides an adaptive assistive exoskeleton system based on user active vision to implement the above method. The system includes: The smart glasses terminal is equipped with a visual sensor and a voice sensor, and has a built-in edge computing unit for generating and outputting the prior perception information; The hip exoskeleton actuator has a built-in self-developed joint motor and body sensor, which is used to output auxiliary torque and collect the real-time status information of the body. The main controller is communicatively connected to both the smart glasses terminal and the exoskeleton actuator. It integrates a multimodal fusion module and an adaptive assist control module to generate the control decision commands and calculate the target torque.
[0043] Furthermore, the smart glasses terminal further includes: The visual perception module, consisting of an RGB-D camera and a first edge inference chip, is used to run a large visual model and output terrain category semantic labels and spatial geometric attributes in real time. The voice interaction module, consisting of a microphone and a second edge inference chip, is used to run a natural language processing model and output the assist adjustment intention command.
[0044] Furthermore, the hip exoskeleton actuator includes: The joint motor module is a high torque density frameless torque motor with customized physical characteristic parameters, which is self-developed and optimized through integrated structural design and drive algorithm. The sensing module includes a double-sided encoder coaxially integrated with the joint motor for high-bandwidth acquisition of the body's real-time status information.
[0045] This embodiment provides an adaptive assistive exoskeleton system and its control method based on user active vision. The system mainly includes a smart glasses terminal and a hip-assisted exoskeleton actuator. The smart glasses terminal integrates vision and voice perception modules for real-time acquisition of environmental terrain information and user voice commands. The hip-assisted exoskeleton actuator integrates a self-developed high-power-density motor and a bilateral encoder. The system uses a multimodal fusion decision algorithm to integrate environmental semantics, user intent, and propriodynamics information to dynamically generate adaptive assist strategies, achieving a leap from passive response to active prediction. This embodiment can be widely applied to scenarios such as walking assistance, weight-bearing transport, and rehabilitation training, significantly improving walking safety and human-machine collaboration in complex terrain.
[0046] Example 3 Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.
[0047] This embodiment provides an adaptive assisted exoskeleton system and its control method based on user active vision, aiming to solve the problem that existing exoskeleton control technologies rely solely on body sensors for passive response, which cannot predict terrain changes or understand the user's natural language intent, resulting in insufficient human-machine collaboration and safety in complex environments.
[0048] This embodiment introduces a smart glasses terminal as a priori perception entry point, and performs multimodal fusion of the environmental terrain visual recognition results, the user voice command parsing results, and the real-time gait information collected by the exoskeleton encoder to construct an adaptive assist control framework that moves from "passive response" to "active prediction".
[0049] The control method for the adaptive assistive exoskeleton system based on user active vision provided in this embodiment has the following core process: First, the smart glasses terminal acquires images of the surrounding environment and picks up user voice commands. Simultaneously, the dual-sided encoder built into the exoskeleton's hip joint motors acquires the angle difference signal between the two leg motors in real time. After filtering out high-frequency noise, this angle difference signal is input to an adaptive oscillator for gait phase recognition. Based on the periodic changes in the leg angle difference, the adaptive oscillator extracts and outputs the precise phase angle of the current gait in real time, providing a precise time reference for the subsequent timing control of the assist torque.
[0050] Secondly, the large-scale visual model deployed inside the smart glasses terminal performs semantic segmentation and depth estimation on the terrain image in front, outputting terrain category semantic labels and their spatial geometric attributes. Simultaneously, the deployed natural language processing model performs semantic understanding on the continuous speech stream emitted by the user, extracting qualitative descriptive commands from the user regarding assistance intensity or response speed, and mapping them to quantitative correction values for control parameters.
[0051] Next, the terrain recognition results, speech parsing results, and gait phase information output by the adaptive oscillator are time-aligned and multimodal fused. The state discriminator then generates control decision instructions to characterize the wearer's current comprehensive assistance needs. The core of these decision instructions is to determine which assistance strategy should be matched to the user's gait phase under the current terrain conditions, and to dynamically adjust the assistance parameters based on the user's personalized instructions.
[0052] Finally, based on control decision commands, the adaptive assist algorithm calculates and outputs the target torque in real time, driving the exoskeleton joint motors to execute the target torque within the precise phase interval of the gait cycle. When the state discriminator detects terrain changes such as ascending stairs or slopes ahead, the algorithm can pre-increase the amplitude of the assist torque in the hip flexion direction at the end of the support phase before the user's lower limbs contact the terrain, i.e., within the corresponding phase interval output by the adaptive oscillator, one or more gait cycles in advance, thus achieving predictive assist. Throughout the process, the status of both leg motors is fed back to the state discriminator in real time, forming a closed-loop control to ensure continuous and stable assist output.
[0053] This embodiment also provides an adaptive assistive exoskeleton system based on user active vision, which adopts the above-described control method and mainly includes the following modules: The smart glasses terminal is equipped with visual and voice sensors and a built-in edge computing unit. The visual perception module consists of an RGB-D camera and a first edge inference chip, which is used to run a large visual model and output terrain category semantic labels and spatial geometric attributes in real time. The voice interaction module consists of a bone conduction microphone and a second edge inference chip, which is used to run a natural language processing model, parse the user's voice, and output instructions to help adjust intentions.
[0054] Hip exoskeleton actuator: Each leg's hip joint integrates a self-developed high-torque-density frameless torque motor. This motor has a built-in dual-sided encoder for high-precision acquisition of position information at both ends of the motor, thereby calculating the angle difference between the two leg motors, which serves as the sole proprioceptive sensing source for gait perception. A torque sensor is also integrated inside the joint to provide feedback on the actual output torque, forming a closed-loop feedback circuit together with the motor status.
[0055] Embedded main controller: Communicates with both the smart glasses terminal and the exoskeleton actuator, and integrates a filter module, an adaptive oscillator module, a terrain recognition module, a state discriminator module, and an adaptive assist control module. The filter module denoises the original angle difference signal from the encoder; the adaptive oscillator module extracts the real-time gait phase from the denoised angle difference signal; the terrain recognition module receives and caches terrain semantic information from the smart glasses; the state discriminator performs multimodal fusion of gait phase, terrain information, and user commands to generate control decision commands; and the adaptive assist control module, based on these decision commands and combined with motor physical characteristic parameters, calculates and outputs the target torque in real time to drive the joint motors.
[0056] This embodiment introduces smart glasses as the front-end entry point for environmental perception and voice interaction, enabling the exoskeleton system to possess "predictive" capabilities for the first time. Predictive assistance based on terrain semantic information can proactively adjust torque output before the wearer encounters changes in terrain, eliminating the assistance response delay of traditional solutions in complex terrain and fundamentally improving the safety and smoothness of walking.
[0057] This embodiment utilizes an adaptive oscillator to perform gait phase recognition on the angle difference signal of the dual-leg encoder. It can accurately obtain the gait time reference without the need for additional IMU or plantar pressure sensors, which simplifies the system hardware architecture, reduces the sensitivity to sensor installation location and wear consistency, and enhances the robustness and convenience of the system in actual deployment.
[0058] This embodiment uses large-scale model-driven natural language understanding as a human-computer interaction channel, enabling wearers to adjust the exoskeleton's assistive performance in real time using the most natural way of speaking in daily life. This truly achieves hands-free operation and intuitive interaction, significantly reducing the barrier to entry and enhancing the personalized experience.
[0059] This embodiment achieves a complete independent technology closed loop from the bottom actuator to the top decision algorithm through the software and hardware co-optimization of the self-developed motor and adaptive assist algorithm. This ensures high precision, low hysteresis and high efficiency of torque output, providing an indispensable hardware foundation for the above-mentioned intelligent functions and building a solid technical barrier.
[0060] Example 4 Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.
[0061] This embodiment provides an adaptive assistive exoskeleton system based on user active vision, comprising three main parts: a smart glasses terminal, a hip exoskeleton actuator, and an embedded main controller. The three parts interact in real time through a high-speed wired or wireless communication link.
[0062] As the system's prior perception entry point, the core mission of smart glasses is to acquire environmental information and user intent before the wearer interacts with the terrain. At the hardware level, the terminal integrates an RGB-D depth camera and a microphone array, and internally carries a neural network inference chip designed specifically for edge computing. During operation, the RGB-D camera continuously captures color and depth images of the terrain in front of the wearer at a fixed frame rate. These image streams are fed in real-time into a large visual model deployed on the edge inference chip. This large visual model is trained on datasets tailored to specific terrain scenarios (including flat ground, ascending stairs, descending stairs, uphill slopes, downhill slopes, thresholds, and gravel surfaces), and possesses both semantic segmentation and depth estimation capabilities.
[0063] The output of the visual big data model is not a simple image annotation, but a structured "terrain semantic label and spatial geometric attribute". For example, when an upward staircase appears about three meters in front of the wearer, the model outputs a structured data packet containing fields such as "terrain type: upward staircase", "distance: 3.2 meters", "estimated step height: 15 centimeters", and "number of steps: 12". Simultaneously, the microphone continuously picks up the wearer's voice signal. When the wearer utters natural language commands such as "go up the stairs with more force" or "switch to power-saving mode and walk slower", the natural language processing model deployed on the glasses performs endpoint detection, speech recognition, and semantic slot filling on the continuous speech stream, ultimately parsing the vague, qualitative user intent into explicit machine-executable instructions. For example, "go up a little more force" is mapped to "increase the peak torque gain coefficient by 15%", and "walk slower" is mapped to "reduce the target step frequency reference value by 10%". This mechanism of converting natural language into quantitative control parameter correction values is a key technical aspect of this invention for achieving intuitive interaction.
[0064] The hip exoskeleton actuator is the physical carrier through which the system directly acts on the human body. Its main structure includes a load-bearing strap around the waist and a flexible strap around the thighs. Each hip joint integrates a self-developed high-torque-density frameless torque motor, which, together with a reducer, forms the drive joint. Unlike traditional solutions that rely on multiple sensors (such as IMUs, foot pressure insoles, etc.), this embodiment relies solely on the bilateral encoders integrated within each motor for gait perception. Specifically, each motor has a high-precision angle encoder installed on both the stator and rotor ends. The system calculates the real-time angular position of the two ends of the motor (i.e., the human thigh relative to the waist) by reading the angle values from the two encoders in real time. The difference between the motor angle values of the left and right hip joints yields the core signal: the difference in motor angle between the two legs. This signal directly reflects the kinematic characteristics of the alternating leg swings during walking and is a periodic waveform containing rich gait information.
[0065] The embedded main controller is the brain of the system, integrating a complete signal processing and control algorithm pipeline. Its workflow unfolds sequentially according to the signal flow, forming a complete closed loop from raw sensor data to the final output of assist torque.
[0066] First, the filter module receives the raw angle data from the dual-motor encoders. After obtaining the angle difference signal between the two leg motors through differential calculation, a bandpass filter or Kalman filter is used to filter out high-frequency interference introduced by factors such as wearer body shaking, motor cogging effect and sensor quantization noise, and extract a smooth and stable gait cycle waveform.
[0067] Subsequently, the denoised angle difference signal is fed into the adaptive oscillator module. An adaptive oscillator is a dynamic system capable of synchronously extracting the frequency, amplitude, and phase of a periodic input signal, particularly suitable for processing quasi-periodic, time-varying biosignals such as human gait. This module internally maintains a virtual oscillator vibrating at the current gait frequency, continuously synchronizing the oscillator's phase with the input angle difference signal via a phase-locked loop (PLL) mechanism. When the wearer accelerates or decelerates, the oscillator quickly converges to the new gait frequency, consistently outputting a monotonically increasing current gait phase between 0 and 2π. This phase angle precisely marks the wearer's current position within the gait cycle (e.g., a phase of 0 corresponds to the heel striking one foot, and a phase of π corresponds to the heel striking the other foot). This precise time reference provides crucial information for the subsequent phase-locked output of the assist torque.
[0068] Meanwhile, the terrain recognition module continuously receives and caches terrain semantic tag data packets from the smart glasses terminal. This module maintains a terrain status register, always retaining the most recent valid terrain recognition result and its timestamp, for subsequent decision-making modules to query at any time.
[0069] Once the information from the aforementioned modules is ready, the state discriminator, acting as the core hub for multimodal information fusion, begins its operation. Within each control cycle, the state discriminator simultaneously reads input from three dimensions: the current gait phase from the adaptive oscillator, the terrain semantic label and spatial geometric attributes from the terrain recognition module, and the voice analysis control parameter correction value from the smart glasses. The state discriminator internally employs a set of rule-based and state machine-based decision-making logic. Its primary task is to determine whether a predictive assistance strategy needs to be triggered. For example, when the terrain label in the register is "uphill stairs," and the "distance" parameter in its spatial attributes gradually decreases over time to a preset threshold (e.g., 0.8 meters, meaning the wearer is about to take their first step onto the stairs), the state discriminator marks a "preloaded" phase window within the current gait cycle. This window is typically set at the end of the support phase of the leg about to lift, i.e., when the phase angle output by the adaptive oscillator is within the push-off range of that leg. When the gait phase enters this window, the state discriminator generates a control decision command, instructing the assist algorithm to additionally superimpose a buckling assist bias torque of a specified amplitude within this phase window. Simultaneously, if the speech analysis module has previously output a correction value of "increasing the peak assist torque gain coefficient by 15%", this correction value will also be incorporated into this control decision command, making the calculated target torque more aligned with the user's personalized expectations.
[0070] Subsequently, the control decision command is transmitted to the adaptive assist control module. The core of this module is a self-developed adaptive assist algorithm. The algorithm receives parameters specified in the control decision command, such as the assist mode, torque amplitude bias, and phase window start and end points. Combined with the real-time gait phase output from the adaptive oscillator, it calculates the target torque value that should be output at the current moment. This torque value is constrained and optimized by the motor's physical characteristic model (e.g., based on the self-developed motor's torque density curve, torque response bandwidth, and current-torque mapping relationship, the target torque is converted into a precise current command). Because the structural design of the self-developed motor (such as winding parameters and magnetic circuit topology) and the algorithm are co-optimized, the assist algorithm can fully utilize the motor's high-bandwidth response characteristics to output a steeply rising torque pulse within an extremely short phase window, achieving highly dynamic transient assist. The current command is applied to the joint motor through the driver. The motor's output torque is amplified by a harmonic reducer and then smoothly applied to the wearer's hip joint, completing one full assist movement.
[0071] Throughout the process, the dual-sided encoder inside the motor continuously feeds back the motor's actual position, speed, and output torque to the main controller, comparing and verifying them with the phase estimated by the adaptive oscillator and the expected state by the state discriminator. If an anomaly is detected (such as torque output deviation exceeding a threshold, gait phase synchronization loss, etc.), the system will immediately degrade to the basic passive assist mode to ensure the wearer's absolute safety. Through this complete closed-loop chain of "perception-recognition-fusion-decision-execution-feedback," this embodiment achieves a fundamental leap from traditional passive responsive exoskeletons to active predictive, personalized intelligent exoskeletons.
[0072] This invention achieves a leap from passive response to proactive prediction through multimodal information fusion, significantly improving the intelligence, security, and smoothness of human-machine collaboration.
[0073] In the description of this application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0074] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A control method for an adaptive assistive exoskeleton system based on user active vision, characterized in that, Includes the following steps: Acquire structured environmental information representing changes in the terrain ahead, collected by wearable vision sensors; Acquire user natural language commands collected by the microphone that qualitatively describe the assist effect, and parse them into quantitative correction values for control parameters; The phase of the current gait is calculated in real time based on the angle signals output by the encoders built into the joints on both sides of the exoskeleton robot. By integrating the structured environmental information, the quantitative correction value, and the phase of the current gait, a control decision command is generated; wherein, when the structured environmental information indicates a change in the terrain ahead, the control decision command is configured to adjust the assist strategy in advance before the exoskeleton robot reaches the change in terrain. Based on the control decision commands and real-time gait phase, the joint motors of the exoskeleton robot are driven to output target torque to assist the user's movement.
2. The control method for the adaptive assistive exoskeleton system based on user active vision according to claim 1, characterized in that, The structured environment information is obtained by processing the collected images of the environment in front of the device using a visual deep learning model deployed on the smart glasses terminal, so as to output the structured environment information containing terrain category semantic labels and their spatial geometric attributes.
3. The control method for the adaptive assistive exoskeleton system based on user active vision according to claim 2, characterized in that, The visual deep learning model is a large-scale visual model; The processing of the foreground environment image specifically involves semantic segmentation and depth estimation of the foreground environment image.
4. The control method for the adaptive assistive exoskeleton system based on user active vision according to claim 1, characterized in that, The phase of the current gait is calculated in real time by: calculating the angle difference signal of the encoder built into the bilateral joints of the exoskeleton robot, and inputting the angle difference signal into the adaptive oscillator for calculation.
5. The control method for the adaptive assistive exoskeleton system based on user active vision according to claim 1, characterized in that, The advance adjustment of the assist strategy is as follows: when it is detected that there is an upward staircase or slope ahead, the assist torque amplitude in the direction of hip flexion is increased in advance at the end of the support phase before the user's lower limbs contact the terrain.
6. The control method for the adaptive assistive exoskeleton system based on user active vision according to claim 1, characterized in that, Parsing the user's natural language instructions specifically involves: performing semantic parsing through a natural language processing model, and mapping instructions that provide a qualitative description of the assistance effect to quantitative correction values for at least one control parameter in the assistance algorithm.
7. The control method for the adaptive assistive exoskeleton system based on user active vision according to claim 1, characterized in that, The target torque output of the drive joint motor is specifically achieved by: coordinating and optimizing the generation of the target torque based on the physical characteristic parameters of the joint motor.
8. An adaptive assistive exoskeleton system based on user active vision, characterized in that, include: Wearable visual sensors and microphones; Hip exoskeleton actuator, which includes bilateral joint motors with built-in encoders; A controller configured to perform the control method for an adaptive assistive exoskeleton system based on user active vision as described in any one of claims 1 to 7.
9. The adaptive assistive exoskeleton system based on user active vision according to claim 8, characterized in that, The wearable visual sensor and microphone are integrated into a smart glasses terminal, which has a built-in edge computing unit for running visual deep learning models and natural language processing models.
10. The adaptive assistive exoskeleton system based on user active vision according to claim 8, characterized in that, The bilateral joint motors of the hip exoskeleton actuator are high torque density frameless torque motors.
