Control method of auxiliary walking system based on large visual model (VLM) and exoskeleton system

Abstract

A control method of an auxiliary walking system based on a visual large model (VLM) is applied to an exoskeleton system suitable for being worn by an operator and comprises the following steps of data collection and fusion, perception and analysis, dynamic path planning and strategy generation, humanized interactive design and learning and adaptive optimization. Data feedback in a VLM processing walking process is utilized, model parameters are automatically adjusted through an online learning mechanism in a deep neural network, and the understanding and adaptive capacity to various environments is optimized; meanwhile, according to user feedback and use habits, weights of all dimensions are adjusted in a personalized mode so as to achieve long-term performance optimization. By fusing multi-element perception information and intelligent analysis, the environmental adaptability and user interaction quality of the exoskeleton are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent auxiliary equipment, and in particular to an auxiliary walking system based on a visual large model (VLM) and a control method thereof, aiming to provide operators with more intelligent and personalized walking assistance, and to achieve real-time environmental perception, dynamic path planning and humanized interactive experience by integrating multimodal information such as vision, hearing and language, and is particularly suitable for the control and optimization of exoskeleton robots. Background Art

[0002] With the trend of population aging and the growing demand for industrial automation, the field of assisted walking technology and intelligent robots is undergoing unprecedented changes. Although traditional walking aids such as crutches and wheelchairs have alleviated the life difficulties of people with limited mobility to a certain extent, they still have great limitations in adaptability to complex environments, satisfaction of personalized needs, and intelligence. At the same time, the rise of exoskeleton robot technology has opened up new possibilities for improving the quality of life of people with limited mobility and enhancing human ability to work in special environments.

[0003] Exoskeleton robots, also known as powered exoskeletons or wearable robots, are devices that combine ergonomic principles, integrate with the human body through mechanical structures, and assist or enhance human movement capabilities. Early exoskeleton designs were mainly focused on the military field, used to enhance soldiers' load-bearing capacity and travel speed. In recent years, with the rapid progress of materials science, sensor technology, battery life, and artificial intelligence algorithms, exoskeleton technology has begun to be widely used in medical rehabilitation, industrial operations, emergency rescue and other fields.

[0004] Although it has been developed for decades, there are still many challenges or new demands in the context of rapid technological development: Environmental adaptability: The real world environment is complex and changeable, and the exoskeleton robot needs to be able to work stably under different terrains (such as flat roads, stairs, slopes, etc.), lighting conditions, and climate conditions. This requires the system to accurately perceive the environment and quickly adjust the walking strategy according to environmental changes.

[0005] Intelligent human-computer interaction: Traditional exoskeleton systems often rely on simple manual control or preset programs and cannot respond to changes in user intentions and external emergencies in real time. Modern exoskeleton systems need to achieve intelligent human-computer interaction, understand the user's action intentions, and even predict the user's next action, so as to provide more natural and smooth walking assistance.

[0006] Personalized needs: Different users have different heights, weights, health conditions, and personal habits, and their needs for assistive devices are also different. How to achieve a high degree of personalized customization in design to meet the specific needs of different users is the key to improving user satisfaction and equipment efficiency.

[0007] Data processing and learning capabilities: Exoskeleton robots need to process a large amount of multimodal data, including vision, hearing, biomechanics, etc., requiring the system to have efficient data processing capabilities. In addition, learning from user usage habits and feedback and continuously optimizing their own performance is an important way to improve the intelligence level of the system.

[0008] The assisted walking system based on the visual large model (VLM) came into being in this context. It aims to build an intelligent exoskeleton system that can accurately perceive the dynamics of the environment, deeply understand user needs, and self-learn and optimize by deeply integrating multiple information such as vision, language, and biomechanics. This technology can not only significantly improve the practicality and user experience of exoskeleton robots, but also open up a new path for promoting the development of intelligent assistance technology. Summary of the invention

[0009] A control method for an assisted walking system based on a visual large model (VLM), the method being applied to an exoskeleton system (2) suitable for being worn by an operator (1), the system being integrated with a sensor component (3) mounted on a robot body or attached to the operator (1), comprising the following steps: Data collection and fusion: The sensor assembly (3) is used to collect multimodal data in real time, and objects in visual images are classified, identified and located through VLM. Depth information assists in structural analysis of three-dimensional road conditions (4), and voice commands and environmental sound features are integrated to perceive environmental dynamics in multiple dimensions.

[0010] Perception and analysis: Based on the pre-trained visual-language association knowledge base, identify physical environment features and situational context, including but not limited to analyzing the operator's (1) behavioral intention and identifying the textual meaning of safety warning signs.

[0011] Dynamic path planning and strategy generation: Combined with the user's personalized demand model, the dimensional weights of the walking strategy are dynamically adjusted; then the optimized walking path and exoskeleton motion parameters are automatically generated; during the walking process, VLM adjusts the strategy based on real-time situational feedback.

[0012] Humanized interactive design: VLM generates natural language feedback and visual prompts to convey walking suggestions, safety warnings or other necessary information to users in an easy-to-understand manner based on user preferences and current context.

[0013] Learning and Adaptive Optimization: The system continuously uses VLM to process data feedback during walking, and automatically adjusts model parameters through the online learning mechanism in the deep neural network to optimize the understanding and adaptability to various environments.

[0014] The assisted walking system driven by the large visual model significantly improves the environmental adaptability and user interaction quality of the exoskeleton by integrating multi-sensory information and intelligent analysis. It not only realizes comprehensive environmental perception and situational understanding to ensure walking safety and efficiency, but also dynamically optimizes paths and actions according to the personalized needs of users. In addition, with humanized interactive design and continuous self-learning capabilities, it ensures that the system can provide a safe and comfortable assisted walking experience in complex environments. At the same time, it continues to evolve to meet the long-term use needs of users, leading assisted walking technology into a new stage of intelligence and personalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the system of the present invention.

[0016] Figure 2 The overall flow chart of the control method of the present invention.

[0017] Figure 3 The structure and process of the pre-training knowledge base of the present invention are schematically shown.

[0018] Figure 4 Schematic diagram of the classification of sensor components of the present invention. DETAILED DESCRIPTION Specific embodiments

[0019] A control method for an assisted walking system based on a visual large model (VLM), the method being applied to an exoskeleton system (2) suitable for being worn by an operator (1), the system being integrated with a sensor component (3) mounted on a robot body or attached to the operator (1), comprising the following steps: Data collection and fusion: The sensor assembly (3) is used to collect multimodal data in real time, and objects in visual images are classified, identified and located through VLM. Depth information assists in structural analysis of three-dimensional road conditions (4), and voice commands and environmental sound features are integrated to perceive environmental dynamics in multiple dimensions.

[0020] In the assisted walking system based on the visual large model (VLM), the algorithms and models involved in implementing the data acquisition and fusion steps include but are not limited to the following aspects: Visual data processing and recognition: The YOLO target detection model is used to locate objects in images in real time, which is crucial for obstacle avoidance and path planning.

[0021] Depth perception and 3D reconstruction: 3D point cloud processing algorithms (specifically using PCL, Point Cloud Library) process and analyze point cloud data generated by depth sensors for accurate analysis of 3D road conditions (4) structures.

[0022] Audio Data Processing and Speech Recognition: Noise suppression and speech enhancement algorithm: Use Wiener filtering to remove background noise and improve the clarity of speech signals.

[0023] Speech recognition model: Converts speech signals into text to support recognition of operator (1) voice commands.

[0024] Keyword recognition technology: quickly identify specific voice commands or emergency signals to improve system response speed. Multimodal data fusion: Multimodal fusion network (Transformer-based multimodal fusion model): integrates visual, depth, and audio data, and processes them through deep learning models to improve the comprehensiveness and accuracy of environmental perception.

[0025] Reinforcement learning algorithm (DQN): adjust walking strategies based on environmental feedback, optimize path planning and motion parameters, and improve walking efficiency and safety.

[0026] Situational understanding model: Based on the pre-trained vision-language model (CLIP), it understands environmental features and operator (1) intentions to improve the system's intelligent decision-making level.

[0027] Based on the pre-trained visual-language association knowledge base, identify physical environment features and situational context, including but not limited to analyzing the operator's (1) behavioral intention and identifying the textual meaning of safety warning signs;.

[0028] The behavioral intention of the operator (1) refers to the behavioral response of the operator (1) in a specific environment, road condition, or situation, which is stored in the pre-trained knowledge base. For example: On a bumpy road with fine pebbles, the operator (1) will slow down and reduce the step distance many times. The system learns this behavior pattern through multiple trainings; When climbing stairs, the hip joint will have an instantaneous large torque output and the joint angle is large. At this time, it is also necessary to switch the walking assistance system to a high-level assistance mode to adapt to this behavior. After multiple trainings, the system can not only identify stairs, but also start the high torque mode at a specific time point after identifying stairs.

[0029] The perception and analysis stage is the core intelligent processing part of the assisted walking system based on the vision-language large model (VLM). It deeply integrates visual and language information to build a capability framework that can highly understand the environment and user intentions.

[0030] The further explanation of this step is the specific implementation method five.

[0031] The pre-trained visual-language association knowledge base is collaboratively trained in a semi-supervised classification manner by an operator (1) wearing an exoskeleton system (2) and carrying the sensor assembly (3), wherein the sensor assembly (3) is divided into at least two sets of independent perception modules, which are respectively responsible for collecting biomechanical data and environmental information of different dimensions, and is characterized by comprising the following steps: Multimodal data collection and preprocessing: When the operator (1) wears the exoskeleton system (2), the first perception module collects the operator's (1) motion data in real time; the second perception module simultaneously obtains external environment data; Collaborative view creation and initial learning: Based on the preprocessed data, at least two independent and conditionally independent data views are constructed, each of which represents the response to the interaction between the exoskeleton operator (1) and the environment. Using the initial labeled data set, including the correct action demonstration of the operator (1), at least two preliminary learning models are trained accordingly. Iterative collaborative training: While the operator (1) is operating, each learning model takes turns to make predictions and infer labels for the unlabeled data set. Based on the confidence of the model prediction and the consistency with the predictions of other views, highly reliable unlabeled samples are selected from each view and added to the training set of the corresponding view. Subsequently, all models are iteratively updated using the expanded training set until a predetermined performance improvement threshold or an upper limit on the number of iterations is reached. Performance evaluation and strategy adjustment: After each training iteration, the overall performance of the exoskeleton system (2) is evaluated to dynamically adjust the collaborative training strategy.

[0032] Dynamic path planning and strategy generation: Combined with the user's personalized demand model, the dimensional weights of the walking strategy are dynamically adjusted; then the optimized walking path and exoskeleton motion parameters are automatically generated; during the walking process, VLM adjusts the strategy based on real-time situational feedback.

[0033] Dynamic path planning and strategy generation are key manifestations of the intelligent level of the assisted walking system. It closely focuses on the user's personalized needs and real-time environmental changes to ensure that the walking process is both safe and efficient. This process can be divided into the following core links: 1. Integration of user personalized demand model The system first integrates the specific demand model of each user, which is established based on multi-dimensional information such as the user's physiological conditions, habitual preferences, health status, etc. These personalized parameters are used as weight factors in the dynamic adjustment of walking strategies. For example, for users with poor balance ability, the system will increase the consideration weight of stability to ensure that the path planning is smoother and reduce sharp turns or steep sections.

[0034] 2. Real-time situation analysis and strategy optimization Using advanced machine learning algorithms, the system continuously analyzes real-time changes in the walking environment, including obstacle movement, changes in crowd density, lighting conditions, etc. VLM plays a bridging role in this link, not only identifying these changes, but also understanding their impact on users, and then dynamically adjusting the priority of walking strategies. For example, when a temporary construction area is detected ahead, the exoskeleton's motion parameters are adjusted to adapt to new walking needs.

[0035] 3. Automatic path generation and exoskeleton control. The exoskeleton's motion parameters will be automatically adjusted according to the path characteristics and user status to ensure that the walking action is ergonomic and can effectively assist users in overcoming the terrain challenges they encounter.

[0036] 4. Iterative strategy adjustment and feedback learning During the walking process, the system continuously receives feedback information from sensors, including the user's biological signals, the working status of the exoskeleton, and environmental changes, and uses this as a basis to fine-tune the walking strategy in real time.

[0037] Humanized interactive design: Use VLM to generate natural language feedback and visual prompts to convey walking suggestions, safety warnings or other necessary information to users in an easy-to-understand manner based on user preferences and current context; Learning and Adaptive Optimization: The system continuously uses VLM to process data feedback during walking, and automatically adjusts model parameters through the online learning mechanism in the deep neural network to optimize the understanding and adaptability to various environments. At the same time, based on user feedback and usage habits, the weights of each dimension are adjusted in a personalized manner to achieve long-term performance optimization.

[0038] At this stage, learning and adaptive optimization become the core driving force for improving user experience.

[0039] The system's built-in VLM records and analyzes real-time data about the user's walking, such as gait characteristics, changes in the surrounding environment, etc. This means that as users frequently use the system in different scenarios, the system can autonomously learn and quickly adapt to diverse environmental conditions, and can provide more accurate and personalized navigation and safety recommendations.

[0040] The speech recognition system also includes a target speech recognition module for speaker orientation in speech recognition. In actual use, the voices of multiple people may be collected by the system. If a person other than the speaker makes a commanding speech, the system will mistake it for a command, which will cause interference to the operator (1) and even danger.

[0041] Therefore, it is necessary to use a voiceprint recognition module to identify the specific speaker orientation.

[0042] Of course, there are other auxiliary technical means to achieve this demand: Microphone array: By comparing the time difference or phase difference of the sound signals received by different microphones, the direction of the sound source can be calculated. Beamforming technology: uses a microphone array to capture sound signals and processes these signals through algorithms to focus on enhancing sounds from a specific direction while suppressing noise and interference from other directions. It can significantly improve the speech clarity of the target speaker, thereby improving the recognition rate.

[0043] Speaker orientation refers to operator (1); and only operator (1) can be identified as the speaker orientation that issues control instructions to the system; control instructions include but are not limited to: Change your pace; lChange the power level; lChange the power assist mode; Start and stop the device.

[0044] Environmental sounds include traffic sound signals and interfering speakers.

[0045] Including, through vibration feedback, the device provides warning information when receiving a vehicle approaching, etc.

[0046] The dimensions include the following specific dimensions: (1) Walking speed; (2) Step length; (3) level of assistance; (4) hip and knee joint angles; (5) Maximum and minimum torque output values; (6) joint elastic impedance angle and impedance level; The weights for the above dimensions, in different environmental modes, will be combined with the personalized demand model to set different weight values ​​in real time or quantitatively.

[0047] An exoskeleton system, characterized by being equipped with the control method of the visual large model (VLM)-based assisted walking system as described in claim 1 or 6.

[0048] The exoskeleton system includes a sensor component (3) comprising an electromyographic sensor (EMG) (31) attached to the body surface of the operator (1).

[0049] The exoskeleton system (2) is specifically a lower limb exoskeleton robot.

[0050] The behavioral intention of the operator (1) is sensed by the muscle electrical signal sensed by the EMG (31) If certain words are used in the specification and claims to refer to specific components, those skilled in the art should understand that hardware manufacturers may use different terms to refer to the same component. This specification and claims do not use differences in names as a way to distinguish components, but use differences in the functions of components as the criteria for distinction. As mentioned throughout the specification and claims, "including" is an open term and should be interpreted as "including but not limited to". "Approximately" means that within an acceptable error range, those skilled in the art can solve the technical problem within a certain error range and basically achieve the technical effect.

[0051] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0052] The above description shows and describes several preferred embodiments of the present invention, but as mentioned above, it should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention concept described herein through the above teachings or the technology or knowledge of the relevant field. Changes and variations made by those skilled in the art do not depart from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A control method for an assisted walking system based on a visual large model (VLM), the method being applied to an exoskeleton system (2) suitable for being worn by an operator (1), the system being integrated with a sensor component (3) mounted on a robot body or attached to the operator (1), characterized in that: The following steps are involved: Data collection and fusion: The sensor assembly (3) is used to collect multimodal data in real time, and objects in the visual image are classified, identified and located through the VLM. The depth information assists in the structural analysis of the three-dimensional road condition (4), and the voice command and environmental sound characteristics are integrated to perceive the dynamic environment in multiple dimensions. Perception and analysis: Based on the pre-trained visual-language association knowledge base, identify physical environment features and situational context, including but not limited to analyzing the operator's (1) behavioral intention and identifying the textual meaning of safety warning signs; Dynamic path planning and strategy generation: Combined with the user's personalized demand model, dynamically adjust the dimension weights of the walking strategy; Then automatically generate the optimized walking path and exoskeleton motion parameters; Humanized interactive design: Use VLM to generate natural language feedback and visual prompts to convey walking suggestions, safety warnings or other necessary information to users in an easy-to-understand manner based on user preferences and current context; Learning and Adaptive Optimization: The system continuously uses VLM to process data feedback during walking, and automatically adjusts model parameters through the online learning mechanism in the deep neural network to optimize the understanding and adaptability to various environments.

2. The control method according to claim 1, characterized in that The speech also includes a target speech recognition module for speaker orientation in speech recognition.

3. According to the control method of claim 2, the speaker orientation refers to the operator (1); and only the operator (1) can be identified as the speaker orientation that issues control instructions to the system; the control instructions include but are not limited to: Change your pace; Change the power assist level; Change the power assist mode; Start and stop equipment.

4. The control method according to claim 1 or 3, characterized in that Environmental sounds include traffic sound signals and interfering speakers.

5. The control method according to claim 4, wherein the pre-trained visual-language association knowledge base is collaboratively trained in a semi-supervised classification manner by an operator (1) wearing an exoskeleton system (2) and carrying the sensor assembly (3), wherein the sensor assembly (3) is divided into at least two sets of independent perception modules, which are respectively responsible for collecting biomechanical data and environmental information of different dimensions, and wherein The following steps are involved: Multimodal data collection and preprocessing: When the operator (1) wears the exoskeleton system (2), the first perception module collects the motion data of the operator (1) in real time; The second perception module synchronously acquires external environment data; Collaborative view creation and initial learning: Based on the preprocessed data, at least two independent and conditionally independent data views are constructed, each of which represents the response to the interaction between the exoskeleton operator (1) and the environment. Using the initial labeled data set, including the correct action demonstration of the operator (1), at least two preliminary learning models are trained accordingly. Iterative collaborative training: While the operator (1) is operating, each learning model takes turns to make predictions and infer labels for the unlabeled dataset. Based on the confidence of the model prediction and its consistency with the predictions of other views, highly reliable unlabeled samples are selected from each view and added to the training set of the corresponding view. Then, all models are iteratively updated using the expanded training set until a predetermined performance improvement threshold or an upper limit on the number of iterations is reached; Performance evaluation and strategy adjustment: After each training iteration, the overall performance of the exoskeleton system (2) is evaluated to dynamically adjust the collaborative training strategy.

6. The control method according to claim 5, characterized in that The dimensions include the following specific dimensions: Pace; Step length; Assistance level; Hip and knee joint angles; Maximum and minimum torque output values; Joint elastic impedance angle and impedance level; The weights for the above dimensions, in different environmental modes, will be combined with the personalized demand model to set different weight values ​​in real time or quantitatively.

7. Exoskeleton system, characterized by A control method for an assisted walking system based on a visual large model (VLM) as described in claim 1 or 6.

8. The exoskeleton system of claim 8, wherein The sensor assembly (3) includes an electromyographic sensor EMG (31) attached to the body surface of the operator (1).

9. An exoskeleton system as claimed in claim 7 or 8, characterized in that The exoskeleton system (2) is specifically a lower limb exoskeleton robot.

10. The exoskeleton system according to claim 9, characterized in that The behavioral intention of the operator (1) is sensed through the muscle electrical signal sensed by the EMG (31).

Images