Big model-based walking exoskeleton intention trajectory planning method and system

Abstract

The application discloses a walking exoskeleton intention trajectory planning method based on a large model, an inertial sensor is applied to calculate the man-machine deviation angle of each joint of a wearer and a walking exoskeleton, obtain the human body movement intention information of the wearer, and solve the control limit in the human body movement intention aspect; image information on the walking path of the exoskeleton is collected, the collected image is segmented in real time through a visual large model, the boundary information of the obstacles on the walking path is obtained, and the control limit in the obstacle avoidance aspect is solved; according to the control limit in the human body movement intention aspect and the control limit in the obstacle avoidance aspect, the optimal intention trajectory of the walking exoskeleton is obtained, and the walking exoskeleton is controlled in real time to walk according to the optimal intention trajectory. The application has the advantages of higher precision, stronger generalization ability, better man-machine cooperation, higher robustness and better applicability to the man-machine interaction system of the walking exoskeleton; and the man-machine integration effect is realized through a quadratic programming problem.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of walking exoskeleton robots, and particularly relates to a walking exoskeleton intention trajectory planning method and system based on a large model. BACKGROUND

[0002] At present, with the development of robot technology and artificial intelligence technology, lower limb walking exoskeleton technology has also developed greatly. With the advent of a series of large models at home and abroad, using large models to assist robot technology has become a research and application trend.

[0003] Walking exoskeletons belong to wearable human-robot interaction robots and have wide applications in the fields of power assistance, rehabilitation medicine, etc.

[0004] In the application of walking exoskeletons, due to the limitation of the flexibility of the exoskeleton mechanical structure and the deficiency of the walking exoskeleton trajectory planning algorithm, the walking exoskeleton cannot successfully avoid obstacles on the motion trajectory, which seriously threatens the safety of the exoskeleton wearer and poses a challenge to the safety of the walking exoskeleton. How to respect the motion intention of the wearer while ensuring safety and plan a suitable motion trajectory is a problem to be solved. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a walking exoskeleton intention trajectory planning method and system based on a large model, which solves the problem that the existing walking exoskeleton cannot walk safely according to the intention of the wearer.

[0006] The present application adopts the following technical solutions to solve the above technical problems:

[0007] The walking exoskeleton intention trajectory planning method based on a large model applies an inertial sensor to calculate the human-machine bias angle of each joint of the wearer and the walking exoskeleton, obtain the human motion intention information of the wearer, and solve the control limit in terms of human motion intention.

[0008] Image information on the walking path of the exoskeleton is collected, the collected images are segmented in real time through a visual large model, the boundary information of the obstacles on the walking path is obtained, and the control limit in terms of obstacle avoidance is solved.

[0009] According to the control limit in terms of human motion intention and the control limit in terms of obstacle avoidance, the optimal intention trajectory of the walking exoskeleton is obtained, and the walking exoskeleton is controlled in real time to walk according to the optimal intention trajectory.

[0010] The human-machine bias angle information is solved by combining the control Lyapunov function to obtain the control limit based on the human motion intention of the wearer.

[0011] The specific process of obtaining the control constraints on human motion intention is as follows:

[0012] The specific expression of the control Lyapunov function is:

[0013]

[0014] in, represents the controlling Lyapunov function, and Respectively represent the actual coordinates of the center of mass on the x-axis and y-axis on the horizontal plane, and Respectively represent the expected coordinates of the center of mass on the x-axis and y-axis on the horizontal plane;

[0015]

[0016] in, for The first derivative of and Represents the control input in the x-direction and y-direction respectively, which are and The derivative of The conditions that need to be met are the control constraints on human movement intentions, which are expressed as follows:

[0017]

[0018] in, for The coefficient constant of .

[0019] According to the boundary information of the obstacle, the obstacle avoidance control function is applied to solve the control constraints of the obstacle avoidance.

[0020] The specific process of solving the control constraints for obstacle avoidance is as follows:

[0021] The specific expression of the obstacle avoidance control function is:

[0022]

[0023] in, represents the obstacle avoidance control function, and Respectively represent the actual coordinates of the center of mass on the x-axis and y-axis on the horizontal plane, and Respectively represent the x-axis and y-axis coordinates of the obstacle center on the horizontal plane, Indicates the size of the obstacle radius;

[0024]

[0025] wherein, is a first derivative, and respectively represent control inputs in the x direction and the y direction, which are respectively derivatives of and , wherein the condition that B needs to satisfy is the control limit in terms of obstacle avoidance, which is expressed by the following formula:

[0026]

[0027] wherein, is a coefficient constant of .

[0028] The specific method for obtaining the optimal trajectory of the walking exoskeleton is as follows:

[0029] The control limit in terms of human motion intention and the control limit in terms of obstacle avoidance are taken as the limits of the quadratic programming problem, the two modes are fused, and the optimal control input, i.e., the optimal intention trajectory, is obtained by solving the quadratic programming problem model.

[0030] The process of processing the image by the visual large model is as follows:

[0031] The collected image and the obtained prompt information are input into the visual large model, respectively pass through the image encoder and the prompt encoder, enter the mask decoder, output the effective mask, and obtain the boundary information of the obstacle on the walking path of the walking exoskeleton.

[0032] In order to further solve the problem that the flexibility of the mechanical structure of the walking exoskeleton is limited, the walking exoskeleton cannot successfully avoid the obstacle on the motion trajectory, and the safety of the wearer is caused, the application also provides a walking exoskeleton human-computer interaction system based on a large model, and the specific technical scheme is as follows:

[0033] The walking exoskeleton human-computer interaction system based on the large model comprises an inertial sensor module, an image acquisition module, a large model image segmentation module, a data processing module and a human-computer cooperative control module, wherein,

[0034] The inertial sensor module comprises a plurality of inertial sensors arranged on the center of mass of the wearer and the exoskeleton, and the inertial sensors acquire the position, speed, angle and angular velocity information of the center of mass of the wearer and the exoskeleton in real time;

[0035] The image acquisition module comprises a visual camera arranged on the walking exoskeleton;

[0036] The large model image segmentation module segments the image information collected by the visual camera, identifies the obstacle, and obtains the boundary information of the obstacle image;

[0037] a data processing module, applying the method to obtain an optimal intention trajectory of the walking exoskeleton;

[0038] a human-machine collaborative control module for collaborative control of the wearer and the walking exoskeleton, so that the walking exoskeleton walks according to the optimal intention trajectory.

[0039] The wearer and the walking exoskeleton are connected through an admittance model and an adaptive method to form the human-machine collaborative control module.

[0040] A computer-readable storage medium stores computer-readable instructions thereon, and the computer-readable instructions invoke all or part of the steps of the method when executed by a processor.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1. The present application introduces a visual large model technology into a walking exoskeleton, and uses a visual large model to segment obstacle image information on a walking exoskeleton motion path, which has higher precision and stronger generalization ability compared to traditional image segmentation methods. At the same time, the visual large model technology is simpler to deploy and has better stability, and is more suitable for a human-machine interaction system such as a walking exoskeleton robot.

[0043] 2. The present application only uses an inertial sensor to identify human motion intention, and does not use force, electromyography, or other sensors. Compared to other sensors, the stability and accuracy of the inertial sensor are much higher, so the human-machine collaboration of the present application is better, the robustness is higher, and the present application is more suitable for a human-machine interaction system such as a walking exoskeleton.

[0044] 3. The present application uses a control Lyapunov function and an obstacle avoidance control function to introduce wearer intention and obstacle information into the solving process of the optimal trajectory through a quadratic programming problem, realizes the fusion of two modes, greatly respects the wearer's motion intention under the premise of ensuring the safety of the walking exoskeleton, and realizes the effect of human-machine integration. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flowchart of a walking exoskeleton intention trajectory planning method based on a large model of the present application.

[0046] Figure 2 A structure diagram of a visual large model SAM of the present application.

[0047] Figure 3 A structure and adaptive process diagram of a wearer-walking exoskeleton system.

[0048] Figure 4 A schematic diagram of optimal trajectory planning.

[0049] Figure 5 An automatic labeling process diagram for a SAM base model.

[0050] Figure 6 A data engine and data set diagram for SAM.

[0051] Figure 7 A diagram for the participation of a CLIP large model in detection. DETAILED DESCRIPTION

[0052] The structure and working process of the present application will be further described below in conjunction with the drawings.

[0053] The purpose of the present application is to provide a large model-based walking exoskeleton intention trajectory planning method. Visual large model technology is introduced into the trajectory planning of a walking exoskeleton robot, so that the walking exoskeleton can respect the wearer's motion intention to the greatest extent under the premise of ensuring safety. The wearer's motion intention is obtained only with an inertial sensor, and the wearer and the exoskeleton are linked with a mobility model; the image of an obstacle on the motion trajectory is collected with a camera, and the boundary of the obstacle is segmented with a visual large model. The wearer's motion intention and the boundary information of the obstacle are informationally fused to obtain the optimal motion trajectory.

[0054] The large model-based walking exoskeleton intention trajectory planning method applies an inertial sensor to calculate the human-machine bias angle of each joint of the wearer and the walking exoskeleton, obtain the human body motion intention information of the wearer, and solve the control limit in terms of human body motion intention;

[0055] Image information on the walking path of the exoskeleton is collected, the collected image is segmented in real time through a visual large model, the boundary information of the obstacle on the walking path is obtained, and the control limit in terms of obstacle avoidance is solved;

[0056] According to the control limit in terms of human body motion intention and the control limit in terms of obstacle avoidance, the optimal intention trajectory of the walking exoskeleton is obtained, and the walking exoskeleton is controlled in real time according to the optimal intention trajectory.

[0057] Specific embodiments, as shown in Figures 1 to 7

[0058] A large model-based walking exoskeleton intention trajectory planning method, specifically comprising the following steps:

[0059] Step 1: Find the center of mass of the wearer and the walking exoskeleton, arrange an inertial sensor on the center of mass of the wearer and the walking exoskeleton, and collect the position, speed, angle, angular velocity, etc. of the center of mass of the wearer and the walking exoskeleton with the inertial sensor; the collected information is filtered through a Gaussian filter to remove the noise of the inertial sensor acquisition signal.​

[0060] Only inertial sensors are used, and no other sensors are used, and a mobility model and an adaptive method are used to link the wearer and the exoskeleton.

[0061] Step 2: Image information on the walking path is collected using a camera. The camera is arranged on the walking exoskeleton. A binocular camera can be selected to extract depth information of the image on the walking path to achieve better and more accurate obstacle avoidance. A monocular camera can also achieve the effect of obstacle avoidance. When arranging the camera, in order to prevent the image collection effect from being poor due to shaking during walking, a lens stabilizer is used to suppress the shaking of the lens to obtain a better image collection effect.

[0062] Step 3: The wearer and the walking exoskeleton are linked by a mobility model and an adaptive method, and the information obtained by the inertial sensor is calculated to the rotation angle information of each joint through the solution of the Jacobian matrix. The specific form of the mobility model is:

[0063]

[0064] wherein, represents a mass matrix, represents a centrifugal force and Coriolis force matrix, represents a gravity matrix, , , respectively represent position, velocity, and acceleration, represents a control input.

[0065] The adaptive method is to estimate the uncertain or time-varying parameters in the model, that is, to correct the current parameters according to the system deviation that occurs at the past time.

[0066] Step 4: The sampled image is segmented using a large visual model (this embodiment uses the image segmentation large model Segment Anything Model (SAM)). The input of the SAM large model is an image and a prompt information, and the prompt information is obtained by solving the human-machine deviation angle. The image and the prompt information pass through an image encoder and a prompt encoder, respectively, into a mask decoder, and then output an effective mask. The specific structure of the SAM large model is as shown in Figure 2 The mask here refers to the processing result of a specific region of the image, that is, the recognition result of the obstacle region on the path, so that the boundary information of the obstacle can be obtained.

[0067] In the use of SAM large model for image segmentation, the host computer needs to have good computing performance to ensure the real-time performance of the operation. At the same time, in order to strengthen the detection effect, the CLIP large model can be used to strengthen the performance and generalization ability of the detection. At the same time, in order to ensure the real-time performance and effectiveness of the system, the Mobile-SAM large model is used, which can better guarantee the real-time performance and effectiveness. Finally, the boundary information of the obstacles on the walking exoskeleton walking path is obtained by using the SAM large model for segmentation. Using the Mobile-SAM large model operation, the speed is faster, and the segmentation effect is almost the same as that of the SAM large model, which can better meet the system requirements. The CLIP large model can be used for further detection of the segmented image to enhance the performance and generalization ability of the detection. On the basis of the original model, according to the actual situation of the walking exoskeleton human-computer interaction system, some weights of the SAM large model are fine-tuned through the training of the image to achieve better results.

[0068] Step 5: The human-machine deviation angle information obtained by the inertial sensor is combined with the control Lyapunov function to obtain the control limit based on the motion intention of the wearer. The human-machine deviation angle information obtained by the inertial sensor is combined with the human-machine deviation angle information, and when the human-machine deviation angle is large, that is, when the wearer has a large turning intention, the object on the front path is considered as an obstacle, and the visual large model is used for boundary segmentation of the obstacle.

[0069] The human-machine deviation angle is obtained by subtracting the turning angle of the inertial sensor arranged on the wearer from the turning angle of the inertial sensor arranged on the exoskeleton.

[0070] The control Lyapunov function is in the form of a sliding film, which can make the motion path continuously tend to the intended trajectory of the wearer. When the motion path deviates greatly from the intended trajectory of the wearer, there will be a large control output to pull the motion path back to the intended trajectory of the wearer.

[0071] The specific representation of the control Lyapunov function is:

[0072]

[0073] wherein, represents the control Lyapunov function, and respectively represent the actual coordinates of the center of mass on the x-axis and y-axis of the horizontal plane, and respectively represent the expected coordinates of the center of mass on the x-axis and y-axis of the horizontal plane;

[0074]

[0075] wherein, is derivative of and respectively represent the control input in x direction and y direction, which are the derivative of and respectively, The condition that needs to be met is the control limit in terms of human motion intention, which is expressed as follows:

[0076]

[0077] wherein, is a coefficient constant of .

[0078] Step 6: The obstacle boundary information obtained by the visual large model is combined with the obstacle avoidance control function to obtain the control limit based on the obstacle boundary information. The obstacle avoidance control function is in the form of a synovial membrane, which can make the motion path avoid the boundary of the obstacle. When the motion path is very close to the boundary of the obstacle, there will be a larger control output to push the motion path away from the boundary of the obstacle.

[0079] The specific representation of the obstacle avoidance control function is as follows:

[0080]

[0081] wherein, represents the obstacle avoidance control function, and respectively represent the actual coordinates of the center of mass on the x-axis and y-axis of the horizontal plane, and respectively represent the coordinates of the center of the obstacle on the x-axis and y-axis of the horizontal plane, represents the size of the obstacle radius;

[0082]

[0083] wherein, is the first derivative of , and respectively represent the control input in x direction and y direction, which are the derivative of and respectively, wherein the condition that B needs to meet is the control limit in terms of obstacle avoidance, which is expressed as follows:

[0084]

[0085] wherein, is a coefficient constant of .

[0086] Step 7: The control restrictions based on the wearer's motion intention and the control restrictions based on the obstacle boundary information are combined as the restrictions of a quadratic programming problem, the two modalities are fused, and the optimal control input, i.e., the optimal intention trajectory, is obtained by solving the quadratic programming problem model, so that the walking exoskeleton can greatly respect the wearer's motion intention while ensuring the safety of the wearer.

[0087] The large model-based walking exoskeleton human-computer interaction system comprises an inertial sensor module, a human-computer collaborative control module, a visual camera module, a large model image segmentation module, a control and obstacle avoidance function module, and an optimal trajectory solving module.

[0088] The inertial sensor module is an inertial sensor arranged on the center of mass of the wearer and the exoskeleton. The inertial sensor obtains the position, speed, angle, and angular velocity information of the center of mass of the wearer and the exoskeleton in real time, which will be used in human-computer collaborative control.

[0089] The human-computer collaborative control module is used to cooperatively control the wearer and the walking exoskeleton, and links the wearer and the walking exoskeleton by using a mobility model, so that the walking exoskeleton can follow the wearer's intention.

[0090] The visual camera module is a camera arranged on the walking exoskeleton. If depth information is needed, a binocular camera can be arranged. When arranging the camera, a lens stabilizer is used to stabilize the camera to obtain better visual image acquisition effect in order to prevent the camera from shaking with the exoskeleton.

[0091] The large model image segmentation module is a fusion of visual large model technology and walking exoskeleton. The obstacle image obtained by the camera is segmented by using the visual segmentation model SAM model to obtain the boundary information of the obstacle image. In the segmentation process, a model with strong generalization ability that has been trained is used to make the segmentation task efficient and accurate. When the human-computer deviation angle changes greatly, it is considered that an obstacle appears in front, and the obstacle is segmented.

[0092] The control and obstacle avoidance function module uses the form of Lyapunov function and combines the method of sliding film to construct the control Lyapunov function and the obstacle avoidance control function. The control restriction obtained by the control Lyapunov function can make the walking exoskeleton respect the wearer's motion intention, and the control restriction obtained by the obstacle avoidance control function can make the walking exoskeleton effectively avoid the obstacle in front and ensure the safety of the walking exoskeleton.

[0093] The optimal trajectory solving module is a quadratic programming model integrating the solving of the optimal trajectory, combining the control limit obtained by the control Lyapunov function and the control limit obtained by the obstacle avoidance control function, and solving the optimal trajectory to make the planned trajectory respect the motion intention of the wearer and safely avoid the obstacles on the front motion trajectory, realizing the fusion of the two modes.

[0094] A computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions are executed by the processor to call all or part of the steps of the method.

[0095] The above functions, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0096] The above is the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A walking exoskeleton intention trajectory planning method based on a large model, characterized by: Inertial sensors are used to calculate the human-machine deflection angle of each joint of the wearer and the walking exoskeleton. The human-machine deflection angle is obtained by subtracting the steering angle of the inertial sensors on the wearer's body from the steering angle on the horizontal plane of the inertial sensors on the exoskeleton. The wearer's human motion intention information is obtained and the human-machine deflection angle information is solved by combining the control Lyapunov function to solve the control constraints of the human motion intention. The specific process is as follows: The specific expression of the control Lyapunov function is: ， in, represents the controlling Lyapunov function, and Respectively represent the actual coordinates of the center of mass on the x-axis and y-axis on the horizontal plane, and Respectively represent the expected coordinates of the center of mass on the x-axis and y-axis on the horizontal plane; ， in, for The first derivative of and Represents the control input in the x-direction and y-direction respectively, which are and The derivative of The conditions that need to be met are the control constraints on human movement intentions, which are expressed as follows: ， in, for The coefficient constant of ; The exoskeleton collects image information along its path, segments it in real time using a large visual model, and obtains boundary information for obstacles along the path. Based on this information, an obstacle avoidance control function is applied to calculate the control constraints for obstacle avoidance. The specific process is as follows: The specific expression of the obstacle avoidance control function is: ， in, represents the obstacle avoidance control function, and Respectively represent the actual coordinates of the center of mass on the x-axis and y-axis on the horizontal plane, and Respectively represent the x-axis and y-axis coordinates of the obstacle center on the horizontal plane, Indicates the size of the obstacle radius; ， in, for The first derivative of and Represents the control input in the x-direction and y-direction respectively, which are and Here, the condition that B needs to satisfy is the control restriction of obstacle avoidance, which is expressed by the following formula: ， in, for The coefficient constant of ; Based on the control constraints of human motion intention and obstacle avoidance, the optimal intention trajectory of the walking exoskeleton is obtained, and the walking exoskeleton is controlled in real time to walk according to the optimal intention trajectory. The specific method for obtaining the optimal intention trajectory of the walking exoskeleton is as follows: The control constraints on human motion intention and obstacle avoidance are taken as constraints of the quadratic programming problem. The two modes are fused and solved using the quadratic programming problem model to obtain the optimal control input, that is, the optimal intention trajectory.

2. The large-model-based walking exoskeleton intention trajectory planning method according to claim 1 is characterized in that: The process of the visual large model processing the image is as follows: The captured images and the obtained prompt information are input into the visual large model, pass through the image encoder and prompt encoder respectively, enter the mask decoder, output the valid mask, and obtain the boundary information of obstacles on the walking path of the walking exoskeleton.

3. A walking exoskeleton human-computer interaction system based on a large model, characterized by: It includes inertial sensor module, image acquisition module, large model image segmentation module, data processing module, and human-machine collaborative control module; among them, The inertial sensor module includes several inertial sensors arranged at the center of mass of the wearer and the exoskeleton. The inertial sensors obtain the position, speed, angle, and angular velocity information of the wearer and the exoskeleton in real time. An image acquisition module, comprising a visual camera arranged on the walking exoskeleton; Large model image segmentation module, which segments the image information collected by the visual camera, identifies obstacles, and obtains the boundary information of the obstacle image; A data processing module, applying the method of claim 1 or 2 to obtain the optimal intended trajectory of the walking exoskeleton; The human-machine collaborative control module is used to collaboratively control the wearer and the walking exoskeleton so that the walking exoskeleton can walk according to the optimal intended trajectory.

4. The large-scale model-based walking exoskeleton human-computer interaction system according to claim 3 is characterized in that: The wearer and the walking exoskeleton are connected through the admittance model and adaptive method to form a human-machine collaborative control module.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, call all or part of the steps of the method according to claim 1 or 2.

Images