An impedance iterative learning control method for a human-guided vision-force fusion manipulator

Through the iterative learning and control method of robotic arm impedance of vision-force fusion under human guidance, combined with vision sensors and force sensors, using image features and motion dynamic primitive technology, the adaptive flexible assembly of robotic arm in the manufacturing of complex components is achieved, solving the problem of solving the problem of The stability and accuracy of traditional robotic arms in intensive contact scenarios are problematic.

CN118544362BActive Publication Date: 2024-12-06MAISI ZHIDU (HUNAN) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411008799.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-06
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

In the manufacturing of complex components, existing robotic arms are difficult to flexibly respond to changes in the operating objects and manufacturing environments, especially in intensive contact scenarios, and high-precision control cannot be stable, and traditional single visual sensing data cannot meet the needs of refined operations.

Method used

Using the human-guided vision-force fusion robot arm impedance iterative learning control method, the visual sensor and force sensor perceive environmental changes, combined with image features and motion dynamic primitive technology, an impedance iterative learning controller based on image features is designed and the impedance control is updated. Parameters to realize the adaptive flexible assembly of the robot arm.

Benefits of technology

It improves the adaptability and intelligence of the robotic arms in the manufacturing of complex components, can flexibly respond to a variety of manufacturing tasks, enhances assembly stability and accuracy in intensive contact scenarios, and reduces the uncertain dynamic impact on the environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118544362B_ABST
    Figure CN118544362B_ABST
Patent Text Reader

Abstract

The invention discloses an impedance iterative learning control method for a vision-force fusion manipulator under human guidance. The method analyzes the robot-environment interaction dynamics equation and solves the visual servo acceleration model, and combines the above equations to establish a human-manipulator-environment interaction dynamics model in an image feature space; collects image feature positions and velocity curves of a robot guided to complete an assembly task, and encodes and generalizes them using motion dynamic primitives; designs an impedance iterative learning controller based on image feature tracking errors as control inputs, learns the impedance characteristics of the robot during contact operations guided by the human, identifies unknown contact dynamics under the interaction between the robot and the environment, offsets identified contact interference in the feature space, and realizes flexible assembly operations; solves the problems of human-manipulator-environment coupled nonlinear dynamics, unknown contact dynamics in intensive contact assembly tasks, and difficulty in generalizing assembly scenarios in existing assembly operations, which require re-learning for different scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of intelligent manufacturing technology, and in particular relates to an impedance iterative learning control method for a vision-force fusion robot arm under human guidance. Background Art

[0002] The manufacturing process of complex parts, such as aviation wall panel assembly and precision electronic manufacturing, often has problems such as small working space, numerous assembly processes, and dynamic environmental changes. Most of them are manual operations or robots use fixed algorithms to complete single repetitive actions. Faced with the changes in the operating objects and manufacturing environment and the complex unstructured manufacturing environment, traditional robot manufacturing cannot flexibly respond to the intelligent needs of the modern robot manufacturing industry.

[0003] At present, vision has been widely introduced into industrial manufacturing. The conversion relationship between the end velocity of the robot arm and the angular velocity of the joint is: the visual sensor can perceive the surrounding environment information and can flexibly adapt to scene changes. Therefore, the visual sensor can be used to detect the working environment, locate the workpiece position, or process the workpiece quality inspection. In addition, considering that the actual assembly tasks are usually contact-rich tasks, robots that rely on single visual sensor data cannot meet the requirements of refined operations. For example, when assembling aviation wall panels, since the wall panels are weak rigid components, the high-rigidity robot will vibrate when it contacts the wall panels, affecting the assembly quality, and applying too much force may damage the workpiece. Force sensors can help robots perceive contact force information when interacting with the environment and help robots identify the interaction dynamics when contacting the environment. Maintaining the stability of contact tasks is another difficult problem to be solved. The coupled dynamics between the robot and the environment will cause the robot to be unable to stably transition from the free state to the contact state with the workpiece, and achieve high-precision control effects. In response to the above problems, multi-sensor information fusion is considered to supplement the insufficiency of contact operation perception. Therefore, we introduced vision sensors and force sensors to perceive environmental changes and contact forces, and realized the vision-force fusion adaptive flexible assembly of the robotic arm to solve problems such as the dynamic environment of complex component manufacturing and nonlinear contact coupling, improve the adaptability and intelligence of the robot, and flexibly respond to various manufacturing task requirements.

[0004] In summary, how to integrate multi-sensory information with heterogeneous data dimensions and mismatched frequencies, and how to overcome the coupled interaction dynamics between humans, robotic arms, and the environment to achieve robot adaptive assembly in intensive contact scenarios are relatively difficult problems currently existing in the field of intelligent manufacturing. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a human-guided vision-force fusion manipulator impedance iterative learning control method. The present invention introduces visual sensors and force sensors to perceive environmental changes and contact forces, and realizes the vision-force fusion adaptive flexible assembly of the manipulator to solve the problems of complex component manufacturing environment dynamics, contact nonlinear coupling, etc., improves the adaptability and intelligence of the robot, and flexibly responds to various manufacturing task requirements.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A human-guided vision-force fusion robot arm impedance iterative learning control method, the method comprising the following steps:

[0008] S100: Obtain the coordinate conversion relationship between the camera carried on the end effector of the robotic arm and the end of the robotic arm through hand-eye calibration, and solve the coordinate relationship between the six-dimensional force sensor and the end of the robotic arm through geometric relationship;

[0009] S200: Based on the coordinate transformation relationship between the camera and the end of the robotic arm, the visual servo acceleration model is derived. Combining the dynamic equation of human-robotic arm interaction and the coordinate transformation relationship between the end of the robotic arm and the six-dimensional force sensor, the human-robotic arm-environment interaction dynamics model in the feature space is derived.

[0010] S300: The robot is guided to complete the assembly task by a human-guided-admittance controller based on image features, during which the image feature position and speed data are recorded, and the image feature position and speed curve is encoded and generated using the motion dynamic primitive DMP. The starting point and end point of the curve are adjusted by changing the DMP parameters, and the image feature trajectory curve is regenerated to achieve trajectory generalization;

[0011] S400: Model the human force and environmental contact disturbance, and conduct theoretical stability analysis on the human-robot-environment interaction system;

[0012] S500: Combining the human-robot-environment interaction dynamics model in the feature space derived from S200 and the generalized image feature trajectory curve of S300, an impedance iterative learning law based on image features is designed. The image feature velocity curve is iteratively learned and the impedance control parameters are updated to characterize the human-guided impedance characteristics. At the same time, visual servoing is used to correct the tracking trajectory online. After a certain number of iterative updates, the robot can successfully track the human-guided feature position curve and complete the assembly task.

[0013] Preferably, S100 includes:

[0014] S110: The end effector of the robot arm is equipped with a six-dimensional force sensor and an RGBD camera.

[0015] S120: Use the RGBD camera to capture an image of a calibration plate with N points, and record the corresponding position and posture of the end of the robotic arm at this time; repeat this step for a preset number of times;

[0016] S130: Solve the coordinate transformation relationship between the camera and the end of the robotic arm using the Tsai-Lenz method , contains the rotation matrix from the camera coordinate system to the end effector coordinate system and the translation vector from the camera coordinate system to the end effector coordinate system , the specific form is:

[0017] ;

[0018] S140: Calculate the coordinate transformation relationship between the six-dimensional force sensor and the camera through the flange geometry model equipped with the six-dimensional force sensor and the camera , contains the rotation matrix from the force sensor coordinate system to the camera coordinate system , and the translation vector , the specific form is:

[0019] .

[0020] Preferably, in S200, a visual servo acceleration model is derived according to the coordinate transformation relationship between the camera and the end of the robotic arm, including:

[0021] S210: When a person applies force to the robot arm and causes it to interact with the environment, the human-guided The dynamic characteristics of the connecting rod robot can be described as:

[0022] ;

[0023] in, are the generalized variables of the joint angle, joint angular velocity and joint angular acceleration of the robot arm, represents a symmetric and positive definite inertia matrix, describes the Coriolis and centrifugal force matrices, is the gravity vector, is the joint driving torque vector, is the human-applied torque acting on the end effector, The Jacobian matrix of the robot arm with the human operating coordinate system as the end coordinate system, is the amount of environmental disturbance generated when the robot arm comes into contact with an unknown environment;

[0024] S220: Consider Visual features , the motion of image features and the corresponding camera motion speed The kinematic relationship is:

[0025] ;

[0026] in, represents the image interaction matrix, and are the movement speeds of the camera and the object respectively;

[0027] S230: When the target is stationary, When , the above formula becomes , according to the conversion relationship between the camera coordinate system and the end effector coordinate system , we get the velocity transfer relationship between the two coordinate systems , its specific form is:

[0028] ;

[0029] The speed conversion equation between the camera and the end effector is obtained as follows:

[0030] ;

[0031] At the same time, the conversion relationship between the end velocity of the robot arm and the joint angular velocity is: , Combining the above equations, the kinematic relationship between the velocity of the image feature and the joint velocity is inferred to be:

[0032] ;

[0033] in represents the task Jacobian matrix, is the Jacobian matrix of the robot arm, is the velocity transformation matrix from the end effector to the camera coordinate system;

[0034] S240: Differentiate the equation in S230 with respect to time to obtain the accelerated expression of the image feature:

[0035] ;

[0036] In order to facilitate the concise derivation of subsequent expressions, the above formula is rewritten as:

[0037] ;

[0038] in .

[0039] Preferably, in S200, the human-robotic arm interaction dynamic equation and the coordinate transformation relationship between the end of the robotic arm and the six-dimensional force sensor are combined to derive a human-robotic arm-environment interaction dynamic model in the feature space, including:

[0040] S250: Rearrange the dynamic equations of the n-link robot under human-robot interaction in S210 to obtain:

[0041] ;

[0042] S260: Substitute the image feature velocity expression in S230 and the image feature acceleration equation in S240 into the above equation, combined with the conversion relationship between the human applied force and torque in S110 above, that is, , we get the dynamic equation of human-robot-environment interaction in image space:

[0043] ;

[0044] in ;

[0045] The above formula is a set of The highly nonlinear coupled differential equations corresponding to the image features are considered. , and the transformation equation of the force applied on the force sensor projected to the camera coordinate system is: , we get the human-guided force projection equation:

[0046] ;

[0047] in, The inverse inertia matrix of the robot arm projected to the camera coordinate system;

[0048] S270: Considering the interactive system dynamic equations and the human-guided force projection equations in S260, the human-robot interaction dynamics equations in the feature space are summarized as follows:

[0049] ;

[0050] in, represents the virtual human-guided force acting on the image feature, which comes from the human-guided torque projected into the camera coordinate system. It is the virtual contact disturbance projected into the feature space when the human guides the robot to come into contact with the environment. Represents the virtual gravity, virtual Coriolis force and centrifugal force projected into the image feature space, Represents the virtual control input of the robot in the feature space.

[0051] Preferably, S300 includes:

[0052] S310: The human-robot interaction system in the image feature space is modeled as a spring-mass-damper system. In the image-based visual servoing IBVS setting, its expression is:

[0053] ;

[0054] S320: Define compliant feature errors , Defined as an auxiliary characteristic variable, when no force is applied by a human, will approach 0, which means , at this time we only need to care about the trajectory tracking problem, so we define , using the visual servoing speed controller to track the desired feature, we have:

[0055] ;

[0056] in is a positive constant. In the actual implementation process, only the velocity and acceleration of the image feature points are considered. If the position of the image feature points is given, when a person applies force, , resulting in tracking error If it is not 0, the current feature cannot keep up with the auxiliary feature, that is, it cannot completely keep up with the human-guided teaching trajectory, that is, it cannot achieve the effect of human-guided teaching;

[0057] S330: Using the admittance controller described in S320 to record the image feature position during the human-guided teaching process ,speed And acceleration The data is encoded and generalized through discrete motion dynamic primitives to generate image feature position and velocity curves. The state expression of motion dynamic primitives based on image features is:

[0058] ;

[0059] in is the characteristic target state variable, is the time scaling factor, and Considered as spring and damping coefficients, they are used to tune the performance of the motion dynamic primitive system. is a nonlinear function used to adjust the trajectory shape. is a time-independent phase variable whose expression satisfies the following canonical system:

[0060] ;

[0061] in is the attenuation coefficient, when As time approaches 0, Correspondingly, it also approaches 0. Will reach the attraction point ;

[0062] S340: The expression of the nonlinear function in the above S330 is:

[0063] ;

[0064] Its reason The basis function is a linear combination of basis functions. The basis function expression is:

[0065] ;

[0066] in represents the width of the basis function, represents the center of the basis function, is the initial state value of the characteristic variable, is the weight coefficient of the basis function, which can be obtained by minimizing the error cost function.

[0067] Preferably, S400 includes:

[0068] S410: Model the human guidance projected into the image feature space as:

[0069] ;

[0070] in, represents the image feature tracking error, is the characteristic position variable expected by humans, and is the virtual stiffness and damping matrix of the human body projected into the image feature space;

[0071] S420: Without interference Ideally, let the control input in the image feature space , bringing into the characteristic space human-robot-environment interaction dynamics equation in S270 and the above equation, we have:

[0072] ;

[0073] The above formula means that the human-guided impedance iterative learning system based on image features is stable, and the instability is caused by contact interference; in the above ideal case, the force applied by the person can guide the image features to reach the desired position with an ideal trajectory, that is, , define a reference image feature vector as ,use replace , considering the above formula, we can get:

[0074] ;

[0075] Visual Features The reference trajectory can be obtained through human-guided teaching and can be used for the subsequent visual-impedance iterative learning control strategy design;

[0076] S430: In order to further discuss the stability of the human-computer interaction system based on image features, it is assumed that the environmental contact disturbance is measurable, and a general viscoelastic model of the contact disturbance projected into the image feature space is considered:

[0077] ;

[0078] in, is the virtual parameter of interference, Represents the static position of image features;

[0079] S440: If the external virtual disturbance does not exist, only the virtual uncertain dynamics are compensated, that is, , then according to the expression in S270, we can get:

[0080] ;

[0081] If the above equation holds, the system is stable;

[0082] S450: In the case of virtual contact disturbance, according to the expression in S430, there is:

[0083] ;

[0084] when and When is a positive definite matrix, it can be clearly observed from the above expression that the image features cannot obediently follow the human-guided trajectory as represented by the expression in S420 , interference , and The introduction of makes the contact of the system unstable;

[0085] S460: In actual operation, it is impossible to accurately measure the environmental contact disturbance. Therefore, a feature-based impedance iterative learning controller is designed to help offset the contact dynamics and flexibly complete the assembly task without prior knowledge of the and .

[0086] Preferably, S500 includes:

[0087] S510: In the iterative learning process, the feature space virtual interference is spatially periodic. In the iteration process, the virtual environment contact disturbance in spatial form is:

[0088] ;

[0089] in, For parameters The spatial form satisfies ;

[0090] S520: Based on the above image feature space virtual interference, define the Iteration image feature tracking error , the image feature space impedance controller is designed as:

[0091] ;

[0092] in, , control parameter matrix Keep the impedance parameters consistent for each iteration. The update law is:

[0093] ;

[0094] ;

[0095] ;

[0096] in, represents a positive constant matrix, represents the actual iteration time of iterative learning, represents the upper limit of iteration time, ,in , The robot is currently Dimensional speed;

[0097] S530: The controller , And the expression Substitute the interaction dynamics equations in S270 into:

[0098] ;

[0099] in,

[0100] ;

[0101] By implementing image feature-based impedance iterative learning control in the contact assembly task, When virtual interference is eliminated to ensure the stability of the closed-loop system.

[0102] The present invention aims to propose an impedance iterative learning control method for a vision-force fusion manipulator under human guidance, referring to the fact that humans can perceive changes in the outside world through multi-source sensor information such as vision and force and adjust corresponding actions accordingly. By equipping the manipulator with vision and force sensing equipment, combining force and vision in the feature space, realizing the fusion control of force and vision, analyzing the robot-environment interaction dynamics equation, and solving the visual servo acceleration model, combining the above equations to establish a human-manipulator-environment interaction dynamics model in the image feature space; collecting the image feature position and velocity curve of the robot guiding the assembly task, using motion dynamic primitives to encode and generalize the feature position and velocity curve; and starting from the mechanism of human-environment interaction, designing an impedance iterative learning controller based on image features as control input, learning the impedance characteristics of human operation tasks, and learning the identified contact dynamics by accessing human guidance, thereby offsetting the uncertain dynamics generated by contact, and realizing autonomous flexible operation of the manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is a flow chart of an impedance iterative learning control method of a vision-force fusion manipulator under human guidance in one embodiment of the present invention;

[0104] Figure 2 This is a control block diagram of a vision-force fusion impedance iterative learning algorithm based on image features in one embodiment of the present invention. DETAILED DESCRIPTION

[0105] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0106] In one embodiment, Figure 1 As shown, a human-guided vision-force fusion robot arm impedance iterative learning control method comprises the following steps:

[0107] S100: Obtain the coordinate conversion relationship between the camera carried on the end effector of the robotic arm and the end of the robotic arm through hand-eye calibration, and solve the coordinate relationship between the six-dimensional force sensor and the end of the robotic arm through geometric relationship;

[0108] S200: Based on the coordinate transformation relationship between the camera and the end of the robotic arm, the visual servo acceleration model is derived. Combining the dynamic equation of human-robotic arm interaction and the coordinate transformation relationship between the end of the robotic arm and the six-dimensional force sensor, the human-robotic arm-environment interaction dynamics model in the image feature space is derived.

[0109] S300: The robot is guided to complete the assembly task by a human-guided-admittance controller based on image features, during which the image feature position and speed data are recorded, and the image feature position and speed curve is encoded and generated using the motion dynamic primitive DMP. The starting point and end point of the curve are adjusted by changing the DMP parameters, and the image feature trajectory curve is regenerated to achieve trajectory generalization;

[0110] S400: Model the human force and environmental contact disturbance, and conduct theoretical stability analysis on the human-robot-environment interaction system;

[0111] S500: Combining the human-robot-environment interaction dynamics model in the feature space derived from S200 and the generalized image feature trajectory curve of S300, an impedance iterative learning law based on image features is designed. The image feature velocity curve is iteratively learned and the impedance control parameters are updated to characterize the human-guided impedance characteristics. At the same time, visual servoing is used to correct the tracking trajectory online. After a certain number of iterative updates, the robot can successfully track the human-guided feature position curve and complete the assembly task.

[0112] The above-mentioned human-guided vision-force fusion robot impedance iterative learning control method (1) analyzes the robot-environment interaction dynamics equation and solves the visual servo acceleration model, and combines the above equations to establish the image feature space human-robot-environment interaction dynamics model; (2) collects the image feature position and velocity curve of the human-guided robot to complete the assembly task, and uses motion dynamic primitives to encode and generalize the feature position and velocity curve; (3) designs an impedance iterative learning controller based on image features as control input, learns the impedance characteristics of the human during the task, identifies unknown contact dynamics, offsets the identified interference in the feature space, and realizes flexible assembly operations; and solves the problems of the existing robot arm impedance iterative learning assembly operation under vision-force fusion (1) the existence of human-robot-environment coupled nonlinear dynamics; (2) the existence of unknown contact dynamics in dense contact assembly tasks; (3) and the difficulty in generalizing assembly scenarios, which requires re-learning for different scenarios.

[0113] In one embodiment, S100 includes:

[0114] S110: The end effector of the robot arm is equipped with a six-dimensional force sensor and an RGBD camera.

[0115] S120: Use the RGBD camera to capture an image of a calibration plate with N points, and record the corresponding position and posture of the end of the robotic arm at this time; repeat this step for a preset number of times;

[0116] S130: Solve the coordinate transformation relationship between the camera and the end of the robotic arm using the Tsai-Lenz method , contains the rotation matrix from the camera coordinate system to the end effector coordinate system and the translation vector from the camera coordinate system to the end effector coordinate system , the specific form is:

[0117] ;

[0118] S140: Calculate the coordinate transformation relationship between the six-dimensional force sensor and the camera through the flange geometry model equipped with the six-dimensional force sensor and the camera , contains the rotation matrix from the force sensor coordinate system to the camera coordinate system , and the translation vector , the specific form is:

[0119] .

[0120] Specifically, a six-dimensional force sensor and an RGBD camera are installed on the end effector of the robot arm at the same time. S120 is repeated for a preset number of times, wherein the preset number of times is 12 times.

[0121] In one embodiment, in S200, a visual servo acceleration model is derived according to a coordinate transformation relationship between a camera and a robot end, including:

[0122] S210: When a person applies force to the robot arm and causes it to interact with the environment, the human-guided The dynamic characteristics of the connecting rod robot can be described as:

[0123] ;

[0124] in, are the generalized variables of the joint angle, joint angular velocity and joint angular acceleration of the robot arm, represents a symmetric and positive definite inertia matrix, describes the Coriolis and centrifugal force matrices, is the gravity vector, is the joint driving torque vector, is the human-applied torque acting on the end effector, The Jacobian matrix of the robot arm with the human operating coordinate system as the end coordinate system, is the amount of environmental disturbance generated when the robot arm comes into contact with an unknown environment;

[0125] S220: Solve the visual servo dynamic equations at the acceleration level to characterize the kinematic relationship between the visual features and the joint variables, thereby merging the dynamic equations of the manipulator with the dynamic equations of the visual system; Consider Visual features , the motion of image features and the corresponding camera motion speed The kinematic relationship is:

[0126] ;

[0127] in, represents the image interaction matrix, and are the movement speeds of the camera and the object respectively;

[0128] S230: When the target is stationary, When , the above formula becomes , according to the conversion relationship between the camera coordinate system and the end effector coordinate system , we get the velocity transfer relationship between the two coordinate systems , its specific form is:

[0129] ;

[0130] The speed conversion equation between the camera and the end effector is obtained as follows:

[0131] ;

[0132] At the same time, the conversion relationship between the end velocity of the robot arm and the joint angular velocity is: , Combining the above equations, the kinematic relationship between the velocity of the image feature and the joint velocity is inferred to be:

[0133] ;

[0134] in represents the task Jacobian matrix, is the Jacobian matrix of the robot arm, is the velocity transformation matrix from the end effector to the camera coordinate system; in this formula, is a constant matrix because the camera is rigidly connected to the end effector of the robot;

[0135] S240: Differentiate the equation in S230 with respect to time to obtain the accelerated expression of the image feature:

[0136] ;

[0137] In order to facilitate the concise derivation of subsequent expressions, the above formula is rewritten as:

[0138] ;

[0139] in .

[0140] In one embodiment, in S200, the human-robot interaction dynamics equation and the coordinate transformation relationship between the end of the robot arm and the six-dimensional force sensor are combined to derive a human-robot-environment interaction dynamics model in the feature space, including:

[0141] S250: Rearrange the dynamic equations of the n-link robot under human-robot interaction in S210 to obtain:

[0142] ;

[0143] S260: Substitute the image feature velocity expression in S230 and the image feature acceleration equation in S240 into the above equation, combined with the conversion relationship between the human applied force and torque in S110 above, that is, , we get the dynamic equation of human-robot-environment interaction in image space:

[0144] ;

[0145] in ;

[0146] The above formula is a set of The highly nonlinear coupled differential equations corresponding to the image features are considered. , and the transformation equation of the force applied on the force sensor projected to the camera coordinate system is: , we get the human-guided force projection equation:

[0147] ;

[0148] in, The inverse inertia matrix of the robot arm projected to the camera coordinate system;

[0149] S270: Considering the interactive system dynamic equations and the human-guided force projection equations in S260, the human-robot interaction dynamics equations in the feature space are summarized as follows:

[0150] ;

[0151] in, represents the virtual human-guided force acting on the image feature, which comes from the human-guided torque projected into the camera coordinate system. It is the virtual contact disturbance projected into the feature space when the human guides the robot to come into contact with the environment. Represents the virtual gravity, virtual Coriolis force and centrifugal force projected into the image feature space, Represents the virtual control input of the robot in the feature space.

[0152] In one embodiment, S300 includes:

[0153] S310: The human hand has spring-like properties due to its viscoelasticity and stretching response. Rigid robots can use this property to achieve compliant contact tasks and avoid damage to the work object. The human-robot interaction system in the image feature space is modeled as a spring-mass-damper system. In the image-based visual servo IBVS setting, its expression is:

[0154] ;

[0155] S320: Define compliant feature errors , Defined as an auxiliary characteristic variable, when no force is applied by a human, will approach 0, which means , at this time we only need to care about the trajectory tracking problem, so we define , using the visual servoing speed controller to track the desired feature, we have:

[0156] ;

[0157] in is a positive constant. In the actual implementation, only the velocity and acceleration of the image feature points are considered. If the position of the image feature point is given, when a person applies force, , resulting in tracking error If it is not 0, the current feature cannot keep up with the auxiliary feature, that is, it cannot completely keep up with the human-guided teaching trajectory, that is, it cannot achieve the effect of human-guided teaching;

[0158] S330: Using the admittance controller described in S320 to record the image feature position during the human-guided teaching process ,speed And acceleration The data is encoded and generalized through discrete motion dynamic primitives to generate image feature position and velocity curves. The state expression of motion dynamic primitives based on image features is:

[0159] ;

[0160] in is the characteristic target state variable, is the time scaling factor, and Considered as spring and damping coefficients, they are used to tune the performance of the motion dynamic primitive system. is a nonlinear function used to adjust the trajectory shape. is a time-independent phase variable whose expression satisfies the following canonical system:

[0161] ;

[0162] in is the attenuation coefficient, when As time approaches 0, Correspondingly, it also approaches 0. Will reach the attraction point ;

[0163] S340: The expression of the nonlinear function in the above S330 is:

[0164] ;

[0165] Its reason The basis function is a linear combination of basis functions. The basis function expression is:

[0166] ;

[0167] in represents the width of the basis function, represents the center of the basis function, is the initial state value of the characteristic variable, is the weight coefficient of the basis function, which can be obtained by minimizing the error cost function.

[0168] In one embodiment, in order to illustrate the necessity and rationality of the designed impedance iterative learning controller based on image features, it is necessary to first perform a simple stability analysis on the system. and interference Acts on image features at the same time. It is considered to play a "positive" role, guiding image features to a specified location. Playing a "negative" role, which may cause unstable contact, S400 includes:

[0169] S410: Model the human guidance projected into the feature space as:

[0170] ;

[0171] in, represents the image feature tracking error, is the characteristic position variable expected by humans, and is the virtual stiffness and damping matrix of the human body projected into the image feature space;

[0172] S420: Without interference Ideally, let the control input in the image feature space , bringing into the characteristic space human-robot-environment interaction dynamics equation in S270 and the above equation, we have:

[0173] ;

[0174] The above formula means that the human-guided impedance iterative learning system based on image features is stable, and the instability is caused by contact interference; in the above ideal case, the force applied by the person can guide the image features to reach the desired position with an ideal trajectory, that is, , define a reference image feature vector as ,use replace , considering the above formula, we can get:

[0175] ;

[0176] Visual Features The reference trajectory can be obtained through human-guided teaching and can be used for the subsequent visual-impedance iterative learning control strategy design;

[0177] S430: In order to further discuss the stability of the human-computer interaction system based on image features, it is assumed that the environmental contact disturbance is measurable, and a general viscoelastic model of the contact disturbance projected into the image feature space is considered:

[0178] ;

[0179] in, is the virtual parameter of interference, Represents the static position of image features;

[0180] S440: If the external virtual disturbance does not exist, only the virtual uncertain dynamics are compensated, that is, , then according to the expression in S270, we can get:

[0181] ;

[0182] If the above equation holds, the system is stable;

[0183] S450: In the case of virtual contact disturbance, according to the expression in S430, there is:

[0184] ;

[0185] when and When is a positive definite matrix, it can be clearly observed from the above expression that the image features cannot obediently follow the human-guided trajectory as represented by the expression in S420 , interference , and The introduction of makes the contact of the system unstable;

[0186] S460: In actual operation, it is impossible to accurately measure the environmental contact disturbance. Therefore, a feature-based impedance iterative learning controller is designed to help offset the contact dynamics and flexibly complete the assembly task without prior knowledge of the and .

[0187] Through the stability analysis of S400 above, it can be observed that as long as the appropriate impedance controller parameters are designed so that the impedance controller based on image features can offset the influence of interference, the system can be guaranteed to be stable. Here, the present invention introduces iterative learning control. Iterative learning uses past execution experience to improve future control performance and improves tracking accuracy through multiple iterations, thereby greatly reducing the dependence on the precise model of the system. The combination of iterative learning and impedance control can make the system better adapt to environmental changes. Iterative learning helps the robot arm remember and adapt to changes in repetitive tasks, while impedance control can adapt to immediate interaction and contact tasks.

[0188] In one embodiment, S500 includes:

[0189] S510: In the iterative learning process, the feature space virtual interference is spatially periodic. In the iteration process, the virtual environment contact disturbance in spatial form is:

[0190] ;

[0191] in, For parameters The spatial form satisfies ;

[0192] S520: Based on the above image feature space virtual interference, define the Iteration image feature tracking error , the image feature space impedance controller is designed as:

[0193] ;

[0194] in, , control parameter matrix Keep the impedance parameters consistent for each iteration. The update law is:

[0195] ;

[0196] ;

[0197] ;

[0198] in, represents a positive constant matrix, represents the actual iteration time of iterative learning, represents the upper limit of iteration time, ,in , The robot is currently Dimensional speed;

[0199] S530: The controller , And the expression Substitute the interaction dynamics equations in S270 into:

[0200] ;

[0201] in,

[0202] ;

[0203] By implementing image feature-based impedance iterative learning control in the contact assembly task, When virtual interference is eliminated to ensure the stability of the closed-loop system.

[0204] Specifically, the control block diagram of the entire method is as follows Figure 2 As shown. In summary, the present invention summarizes the following theorem: Considering the feature space human-robot-environment interaction dynamics described in S200, the generalization equation of the human-guided image feature trajectory in S300, the impedance iterative learning control formula in S500 and the impedance parameter iterative update law, the combination of the above equations ensures the system stability in the above S530, eliminates the environmental contact disturbance, and can achieve a certain degree of scene generalization within the visual framework.

[0205] In one embodiment, after S500, the method further includes:

[0206] S600: Analyze and prove the stability of the human-robot-environment interaction system using Lyapunov stability proof theory.

[0207] In one embodiment, S600 includes:

[0208] S610: Define Lyapunov candidate function:

[0209] ;

[0210] ;

[0211] ;

[0212] ;

[0213] ;

[0214] ;

[0215] ;

[0216] ;

[0217] in represents the feature tracking error, , and The composite energy function related to the impedance coefficient estimation error is defined, with subscript The Lyapunov candidate function is related to space but not time. , , The estimated error matrices are , No. Column vector; No. elements; is the symbolic function, They are No. elements;

[0218] S620: Because

[0219] ;

[0220] Similarly, we can get

[0221] ;

[0222] in is a time-dependent integral variable, so it can be inferred that the Lyapunov equation in S610 is positive definite, that is, ;

[0223] S630: At iteration time , feature tracking error ,right Differentiating with respect to time, we obtain:

[0224] ;

[0225] Substituting the equation in S510 into the above formula, we have:

[0226] ;

[0227] Its integral form is:

[0228] ;

[0229] definition:

[0230] ;

[0231] It can be inferred that:

[0232] ;

[0233] S640: Similarly, definition:

[0234] ;

[0235] Bring it into S620 The expressions are:

[0236] ;

[0237] definition:

[0238] ;

[0239] get:

[0240] ;

[0241] Bring it into the above The expressions are:

[0242] ;

[0243] From the above formula, we can get:

[0244] ;

[0245] Similarly, we can get:

[0246] ;

[0247] ;

[0248] S650: Combined with the inequality in S640, we have:

[0249] ;

[0250] Substituting the impedance parameter update law expressed by S520 into the above formula, we have:

[0251] ;

[0252] in, , and They are , and The elements;

[0253] S660: When According to the characteristic tracking error Definition of, and S630 Points form and The definitions are:

[0254] ;

[0255] when When the impedance estimation parameters stop updating, , so the following inequality holds:

[0256] ;

[0257] Similarly, referring to the expression in S650, we have:

[0258] ;

[0259] S670: Comprehensively consider the S650 and S660 The final form of the inequality is, exist and The case is established, because when When , the initial energy of the iterative impedance learning system based on image features is bounded, which means It will decrease monotonically with the increase of the number of iterations. The definition of , which is a positive definite function, yields:

[0260] ;

[0261] exist In the case of:

[0262] ;

[0263] Therefore, based on the S530 expression, we can get:

[0264] ;

[0265] By analyzing the above results, it can be seen that the iterative impedance learning method based on image features can eliminate the unknown environmental contact disturbance, and the robot will follow the trajectory guided by humans with the assistance of the vision system.

[0266] Compared with the prior art, the advantages of the present invention are:

[0267] (1) This paper proposes a human-guided visual impedance fusion framework in the image feature space for the first time. By combining it with image-based visual servoing (IBVS), the human-guided force and contact disturbance are projected into the image feature space, thereby modeling the interactive dynamics of human-robot-environment in the feature space.

[0268] (2) Introducing motion dynamic primitive technology, encoding and generalizing the image feature point trajectory of the human-guided robot to complete assembly, and establishing a generalization framework for vision-force fusion impedance iterative learning control based on image features;

[0269] (3) Based on the above framework, the present invention proposes an impedance iterative learning control technology based on image features to improve the stability and accuracy of the contact task. The impedance parameters are updated by learning the characteristic trajectory guided by humans to offset the contact interference. Finally, the stability of the human-robot-environment interaction system is proved through the Lyapunov stability theory.

[0270] The above is a detailed introduction to the human-guided visual-force fusion robotic arm impedance iterative learning control method provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A human-guided vision-force fusion robot arm impedance iterative learning control method, characterized in that: The method comprises the following steps: S100: Obtain the coordinate conversion relationship between the camera carried on the end effector of the robotic arm and the end of the robotic arm through hand-eye calibration, and solve the coordinate relationship between the six-dimensional force sensor and the end of the robotic arm through geometric relationship; S200: Based on the coordinate transformation relationship between the camera and the end of the robotic arm, the visual servo acceleration model is derived. Combining the dynamic equation of human-robotic arm interaction and the coordinate transformation relationship between the end of the robotic arm and the six-dimensional force sensor, the human-robotic arm-environment interaction dynamics model in the feature space is derived. S300: The robot is guided to complete the assembly task by a human-guided-admittance controller based on image features, during which the image feature position and speed data are recorded, and the image feature position and speed curve is encoded and generated using the motion dynamic primitive DMP. The starting point and end point of the curve are adjusted by changing the DMP parameters, and the image feature trajectory curve is regenerated to achieve trajectory generalization; S400: Model the human force and environmental contact disturbance, and conduct theoretical stability analysis on the human-robot-environment interaction system; S500: Combining the human-robot-environment interaction dynamics model in the feature space derived from S200 and the generalized image feature trajectory curve of S300, an impedance iterative learning law based on image features is designed. The image feature velocity curve is iteratively learned and the impedance control parameters are updated to characterize the human-guided impedance characteristics. At the same time, visual servoing is used to correct the tracking trajectory online. After a certain number of iterative updates, the robot can successfully track the human-guided feature position curve and complete the assembly task.

2. The method according to claim 1, characterized in that S100 includes: S110: The end effector of the robot arm is equipped with a six-dimensional force sensor and an RGBD camera. S120: Use the RGBD camera to capture an image of a calibration plate with N points, and record the corresponding position and posture of the end of the robotic arm at this time; repeat this step for a preset number of times; S130: Solve the coordinate transformation relationship between the camera and the end of the robotic arm using the Tsai-Lenz method , contains the rotation matrix from the camera coordinate system to the end effector coordinate system and the translation vector from the camera coordinate system to the end effector coordinate system , the specific form is: ; S140: Calculate the coordinate transformation relationship between the six-dimensional force sensor and the camera through the flange geometry model equipped with the six-dimensional force sensor and the camera , contains the rotation matrix from the force sensor coordinate system to the camera coordinate system , and the translation vector , the specific form is: 。 3. The method according to claim 2, characterized in that In S200, the visual servo acceleration model is derived based on the coordinate transformation relationship between the camera and the end of the robotic arm, including: S210: When a person applies force to the robot arm and causes it to interact with the environment, the human-guided The dynamic characteristics of the connecting rod robot can be described as: ; in, are the generalized variables of the joint angle, joint angular velocity and joint angular acceleration of the robot arm, represents a symmetric and positive definite inertia matrix, describes the Coriolis and centrifugal force matrices, is the gravity vector, is the joint driving torque vector, is the human-applied torque acting on the end effector, The Jacobian matrix of the robot arm with the human operating coordinate system as the end coordinate system, is the amount of environmental disturbance generated when the robot arm comes into contact with an unknown environment; S220: Consider Visual features , the motion of image features and the corresponding camera motion speed The kinematic relationship is: ; in, represents the image interaction matrix, and are the movement speeds of the camera and the object respectively; S230: When the target is stationary, When , the above formula becomes , according to the conversion relationship between the camera coordinate system and the end effector coordinate system , we get the velocity transfer relationship between the two coordinate systems , its specific form is: ; The speed conversion equation between the camera and the end effector is obtained as follows: ; At the same time, the conversion relationship between the end velocity of the robot arm and the joint angular velocity is: , Combining the above equations, the kinematic relationship between the velocity of the image feature and the joint velocity is inferred to be: ; in represents the task Jacobian matrix, is the Jacobian matrix of the robot arm, is the velocity transformation matrix from the end effector to the camera coordinate system; S240: Differentiate the equation in S230 with respect to time to obtain the accelerated expression of the image feature: ; In order to facilitate the concise derivation of subsequent expressions, the above formula is rewritten as: ; in .

4. The method according to claim 3, characterized in that S200 combines the dynamic equations of human-robot interaction and the coordinate transformation relationship between the end of the robot and the six-dimensional force sensor to derive the human-robot-environment interaction dynamics model in the feature space, including: S250: Rearrange the dynamic equations of the n-link robot under human-robot interaction in S210 to obtain: ; S260: Substitute the image feature velocity expression in S230 and the image feature acceleration equation in S240 into the above equation, combined with the conversion relationship between the human applied force and torque in S110 above, that is, , we get the dynamic equation of human-robot-environment interaction in image space: ; in ; The above formula is a set of The highly nonlinear coupled differential equations corresponding to the image features are considered. , and the transformation equation of the force applied on the force sensor projected to the camera coordinate system is: , we get the human-guided force projection equation: ; in, The inverse inertia matrix of the robot arm projected to the camera coordinate system; S270: Considering the interactive system dynamic equations and the human-guided force projection equations in S260, the human-robot interaction dynamics equations in the feature space are summarized as follows: ; in, represents the virtual human-guided force acting on the image feature, which comes from the human-guided torque projected into the camera coordinate system. It is the virtual contact disturbance projected into the feature space when the human guides the robot to come into contact with the environment. Represents the virtual gravity, virtual Coriolis force and centrifugal force projected into the image feature space, Represents the virtual control input of the robot in the feature space.

5. The method according to claim 4, characterized in that S300 includes: S310: The human-robot interaction system in the image feature space is modeled as a spring-mass-damper system. In the image-based visual servoing IBVS setting, its expression is: ; S320: Define compliant feature errors , Defined as an auxiliary characteristic variable, when no force is applied by a human, will approach 0, which means , at this time we only need to care about the trajectory tracking problem, so we define , using the visual servoing speed controller to track the desired feature, we have: ; in is a positive constant. In the actual implementation process, only the velocity and acceleration of the image feature points are considered. If the position of the image feature points is given, when a person applies force, , resulting in tracking error If it is not 0, the current feature cannot keep up with the auxiliary feature, that is, it cannot completely keep up with the human-guided teaching trajectory, that is, it cannot achieve the effect of human-guided teaching; S330: Using the admittance controller described in S320 to record the image feature position during the human-guided teaching process ,speed And acceleration The data is encoded and generalized through discrete motion dynamic primitives to generate image feature position and velocity curves. The state expression of motion dynamic primitives based on image features is: ; in is the characteristic target state variable, is the time scaling factor, and Considered as spring and damping coefficients, they are used to tune the performance of the motion dynamic primitive system. is a nonlinear function used to adjust the trajectory shape. is a time-independent phase variable whose expression satisfies the following canonical system: ; in is the attenuation coefficient, when As time approaches 0, Correspondingly, it also approaches 0. Will reach the attraction point ; S340: The expression of the nonlinear function in the above S330 is: ; Its reason The basis function is a linear combination of basis functions. The basis function expression is: ; in represents the width of the basis function, represents the center of the basis function, is the initial state value of the characteristic variable, is the weight coefficient of the basis function, which can be obtained by minimizing the error cost function.

6. The method according to claim 5, characterized in that S400 includes: S410: Model the human guidance projected into the image feature space as: ; in, represents the image feature tracking error, is the characteristic position variable expected by humans, and is the virtual stiffness and damping matrix of the human body projected into the image feature space; S420: Without interference Ideally, let the control input in the image feature space , bringing into the characteristic space human-robot-environment interaction dynamics equation in S270 and the above equation, we have: ; The above formula means that the human-guided impedance iterative learning system based on image features is stable, and the instability is caused by contact interference; in the above ideal case, the force applied by the person can guide the image features to reach the desired position with an ideal trajectory, that is, , define a reference image feature vector as ,use replace , considering the above formula, we can get: ; Visual Features The reference trajectory can be obtained through human-guided teaching and can be used for the subsequent visual-impedance iterative learning control strategy design; S430: In order to further discuss the stability of the human-computer interaction system based on image features, it is assumed that the environmental contact disturbance is measurable, and a general viscoelastic model of the contact disturbance projected into the image feature space is considered: ; in, is the virtual parameter of interference, Represents the static position of image features; S440: If the external virtual disturbance does not exist, only the virtual uncertain dynamics are compensated, that is, , then according to the expression in S270, we can get: ; If the above equation holds, the system is stable; S450: In the case of virtual contact disturbance, according to the expression in S430, there is: ; when and When is a positive definite matrix, it can be clearly observed from the above expression that the image features cannot obediently follow the human-guided trajectory as represented by the expression in S420 , interference , and The introduction of makes the contact of the system unstable; S460: In actual operation, it is impossible to accurately measure the environmental contact disturbance. Therefore, a feature-based impedance iterative learning controller is designed to help offset the contact dynamics and flexibly complete the assembly task without prior knowledge of the and .

7. The method according to claim 6, characterized in that S500 includes: S510: In the iterative learning process, the feature space virtual interference is spatially periodic. In the iteration process, the virtual environment contact disturbance in spatial form is: ; in, For parameters The spatial form satisfies ; S520: Based on the above image feature space virtual interference, define the Iteration image feature tracking error , the image feature space impedance controller is designed as: ; in, , control parameter matrix Keep the impedance parameters consistent for each iteration. The update law is: ; ; ; in, represents a positive constant matrix, represents the actual iteration time of iterative learning, represents the upper limit of iteration time, ,in , The robot is currently Dimensional speed; S530: The controller , And the expression Substitute the interaction dynamics equations in S270 into: ; in, ; By implementing image feature-based impedance iterative learning control in the contact assembly task, When virtual interference is eliminated to ensure the stability of the closed-loop system.