Human-guided vision-force fused impedance iterative learning control method for robotic arm

Abstract

A human-guided vision-force fused impedance iterative learning control method for a robotic arm, comprising: analyzing a robot-environment interaction dynamics equation, solving a visual servo acceleration model, and making use of the equation to establish a human-robotic arm-environment interaction dynamics model in an image feature space; acquiring an image feature position and speed curve of a human-guided robot completing an assembly task, and using dynamic movement primitives for coding and generalization; and designing an impedance iterative learning controller which uses image feature tracking errors as control input, learning impedance characteristics when the human-guided robot performs a contact operation, identifying unknown contact dynamics under the interaction between the robot and the environment, and counteracting identified contact interference in the feature space, so as to implement a flexible assembly operation. The control method solves the problems in existing assembly operations that human-robotic arm-environment coupling nonlinear dynamics, unknown contact dynamics of intensive contact assembly tasks and poor generalization of assembly scenarios require relearning for different scenarios, etc.

Description

Human-guided visual-force fusion mechanical arm impedance iterative learning control method

[0001] The present application claims priority to the Chinese patent application filed on July 26, 2024, with the Chinese Patent Office, the number of which is 2024110087999, and the title of which is "Human-guided visual-force fusion mechanical arm impedance iterative learning control method", the whole content or part of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application belongs to the technical field of intelligent manufacturing, in particular relates to a human-guided visual-force fusion mechanical arm impedance iterative learning control method. BACKGROUND

[0003] Complex component manufacturing, such as aviation panel assembly, precision electronic manufacturing, etc., often has problems such as narrow workspace, numerous assembly procedures, dynamic changes in the environment, etc. Most of the work is done by humans or robots using fixed algorithms to complete single repetitive actions. In the face of changes in work objects and manufacturing environments, and complex unstructured manufacturing environments, traditional robot manufacturing cannot flexibly meet the intelligent needs of modern robot manufacturing industry.

[0004] At present, vision has been introduced into industrial manufacturing to a large extent. Visual sensors can perceive surrounding environmental information and flexibly adapt to scene changes, so they can be used to detect work environments, locate workpiece positions, or detect workpiece quality, etc. In addition, considering that real assembly tasks are usually contact-rich tasks, robots relying on single visual sensor data cannot meet the requirements of fine operations. For example, during aviation panel assembly, since the panel is a weakly rigid component, a robot with high rigidity may cause vibration when it contacts the panel, affecting the assembly quality, and too much force may damage the workpiece. Force sensors can help robots perceive contact force information when interacting with the environment, and help robots identify the interaction dynamics when contacting the environment. Maintaining the stability of contact tasks is another problem to be solved. The coupling dynamics between the robot and the environment will cause the robot to be unable to stably transition from a free state to a contact state with the workpiece, and achieve high-precision control effect. To solve the above problems, we consider multi-sensor information fusion to supplement the lack of contact task perception. Therefore, we introduce visual sensors and force sensors to perceive environmental changes and contact forces, and realize visual-force fusion adaptive flexible assembly of the mechanical arm to solve complex component manufacturing environmental dynamics, contact nonlinear coupling, etc., improve the adaptability and intelligence of the robot, and flexibly respond to various manufacturing task requirements.

[0005] In summary, how to fuse multi-sensor information with heterogeneous data dimensions and frequency mismatch, and how to overcome the coupled interaction dynamics between human-robot-environment to achieve adaptive robot assembly in dense contact scenarios are the current difficult problems in the field of intelligent manufacturing. SUMMARY

[0006] In view of the above technical problems, the present application provides a human-guided visual-force fusion robot impedance iterative learning control method. The present application introduces a visual sensor and a force sensor to perceive environmental changes and contact forces, realizes visual-force fusion adaptive flexible assembly of the robot to solve problems such as complex component manufacturing environment dynamics, contact nonlinear coupling, and improves the adaptability and intelligence of the robot to flexibly cope with various manufacturing task requirements.

[0007] The technical scheme adopted by the present application to solve its technical problems is:

[0008] A human-guided visual-force fusion robot impedance iterative learning control method, the method comprising the following steps:

[0009] S100: Obtain the coordinate conversion relationship between the camera mounted on the robot end effector and the robot end through hand-eye calibration, and solve the coordinate relationship between the six-dimensional force sensor and the robot end through geometric relationship;

[0010] S200: Derive a visual servo acceleration model according to the coordinate conversion relationship between the camera and the robot end, combine the human-robot interaction dynamic equation and the coordinate conversion relationship between the robot end and the six-dimensional force sensor, and derive a human-robot-environment interaction dynamics model in the feature space;

[0011] S300: Guide the robot to complete the assembly task through the human-guided-inertial controller based on image features, record the image feature position and velocity data during the process, use the motion dynamic primitive DMP to encode and generate the image feature position and velocity curve, and adjust the starting point and end point position of the curve by changing the DMP parameters, regenerate the image feature trajectory curve, and realize trajectory generalization;

[0012] S400: Model the force on the human and the environmental contact disturbance, and theoretically analyze the stability of the human-robot-environment interaction system;

[0013] S500: combine the human-robot-environment interaction dynamics model in the feature space derived in S200 and the generalized image feature trajectory curve in S300, design an image feature based impedance iterative learning law, update the impedance control parameters through iterative learning of the image feature velocity curve to represent the human-guided impedance characteristics, and simultaneously correct the tracking trajectory online using visual servoing. After a certain number of iterations, the robot arm can successfully track the human-guided feature position curve and complete the assembly task.

[0014] Preferably, S100 includes:

[0015] S110: a six-axis force sensor and an RGBD camera are mounted on the robot arm end effector at the same time;

[0016] S120: use the RGBD camera to take an image of the calibration board with N points, and record the corresponding robot arm end pose at this time; repeat this step a predetermined number of times;

[0017] S130: solve the coordinate transformation relationship between the camera and the robot arm end by Tsai-Lenz method e T c , including the rotation matrix of the camera coordinate system to the end effector coordinate system e R c and the translation vector of the camera coordinate system to the end effector coordinate system e t c , the specific form is:

[0018] S140: calculate the coordinate transformation relationship between the six-axis force sensor and the camera c T h , including the rotation matrix of the force sensor coordinate system to the camera coordinate system c R h and the translation vector c t h , the specific form is:

[0019] Preferably, in S200, the visual servoing acceleration model is derived according to the coordinate transformation relationship between the camera and the robot arm end, including:

[0020] S210: when the human applies force to the robot arm and interacts with the environment, the dynamics of the human-guided n-link robot arm in joint space can be described as:

[0021] where, are the generalized variables of the joint angle, joint angular velocity and joint angular acceleration of the robot arm, B(q)∈R n×nThis represents a symmetric and positive definite inertia matrix. The Coriolis force and centrifugal force matrices are described, G(q)∈R n It is the gravity vector, τ∈R n It is the joint driving torque vector. J is the torque applied by the human to the end effector. h The Jacobian matrix of the robotic arm, where the operator's coordinate system is the end-effector coordinate system, is d∈R. n It is the amount of environmental disturbance generated when the robotic arm comes into contact with the unknown environment;

[0022] S220: Consider m visual features s∈R m The motion of image features and the corresponding camera motion speed c v∈R 6 The kinematic relationship is as follows:

[0023] Among them, L s ∈R m×6 Represents the image interaction matrix. c v c and c v o These are the speeds of the camera and the object, respectively.

[0024] S230: When the target is stationary... c v o When = 0, the above expression becomes Based on the transformation relationship between the camera coordinate system and the end effector coordinate system e T c The velocity transmission relationship in the two coordinate systems is obtained. e M c ∈R 6×6 Its specific form is as follows:

[0025] The equation for the velocity conversion relationship between the camera and the end effector is: v e = e M c v c ;

[0026] Meanwhile, the conversion relationship between the end effector velocity and the joint angular velocity of the robotic arm is as follows: Based on the above equations, the kinematic relationship between the velocity representing image features and the joint velocity can be deduced as follows:

[0027] J s ∈R m×n The Jacobian matrix representing the task, J e ∈R 6×n It is the Jacobian matrix of the robotic arm. It is the velocity transformation matrix from the end effector to the camera coordinate system;

[0028] S240: Differentiating the equation in S230 with respect to time yields an accelerated form of the image features:

[0029] To facilitate the concise derivation of subsequent expressions, the above expression is rewritten as:

[0030] in

[0031] Preferably, in S200, by combining the human-robotic arm interaction dynamic equations and the coordinate transformation relationship between the robotic arm end effector and the six-dimensional force sensor, a human-robotic arm-environment interaction dynamic model in the feature space is derived, including:

[0032] S250: Rearranging the dynamic equations of the n-link robotic arm under the human-robotic arm interaction in S210, we have:

[0033] S260: Substitute the image feature velocity expression from S230 and the image feature acceleration equation from S240 into the above equation, and combine this with the force and torque conversion relationship applied by the person in S110 above, i.e. The dynamic equations for the human-robotic arm-environment interaction in image space are obtained as follows:

[0034] in

[0035] The above equation is a set of highly nonlinear coupled differential equations corresponding to m image features. Consider the equation... And the transformation equation for the force projection applied to the force sensor into the camera coordinate system is: The projection equation of the human guiding force is obtained: J s B(q) -1 τ h =L s B c (q) -1 f ch ;

[0036] in, The inverse inertia matrix is ​​the projection of the robotic arm onto the camera coordinate system.

[0037] S270: Considering the dynamic equations of the interactive system and the human-guided force projection equations in S260, the human-robotic arm interaction dynamic equations in the characteristic space are summarized as follows:

[0038] Among them, f sh=L s B c (q) -1 f ch This represents the virtual human guide force acting on image features, which originates from the human guide torque projected onto the camera coordinate system, d. s =J s B(q) -1 d is the virtual contact perturbation projected into the feature space when the robot, guided by a human, comes into contact with the environment; b s =J s B(q) -1 b represents the virtual gravity, virtual Coriolis force, and centrifugal force in the characteristic space, u s =J s B(q) -1 τ represents the virtual control input of the robotic arm in the feature space.

[0039] Preferably, S300 includes:

[0040] S310: The human-robotic arm interaction system in the image feature space is modeled as a spring-mass-damped system. Under the image-based visual servoing IBVS setting, its expression is:

[0041] S320: Define compliance characteristic error Defined as an auxiliary feature variable, when no one applies a force. It will approach 0, which means s d =s c At this point, we only need to focus on the trajectory tracking problem; therefore, we define e. s =s c -s, using a visual servo speed controller to track desired features, has:

[0042] Where λ is a positive constant, in actual execution, only the velocity and acceleration of image feature points are considered. Given the position of the image feature points, when a person applies a force, f sh ≠0, resulting in a tracking error e s If the value is not 0, the current feature cannot keep up with the auxiliary feature, that is, it cannot fully keep up with the human-guided teaching trajectory, and thus cannot achieve the effect of human-guided teaching.

[0043] S330: Utilizes the admittance controller described in S320 to record the image feature positions s during the human-guided teaching process. h ,speed and acceleration The data is encoded and generalized using discrete motion dynamic primitives to generate image feature position and velocity curves. The state expression of the motion dynamic primitive based on image features is as follows:

[0044] Where g is the feature target state variable, γ is the time scaling factor, and α s and β s Considered as the spring and damping coefficients, used to adjust the performance of the motion dynamic primitive system, F(x) is a nonlinear function used to adjust the trajectory shape, where x is a time-independent phase variable, and its expression satisfies the following canonical system:

[0045] Where c is the decay coefficient, and as x approaches 0 over time, F(x) also approaches 0 accordingly. It will reach the attraction point [g,0];

[0046] S340: The expression for the nonlinear function in S330 above is:

[0047] It is composed of a linear combination of N basis functions, and the expression for the basis functions is f. i (x)=exp(-l i (x-μ i ) 2 ), i = 1, 2, ..., N;

[0048] Among them l i μ represents the width of the basis functions. i Let θ represent the center of the basis functions, s0 be the initial state value of the characteristic variables, and Θ be the initial state value of the characteristic variables. i These are the weighting coefficients of the basis functions, which can be obtained by minimizing the error cost function.

[0049] Preferably, S400 includes:

[0050] S410: Modeling human guidance projected onto the image feature space as

[0051] Among them, e sh =ss h ∈R m Indicates the image feature tracking error, s h ∈R m It is the characteristic location variable expected by humans, K sh ∈R m×m and K shv ∈R m×m It is the virtual stiffness and damping matrix of the human body projected onto the image feature space; S420: in the absence of interference, i.e., d s In the ideal case where = 0, the control input in the image feature space Substituting the characteristic space human-robotic arm-environment interaction dynamics equations from S270 and the above equation, we have:

[0052] The above equation implies that the human-guided impedance iterative learning control system based on image features is stable; instability is caused by contact disturbances. Under the ideal conditions described above, the force applied by the person can guide the image features to the desired position along an ideal trajectory, i.e., e. sh ≈0, define a reference image feature vector as s r ∈R m , using s r Replace s h Considering the above equation, we can obtain:

[0053] Visual features s r The reference trajectory can be obtained through human-guided teaching and can be used for subsequent visual impedance iterative learning control strategy design;

[0054] S430: To further discuss the stability of image feature-based human-computer interaction systems, it is assumed that environmental contact disturbances are measurable, and a general viscoelastic model of contact disturbances projected onto the image feature space is considered:

[0055] Among them, K s ,K sv ∈R m×m s is a virtual parameter for interference. e ∈R m Indicates the stationary position of image features;

[0056] S440: If the external virtual disturbance does not exist, only the virtual uncertain dynamics are compensated, i.e. Based on the expression in S270, we can conclude that:

[0057] If the above equation holds true, then the system is stable;

[0058] S450: In the case of virtual contact disturbance, according to the expression in S430, we have:

[0059] When K s and K sv When the matrix is ​​positive definite, it can be clearly observed from the above expression that image features do not obediently follow the human-guided trajectory s as represented by the expression in S420. r Interference items -K s s and K s s e The introduction of this makes the system's contact unstable;

[0060] S460: In practical operation, it is impossible to accurately measure environmental contact disturbances. Therefore, a feature-based impedance iterative learning controller is designed to help counteract contact dynamics while flexibly completing assembly tasks without prior knowledge of d. s and s h .

[0061] Preferably, S500 includes:

[0062] S510: During iterative learning, the virtual disturbance in the feature space is spatially periodic. In the i-th iteration, the spatial form of the virtual environment contact disturbance is:

[0063] Among them, K sp ,K svp ∈R m×m For parameter K s ,K sv The spatial form satisfies K sp,i =K sp,i-1 ,K svp,i =K svp,i-1 ,s e,i =s e,i-1 ;

[0064] S520: Based on the above virtual interference in the image feature space, define the image feature tracking error e in the i-th iteration. s,i =s i -s r,i The image feature spatial impedance controller is designed as follows:

[0065] in, Control parameter matrix K e ∈R m×m The impedance parameter remains consistent for each iteration. The update law is:

[0066] Among them, B s B sv B sp ∈R m×m Represents a matrix of positive constants. This represents the actual iteration time of iterative learning. Represents the upper limit of iteration time, Λ=diag{λ 1 ,...,λ m}∈R m×m , where λ i =|v i | -1 v i It is the robot's current velocity in the i-th dimension;

[0067] S530: The above controller u s,i d s,i and expressions Substituting the interaction dynamics equations in S270, we have:

[0068] in,

[0069] By implementing image feature-based impedance iterative learning control in contact assembly tasks, when At that time, virtual interference can be eliminated. s To mitigate the impact and ensure the stability of the closed-loop system.

[0070] This invention aims to propose an impedance iterative learning control method for a human-guided vision-force fusion robotic arm. It references the human ability to perceive changes in the external environment and adjust actions accordingly through multi-source sensor information such as vision and force. By equipping the robotic arm with vision and force sensors, force and vision are combined in the feature space to achieve fused force and vision control. The invention analyzes the robot-environment interaction dynamics equations and solves the visual servo acceleration model. Based on these equations, a human-robotic arm-environment interaction dynamics model is established in the image feature space. Image feature position and velocity curves of human-guided robot assembly tasks are collected, and motion dynamic primitives are used to encode and generalize the feature position and velocity curves. Furthermore, based on the mechanism of human-environment interaction, an impedance iterative learning controller based on image features as control input is designed to learn the impedance characteristics during human task execution. By incorporating human guidance, the contact dynamics of the robot are learned and recognized, thereby offsetting the uncertain dynamics generated by contact and achieving autonomous and flexible operation of the robotic arm. Attached Figure Description

[0071] Figure 1 is a flowchart of an impedance iterative learning control method for a vision-force fusion robotic arm under human guidance according to an embodiment of the present invention;

[0072] Figure 2 is a control block diagram of a vision-force fusion impedance iterative learning algorithm based on image features in one embodiment of the present invention. Detailed Implementation

[0073] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0074] In one embodiment, as shown in Figure 1, a human-guided vision-force fusion robotic arm impedance iterative learning control method includes the following steps:

[0075] S100: The coordinate transformation relationship between the camera mounted on the end effector of the robotic arm and the end effector of the robotic arm is obtained through hand-eye calibration, and the coordinate relationship between the six-dimensional force sensor and the end effector of the robotic arm is solved through geometric relationships;

[0076] S200: Based on the coordinate transformation relationship between the camera and the robotic arm end effector, a visual servo acceleration model is derived. Combining the human-robotic arm interaction dynamic equation and the coordinate transformation relationship between the robotic arm end effector and the six-dimensional force sensor, a human-robotic arm-environment interaction dynamic model in the image feature space is derived.

[0077] S300: The robot is guided to complete the assembly task by a human-guided admittance controller based on image features. During the process, image feature position and velocity data are recorded. The DMP primitive is used to encode and generate image feature position and velocity curves. By changing the DMP parameters, the starting point and ending point of the curve are adjusted, and the image feature trajectory curve is regenerated to achieve trajectory generalization.

[0078] S400: Models human forces and environmental contact disturbances, and performs theoretical stability analysis on the human-robotic arm-environment interaction system;

[0079] S500: Combining the human-robotic arm-environment interaction dynamics model in the feature space derived by S200 and the generalized image feature trajectory curve of S300, an impedance iterative learning law based on image features is designed. By iteratively learning the image feature velocity curve, the impedance control parameters are updated to characterize the human guide impedance characteristics. At the same time, visual servoing is used to correct the tracking trajectory online. After iterative updates reach a certain number of times, the robotic arm can successfully track the human guide feature position curve and complete the assembly task.

[0080] The above-mentioned human-guided vision-force fusion robotic arm impedance iterative learning control method (1) analyzes the robot-environment interaction dynamic equation and solves the visual servo acceleration model, and establishes a human-robotic arm-environment interaction dynamic model in the image feature space by combining the above equations; (2) collects the image feature position and velocity curves of the robot guided by the human to complete the assembly task, and encodes and generalizes the feature position and velocity curves using motion dynamic primitives; (3) designs an impedance iterative learning controller based on image features as control input, learns the impedance characteristics of the human when performing the task, identifies unknown contact dynamics, cancels the identified interference in the feature space, and realizes flexible assembly operation; solves the problems in the existing human-robotic arm-environment coupling nonlinear dynamics in the impedance iterative learning assembly operation under vision-force fusion of robotic arms (1) the existence of human-robotic arm-environment coupling nonlinear dynamics; (2) the existence of unknown contact dynamics in dense contact assembly tasks; (3) the difficulty in generalizing the assembly scene and the need to relearn for different scenes.

[0081] In one embodiment, S100 includes:

[0082] S110: A six-dimensional force sensor and an RGBD camera are simultaneously installed on the end effector of the robotic arm;

[0083] S120: Use an RGBD camera to capture an image of the calibration board with N points and record the corresponding end-effector pose; repeat this step a preset number of times.

[0084] S130: The coordinate transformation relationship between the camera and the robotic arm end effector is solved using the Tsai-Lenz method. e T c The rotation matrix from the camera coordinate system to the end effector coordinate system. e R c Translation vector from camera coordinate system to end effector coordinate system e t c The specific form is as follows:

[0085] S140: Calculate the coordinate transformation relationship between the six-dimensional force sensor and the camera using a flange geometric model equipped with a six-dimensional force sensor and camera. c T h Includes the rotation matrix from the force sensing coordinate system to the camera coordinate system. c R h and translation vector c t h The specific form is as follows:

[0086] Specifically, a six-dimensional force sensor and an RGBD camera are simultaneously installed on the end effector of the robotic arm. The S120 sequence is repeated a preset number of times, where the preset number of times is 12.

[0087] In one embodiment, step S200 derives a visual servo acceleration model based on the coordinate transformation relationship between the camera and the robotic arm's end effector, including:

[0088] S210: When a human applies a force to a robotic arm and causes it to interact with the environment, the dynamic characteristics of a human-guided n-link robotic arm in the joint space can be described as follows:

[0089] in, These are the generalized variables of the robotic arm's joint angles, joint angular velocities, and joint angular accelerations, respectively, B(q)∈R. n×n This represents a symmetric and positive definite inertia matrix. The Coriolis force and centrifugal force matrices are described, G(q)∈R n It is the gravity vector, τ∈R n It is the joint driving torque vector. J is the torque applied by the human to the end effector.h The Jacobian matrix of the robotic arm, where the operator's coordinate system is the end-effector coordinate system, is d∈R. n It is the amount of environmental disturbance generated when the robotic arm comes into contact with the unknown environment;

[0090] S220: Solve the visual servo dynamic equations for the acceleration level to characterize the kinematic relationship between visual features and joint variables, thereby merging the dynamic equations of the robotic arm and the vision system; consider m visual features s∈R m The motion of image features and the corresponding camera motion speed c v∈R 6 The kinematic relationship is as follows:

[0091] Among them, L s ∈R m×6 Represents the image interaction matrix. c v c and c v o These are the speeds of the camera and the object, respectively.

[0092] S230: When the target is stationary... c v o When = 0, the above expression becomes Based on the transformation relationship between the camera coordinate system and the end effector coordinate system e T c The velocity transmission relationship in the two coordinate systems is obtained. e M c ∈R 6×6 Its specific form is as follows:

[0093] The equation for the velocity conversion relationship between the camera and the end effector is: v e = e M c v c ;

[0094] Meanwhile, the conversion relationship between the end effector velocity and the joint angular velocity of the robotic arm is as follows: Based on the above equations, the kinematic relationship between the velocity representing image features and the joint velocity can be deduced as follows:

[0095] J s ∈R m×n The Jacobian matrix representing the task, J e ∈R 6×n It is the Jacobian matrix of the robotic arm. This is the velocity transformation matrix from the end effector to the camera coordinate system; in this formula, c M eIt is a constant matrix because the camera is rigidly connected to the robot's end effector;

[0096] S240: Differentiating the equation in S230 with respect to time yields an accelerated form of the image features:

[0097] To facilitate the concise derivation of subsequent expressions, the above expression is rewritten as:

[0098] in

[0099] In one embodiment, S200 combines the human-robotic arm interaction dynamic equations and the coordinate transformation relationship between the robotic arm end effector and the six-dimensional force sensor to derive a human-robotic arm-environment interaction dynamic model in the feature space, including:

[0100] S250: Rearranging the dynamic equations of the n-link robotic arm under the human-robotic arm interaction in S210, we have:

[0101] S260: Substitute the image feature velocity expression from S230 and the image feature acceleration equation from S240 into the above equation, and combine this with the force and torque conversion relationship applied by the person in S110 above, i.e. The dynamic equations for the human-robotic arm-environment interaction in image space are obtained as follows:

[0102] in

[0103] The above equation is a set of highly nonlinear coupled differential equations corresponding to m image features. Consider the equation... And the transformation equation for the force projection applied to the force sensor into the camera coordinate system is: The projection equation of the human guiding force is obtained: J s B(q) -1 τ h =L s B c (q) -1 f ch ;

[0104] in, The inverse inertia matrix is ​​the projection of the robotic arm onto the camera coordinate system.

[0105] S270: Considering the dynamic equations of the interactive system and the human-guided force projection equations in S260, the human-robotic arm interaction dynamic equations in the characteristic space are summarized as follows:

[0106] Among them, f sh=L s B c (q) -1 f ch This represents the virtual human guide force acting on image features, which originates from the human guide torque projected onto the camera coordinate system, d. s =J s B(q) -1 d is the virtual contact perturbation projected into the feature space when the robot, guided by a human, comes into contact with the environment; b s =J s B(q) -1 b represents the virtual gravity, virtual Coriolis force, and centrifugal force projected onto the image feature space, u s =J s B(q) -1 τ represents the virtual control input of the robotic arm in the feature space.

[0107] In one embodiment, S300 includes:

[0108] S310: The human hand, due to the viscoelasticity and stretching response of its muscles, possesses spring-like properties. Rigid robots can leverage this property to achieve compliant contact tasks, avoiding damage to the workpiece. The human-robotic arm interaction system in image feature space is modeled as a spring-mass-damping system. Under image-based visual servoing IBVS settings, its expression is:

[0109] S320: Define compliance characteristic error Defined as an auxiliary feature variable, when no one applies a force. It will approach 0, which means s d =s c At this point, we only need to focus on the trajectory tracking problem; therefore, we define e. s =s c -s, using a visual servo speed controller to track desired features, has:

[0110] Where λ is a positive constant, in actual execution, only the velocity and acceleration of image feature points are considered. Given the positions of image feature points, when a person applies a force, f... sh ≠0, resulting in a tracking error e s If the value is not 0, the current feature cannot keep up with the auxiliary feature, that is, it cannot fully keep up with the human-guided teaching trajectory, and thus cannot achieve the effect of human-guided teaching.

[0111] S330: Utilizes the admittance controller described in S320 to record the image feature positions s during the human-guided teaching process. h ,speed and acceleration The data is encoded and generalized using discrete motion dynamic primitives to generate image feature position and velocity curves. The state expression of the motion dynamic primitive based on image features is as follows:

[0112] Where g is the feature target state variable, γ is the time scaling factor, and α s and β s Considered as the spring and damping coefficients, used to adjust the performance of the motion dynamic primitive system, F(x) is a nonlinear function used to adjust the trajectory shape, where x is a time-independent phase variable, and its expression satisfies the following canonical system:

[0113] Where c is the decay coefficient, and as x approaches 0 over time, F(x) also approaches 0 accordingly. It will reach the attraction point [g,0];

[0114] S340: The expression for the nonlinear function in S330 above is:

[0115] It is composed of a linear combination of N basis functions, and the expression for the basis functions is f. i (x)=exp(-l i (x-μ i ) 2 ), i = 1, 2, ..., N;

[0116] Among them l i μ represents the width of the basis functions. i Let θ represent the center of the basis functions, s0 be the initial state value of the characteristic variables, and Θ be the initial state value of the characteristic variables. i These are the weighting coefficients of the basis functions, which can be obtained by minimizing the error cost function.

[0117] In one embodiment, to illustrate the necessity and rationality of the designed image feature-based impedance iterative learning controller, a simple stability analysis of the system is required first. In the feature space, the virtual human guiding force f... sh and interference d s It also acts on image features. sh It is considered to play a "positive" role, directing image features to a specified location. Conversely, d s Having a "negative" effect, potentially leading to unstable contact, the S400 includes:

[0118] S410: Modeling human guidance forces projected onto the feature space as

[0119] Among them, e sh =ss h ∈Rm Indicates the image feature tracking error, s h ∈R m It is the characteristic location variable expected by humans, K sh ∈R m×m and K shv ∈R m×m It is the virtual stiffness and damping matrix of the human body projected onto the image feature space; S420: in the absence of interference, i.e., d s In the ideal case where = 0, the control input in the image feature space Substituting the characteristic space human-robotic arm-environment interaction dynamics equations from S270 and the above equation, we have:

[0120] The above equation implies that the human-guided impedance iterative learning control system based on image features is stable; instability is caused by contact disturbances. Under the ideal conditions described above, the force applied by the person can guide the image features to the desired position along an ideal trajectory, i.e., e. sh ≈0, define a reference image feature vector as s r ∈R m , using s r Replace s h Considering the above equation, we can obtain:

[0121] Visual features s r The reference trajectory can be obtained through human-guided teaching and can be used for subsequent visual impedance iterative learning control strategy design;

[0122] S430: To further discuss the stability of image feature-based human-computer interaction systems, it is assumed that environmental contact perturbations are measurable, and a general viscoelastic model of contact perturbations projected onto the feature space is considered:

[0123] Among them, K s ,K sv ∈R m×m s is a virtual parameter for interference. e ∈R m Indicates the stationary position of image features;

[0124] S440: If the external virtual disturbance does not exist, only the virtual uncertain dynamics are compensated, i.e. Based on the expression in S270, we can conclude that:

[0125] If the above equation holds true, then the system is stable;

[0126] S450: In the case of virtual contact disturbance, according to the expression in S430, we have:

[0127] When K s and K sv When the matrix is ​​positive definite, it can be clearly observed from the above expression that image features do not obediently follow the human-guided trajectory s as represented by the expression in S420. r Interference items -K s s and K s s e The introduction of this makes the system's contact unstable;

[0128] S460: In practical operation, it is impossible to accurately measure environmental contact disturbances. Therefore, a feature-based impedance iterative learning controller is designed to help counteract contact dynamics while flexibly completing assembly tasks without prior knowledge of d. s and s h .

[0129] Through the stability analysis of the S400 described above, it can be observed that as long as the impedance controller parameters are designed appropriately so that the image feature-based impedance controller can counteract the effects of interference, the system can be guaranteed to tend towards stability. Here, this invention introduces iterative learning control. Iterative learning utilizes past execution experience to improve future control performance, increasing tracking accuracy through multiple iterations, thus significantly reducing the dependence on a precise system model. The combination of iterative learning and impedance control allows the system to better adapt to environmental changes. Iterative learning helps the robotic arm remember and adapt to changes in repetitive tasks, while impedance control can adapt to immediate interactive and contact tasks.

[0130] In one embodiment, S500 includes:

[0131] S510: During iterative learning, the virtual disturbance in the feature space is spatially periodic. In the i-th iteration, the spatial form of the virtual environment contact disturbance is:

[0132] Among them, K sp ,K svp ∈R m×m For parameter K s ,K sv The spatial form satisfies K sp,i =K sp,i-1 ,K svp,i =K svp,i-1 ,s e,i =s e,i-1 ;

[0133] S520: Based on the above virtual interference in the image feature space, define the image feature tracking error e in the i-th iteration. s,i =si -s r,i The image feature spatial impedance controller is designed as follows:

[0134] in, Control parameter matrix K e ∈R m×m The impedance parameter remains consistent for each iteration. The update law is:

[0135] Among them, B s B sv B sp ∈R m×m Represents a matrix of positive constants. Λ represents the actual iteration time of iterative learning, T represents the upper limit of iteration time, and Λ = diag{λ} 1 ,...,λ m}∈R m×m , where λ i =|v i | -1 v i It is the robot's current velocity in the i-th dimension;

[0136] S530: The above controller u s,i d s,i and expressions Substituting the interaction dynamics equations in S270, we have:

[0137] in,

[0138] By implementing image feature-based impedance iterative learning control in contact assembly tasks, when At that time, virtual interference can be eliminated. s To mitigate the impact and ensure the stability of the closed-loop system.

[0139] Specifically, the control block diagram of the entire method is shown in Figure 2. In summary, this invention concludes with the following theorem: Considering the human-robotic arm-environment interaction dynamics described in S200, the generalization equation of the human-guided image feature trajectory in S300, the impedance iterative learning control formula and the impedance parameter iterative update law in S500, the combination of the above equations ensures the system stability in S530, eliminates environmental contact disturbances, and simultaneously enables a certain degree of scene generalization within the visual framework.

[0140] In one embodiment, S500 is followed by:

[0141] S600: The stability of the human-robotic arm-environment interaction system is analyzed and proven using Lyapunov's stability proof theory.

[0142] In one embodiment, S600 includes:

[0143] S610: Define Lyapunov candidate functions: V i =V e,i +V x,i +V xv,i +V xp,i , ξ=[ξ 1 ,...,ξ m ],dξ=[dξ 1 ,...,dξ m ],

[0144] Where V e,i V represents the feature tracking error. x,i V xp,i and V xv,i A composite energy function related to the impedance coefficient estimation error is defined, where the subscript x represents that the Lyapunov candidate function is spatially dependent but time-independent. These are the estimation error matrices. The k-th column vector; for The k-th element; sgn(·) is the sign function. They are B s B sv B sp The kth element;

[0145] S620: Because

[0146] Similarly, it is possible to obtain

[0147] Where ρ is a time-dependent integral variable, it can be inferred that the Lyapunov equation in S610 is positive definite, i.e., V i >0;

[0148] S630: At iteration time t∈[0,T] i Feature tracking error ε s,i =e s,i , for V e,i Taking the derivative with respect to time, we get:

[0149] Substituting the equation from S510 into the above equation, we get:

[0150] Its integral form is:

[0151] Definition: ΔV e,i =V e,i -V e,i-1 ,

[0152] It can be inferred that:

[0153] S640: Similarly, define: ΔV x,i =V x,i -V x,i-1 , ΔV xv,i =V xv,i -V xv,i-1 , ΔV xp,i =V xp,i -V xp,i-1 Substituting V into S620 x,i The expressions are:

[0154] definition:

[0155] get:

[0156] Substitute it into the above ΔV x,i The expressions are:

[0157] From the above formula, we can obtain:

[0158] Similarly, we can obtain:

[0159] S650: Combining the inequalities in S640, we have:

[0160] Substituting the impedance parameter update law expressed in S520 into the above equation, we get:

[0161] in, and They are s i ∈R m and The k-th element in;

[0162] S660: When At that time, based on the feature tracking error ε s,i The definition of V in S630 e,iIntegral form and ΔV e,i The definitions are as follows:

[0163] when At this point, the impedance estimation parameters stop updating. Therefore, the following inequality holds:

[0164] Similarly, referring to the expression in S650, we have:

[0165] S670: Taking into account the ΔV in S650 and S660 i The final form of the inequality, ΔV i ≤0 in t∈[0,T i ]and This holds true under all circumstances, because when ΔV i When V < 0, the initial energy of the iterative impedance learning system based on image features is bounded, which means that V i It will monotonically decrease with the increase of the number of iterations, according to V in S610. i The definition of , which is a positive definite function, yields:

[0166] In ΔV i When = 0, we have:

[0167] Therefore, based on the S530 expression, we can obtain:

[0168] Analysis of the above results shows that the iterative impedance learning method based on image features can eliminate contact disturbances in the unknown environment, and the robot will track the trajectory guided by the human with the assistance of the vision system.

[0169] Compared with the prior art, the advantages of the present invention are as follows:

[0170] (1) This invention proposes a human-guided visual impedance fusion framework in the image feature space for the first time. By combining it with image-based visual servoing (IBVS), human-guided forces and contact disturbances are projected into the image feature space, thereby modeling the dynamic interaction between human, robotic arm and environment in the feature space.

[0171] (2) Introduce motion dynamic primitives to encode and generalize the image feature point trajectory of human-guided robot to complete assembly, and establish a vision-force fusion impedance iterative learning control generalization framework based on image features;

[0172] (3) Based on the above framework, this invention proposes an impedance iterative learning control technology based on image features to improve the stability and accuracy of contact tasks. The impedance parameters are updated by learning the feature trajectory guided by humans to counteract contact interference. Finally, the stability of the human-robotic arm-environment interaction system is proved by Lyapunov stability theory.

[0173] The foregoing has provided a detailed description of the human-guided vision-force fusion robotic arm impedance iterative learning control method provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A human-guided visual-force fusion robotic arm impedance iterative learning control method, characterized in that, The method comprises the following steps: S100: Obtain the coordinate conversion relationship between the camera carried on the end effector of the robot arm and the end of the robot arm through hand-eye calibration, and solve the coordinate relationship between the six-dimensional force sensor and the end of the robot arm through geometric relationship; S200: Derive a visual servo acceleration model according to the coordinate conversion relationship between the camera and the end of the robot arm, combine a human-robot arm interaction dynamic equation and the coordinate conversion relationship between the end of the robot arm and the six-dimensional force sensor, and derive a human-robot-environment interaction dynamics model in a feature space; S300: Guide the robot to complete an assembly task through a human-guide-navigator controller based on image features, record image feature position and speed data during the process, encode and generate image feature position and speed curves by using a motion dynamic primitive DMP, adjust the starting point and the end point position of the curve by changing the DMP parameters, regenerate the image feature trajectory curve, and realize trajectory generalization; S400: Model the force acting on the human and the environmental contact disturbance, and theoretically analyze the stability of the human-robot-environment interaction system; S500: Combine the human-robot-environment interaction dynamics model in the feature space derived in S200 and the generalized image feature trajectory curve in S300, design an impedance iterative learning law based on image features, update the impedance control parameters by iteratively learning the image feature speed curve, represent the human-guide impedance characteristics, simultaneously correct the tracking trajectory online by using visual servoing, and after a certain number of iterations, the robot arm can successfully track the human-guide feature position curve and complete the assembly task.

2. The method of claim 1, wherein, S100 comprises: S110: A six-dimensional force sensor and an RGBD camera are simultaneously installed on the end effector of the robot arm; S120: An image of a calibration board with N points is captured by using the RGBD camera, and the corresponding robot arm end pose at this time is recorded; the step is repeated a preset number of times; S130: Solve the coordinate transformation relationship between the camera and the end of the robot arm by Tsai-Lenz method e T c , including a rotation matrix from the camera coordinate system to the end effector coordinate system e R c , and a translation vector from the camera coordinate system to the end effector coordinate system e t c , specifically: S140: Calculate the coordinate conversion relationship between the six-dimensional force sensor and the camera through the flange geometry model equipped with the six-dimensional force sensor and the camera c T h , including the rotation matrix of the force sensing coordinate system to the camera coordinate system c R h , and the translation vector c t h , the specific form is:

3. The method of claim 2, wherein, In S200, the visual servo acceleration model is derived according to the coordinate conversion relationship between the camera and the end of the robot arm, comprising: S210: When a human exerts a force on the robotic arm and causes it to interact with the environment, the dynamics of the human-guided n-link robotic arm in joint space can be described as: wherein q, q, q are the generalized variables of the joint angle, joint angular velocity and joint angular acceleration of the robot arm respectively, B(q) e R n×n represents a symmetric and positive definite inertia matrix, The Coriolis and centrifugal force matrices, G(q) e R n is the gravity vector, τ e R n is the joint drive torque vector, is the torque exerted on the end effector by a human, J h is the Jacobian matrix of the robot with the end coordinate system as the coordinate system of the operator, d∈R n is the environmental disturbance generated when the robot contacts with the unknown environment; S220: Consider m visual features s ∈ R m , the kinematic relationship between the motion of image features and the corresponding camera motion velocity c v ∈ R 6 is: where L s ∈ R m×6 represents the image interaction matrix, c v c and c v o are the motion velocities of the camera and the object, respectively; S230: When the target is stationary, i.e. c v o = 0, the above equation becomes According to the conversion relationship between the camera coordinate system and the end effector coordinate system e T c , the velocity transmission relationship in the two coordinate systems is obtained e M c ∈R 6×6 , and the specific form is: The velocity conversion relationship equation between the camera and the end effector is obtained as follows: v e = e M c v c ; Meanwhile, the conversion relationship between the end speed of the mechanical arm and the joint angular velocity is From the above equations, the kinematic relationship between the image feature velocity and the joint velocity is inferred as: where J s ∈ R m×n represents the task Jacobian matrix, J e ∈ R 6×n is the manipulator Jacobian matrix, is a velocity transformation matrix of the end effector to the camera coordinate system; S240: Differentiate the equation in S230 with respect to time to obtain an accelerated form expression of the image feature: For the sake of subsequent concise derivation of expressions, the above equation is rewritten as: wherein 4. The method of claim 3, wherein, In S200, the human-robot-environment interaction dynamics model in the feature space is derived by combining the human-robot arm interaction dynamic equation and the coordinate conversion relationship between the end of the robot arm and the six-dimensional force sensor, comprising: S250: The n-link robot dynamics equation under human-robot interaction of S210 is rearranged, and there is: S260: Substitute the image feature velocity expression in S230 and the image feature acceleration equation in S240 into the above formula, combine the human applied force and torque conversion relationship in S110 above, i.e. The human-robot-environment interaction dynamic equation in the image space is obtained: wherein The above equations are a set of highly nonlinear coupled differential equations corresponding to m image features, taking into account equation J s = L s c M e J e and the conversion equation for the force projection on the force sensor to the camera coordinate system is The human-guide force projection equation is obtained as follows: J s B(q) -1 τ h = L s B c (q) -1 f ch ; wherein, is the inverse inertia matrix of the robot arm projected into the camera coordinate system; S270: Considering the interaction system dynamic equation in S260 and the human force projection equation, the human-robot interaction dynamics equation in the feature space is summarized as: where f sh = L s B c (q) -1 f ch represents the virtual human-induced force acting on the image feature, which comes from the human-induced torque projected into the camera coordinate system, d s = J s B(q) -1 d is the virtual contact disturbance projected into the feature space when the human-induced robot makes contact with the environment, b s = J s B(q) -1 b represents the virtual gravity, virtual Coriolis force, and centrifugal force projected into the image feature space, u s = J s B(q) -1 τ represents the virtual control input of the manipulator in the feature space.

5. The method of claim 4, wherein, S300 comprises: S310: Model the human-robot interaction system in the image feature space as a spring-mass-damper system, which is expressed as: S320: define a compliance feature error defined as an auxiliary characteristic variable when no force is applied by a person, will approach 0, which means s d = s c At this point, we only need to concern about the tracking problem, thus, define e s = s c -s, track the desired feature using visual servoing speed controller, have: where λ is a constant, in actual implementation, only the image feature point velocity and acceleration are considered, if the image feature point position is given, when the person applies force, f sh ≠ 0, causes tracking error e s ≠ 0, the current feature cannot follow the auxiliary feature, that is, cannot completely follow the human guide teaching trajectory, that is, cannot achieve the effect of human guide teaching. S330: Record the image feature position s in the human guide teaching process using the admittance controller set forth in S320 h , velocity and acceleration Data, and encode and generalize them by discrete motion dynamics primitives, generate image feature position and velocity curves, and based on the image feature motion dynamics primitives state expression is: where g is the characteristic target state variable, γ is a time scaling factor, α s and β s are considered as spring and damping coefficients to adjust the performance of the motion dynamic primitive system, F(x) is a nonlinear function to adjust the trajectory shape, x is a phase variable independent of time, whose expression satisfies the following regular system: where c is an attenuation coefficient, and F(x) also tends to 0 as x tends to 0 over time, when The attraction point [g, 0] will be reached; S340: The expression of the non-linear function in S330 above is: It is composed of N basis functions, and the basis function expression is f i (x) = exp(-l i (x - μ i ) 2 ), i = 1, 2,..., N, where l i denotes the width of the basis function, μ i denotes the center of the basis function, s0is the initial state value of the characteristic variable, Θ i is the weight coefficient of the basis function, which can be obtained by minimizing the error cost function.

6. The method of claim 5, wherein, S400 comprises: S410: model the human guidance projected to the image feature space as: where e sh = s - s h ∈ R m denotes the image feature tracking error, s h ∈ R m is the human desired feature position variable, K sh ∈ R m×m and K shv ∈ R m×m are the virtual human stiffness and damping matrices projected to the image feature space; S420: In the ideal case where there is no interference, i.e. d s = 0, let the control input in the image feature space With the feature space human-robot-environment interaction dynamics equation brought into S270 and the above equation, there are: The above formula means that the human impedance iterative learning system based on image features is stable, and instability is caused by contact interference; in the above ideal case, the human-applied force can guide the image features to reach the desired position along the ideal trajectory, i.e. e sh ≈0, define a reference image feature vector as s r ∈R m , replace s r with s h , considering the above formula, we can get: Visual feature s r The reference trajectory of s can be obtained by human demonstration and can be used for subsequent visual-impedance iterative learning control strategy design. S430: To further discuss the stability of the image feature based human-computer interaction system, it is assumed that the environmental contact disturbance is measurable, and a general viscoelastic model of the contact disturbance projected to the image feature space is considered: where K s , sv ∈ R m×m is a virtual parameter of interference, s e ∈ R m represents the static position of image features; S440: If the external virtual disturbance is not present, only the virtual uncertain dynamics is compensated, i.e. Then, according to the expression in S270, we can get: If the above formula is established, the system is stable; S450: In the case of a virtual contact perturbation, according to the expression in S430, there is: When K s and K sv are positive definite matrices, it can be very clear from the above expressions that the image features do not follow the human-guided trajectory s r compliantly as characterized by the expression in S420, with the interference term - K s s and K s s e The introduction of s and K makes the contact of the system unstable; S460: In practice, the environmental contact disturbance cannot be measured accurately, therefore, a feature-based impedance iterative learning controller is designed to help to offset the contact dynamics while completing the assembly task flexibly without prior knowledge of d s and s h .

7. The method of claim 6, wherein, S500 comprises: S510: In the iterative learning process, the feature space virtual disturbance is spatially periodic, and in the i-th iteration process, the spatial form of the virtual environment contact disturbance is: where K sp , K svp ∈R m×m is a parameter K s , K sv of the spatial form, satisfying K sp,i = K sp,i-1 , K svp,i = K svp,i-1 , s e,i = s e,i-1 ; S520: define the image feature tracking error e of the i-th iteration based on the above-mentioned virtual disturbance in the image feature space s,i = s i -s r,i , the image feature space impedance controller is designed as: wherein Control parameter matrix K e ∈ R m×m Consistent with each iteration process, impedance parameter The update law is: where B s ,B sv ,B sp ∈R m×m denotes a normal matrix, representative of the actual iteration time of the iteration learning, Λ = diag{λ 1 ,...,λ m} ∈ R m×m represents the upper bound of the iteration time, where λ i = |v i | -1 , v i is the current i-th dimension velocity of the robot. S530: The controller u s,i , d s,i and the expression With the interactive dynamics equation brought into S270, there are: wherein By implementing image feature based impedance iterative learning control in contact assembly tasks, when At this time, the influence of the virtual interference d s can be eliminated, ensuring the stability of the closed-loop system.

Images