Optimization-based robot variable stiffness vision-impedance control method and system

Abstract

The invention discloses a robot variable stiffness vision-impedance control method and system based on optimization, and the method comprises the steps: building a coordinate transformation relation between a camera and a six-dimensional force sensor and a coordinate transformation relation between the camera and the tail end of a robot, calibrating the six-dimensional force sensor, and analyzing the interaction dynamics relation between the robot and the environment. A robot-environment interaction mathematical model is constructed, and a feature space human-robot-environment interaction dynamical model is established by solving a visual servo acceleration model and combining a dynamical equation of the robot in the assembly process; establishing a feature space human-guided dynamics model as a planner to obtain a reference feature trajectory implying human recognition contact dynamics; a characteristic space variable stiffness impedance controller based on QP online planning is designed, impedance parameters can be dynamically adjusted in the assembly process, interference force caused by environment uncertainty is compensated in real time, constant force is kept in a rich contact task, and it is ensured that the robot can stably complete the assembly task with high precision.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology, and in particular relates to an optimized visual-impedance control method and system for variable stiffness of robots. Background Technology

[0002] As industrial manufacturing moves towards higher precision, higher efficiency, and greater flexibility, the manufacturing of complex components, such as aerospace panel assembly and precision electronics manufacturing, places higher demands on the intelligence level of production systems. However, the manufacturing process is often constrained by many factors, such as limited workspace, numerous assembly steps, and dynamic environmental changes, making it difficult for traditional manufacturing models to meet the needs of modern industry.

[0003] Currently, the manufacturing industry mainly relies on two methods for production: manual labor, which is flexible but inefficient, labor-intensive, and heavily influenced by the experience and skill level of the operators; and robotic production lines, which, while enabling high-efficiency production, often rely on fixed algorithms only for repetitive tasks. When faced with changing production environments, unstructured manufacturing scenarios, and complex assembly tasks, the adaptability and intelligence of traditional robotic manufacturing methods are insufficient. Therefore, improving robots' perception of the manufacturing environment, enabling them to autonomously adapt to changes, and enhancing their precision operation capabilities have become urgent problems to be solved in the field of intelligent manufacturing.

[0004] In intelligent manufacturing, vision technology has been widely applied to tasks such as workpiece positioning and quality inspection. Robots perceive environmental information through vision sensors, enabling them to identify the position, posture, and assembly quality of target objects. However, real-world manufacturing tasks are often highly contact-intensive, and visual information alone cannot meet the requirements of precise operation. For example, in the assembly of aerospace panels, which are weakly rigid components, if the robot relies solely on visual positioning for assembly, insufficient control precision during contact may cause vibrations, affecting assembly quality, and could even damage the workpiece due to excessive contact force.

[0005] On the other hand, robots face the challenges of unstructured environments and their own dynamic constraints when performing contact-intensive tasks. Due to the coupled dynamic relationship between the robot and the workpiece, instability in contact forces can lead to task failure. Therefore, in addition to visual information, force sensing is needed to acquire contact force information during robot-environment interaction, thereby optimizing robot control strategies and improving operational stability and adaptability.

[0006] In summary, in the field of intelligent manufacturing, effectively integrating multimodal sensing information (such as vision and force) and overcoming the complex coupling dynamics between humans, robots, and the environment to achieve high-precision adaptive assembly is a core technical challenge that urgently needs to be addressed. To this end, this invention proposes an optimized robot variable stiffness vision-impedance control method to improve the robot's adaptability in complex manufacturing environments. Summary of the Invention

[0007] To address the above technical problems, this invention provides an optimized visual-impedance control method and system for variable stiffness of robots.

[0008] The technical solution adopted by this invention to solve its technical problem is: An optimization-based visual-impedance control method for variable stiffness of a robot, the method comprising the following steps: S100: Establish coordinate transformation relationships between the camera, the six-dimensional force sensor, and the robot end effector. Simultaneously, calibrate the six-dimensional force sensor and compensate for the sensor's own zero drift and tool gravity through a force calibration algorithm. S200: Considering the dynamic characteristics of the robot under force, including the guiding force applied by the human and the reaction force of the environment, and combining the robot's kinematics and dynamic equations, as well as the visual servo acceleration equation, a complete human-robot-environment interaction dynamic model in the feature space is constructed. S300: Based on the dynamic model, a human guidance force model is established in the feature space. Visual features are associated with robot motion and a reference trajectory is generated. A variable stiffness visual impedance controller is designed. The stiffness matrix is ​​optimized online through QP to ensure constant contact force and satisfy torque constraints. The system equations are derived to achieve external force convergence. An energy function is constructed to analyze stability. S400: Based on the stability analysis results, online optimization based on energy tank stiffness enhancement is introduced. The energy tank state equation and upper and lower limit constraints are constructed to limit active energy injection. A quadratic programming QP model is established to optimize stiffness parameters in real time under passive constraints and ensure that the total energy is bounded. Passive analysis is performed by constructing a system storage function. S500: The robot assembles gears using a human-guided robot. It integrates six-dimensional force sensing and RealSense visual data, extracts feature point trajectory errors in real time based on the AprilTag algorithm, drives QP optimization to dynamically adjust variable stiffness parameters, and combines Jacobi pseudo-inverse mapping to the joint space to drive the robot arm movement, achieving compliant trajectory tracking.

[0009] Preferably, in S100, the coordinate transformation relationship between the camera, the six-dimensional force sensor, and the robot end effector is established, including: S110: A six-dimensional force sensor and an RGB-D camera are installed at the end of the robotic arm to ensure that the camera's field of view is unobstructed; S120: Use the end-effector camera to capture an image of a calibration plate with known geometric features and record the corresponding end-effector pose at that time; S130: Repeat S120 a preset number of times under different positions and postures, take images of the calibration board, and simultaneously record the pose data of the robot's end effector; S140: By analyzing the captured calibration board images and recorded end-effector pose data, the rotation matrix of the camera relative to the robotic arm end effector is calculated. Translation vector By using the Tsai-Lenz method combined with hand-eye calibration technology, the coordinate transformation relationship between the camera and the robotic arm end effector was solved. Specifically: ; S150: Using modeling software, an assembly model of the flange, six-dimensional force sensor, and camera is constructed. The rotation matrix of the six-dimensional force sensor relative to the camera is calculated through geometric analysis. Translation vector The coordinate transformation relationship between the six-dimensional force sensor and the camera was obtained. Specifically: .

[0010] Preferably, S200 includes: S210: When a human applies forces to a robot and causes it to interact with its environment, in the joint space, the human-guided... The dynamic characteristics of the linkage robot are described as follows: ; in, These are the generalized variables of the robot's joint angles, joint velocities, and joint accelerations. This represents a symmetric and positive definite inertia matrix. The Coriolis force and centripetal force matrix are described. It is a gravity vector. It is a joint drive command vector. It is the torque applied by the human to the end effector. It is an external torque; S220: Solve for the visual servo dynamics at the acceleration level to characterize the motion relationship between visual features and joint variables, considering... Image features Motion of image features and associated motion spinors The kinematic relationship is as follows: ; in, It is the speed of image feature generation. It is an image interaction matrix used to correlate the velocity of feature points with the robot's motion spinor. and These represent the motion spins of the camera and the object, respectively. S230: When the object is stationary... In this case, the above equation can be used to characterize the kinematic relationship between the characteristic velocity and the joint velocity through the robot's Jacobian function, that is: ; in The Jacobian matrix represents the task. It is the Jacobian matrix of robots. It is a screw transformation matrix. In this way, the velocity of image features can be expressed as a function of joint velocity, thus providing a basis for subsequent control strategy design. S240: Differentiating the equation in S230 with respect to time yields an accelerated form of the image features: ; To facilitate the concise derivation of subsequent expressions, the above expression is rewritten as: ; in, It is the acceleration of image features. Here is the acceleration coupling matrix; S250: The dynamic equations of the human-guided n-link robot in S210 are rearranged to derive the dynamic characteristics of the human-guided robot in the feature space: ; Substituting the velocity expressions for the characteristic points in S230 and the acceleration expressions for the characteristic points in S240 into the dynamic characteristic formula, we obtain the dynamic equations related to the characteristic motion: ; in This is a comprehensive force term; the above formula is a set of terms related to... Nonlinear, highly coupled differential equations corresponding to each image feature; S260: By introducing the task Jacob coefficient Consider a projection matrix that represents the torque applied by the human in the human-tool coordinate system. Transform to the camera coordinate system, therefore we have , Substituting into the dynamic equations related to the features above, we get: ; in, This represents a virtual force acting on image features, originating from a human-guided torque projected onto the camera coordinate system. It is a virtual external force in the characteristic space. Represents virtual gravity, virtual Coriolis force, and centripetal force in characteristic space. Represents the damping matrix. This represents the characteristic space impedance controller, which is the virtual control input for the robot in the characteristic space. This represents the torque applied by the human in the camera coordinate system.

[0011] Preferably, S300 includes: S310: Virtual human guidance force in feature space And acting on image features, for ease of analysis, the human guiding force projected onto the feature space is modeled as: ; in, These represent the velocity and acceleration variables of human guiding characteristics, respectively. Represents the damping matrix; Based on the above equation, the visual feature variables and the robot's end effector can move in the same direction as the external torque guided by the human, and at the same time, the characteristic trajectory guided by the human can be obtained. This trajectory serves as a reference trajectory for subsequent controller design; S320: A variable-stiffness visual impedance controller based on the S310 is designed and operates in the feature space. The stiffness is adjusted using an online optimized QP method to ensure that the robot end effector maintains a constant force when performing contact-intensive tasks. A feature space impedance controller is designed. Its form is: ; in, It is the feature tracking error vector. The rate of change of feature tracking error. and These represent the stiffness and damping matrices for impedance control, respectively. S330: Substituting the formulas in S310 and S320 into S260, we get: ; Observing the above formula, we can further deduce: ; Since the performance of an impedance system depends on its parameters, it is desirable that the virtual external force in the characteristic space converges to the desired external force. ; S340: To solve the force convergence problem in S320, the following QP problem is constructed for adjusting the stiffness of the impedance system online: ; ; in, and These represent the minimum and maximum allowable stiffness, respectively. This indicates the maximum external torque that the robot can apply. and These are weighted matrices used to adjust the external torque and minimum stiffness, respectively. S350: After completing the above controller optimization, to facilitate subsequent stability analysis, the energy function of the characteristic space impedance system is defined as follows: ; Its derivative with respect to time is: ; Due to the item The sign of the impedance system is unknown beforehand, and it is not feasible to define a suitable energy storage function for the relative ports. Therefore, it is impossible to guarantee the impedance system with respect to the ports. The passive nature of.

[0012] Preferably, S400 includes: S410: To ensure the stability of the characteristic space variable impedance system, firstly, the variable stiffness... Rewritten as ,in and Let the constant stiffness component and the time-varying stiffness component be respectively. Without loss of generality, let ; Subsequently, the concept of a virtual energy tank is introduced to separate the potential active behavior of time-varying stiffness components, whose energy is defined as... ,in Indicate its dynamic state: ; in, This represents the rate of change of the dynamic state of the energy tank. For energy flow switching parameters, It is an energy exchange term; in Defined as: ; in This indicates the lower limit of the energy tank. For the spacetime stiffness matrix, in addition, to ensure that the total energy of the entire interconnected system remains bounded, when a certain energy tank upper limit is reached... To prevent the energy tank from overloading, the following methods should be used: ; S420: Therefore, online stiffness optimization under passive constraints, incorporating the energy tank constraint into the optimization problem, can be expressed as the following QP problem: ; ; S430: The final characteristic space impedance controller is described as follows: ; The expression for the virtual external force in the characteristic space is modified as follows: ; By implementing a QP-based online stiffness optimization feature-space variable stiffness visual impedance controller, as described in the two equations above, It can be dynamically adjusted to maintain a constant force in contact-intensive tasks, and the energy tank initializes to meet the following conditions. When the virtual energy tank's energy When the stiffness is constant, it can be continuously optimized online; conversely, when... At that time, the stiffness degenerates into The controller is simplified to a constant stiffness impedance controller; S440: For the modified energy tank constraint controller and virtual external force equations, i.e., equations in S420 and S430, the energy tank dynamic state is shown in equation S410, and the storage function of the entire interconnected system. Selected as: ; Total Energy Differential with respect to time It is given by the following formula: ; because Therefore: ; Therefore, we can conclude that: ; The above equation indicates the existence of the following passive condition: ; in, The value of the storage function at time t. The initial stored function value. For integration variables, For power exchange between the system and the external environment; The above passive conditional inequalities prove that the modified enhanced control framework can achieve variable stiffness in the visual impedance model while safely interacting with any passive environment.

[0013] Preferably, S500 includes: S510: A six-dimensional force sensor is installed between the handle and the flange to measure the force and torque applied by the user in real time; a RealSense D435i camera is installed at the front of the flange to capture image information of the gears and gear shafts; the end effector of the robot is connected to the six-dimensional force sensor through the flange to ensure the measurement accuracy and stability of the force sensor. S520: During the experiment, the operator guides the robot to assemble the gear into the gear shaft. At the same time, the RGB-D camera captures the image information of the gear and gear shaft in real time and records the corresponding four sets of feature point trajectory data. These feature point trajectory data include the relative position and attitude information of the gear and gear shaft, which are used for subsequent visual servo control. By recording these data, the operator's skills and dynamic characteristics in the human-guided assembly process are analyzed, providing a reference for the variable stiffness visual impedance control strategy. S530: Based on the feature point trajectory data recorded during human guidance, initialize the parameters of the variable stiffness visual impedance controller. These parameters include the virtual stiffness matrix. Virtual damping matrix Expected contact force ; S540: Feature Point Extraction and Tracking. The RealSense D435i camera acquires images of gears and gear shafts in real-time at a frequency of 30Hz. The AprilTag algorithm is used to identify four key feature points from the images of the gears and gear shafts. The positional information of these feature points will be used as input to the visual servo controller. The error between the current feature point position and the trajectory of the hand-operated indicator is calculated. ,in This is the current location of the feature point. It is a hand-guided display trajectory; by monitoring the error of the feature points in real time, the controller dynamically adjusts the robot's motion trajectory to ensure that the feature points can closely follow the hand-guided trajectory; S550: Variable stiffness visual impedance control based on feature point error And a variable stiffness strategy based on QP optimization to dynamically adjust the virtual stiffness matrix Virtual damping matrix The variable stiffness strategy is as follows: ; ; By using variable stiffness parameters, the controller can respond to contact interference that occurs during the assembly process in real time, ensuring that the robot exhibits compliance during contact and avoiding damage to gears and gear shafts. S560: Calculate the control signal based on the updated impedance parameters and characteristic point error information: ; Among them, control signals Used to guide the movement of the robot's end effector, enabling feature points to closely follow the hand-guided teaching trajectory; S570: Control signals in the feature space Converted into control signals for joint space: ; in It is the pseudo-inverse of the Jacobian matrix and the inverse inertial matrix. By converting the control signal from the feature space to the joint space, the robot can adjust the motion of the end effector according to the calculated joint space control signal. S580: The robot controls the joint space based on the calculated signals. Adjust the movement of the end effector so that the feature points can closely follow the reading trajectory guided by the hand.

[0014] An optimized robot variable stiffness vision-impedance control system includes a Sawyer robot and an end effector. The end effector includes a flange, a six-dimensional force sensor, and a depth camera. The flange is used to integrate the six-dimensional force sensor and the camera at the end of the robot. The six-dimensional force sensor is used to sense the force applied by the human, and the depth camera is used to detect feature points. The Sawyer robot is used to execute the steps of an optimized robot variable stiffness vision-impedance control method.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of an optimized robot variable stiffness vision-impedance control method.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an optimized visual-impedance control method for variable stiffness of a robot.

[0017] The aforementioned optimization-based robot variable stiffness vision-impedance control method and system analyzes the interactive dynamics between the robot and the environment, constructs a mathematical model of robot-environment interaction, and establishes a feature space human-robot-environment interactive dynamic model by solving the visual servo acceleration model and combining it with the robot's dynamic equations during assembly. A feature space human-guided dynamic model is established as a planner to obtain the reference feature trajectory implicitly representing human recognition contact dynamics. Furthermore, a QP-based optimization method is introduced to design a QP-based online planning feature space variable stiffness impedance controller, which can dynamically adjust impedance parameters during assembly and compensate for interference forces caused by environmental uncertainties in real time, maintaining constant end-effector contact force. This improves the stability and accuracy of contact-rich tasks, ultimately enabling the robot to complete assembly operations stably, intelligently, flexibly, and with high precision. Attached Figure Description

[0018] Figure 1 This is a flowchart of an optimized robot variable stiffness vision-impedance control method according to an embodiment of the present invention; Figure 2 This is a control block diagram of a feature-based human conduction impedance learning adaptive control algorithm in one embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] In one embodiment, such as Figure 1 and Figure 2 As shown, an optimized vision-impedance control method for variable stiffness of a robot is proposed, the method comprising the following steps: S100: Establish coordinate transformation relationships between the camera, the six-dimensional force sensor, and the robot end effector to ensure the effective fusion of subsequent visual and force information; at the same time, calibrate the six-dimensional force sensor by using a force calibration algorithm to compensate for the sensor's own zero drift and tool gravity, thereby improving its measurement accuracy and ensuring the reliability of force data. S200: Considering the dynamic characteristics of the robot under force, including the guiding force applied by the human and the reaction force of the environment, and combining the robot's kinematics and dynamic equations, as well as the visual servo acceleration equation, a complete human-robot-environment interaction dynamic model in the feature space is constructed. S300: Based on the dynamic model, a human guidance force model is established in the feature space. Visual features are associated with robot motion and a reference trajectory is generated. A variable stiffness visual impedance controller is designed. The stiffness matrix is ​​optimized online through QP (Quadratic Programming) to ensure that the contact force is constant and the torque constraint is satisfied. The system equation is derived to achieve the convergence of external forces. The energy function is constructed to analyze the stability. S400: Based on the stability analysis results, online optimization based on energy tank stiffness enhancement is introduced. The energy tank state equation and upper and lower limit constraints are constructed to limit active energy injection. A quadratic programming QP model is established to optimize stiffness parameters in real time under passive constraints and ensure that the total energy is bounded. Passive analysis is performed by constructing a system storage function. S500: The robot assembles gears using a human-guided robot. It integrates six-dimensional force sensing and RealSense visual data, extracts feature point trajectory errors in real time based on the AprilTag algorithm, drives QP optimization to dynamically adjust variable stiffness parameters, and combines Jacobi pseudo-inverse mapping to the joint space to drive the robot arm movement, achieving compliant trajectory tracking.

[0021] Specifically, in response to the following problems in robot vision-force sensing fusion human-guided learning assembly: (1) nonlinear dynamics of human-robot-environment interaction coupling; (2) heterogeneous vision-force sensing data, which are difficult to fuse; and (3) contact uncertainty dynamic problems generated in assembly tasks, this invention proposes an optimization-based robot variable stiffness vision-impedance control method.

[0022] When performing assembly tasks, humans naturally combine visual and force information and adjust their actions based on environmental feedback. Inspired by this, this invention integrates visual and force sensors into a robot and, by mimicking human interaction and learning mechanisms, enables the robot to learn human impedance characteristics. It fuses visual and force data within a feature space, eliminating the inherent differences in perception modes and data dimensionality between visual and force sensors. By analyzing the dynamic relationship between the robot and its environment and constructing a mathematical model of robot-environment interaction, and by solving the visual servo acceleration model, combined with the robot's dynamic equations during assembly, a feature space human-robot-environment interaction dynamic model is established. A feature space human-guided dynamic model is also established as a planner to obtain reference feature trajectories that implicitly represent human contact dynamics. Furthermore, a QP-based optimization method is introduced to design a QP-based online planning feature space variable stiffness impedance controller. This controller can dynamically adjust impedance parameters during assembly and compensate for interference forces caused by environmental uncertainties in real time, maintaining constant end-effector contact force. This improves the stability and accuracy of contact-rich tasks, ultimately enabling the robot to complete assembly operations stably, intelligently, flexibly, and with high precision.

[0023] In one embodiment, S100 establishes the coordinate transformation relationship between the camera, the six-dimensional force sensor, and the robot end effector, including: S110: A six-dimensional force sensor and an RGB-D camera are installed at the end of the robotic arm to ensure that the camera's field of view is unobstructed; S120: Use the end-effector camera to capture an image of a calibration board with known geometric features (such as N points) and record the corresponding end-effector pose at this time; the feature points on the calibration board should have known geometric relationships so that the pose of the camera relative to the calibration board can be determined by the image processing algorithm; S130: To improve the accuracy and reliability of the calibration, repeat the S120 preset number of times (12 times) at different positions and orientations, take images of the calibration board, and simultaneously record the pose data of the robot end effector; this helps to reduce noise interference and improve the stability of the solution. S140: By analyzing the captured calibration board images and recorded end-effector pose data, the rotation matrix of the camera relative to the robotic arm end effector is calculated. Translation vector By using the Tsai-Lenz method combined with hand-eye calibration technology, the coordinate transformation relationship between the camera and the robotic arm end effector was solved. Specifically: ; S150: Using modeling software, an assembly model of the flange, six-dimensional force sensor, and camera is constructed. The rotation matrix of the six-dimensional force sensor relative to the camera is calculated through geometric analysis. Translation vector The coordinate transformation relationship between the six-dimensional force sensor and the camera was obtained. Specifically: .

[0024] In one embodiment, S200 includes: S210: When a human applies forces to a robot and causes it to interact with its environment, in the joint space, the human-guided... The dynamic characteristics of the linkage robot are described as follows: ; in, These are the generalized variables of the robot's joint angles, joint velocities, and joint accelerations. This represents a symmetric and positive definite inertia matrix. The Coriolis force and centripetal force matrix are described. It is a gravity vector. It is a joint drive command vector. It is the torque applied by the human to the end effector. It is an external torque; S220: Solve for the visual servo dynamics at the acceleration level to characterize the motion relationship between visual features and joint variables, considering... Image features Motion of image features and associated motion spinors The kinematic relationship is as follows: ; in, It is the speed of image feature generation. It is an image interaction matrix used to correlate the velocity of feature points with the robot's motion spinor. and These represent the motion spins of the camera and the object, respectively. Therefore, the difference between the two reflects the relative motion between them. S230: When the object is stationary... In this case, the above equation can be used to characterize the kinematic relationship between the characteristic velocity and the joint velocity through the robot's Jacobian function, that is: ; in The Jacobian matrix represents the task. It is the Jacobian matrix of robots. It is the spinor transformation matrix (in this formula, It is a constant matrix (because the camera is rigidly mounted on the robot's end effector). In this way, the velocity of the image features can be expressed as a function of the joint velocity, thus providing a basis for the design of subsequent control strategies. S240: Differentiating the equation in S230 with respect to time yields an accelerated form of the image features: ; To facilitate the concise derivation of subsequent expressions, the above expression is rewritten as: ; in, It is the acceleration of image features. Here is the acceleration coupling matrix; S250: The dynamic equations of the human-guided n-link robot in S210 are rearranged to derive the dynamic characteristics of the human-guided robot in the feature space: ; Substituting the velocity expressions for the characteristic points in S230 and the acceleration expressions for the characteristic points in S240 into the dynamic characteristic formula, we obtain the dynamic equations related to the characteristic motion: ; in This is a comprehensive force term; the above formula is a set of terms related to... Nonlinear, highly coupled differential equations corresponding to each image feature; S260: By introducing the task Jacob coefficient Consider a projection matrix that represents the torque applied by the human in the human-tool coordinate system. Transform to the camera coordinate system, therefore we have , Substituting into the dynamic equations related to the features above, we get: ; in, This represents a virtual force acting on image features, originating from a human-guided torque projected onto the camera coordinate system. It is a virtual external force in the characteristic space. Represents virtual gravity, virtual Coriolis force, and centripetal force in characteristic space. Represents the damping matrix. This represents the characteristic space impedance controller, which is the virtual control input for the robot in the characteristic space. This represents the torque applied by the human in the camera coordinate system.

[0025] In one embodiment, S300 includes: S310: Virtual human guidance force in feature space And acting on image features, for ease of analysis, the human guiding force projected onto the feature space is modeled as: ; in, These represent the velocity and acceleration variables of human guiding characteristics, respectively. Represents the damping matrix; Based on the above equation, the visual feature variables and the robot's end effector can move in the same direction as the external torque guided by the human, and at the same time, the characteristic trajectory guided by the human can be obtained. This trajectory serves as a reference trajectory for subsequent controller design; S320: A variable-stiffness visual impedance controller based on the S310 is designed and operates in the feature space. The stiffness is adjusted using an online optimized QP method to ensure that the robot end effector maintains a constant force when performing contact-intensive tasks. A feature space impedance controller is designed. Its form is: ; in, It is the feature tracking error vector. The rate of change of feature tracking error. and These represent the stiffness and damping matrices for impedance control, respectively. S330: Substituting the formulas in S310 and S320 into S260, we get: ; Observing the above formula, we can further deduce: ; Since the performance of an impedance system depends on its parameters, it is desirable that the virtual external force in the characteristic space converges to the desired external force. ; S340: To solve the force convergence problem in S320, the following QP problem is constructed for adjusting the stiffness of the impedance system online: ; ; in, and These represent the minimum and maximum allowable stiffness, respectively. This indicates the maximum external torque that the robot can apply. and These are weighted matrices used to adjust the external torque and minimum stiffness, respectively. S350: After completing the above controller optimization, to facilitate subsequent stability analysis, the energy function of the characteristic space impedance system is defined as follows: ; Its derivative with respect to time is: ; Due to the item The sign of the impedance system is unknown beforehand, and it is not feasible to define a suitable energy storage function for the relative ports. Therefore, it is impossible to guarantee the impedance system with respect to the ports. The passive nature of.

[0026] In one embodiment, S400 includes: S410: To ensure the stability of the characteristic space variable impedance system, firstly, the variable stiffness... Rewritten as ,in and Let the constant stiffness component and the time-varying stiffness component be respectively. Without loss of generality, let ; Subsequently, the concept of a virtual energy tank is introduced to separate the potential active behavior of time-varying stiffness components, whose energy is defined as... ,in Indicate its dynamic state: ; in, This represents the rate of change of the dynamic state of the energy tank. For energy flow switching parameters, It is an energy exchange term; in Defined as: ; in This indicates the lower limit of the energy tank. For the spacetime stiffness matrix, in addition, to ensure that the total energy of the entire interconnected system remains bounded, when a certain energy tank upper limit is reached... To prevent the energy tank from overloading, the following methods should be used: ; S420: Therefore, online stiffness optimization under passive constraints, incorporating the energy tank constraint into the optimization problem, can be expressed as the following QP problem: ; ; S430: The final characteristic space impedance controller is described as follows: ; The expression for the virtual external force in the characteristic space is modified as follows: ; By implementing a QP-based online stiffness optimization feature-space variable stiffness visual impedance controller, as described in the two equations above, It can be dynamically adjusted to maintain a constant force in contact-intensive tasks, and the energy tank initializes to meet the following conditions. When the virtual energy tank's energy When the stiffness is constant, it can be continuously optimized online; conversely, when... At that time, the stiffness degenerates into The controller is simplified to a constant stiffness impedance controller; S440: For the modified energy tank constraint controller and virtual external force equations, i.e., equations in S420 and S430, the energy tank dynamic state is shown in equation S410, and the storage function of the entire interconnected system. Selected as: ; Total Energy Differential with respect to time It is given by the following formula: ; because Therefore: ; Therefore, we can conclude that: ; The above equation indicates the existence of the following passive condition: ; in, The value of the storage function at time t. The initial stored function value. For integration variables, For power exchange between the system and the external environment; The above passive conditional inequalities prove that the modified enhanced control framework can achieve variable stiffness in the visual impedance model while safely interacting with any passive environment.

[0027] In one embodiment, S500 includes: S510: A six-dimensional force sensor is installed between the handle and the flange to measure the force and torque applied by the user in real time; a RealSense D435i camera is installed at the front of the flange to capture image information of the gears and gear shafts; the end effector of the robot is connected to the six-dimensional force sensor through the flange to ensure the measurement accuracy and stability of the force sensor. S520: During the experiment, the operator guides the robot to assemble the gear into the gear shaft. At the same time, the RGB-D camera captures the image information of the gear and gear shaft in real time and records the corresponding four sets of feature point trajectory data. These feature point trajectory data include the relative position and attitude information of the gear and gear shaft, which are used for subsequent visual servo control. By recording these data, the operator's skills and dynamic characteristics in the human-guided assembly process are analyzed, providing a reference for the variable stiffness visual impedance control strategy. S530: Based on the feature point trajectory data recorded during human guidance, initialize the parameters of the variable stiffness visual impedance controller. These parameters include the virtual stiffness matrix. Virtual damping matrix Expected contact force ; S540: Feature Point Extraction and Tracking. The RealSense D435i camera acquires images of gears and gear shafts in real-time at a frequency of 30Hz. The AprilTag algorithm is used to identify four key feature points from the images of the gears and gear shafts. The positional information of these feature points will be used as input to the visual servo controller. The error between the current feature point position and the trajectory of the hand-operated indicator is calculated. ,in This is the current location of the feature point. It is a hand-guided display trajectory; by monitoring the error of the feature points in real time, the controller dynamically adjusts the robot's motion trajectory to ensure that the feature points can closely follow the hand-guided trajectory; S550: Variable stiffness visual impedance control based on feature point error And a variable stiffness strategy based on QP optimization to dynamically adjust the virtual stiffness matrix Virtual damping matrix The variable stiffness strategy is as follows: ; ; By using variable stiffness parameters, the controller can respond to contact interference that occurs during the assembly process in real time, ensuring that the robot exhibits compliance during contact and avoiding damage to gears and gear shafts. S560: Calculate the control signal based on the updated impedance parameters and characteristic point error information: ; Among them, control signals Used to guide the movement of the robot's end effector, enabling feature points to closely follow the hand-guided teaching trajectory; S570: Control signals in the feature space Converted into control signals for joint space: ; in It is the pseudo-inverse of the Jacobian matrix and the inverse inertial matrix. By converting the control signal from the feature space to the joint space, the robot can adjust the motion of the end effector according to the calculated joint space control signal. S580: The robot controls the joint space based on the calculated signals. Adjust the movement of the end effector so that the feature points can closely follow the reading trajectory guided by the hand.

[0028] During assembly, the controller continuously adjusts the robot's end effector trajectory to ensure that feature points closely follow the hand-guided display path. Through adaptive impedance control, the robot automatically adjusts force and position during contact, achieving stable and accurate assembly. This human-guided demonstration ( This method, compared to classical vision servo control, Admittance learning control () It exhibits significant advantages in feature point tracking error, assembly accuracy, and stability, enabling high-precision and stable assembly tasks.

[0029] Compared with the prior art, the advantages of the present invention are as follows: (1) This invention provides a unified vision-force framework that integrates visual and force information in a feature space, effectively solving the data heterogeneity problem between visual information and six-dimensional force information. In traditional methods, visual and force information are difficult to directly integrate and process due to their different data types and sources. However, this invention fully utilizes the high precision of visual information and the high sensitivity of force information by integrating them in the feature space, achieving complementarity between the two. This integration method not only improves the adaptability and robustness of robots in complex tasks, but also enables robots to complete tasks more accurately. For example, in assembly tasks, visual information can help robots quickly locate targets, while force information can provide fine adjustments during contact, ensuring the accuracy and quality of assembly.

[0030] (2) This invention considers the influence of the environment and introduces a human-guided learning control strategy to adaptively counteract identified environmental disturbances. In actual tasks, there is a high degree of nonlinear coupling between humans, robots, and the environment, especially in tasks requiring fine manipulation, such as assembly and drilling. These tasks involve contact dynamics, meaning the contact state between the robot and the environment changes continuously as the operation progresses. By introducing human-guided learning control, this invention can learn the operational skills of human operators in complex tasks and adaptively adjust control parameters to complete high-quality assembly tasks. This strategy not only improves the robot's flexibility and adaptability but also enables it to exhibit human-like flexibility and adaptability in actual operation, ensuring the smooth progress of the task.

[0031] (3) This invention significantly improves the robustness and stability of the robot system in complex environments through a variable stiffness visual impedance control strategy based on QP optimization. In actual operation, the contact dynamics between the robot and the environment are complex and variable. Traditional rigid control methods are prone to excessive contact force when facing such complex environments, thereby damaging the robot or the manipulated object. However, this invention, through variable stiffness impedance control, can dynamically adjust the stiffness and damping characteristics of the robot according to the real-time contact state, so that the robot exhibits compliance during the contact process, avoiding system instability or task failure caused by excessive contact force.

[0032] In one embodiment, an optimized robot variable stiffness visual-impedance control system includes a Sawyer robot and an end effector. The end effector includes a flange, a six-dimensional force sensor, and a depth camera. The flange is used to integrate the six-dimensional force sensor and the camera at the end of the robot. The six-dimensional force sensor is used to sense the force applied by the human, and the depth camera is used to detect feature points. The Sawyer robot is used to perform the steps of an optimized robot variable stiffness visual-impedance control method.

[0033] Specific limitations regarding the optimization-based robot variable stiffness vision-impedance control system can be found in the limitations of the optimization-based robot variable stiffness vision-impedance control method described above, and will not be repeated here. Each module in the aforementioned optimization-based robot variable stiffness vision-impedance control system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0034] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of an optimized robot variable stiffness vision-impedance control method.

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an optimized visual-impedance control method for variable stiffness of a robot.

[0036] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0037] The above provides a detailed description of an optimized visual-impedance control method and system for variable stiffness robots. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of these embodiments are merely illustrative of the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims of this invention.

Claims

1. An optimization-based visual-impedance control method for variable stiffness of a robot, characterized in that, The method includes the following steps: S100: Establish coordinate transformation relationships between the camera, the six-dimensional force sensor, and the robot end effector. Simultaneously, calibrate the six-dimensional force sensor and compensate for the sensor's own zero drift and tool gravity through a force calibration algorithm. S200: Considering the dynamic characteristics of the robot under force, including the guiding force applied by the human and the reaction force of the environment, and combining the robot's kinematics and dynamic equations, as well as the visual servo acceleration equation, a complete human-robot-environment interaction dynamic model in the feature space is constructed. S300: Based on the dynamic model, a human guidance force model is established in the feature space. Visual features are associated with robot motion and a reference trajectory is generated. A variable stiffness visual impedance controller is designed. The stiffness matrix is ​​optimized online through QP to ensure constant contact force and satisfy torque constraints. The system equations are derived to achieve external force convergence. An energy function is constructed to analyze stability. S400: Based on the stability analysis results, online optimization based on energy tank stiffness enhancement is introduced. The energy tank state equation and upper and lower limit constraints are constructed to limit active energy injection. A quadratic programming QP model is established to optimize stiffness parameters in real time under passive constraints and ensure that the total energy is bounded. Passive analysis is performed by constructing a system storage function. S500: The robot assembles gears using a human-guided robot. It integrates six-dimensional force sensing and RealSense visual data, extracts feature point trajectory errors in real time based on the AprilTag algorithm, drives QP optimization to dynamically adjust variable stiffness parameters, and combines Jacobi pseudo-inverse mapping to the joint space to drive the robot arm movement, achieving compliant trajectory tracking.

2. The method according to claim 1, characterized in that, The S100 establishes coordinate transformation relationships between the camera, the six-dimensional force sensor, and the robot's end effector, including: S110: A six-dimensional force sensor and an RGB-D camera are installed at the end of the robotic arm to ensure that the camera's field of view is unobstructed; S120: Use the end-effector camera to capture an image of a calibration plate with known geometric features and record the corresponding end-effector pose at that time; S130: Repeat the S120 preset number of times under different positions and postures, take images of the calibration board, and simultaneously record the pose data of the robot end effector; S140: By analyzing the captured calibration board images and recorded end-effector pose data, the rotation matrix of the camera relative to the robotic arm end effector is calculated. Translation vector By using the Tsai-Lenz method combined with hand-eye calibration technology, the coordinate transformation relationship between the camera and the robotic arm end effector was solved. Specifically: ； S150: Using modeling software, an assembly model of the flange, six-dimensional force sensor, and camera is constructed. The rotation matrix of the six-dimensional force sensor relative to the camera is calculated through geometric analysis. Translation vector The coordinate transformation relationship between the six-dimensional force sensor and the camera was obtained. Specifically: 。 3. The method according to claim 2, characterized in that, S200 includes: S210: When a human applies forces to a robot and causes it to interact with its environment, in the joint space, the human-guided... The dynamic characteristics of the linkage robot are described as follows: ； in, These are the generalized variables of the robot's joint angles, joint velocities, and joint accelerations. This represents a symmetric and positive definite inertia matrix. The Coriolis force and centripetal force matrix are described. It is a gravity vector. It is a joint drive command vector. It is the torque applied by the human to the end effector. It is an external torque; S220: Solve for the visual servo dynamics at the acceleration level to characterize the motion relationship between visual features and joint variables, considering... Image features Motion of image features and associated motion spinors The kinematic relationship is as follows: ； in, It is the speed of image feature generation. It is an image interaction matrix used to correlate the velocity of feature points with the robot's motion spinor. and These represent the motion spins of the camera and the object, respectively. S230: When the object is stationary... In this case, the above equation can be used to characterize the kinematic relationship between the characteristic velocity and the joint velocity through the robot's Jacobian function, that is: ； in The Jacobian matrix represents the task. It is the Jacobian matrix of robots. It is a screw transformation matrix. In this way, the velocity of image features can be expressed as a function of joint velocity, thus providing a basis for subsequent control strategy design. S240: Differentiating the equation in S230 with respect to time yields an accelerated form of the image features: ； To facilitate the concise derivation of subsequent expressions, the above expression is rewritten as: ； in, It is the acceleration of image features. Here is the acceleration coupling matrix; S250: The dynamic equations of the human-guided n-link robot in S210 are rearranged to derive the dynamic characteristics of the human-guided robot in the feature space: ； Substituting the velocity expressions for the characteristic points in S230 and the acceleration expressions for the characteristic points in S240 into the dynamic characteristic formula, we obtain the dynamic equations related to the characteristic motion: ； in This is a combined force term; the above formula is a set of terms related to... Nonlinear, highly coupled differential equations corresponding to each image feature; S260: By introducing the task Jacob coefficient Consider a projection matrix that represents the torque applied by the human in the human-tool coordinate system. Transform to the camera coordinate system, therefore we have , Substituting into the dynamic equations related to the features above, we get: ； in, This represents a virtual force acting on image features, originating from the man-guided torque projected onto the camera coordinate system. It is a virtual external force in the characteristic space. Represents virtual gravity, virtual Coriolis force, and centripetal force in characteristic space. Represents the damping matrix. This represents the characteristic space impedance controller, which is the virtual control input for the robot in the characteristic space. This represents the torque applied by the human in the camera coordinate system.

4. The method according to claim 3, characterized in that, The S300 includes: S310: Virtual human guidance force in feature space And acting on image features, for ease of analysis, the human guiding force projected onto the feature space is modeled as: ； in, These represent the velocity and acceleration variables of human guiding characteristics, respectively. Represents the damping matrix; Based on the above equation, the visual feature variables and the robot's end effector can move in the same direction as the external torque guided by the human, and at the same time, the characteristic trajectory guided by the human can be obtained. This trajectory serves as a reference trajectory for subsequent controller design; S320: A variable-stiffness visual impedance controller based on the S310 is designed and operates in the feature space. The stiffness is adjusted using an online optimized QP method to ensure that the robot end effector maintains a constant force when performing contact-intensive tasks. A feature space impedance controller is designed. Its form is: ； in, It is the feature tracking error vector. The rate of change of feature tracking error. and These represent the stiffness and damping matrices for impedance control, respectively. S330: Substituting the formulas in S310 and S320 into S260, we get: ； Observing the above formula, we can further deduce: ； Since the performance of an impedance system depends on its parameters, it is desirable that the virtual external force in the characteristic space converges to the desired external force. ; S340: To solve the force convergence problem in S320, the following QP problem is constructed for adjusting the stiffness of the impedance system online: ； ； in, and These represent the minimum and maximum allowable stiffness, respectively. This indicates the maximum external torque that the robot can apply. and These are weighted matrices used to adjust the external torque and minimum stiffness, respectively. S350: After completing the above controller optimization, to facilitate subsequent stability analysis, the energy function of the characteristic space impedance system is defined as follows: ； Its derivative with respect to time is: ； Due to the item The sign of the impedance system is unknown beforehand, and it is not feasible to define a suitable energy storage function for the relative ports. Therefore, it is impossible to guarantee the impedance system with respect to the ports. The passive nature of.

5. The method according to claim 4, characterized in that, The S400 includes: S410: To ensure the stability of the characteristic space variable impedance system, firstly, the variable stiffness... Rewritten as ,in and Let the constant stiffness component and the time-varying stiffness component be respectively. Without loss of generality, let ; Subsequently, the concept of a virtual energy tank is introduced to separate the potential active behavior of time-varying stiffness components, whose energy is defined as... ,in Indicate its dynamic state: ； in, This represents the rate of change of the dynamic state of the energy tank. For energy flow switching parameters, It is an energy exchange term; in Defined as: ； in This indicates the lower limit of the energy tank. For the spacetime stiffness matrix, in addition, to ensure that the total energy of the entire interconnected system remains bounded, when a certain energy tank upper limit is reached... To prevent the energy tank from overloading, the following methods should be used: ； S420: Therefore, online stiffness optimization under passive constraints, incorporating the energy tank constraint into the optimization problem, can be expressed as the following QP problem: ； ； S430: The final characteristic space impedance controller is described as follows: ； The expression for the virtual external force in the characteristic space is modified as follows: ； By implementing a QP-based online stiffness optimization feature-space variable stiffness visual impedance controller, as described in the two equations above, It can be dynamically adjusted to maintain a constant force in contact-intensive tasks, and the energy tank initializes to meet the following conditions. When the virtual energy tank's energy When the stiffness is constant, it can be continuously optimized online; conversely, when... At that time, the stiffness degenerates into The controller is simplified to a constant stiffness impedance controller; S440: For the modified energy tank constraint controller and virtual external force equations, i.e., equations in S420 and S430, the dynamic state of the energy tank is shown in equation S410, and the storage function of the entire interconnected system. Selected as: ； Total Energy Differential with respect to time It is given by the following formula: ； because Therefore: ； Therefore, we can conclude that: ； The above equation indicates the existence of the following passive condition: ； in, The value of the storage function at time t. The stored function value at the initial moment. For integration variables, For power exchange between the system and the external environment; The above passive conditional inequalities prove that the modified enhanced control framework can achieve variable stiffness in the visual impedance model while safely interacting with any passive environment.

6. The method according to claim 5, characterized in that, The S500 includes: S510: A six-dimensional force sensor is installed between the handle and the flange to measure the force and torque applied by the user in real time; a RealSense D435i camera is installed at the front of the flange to capture image information of the gears and gear shafts; the end effector of the robot is connected to the six-dimensional force sensor through the flange to ensure the measurement accuracy and stability of the force sensor. S520: During the experiment, the operator guides the robot to assemble the gear into the gear shaft. At the same time, the RGB-D camera captures the image information of the gear and gear shaft in real time and records the corresponding four sets of feature point trajectory data. These feature point trajectory data include the relative position and attitude information of the gear and gear shaft, which are used for subsequent visual servo control. By recording these data, the operator's skills and dynamic characteristics in the human-guided assembly process are analyzed, providing a reference for the variable stiffness visual impedance control strategy. S530: Based on the feature point trajectory data recorded during human guidance, initialize the parameters of the variable stiffness visual impedance controller. These parameters include the virtual stiffness matrix. Virtual damping matrix Expected contact force ; S540: Feature Point Extraction and Tracking. The RealSense D435i camera acquires images of gears and gear shafts in real-time at a frequency of 30Hz. The AprilTag algorithm is used to identify four key feature points from the images of the gears and gear shafts. The positional information of these feature points will be used as input to the visual servo controller. The error between the current feature point position and the trajectory of the hand-operated indicator is calculated. ,in This is the current location of the feature point. It is a hand-guided display trajectory; by monitoring the error of the feature points in real time, the controller dynamically adjusts the robot's motion trajectory to ensure that the feature points can closely follow the hand-guided trajectory; S550: Variable stiffness visual impedance control based on feature point error And a variable stiffness strategy based on QP optimization to dynamically adjust the virtual stiffness matrix. Virtual damping matrix The variable stiffness strategy is as follows: ； ； By using variable stiffness parameters, the controller can respond to contact interference that occurs during the assembly process in real time, ensuring that the robot exhibits compliance during contact and avoiding damage to gears and gear shafts. S560: Calculate the control signal based on the updated impedance parameters and characteristic point error information: ； Among them, control signals Used to guide the movement of the robot's end effector, enabling feature points to closely follow the taught trajectory guided by the human hand; S570: Control signals in the feature space Converted into control signals for joint space: ； in It is the pseudo-inverse of the Jacobian matrix and the inverse inertial matrix. By converting the control signal from the feature space to the joint space, the robot can adjust the motion of the end effector according to the calculated joint space control signal. S580: The robot controls the joint space based on the calculated signals. Adjust the movement of the end effector so that the feature points can closely follow the reading trajectory guided by the hand.

7. An optimized robot variable stiffness vision-impedance control system, characterized in that, The device includes a Sawyer robot and an end effector. The end effector comprises a flange, a six-dimensional force sensor, and a depth camera. The flange is used to integrate the six-dimensional force sensor and the camera at the end of the robot. The six-dimensional force sensor is used to sense the force applied by a person, and the depth camera is used to detect feature points. The Sawyer robot is used to perform the steps of the method as described in any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images