Robust visual constraint control method, system and computer device for manipulator

Abstract

The present invention discloses a robust visual constraint control method, system and computer equipment for a manipulator. The dynamic model of the visual servo system as a whole is established by combining the dynamic model of the manipulator, the visual servo kinematic model and the hand-eye calibration relationship between the camera and the manipulator. The visual servo kinematic model is combined to obtain the constraint satisfaction condition of constraining the visual servo feature error within a preset range by designing a time-varying asymmetric control obstacle function, and the optimal visual servo speed controller is designed. The robot dynamics model is combined to obtain the constraint satisfaction condition of constraining the joint angular velocity within a preset range by using the time-varying asymmetric control obstacle function. At the same time, the optimal torque controller is designed in combination with the visual servo feature error constraint satisfaction condition. Combined with quadratic programming, the torque control signal that satisfies the performance constraint is obtained to drive the robot to the desired feature position. The success rate of visual servo tasks is improved, and high-precision visual guided assembly operations of the manipulator are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing, and in particular relates to a robust visual constraint control method, system and computer equipment for a manipulator. Background Art

[0002] In the field of industrial production, there are a large number of small-batch, multi-variety production tasks. Robot systems face challenges such as unstructured manufacturing environment, small working space, variable workstations, and mixed production of multiple varieties and multiple processes. In existing traditional industrial production lines, robotic arms do not have visual guidance functions. Most of the work of robotic arms is to follow set procedures and follow specified paths to complete repetitive work. When the task needs to be changed or the target object deviates from the instruction setting, the robotic arm cannot be adjusted in time according to changes in the environment, which greatly reduces the actual value of the industrial robotic arm.

[0003] In recent years, visual servo technology has been proposed and widely used in logistics robots, service robots, surgical robots, industrial robots and other fields. Visual servo systems can achieve high-precision positioning and recognition, enabling robots to perform precise tasks such as assembly, welding and inspection; and can automatically adjust posture and position errors, reduce the robot system's dependence on precise position, and reduce the difficulty of control; in addition, visual servo control has good adaptability and can adapt to workpieces of different sizes and shapes without complex reprogramming or tool replacement. Therefore, the introduction of visual servo control for unstructured manufacturing environments can enhance the robot's flexibility and adaptability to the environment through visual perception of the environment.

[0004] Visual servo control can be summarized into three categories. According to the different control input variables, visual servo can be divided into image-based visual servo (IBVS) and position-based visual servo (PBVS). Combining 3D posture and 2D image feature signals as control input variables is defined as hybrid visual servo (HVS). The PBVS method relies on the 3D position of the camera to adjust the posture of the end effector in Cartesian space. Due to its global asymptotic stability, this method is suitable for industrial robot manipulators. However, since the PBVS method is inevitably affected by camera calibration errors and robot modeling errors, it is difficult to achieve high-precision control. In contrast, IBVS directly uses image feature variables as control input, avoiding the shortcomings of PBVS and achieving precise control. The realization of the IBVS task directly depends on minimizing the error between the desired image features and the current image features. Therefore, ensuring that the visual features remain within the camera field of view (FoV) is a hard requirement and the key to ensuring the completion of the visual servo task. If some visual features disappear from the camera's field of view, it will cause visual servoing to fail.

[0005] Therefore, the present invention aims to propose a robust visual constraint control method for a manipulator to ensure that visual features are always kept within the camera field of view during the robot's operation task, so as to improve the success rate of visual servoing tasks and realize high-precision vision-guided assembly operations of the manipulator. Summary of the invention

[0006] In view of the above technical problems, the present invention provides a robust visual constraint control method, system and computer device for a manipulator.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A robust visual constraint control method for a manipulator, the method comprising the following steps:

[0009] S100: Calibrate the coordinate transformation relationship between the robot arm and the camera using the hand-eye calibration method;

[0010] S200: Combine the robot arm dynamics model, visual servo kinematics model, and the hand-eye calibration relationship between the camera and the robot arm to establish the overall dynamic model of the visual servo system;

[0011] S300: Combining the visual servoing kinematics model, by designing a time-varying asymmetric control obstacle function, the constraint satisfaction condition for constraining the visual servoing feature error within a preset error range is obtained, and the optimal visual servoing speed controller is designed;

[0012] S400: The optimal visual servo speed controller solves the robot reference joint angular velocity through the speed conversion matrix and the robot inverse kinematics, and defines the joint angular velocity tracking error. Combined with the robot arm dynamics model, the time-varying asymmetric control obstacle function is used to obtain the constraint satisfaction condition that constrains the joint angular velocity within the preset joint angular velocity range. At the same time, combined with the visual servo characteristic error constraint satisfaction condition in S300, the optimal torque controller is designed, and combined with quadratic programming, the torque control signal that meets the performance constraint is obtained;

[0013] S500: Send the torque control signal to the robot actuator to drive the robot to reach the desired feature position, and ensure that the feature point trajectory does not exceed the camera field of view, while ensuring the smoothness of the trajectory.

[0014] Preferably, S100 includes:

[0015] S110: prepare a calibration plate with N points;

[0016] S120: placing the calibration plate under the camera, evenly capturing 12 groups of calibration plate images under different camera postures with the calibration point as the center, and recording the corresponding 12 groups of robot arm end postures;

[0017] S130: The coordinate transformation relationship between the camera and the end of the robotic arm is solved by the Tsai-Lenz method and the hand-eye calibration coordinate transformation relationship , specifically:

[0018] ;

[0019] in, Indicates the coordinate transformation relationship between the camera and the end of the robotic arm. and are the rotation matrix and translation vector from the camera coordinate system to the robot end coordinate system respectively.

[0020] Preferably, S200 includes:

[0021] S210: Define the dynamic model of the n-link rigid robot system, specifically:

[0022] ;

[0023] in, is the mass inertia matrix of the robot, Represents the robot's centripetal force and Coriolis force matrix, represents the robot gravity matrix, Respectively represent the robot's joint angle position, joint angle velocity and joint angle acceleration variables; Input torque variables for robot control;

[0024] The dynamic model of the robot arm is expressed as a state equation:

[0025] ;

[0026] S220: Consider a camera rigidly connected to the end of the robotic arm, image feature points The speed of the camera There is a kinematic relationship between them, specifically:

[0027] ;

[0028] in represents the velocity of the i-th image feature point, Represents the total number of image feature points, represents the image interaction matrix, Represents the depth information of image feature points;

[0029] S230: Considering the conversion relationship between the end velocity of the robot arm and the angular velocity of the robot arm joint, specifically:

[0030] ;

[0031] in, represents the Jacobian matrix of the robot arm, Represents the speed of the end effector of the robot arm;

[0032] According to the coordinate transformation matrix between the end of the robot and the camera , the speed conversion equation between the end effector speed and the camera speed can be obtained, specifically:

[0033] ;

[0034] in, represents the camera speed, is the speed conversion matrix between the camera speed and the end effector speed, specifically:

[0035] ;

[0036] in Represents the rotation matrix from the robot end coordinate system to the camera coordinate system, The antisymmetric matrix representing the translation vector from the robot end coordinate system to the camera coordinate system;

[0037] S240: Combining the image feature point velocity and camera velocity equation in S220, a visual servo kinematics model is obtained, specifically:

[0038] ;

[0039] in, is the task Jacobian matrix;

[0040] S250: Combining the robot arm dynamics model in S210 with the visual servo kinematics model in S240, the overall dynamic equation of the visual servo system is obtained, which is:

[0041] .

[0042] Preferably, S300 includes:

[0043] S310: Define image feature error:

[0044] ;

[0045] in, is the desired feature point, Represents the image feature error signal; for the speed control of the camera coordinate system, the following nominal control law is designed :

[0046] ;

[0047] in, is a normal number, Pseudo-inverse matrix of the image interaction matrix;

[0048] The image feature error change rate is:

[0049] ;

[0050] in represents the derivative of the image feature error with respect to time, is the expected image feature velocity variable;

[0051] S320: Define tracking signal performance specifications , which is an exponentially decreasing function, specifically:

[0052] ;

[0053] in, The convergence speed of the error performance function can be adjusted. Represent the maximum allowable error and steady-state error respectively;

[0054] definition and , ,in, and are the lower and upper bound functions of the feature error, respectively. and is a positive number; when When , the performance function is asymmetric;

[0055] S330: Define the zero-point control barrier function to ensure ,definition

[0056] ;

[0057] in is the first image feature error vector elements, and Represents the image feature errors The lower and upper bounds of the zero-point control barrier function;

[0058] Define the error transformation vector and ,in , , ;

[0059] Will and Substituting in the error transformation, we can get the following:

[0060] ;

[0061] S340: Design conditions that meet performance constraints, when and hour, ;

[0062] Therefore, to meet the predetermined performance constraints, it is equivalent to designing the controller to ensure and , definition:

[0063] ;

[0064] ;

[0065] in, and Respectively represent and The control barrier function of and Represents the image feature errors Zero-point control barrier function and Derivatives with respect to time; and Represent the upper and lower bound function vectors of image feature error respectively and Derivatives with respect to time; and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ;

[0066] Will Substitution and From the expression of , we can get:

[0067] ;

[0068] ;

[0069] S350: When and When, there is ;when and When, there is ; Combined with the nominal speed control law , use the following quadratic programming algorithm to obtain the speed control variables in the camera coordinate system that meet the field of view constraints As an optimal visual servoing speed controller:

[0070] .

[0071] Preferably, S400 includes:

[0072] S410: Based on the velocity transfer equation between the end effector velocity and the camera velocity And the speed control variable in the camera coordinate system that satisfies the field of view constraint , and get the expected end effector velocity:

[0073] ;

[0074] According to the mapping relationship between the end effector speed and the joint speed, the reference joint angular velocity can be obtained. for:

[0075] ;

[0076] in, is the pseudo-inverse matrix of the Jacobian matrix of the robot;

[0077] Define joint velocity tracking error :

[0078] ;

[0079] in, is the joint angular velocity of the robot;

[0080] Defining tracking signal performance specifications , which is an exponentially decreasing function, specifically:

[0081] ;

[0082] in, The convergence speed of the error performance function can be adjusted. They represent the maximum allowable error and steady-state error of the joint angular velocity error respectively;

[0083] definition and , ,in, and are the lower and upper bound functions of the joint angular velocity error, respectively. and is a positive number; when When , the performance function is asymmetric;

[0084] Define the zero-point control barrier function for the joint angular velocity tracking error to ensure ,definition:

[0085] ;

[0086] in is the joint angular velocity tracking error vector elements, and Represents the joint angular velocity error The lower and upper bounds of the zero-point control barrier function;

[0087] Define the error transformation vector and ,in , ;Will and , Substituting the expression into the error transformation, the error transformation is as follows:

[0088] ;

[0089] The design meets the performance constraints when and hour, ;

[0090] Therefore, to meet the predetermined performance constraints, it is equivalent to designing the controller to ensure and ,definition:

[0091] ;

[0092] in, and Respectively about and The control barrier function is and Is an extended class Function; when and hour, and ; Therefore, to achieve the joint angular velocity error constraint, the control input Should meet and ,Right now:

[0093] ;

[0094] in, and Represents the control obstacle function of the joint angular velocity error and Differentiation of time; and Represent the upper and lower bound function vectors of image feature error respectively and The first derivative of ; is the expected joint angular acceleration; is the joint angular acceleration, and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ;

[0095] Because for any positive number ,have:

[0096] ;

[0097] in, is the control obstacle function of the field of view constraint and The minimum value of is expressed as

[0098] ;

[0099] In order to integrate the control obstacle function of the field of view constraint into the robot dynamics, the following energy-based obstacle function is proposed :

[0100] ;

[0101] when hour, ,because is the control obstacle function of the field of view constraint and The minimum value of , so we have and , that is, satisfying the field of view constraint; therefore, to satisfy the predetermined field of view constraint is equivalent to designing a controller to ensure ,definition:

[0102] ;

[0103] in, Is an extended class Function, that is, is a strictly increasing function and satisfies, ;

[0104] when hour, Therefore, to realize the joint angular velocity error constraint, combined with the robot dynamics model , control input Should meet ,Right now:

[0105] ;

[0106] The kinetic model ,replace and The joint angular acceleration in , and obtain the torque control signal that satisfies the field of view constraint and joint angular velocity performance constraint , that is:

[0107] ;

[0108] in, is a nominal controller, which is usually the controller designed without considering performance constraints.

[0109] Preferably, S500 includes:

[0110] S510: Using the image processing library installed on the industrial computer, extract the coordinates of the feature corner points of the captured QR code and write a robust visual servo controller program;

[0111] S520: The calculated torque control signal is transmitted to the SDK programming platform on the same industrial computer in the form of a ROS topic to write a robot torque control program;

[0112] S530: By properly adjusting parameters, the robot reaches the specified position.

[0113] The robust visual constraint control system for the manipulator includes a coordinate conversion module, a visual servo system dynamic model building module, a constraint satisfaction condition determination module, a control signal determination module, and a robot drive module.

[0114] The coordinate conversion module is used to calibrate the coordinate conversion relationship between the robot arm and the camera using the hand-eye calibration method;

[0115] The visual servo system dynamic model building module is used to combine the robot arm dynamics model, the visual servo kinematics model, and the hand-eye calibration relationship between the camera and the robot arm to build the overall dynamic model of the visual servo system;

[0116] The constraint satisfaction condition determination module is used to combine the visual servoing kinematics model, obtain the constraint satisfaction condition for constraining the visual servoing feature error within a preset error range by designing a time-varying asymmetric control obstacle function, and design the optimal visual servoing speed controller;

[0117] The control signal determination module is used to solve the robot reference joint angular velocity according to the optimal visual servo speed controller through the speed conversion matrix and the robot inverse kinematics, and define the joint angular velocity tracking error. In combination with the mechanical arm dynamics model, the constraint satisfaction condition for constraining the joint angular velocity within the preset joint angular velocity range is obtained through the time-varying asymmetric control obstacle function. At the same time, the optimal torque controller is designed in combination with the visual servo characteristic error constraint satisfaction condition in the constraint satisfaction condition determination module, and the torque control signal that meets the performance constraint is obtained in combination with the quadratic programming.

[0118] The robot drive module is used to send the torque control signal to the robot actuator to drive the robot to the desired feature position and ensure that the feature point trajectory does not exceed the camera field of view while ensuring the smoothness of the trajectory.

[0119] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a robust visual constraint control method for a manipulator when executing the computer program.

[0120] The above-mentioned robust visual constraint control method, system and computer equipment for the manipulator adopt a constraint satisfaction control scheme with a time-varying asymmetric control obstacle function to design an image-based visual servoing (IBVS) constraint controller and an optimal torque constraint controller, and adopt a time-varying control obstacle function to ensure that the feature point tracking error and the joint angular velocity tracking error converge to a strictly specified range, and ensure the transient response of the tracking error, so that the visual features are always retained in the camera's field of view during the robot's operation task, so as to improve the success rate of the visual servoing task and realize the high-precision visual guided assembly operation of the manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 The flowchart is a robust visual constraint control method for a manipulator in one embodiment of the present invention.

[0122] Figure 2 This is a control block diagram of a robot vision robust constraint algorithm based on CBF in one embodiment of the present invention. DETAILED DESCRIPTION

[0123] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0124] In one embodiment, Figure 1 and Figure 2 As shown, a robust visual constraint control method for a manipulator, the method comprising the following steps:

[0125] S100: Calibrate the coordinate transformation relationship between the robot arm and the camera using the hand-eye calibration method;

[0126] S200: Combine the robot arm dynamics model, visual servo kinematics model, and the hand-eye calibration relationship between the camera and the robot arm to establish the overall dynamic model of the visual servo system;

[0127] S300: Combining the visual servoing kinematics model, by designing a time-varying asymmetric control obstacle function, the constraint satisfaction condition for constraining the visual servoing feature error within a preset error range is obtained, and the optimal visual servoing speed controller is designed;

[0128] S400: The optimal visual servo speed controller solves the robot reference joint angular velocity through the speed conversion matrix and the robot inverse kinematics, and defines the joint angular velocity tracking error. Combined with the robot arm dynamics model, the time-varying asymmetric control obstacle function is used to obtain the constraint satisfaction condition that constrains the joint angular velocity within the preset joint angular velocity range. At the same time, combined with the visual servo characteristic error constraint satisfaction condition in S300, the optimal torque controller is designed, and combined with quadratic programming, the torque control signal that meets the performance constraint is obtained;

[0129] S500: Send the torque control signal to the robot actuator to drive the robot to reach the desired feature position, and ensure that the feature point trajectory does not exceed the camera field of view, while ensuring the smoothness of the trajectory.

[0130] The present invention proposes a robust visual constraint control method for a manipulator, which aims to solve the problem of visual feature loss during the operation of a visual servoing system, resulting in the failure of the visual servoing task. A visual servoing field of view constraint controller is designed by an asymmetric control obstacle function, and a strictly formulated error range is defined for the feature coordinate error, thereby ensuring the transient response of the feature tracking error and achieving arbitrary tracking accuracy.

[0131] In one embodiment, S100 includes:

[0132] S110: prepare a calibration plate with N points;

[0133] S120: placing the calibration plate under the camera, evenly capturing 12 groups of calibration plate images under different camera postures with the calibration point as the center, and recording the corresponding 12 groups of robot arm end postures;

[0134] S130: The coordinate transformation relationship between the camera and the end of the robotic arm is solved by the Tsai-Lenz method and the hand-eye calibration coordinate transformation relationship , specifically:

[0135] ;

[0136] in, Indicates the coordinate transformation relationship between the camera and the end of the robotic arm. and are the rotation matrix and translation vector from the camera coordinate system to the robot end coordinate system respectively.

[0137] In one embodiment, Figure 2 As shown, S200 includes:

[0138] S210: Define the dynamic model of the n-link rigid robot system, specifically:

[0139] ;

[0140] in, is the mass inertia matrix of the robot, Represents the robot's centripetal force and Coriolis force matrix, represents the robot gravity matrix, Respectively represent the robot's joint angle position, joint angle velocity and joint angle acceleration variables; Input torque variables for robot control;

[0141] The dynamic model of the robot arm is expressed as a state equation:

[0142] ;

[0143] S220: Consider a camera rigidly connected to the end of the robotic arm, image feature points The speed of the camera There is a kinematic relationship between them, specifically:

[0144] ;

[0145] in represents the velocity of the i-th image feature point, Represents the total number of image feature points, represents the image interaction matrix, Represents the depth information of image feature points;

[0146] S230: Considering the conversion relationship between the end velocity of the robot arm and the angular velocity of the robot arm joint, specifically:

[0147] ;

[0148] in, represents the Jacobian matrix of the robot arm, Represents the speed of the end effector of the robot arm;

[0149] According to the coordinate transformation matrix between the end of the robot and the camera , the speed conversion equation between the end effector speed and the camera speed can be obtained, specifically:

[0150] ;

[0151] in, represents the camera speed, is the speed conversion matrix between the camera speed and the end effector speed, specifically:

[0152] ;

[0153] in Represents the rotation matrix from the robot end coordinate system to the camera coordinate system, The antisymmetric matrix representing the translation vector from the robot end coordinate system to the camera coordinate system;

[0154] S240: Combining the image feature point velocity and camera velocity equation in S220, a visual servo kinematics model is obtained, specifically:

[0155] ;

[0156] in, is the task Jacobian matrix;

[0157] S250: Combining the robot arm dynamics model in S210 with the visual servo kinematics model in S240, the overall dynamic equation of the visual servo system is obtained, which is:

[0158] .

[0159] In one embodiment, S300 includes:

[0160] S310: Define image feature error:

[0161] ;

[0162] in, is the desired feature point, Represents the image feature error signal; for the speed control of the camera coordinate system, the following nominal control law is designed :

[0163] ;

[0164] in, is a normal number, Pseudo-inverse matrix of the image interaction matrix;

[0165] The image feature error change rate is:

[0166] ;

[0167] in represents the derivative of the image feature error with respect to time, is the expected image feature velocity variable;

[0168] S320: Define tracking signal performance specifications , which is an exponentially decreasing function, specifically:

[0169] ;

[0170] in, The convergence speed of the error performance function can be adjusted. Represent the maximum allowable error and steady-state error respectively;

[0171] definition and , ,in, and are the lower and upper bound functions of the feature error, respectively. and is a positive number; when When , the performance function is asymmetric;

[0172] S330: Define the zero-point control barrier function to ensure ,definition

[0173] ;

[0174] in is the first image feature error vector elements, and Represents the image feature errors The lower and upper bounds of the zero-point control barrier function;

[0175] Define the error transformation vector and ,in , , ;

[0176] Will and Substituting in, the error transformation can be obtained as follows:

[0177] ;

[0178] S340: Design conditions that meet performance constraints, when and hour, ;

[0179] Therefore, to meet the predetermined performance constraints, it is equivalent to designing the controller to ensure and ,definition:

[0180] ;

[0181] ;

[0182] in, and Respectively represent and The control barrier function of and Represents the image feature errors Zero-point control barrier function and Derivatives with respect to time; and Represent the upper and lower bound function vectors of image feature error respectively and Derivatives with respect to time; and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ;

[0183] Will Substitution and From the expression of , we can get:

[0184] ;

[0185] ;

[0186] S350: When and When, there is ;when and When, there is ; Combined with the nominal speed control law , use the following quadratic programming algorithm to obtain the speed control variables in the camera coordinate system that meet the field of view constraints As the optimal visual servoing speed control law (CBF-IBVS control law):

[0187] .

[0188] Furthermore, when and When, there is ;when and When, there is , the proof process is as follows:

[0189] Assume existence , so that .because ,therefore, must go from non-negative to negative at some point. The continuity of , , so that Consider The derivative at .Depend on and ,have to:

[0190] ;

[0191] because When , its instantaneous rate of change is non-negative, that is, This means that hour, will not jump from non-negative to negative. Therefore, when and hour, exist Always satisfied Similarly, when and hour, exist Always satisfied That is, when and When , the field of view meets the predetermined performance constraints. In addition, due to A strictly increasing function that satisfies, , so when and When , Available , so when and hour, This means that even if the image feature error signal does not meet the constraints at the beginning, it will converge to the boundary that meets the constraints, and once it enters the boundary of the performance constraints, so: and When, there is Similarly, when and When, there is . and By choosing a sufficiently large , and To guarantee.

[0192] In one embodiment, Figure 2 As shown, S400 includes:

[0193] S410: Based on the velocity transfer equation between the end effector velocity and the camera velocity And the speed control variable in the camera coordinate system that satisfies the field of view constraint , and get the expected end effector velocity:

[0194] ;

[0195] According to the mapping relationship between the end effector speed and the joint speed, the reference joint angular velocity can be obtained. for:

[0196] ;

[0197] in, is the pseudo-inverse matrix of the Jacobian matrix of the robot;

[0198] Define joint velocity tracking error :

[0199] ;

[0200] in, is the joint angular velocity of the robot;

[0201] Defining tracking signal performance specifications , which is an exponentially decreasing function, specifically:

[0202] ;

[0203] in, The convergence speed of the error performance function can be adjusted. They represent the maximum allowable error and steady-state error of the joint angular velocity error respectively;

[0204] definition and , ,in, and are the lower and upper bound functions of the joint angular velocity error, respectively. and is a positive number; when When , the performance function is asymmetric;

[0205] Define the zero-point control barrier function for the joint angular velocity tracking error to ensure ,definition:

[0206] ;

[0207] in is the joint angular velocity tracking error vector elements, and Represents the joint angular velocity error The lower and upper bounds of the zero-point control barrier function;

[0208] Define the error transformation vector and ,in , ;Will and , Substituting the expression into the error transformation, the error transformation is as follows:

[0209] ;

[0210] The design meets the performance constraints when and hour, ;

[0211] Therefore, to meet the predetermined performance constraints, it is equivalent to designing the controller to ensure and ,definition:

[0212] ;

[0213] in, and Respectively about and The control barrier function is and Is an extended class Function; when and hour, and ; Therefore, to achieve the joint angular velocity error constraint, the control input Should meet and ,Right now:

[0214] ;

[0215] in, and Represents the control obstacle function of the joint angular velocity error and Differentiation of time; and Represent the upper and lower bound function vectors of image feature error respectively and The first derivative of ; is the expected joint angular acceleration; is the joint angular acceleration, and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ;

[0216] Because for any positive number ,have:

[0217] ;

[0218] in, is the control obstacle function of the field of view constraint and The minimum value of is expressed as

[0219] ;

[0220] In order to integrate the control obstacle function of the field of view constraint into the robot dynamics, the following energy-based obstacle function is proposed :

[0221] ;

[0222] when hour, ,because is the control obstacle function of the field of view constraint and The minimum value of , so we have and , that is, satisfying the field of view constraint; therefore, to satisfy the predetermined field of view constraint is equivalent to designing a controller to ensure ,definition:

[0223] ;

[0224] in, Is an extended class Function, that is, is a strictly increasing function and satisfies, ;

[0225] when hour, Therefore, to realize the joint angular velocity error constraint, combined with the robot dynamics model , control input Should meet ,Right now:

[0226] ;

[0227] The kinetic model ,replace and The joint angular acceleration in , and obtain the torque control signal that satisfies the field of view constraint and joint angular velocity performance constraint (CBF torque controller), that is:

[0228] ;

[0229] in, is a nominal controller, which is usually the controller designed without considering performance constraints.

[0230] In one embodiment, S500 includes:

[0231] S510: Using the image processing library installed on the industrial computer, extract the coordinates of the feature corners of the captured QR code and write a robust visual servo controller program;

[0232] S520: The calculated torque control signal is transmitted to the SDK programming platform on the same industrial computer in the form of a ROS topic to write a robot torque control program;

[0233] S530: By properly adjusting parameters, the robot reaches the specified position.

[0234] The robust visual constraint control system for the manipulator includes a coordinate conversion module, a visual servo system dynamic model building module, a constraint satisfaction condition determination module, a control signal determination module, and a robot drive module.

[0235] The coordinate conversion module is used to calibrate the coordinate conversion relationship between the robot arm and the camera using the hand-eye calibration method;

[0236] The visual servo system dynamic model building module is used to combine the robot arm dynamics model, the visual servo kinematics model, and the hand-eye calibration relationship between the camera and the robot arm to build the overall dynamic model of the visual servo system;

[0237] The constraint satisfaction condition determination module is used to combine the visual servoing kinematics model, obtain the constraint satisfaction condition for constraining the visual servoing feature error within a preset error range by designing a time-varying asymmetric control obstacle function, and design the optimal visual servoing speed controller;

[0238] The control signal determination module calculates the reference joint angular velocity of the robot through the speed conversion matrix and the robot inverse kinematics according to the optimal visual servo speed controller, and defines the joint angular velocity tracking error. Combined with the mechanical arm dynamics model, the constraint satisfaction condition that constrains the joint angular velocity within the preset joint angular velocity range is obtained through the time-varying asymmetric control obstacle function. At the same time, combined with the visual servo characteristic error constraint satisfaction condition in the constraint satisfaction condition determination module, the optimal torque controller is designed, and combined with quadratic programming, the torque control signal that meets the performance constraint is obtained;

[0239] The robot drive module is used to send the torque control signal to the robot actuator to drive the robot to the desired feature position and ensure that the feature point trajectory does not exceed the camera field of view while ensuring the smoothness of the trajectory.

[0240] For the specific definition of a robust visual constraint control system for a manipulator, please refer to the definition of a robust visual constraint control method for a manipulator mentioned above, which will not be repeated here. Each module in the above-mentioned robust visual constraint control system for a manipulator can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0241] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a robust visual constraint control method for a manipulator when executing the computer program.

[0242] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0243] Compared with the prior art, the advantages of the present invention are:

[0244] (1) We innovatively adopt a constraint satisfaction control scheme with a time-varying asymmetric control barrier function to design an image-based visual servoing (IBVS) constraint controller to constrain feature points within the camera field of view. The main contribution of this work is that its performance metric is a priori, designable, and imposed without increasing the complexity of the controller.

[0245] (2) The optimal torque controller innovatively achieves the dual constraint satisfaction conditions of joint angular velocity error and visual servoing feature error by combining the time-varying asymmetric control obstacle function and the quadratic programming algorithm. On this basis, the torque controller works together with the IBVS constraint controller to solve the torque control input signal that meets the performance constraints, avoid the feature point trajectory beyond the camera field of view, and achieve high-precision dynamic performance control.

[0246] (3) A time-varying asymmetric control obstacle function is combined with a performance function to ensure that the feature point tracking error and the joint angular velocity error converge to a strictly specified range and to guarantee the transient steady-state performance of the tracking error without requiring any prior knowledge about the nonlinear visual servo system.

[0247] The above is a detailed introduction to a robust visual constraint control method, system and computer device for a manipulator provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A robust visual constraint control method for a manipulator, characterized in that: The method comprises the following steps: S100: Calibrate the coordinate transformation relationship between the robot arm and the camera using the hand-eye calibration method; S200: Combine the robot arm dynamics model, visual servo kinematics model, and the hand-eye calibration relationship between the camera and the robot arm to establish the overall dynamic model of the visual servo system; S300: In combination with the visual servoing kinematics model, a time-varying asymmetric control obstacle function is designed to obtain a constraint satisfaction condition for constraining the visual servoing feature error within a preset error range, and an optimal visual servoing speed controller is designed; wherein the time-varying asymmetric control obstacle function is specifically: ； in, and Represent the current image features and the expected image features respectively, represents the expected image feature velocity; represents the image interaction matrix, Represents the image feature depth information, Represents the camera speed; and Represent the upper and lower bound function vectors of image feature error respectively and Derivatives with respect to time; and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ; and are the lower bound transformation vector and upper bound transformation vector of the image feature tracking error, respectively. and Respectively represent and The control barrier function of The optimal visual servo speed controller is specifically: ； in, is the nominal speed control law; S400: The optimal visual servo speed controller solves the robot reference joint angular velocity through the speed conversion matrix and the robot inverse kinematics, and defines the joint angular velocity tracking error. Combined with the robot arm dynamics model, the time-varying asymmetric control obstacle function is used to obtain the constraint satisfaction condition that constrains the joint angular velocity within the preset joint angular velocity range. At the same time, combined with the visual servo characteristic error constraint satisfaction condition in S300, the optimal torque controller is designed, and combined with quadratic programming, the torque control signal that meets the performance constraint is obtained; S500: Send the torque control signal to the robot actuator to drive the robot to reach the desired feature position, and ensure that the feature point trajectory does not exceed the camera field of view, while ensuring the smoothness of the trajectory.

2. The method according to claim 1, characterized in that S100 includes: S110: prepare a calibration plate with N points; S120: placing the calibration plate under the camera, evenly capturing 12 groups of calibration plate images under different camera postures with the calibration point as the center, and recording the corresponding 12 groups of robot arm end postures; S130: The coordinate transformation relationship between the camera and the end of the robotic arm is solved by the Tsai-Lenz method and the hand-eye calibration coordinate transformation relationship , specifically: ； in, Indicates the coordinate transformation relationship between the camera and the end of the robotic arm. and are the rotation matrix and translation vector from the camera coordinate system to the robot end coordinate system respectively.

3. The method according to claim 2, characterized in that S200 includes: S210: Define the dynamic model of the n-link rigid robot system, specifically: ； in, is the mass inertia matrix of the robot, Represents the robot's centripetal force and Coriolis force matrix, represents the robot gravity matrix, Respectively represent the robot's joint angle position, joint angle velocity and joint angle acceleration variables; Input torque variables for robot control; The dynamic model of the robot arm is expressed as a state equation: ； S220: Consider a camera rigidly connected to the end of the robotic arm, image feature points The speed of the camera There is a kinematic relationship between them, specifically: ； in represents the velocity of the i-th image feature point, Represents the total number of image feature points; S230: Considering the conversion relationship between the end velocity of the robot arm and the angular velocity of the robot arm joint, specifically: ； in, represents the Jacobian matrix of the robot arm, Represents the speed of the end effector of the robot arm; According to the coordinate transformation matrix between the end of the robot and the camera , the speed conversion equation between the end effector speed and the camera speed can be obtained, specifically: ； in, is the speed conversion matrix between the camera speed and the end effector speed, specifically: ； in Represents the rotation matrix from the robot end coordinate system to the camera coordinate system, The antisymmetric matrix representing the translation vector from the robot end coordinate system to the camera coordinate system; S240: Combining the image feature point velocity and camera velocity equation in S220, a visual servo kinematics model is obtained, specifically: ； in, is the task Jacobian matrix; S250: Combining the robot arm dynamics model in S210 with the visual servo kinematics model in S240, the overall dynamic equation of the visual servo system is obtained, which is: 。 4. The method according to claim 3, characterized in that S300 includes: S310: Define image feature error: ； in, Represents the image feature error signal; for the speed control of the camera coordinate system, the following nominal control law is designed : ； in, is a normal number, Pseudo-inverse matrix of the image interaction matrix; The image feature error change rate is: ； in Represents the derivative of the image feature error with respect to time; S320: Define tracking signal performance specifications , which is an exponentially decreasing function, specifically: ； in, The convergence speed of the error performance function can be adjusted. Represent the maximum allowable error and steady-state error respectively; definition and , ,in, and are the lower and upper bound functions of the feature error, respectively. and is a positive number; when When , the performance function is asymmetric; S330: Define the zero-point control barrier function to ensure ,definition ； in is the first image feature error vector elements, and Represents the image feature errors The lower and upper bounds of the zero-point control barrier function; Define the error transformation vector and ,in , , ; Will and Substituting in the error transformation, we can get the following: ； S340: Design conditions that meet performance constraints, when and hour, ; Therefore, to meet the predetermined performance constraints, it is equivalent to designing the controller to ensure and ,definition: ； ； in, and Represents the image feature errors Zero-point control barrier function and Derivatives with respect to time; Will Substitution and From the expression of , we can get: ； ； S350: When and When, there is ;when and When, there is ; Combine , use the following quadratic programming algorithm to obtain the speed control variables in the camera coordinate system that meet the field of view constraints As an optimal visual servoing speed controller: 。 5. The method according to claim 4, characterized in that S400 includes: S410: Based on the velocity transfer equation between the end effector velocity and the camera velocity And the speed control variables in the camera coordinate system that meet the field of view constraints , and get the expected end effector velocity: ； According to the mapping relationship between the end effector speed and the joint speed, the reference joint angular velocity can be obtained. for: ； in, is the pseudo-inverse matrix of the Jacobian matrix of the robot; Define joint velocity tracking error : ； in, is the joint angular velocity of the robot; Defining tracking signal performance specifications , which is an exponentially decreasing function, specifically: ； in, The convergence speed of the error performance function can be adjusted. They represent the maximum allowable error and steady-state error of the joint angular velocity error respectively; definition and , ,in, and are the lower and upper bound functions of the joint angular velocity error, respectively. and is a positive number; when When , the performance function is asymmetric; Define the zero-point control barrier function for the joint angular velocity tracking error to ensure ,definition: ； in is the joint angular velocity tracking error vector elements, and Represents the joint angular velocity error The lower and upper bounds of the zero-point control barrier function; Define the error transformation vector and ,in , ;Will and , Substituting the expression into the error transformation, the error transformation is as follows: ； The design meets the performance constraints when and hour, ; Therefore, to meet the predetermined performance constraints, it is equivalent to designing the controller to ensure and ,definition: ； in, and Respectively about and The control barrier function is and Is an extended class Function; when and hour, and ; Therefore, to achieve the joint angular velocity error constraint, the control input Should meet and ,Right now: ； in, and Represents the control obstacle function of the joint angular velocity error and Differentiation of time; and Represent the upper and lower bound function vectors of image feature error respectively and The first derivative of ; is the expected joint angular acceleration; is the joint angular acceleration, and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ; Because for any positive number ,have: ； in, is the control obstacle function of the field of view constraint and The minimum value of is expressed as ； In order to integrate the control obstacle function of the field of view constraint into the robot dynamics, the following energy-based obstacle function is proposed : ； when hour, ,because is the control obstacle function of the field of view constraint and The minimum value of , so we have and , that is, satisfying the field of view constraint; therefore, to satisfy the predetermined field of view constraint is equivalent to designing a controller to ensure ,definition: ； in, Is an extended class Function, that is, is a strictly increasing function and satisfies, ; when hour, Therefore, to realize the joint angular velocity error constraint, combined with the robot dynamics model , control input Should meet ,Right now: ； The kinetic model ,replace and The joint angular acceleration in , and obtain the torque control signal that satisfies the field of view constraint and joint angular velocity performance constraint ,Right now: ； in, is a nominal controller, which is usually the controller designed without considering performance constraints.

6. The method according to claim 5, characterized in that S500 includes: S510: Using the image processing library installed on the industrial computer, extract the coordinates of the feature corners of the captured QR code and write a robust visual servo controller program; S520: The calculated torque control signal is transmitted to the SDK programming platform on the same industrial computer in the form of a ROS topic to write a robot torque control program; S530: By properly adjusting parameters, the robot reaches the specified position.

7. A robust visual constraint control system for a manipulator, characterized in that: It includes coordinate conversion module, visual servo system dynamic model building module, constraint satisfaction condition determination module, control signal determination module and robot drive module. The coordinate conversion module is used to calibrate the coordinate conversion relationship between the robot arm and the camera using the hand-eye calibration method; The visual servo system dynamic model building module is used to combine the robot arm dynamics model, the visual servo kinematics model, and the hand-eye calibration relationship between the camera and the robot arm to establish the overall dynamic model of the visual servo system; among them, the time-varying asymmetric control obstacle function is specifically: ； in, and Represent the current image features and the expected image features respectively, represents the expected image feature velocity; represents the image interaction matrix, Represents the image feature depth information, Represents the camera speed; and Represent the upper and lower bound function vectors of image feature error respectively and Derivatives with respect to time; and Is an extended class Function, that is, and is a strictly increasing function and satisfies, and ; and are the lower bound transformation vector and upper bound transformation vector of the image feature tracking error, respectively. and Respectively represent and The control barrier function of The optimal visual servo speed controller is specifically: ； in, is the nominal speed control law; The constraint satisfaction condition determination module is used to combine the visual servoing kinematics model, obtain the constraint satisfaction condition for constraining the visual servoing feature error within a preset error range by designing a time-varying asymmetric control obstacle function, and design the optimal visual servoing speed controller; The control signal determination module calculates the reference joint angular velocity of the robot through the speed conversion matrix and the robot inverse kinematics according to the optimal visual servo speed controller, and defines the joint angular velocity tracking error. Combined with the mechanical arm dynamics model, the constraint satisfaction condition that constrains the joint angular velocity within the preset joint angular velocity range is obtained through the time-varying asymmetric control obstacle function. At the same time, combined with the visual servo characteristic error constraint satisfaction condition in the constraint satisfaction condition determination module, the optimal torque controller is designed, and combined with quadratic programming, the torque control signal that meets the performance constraint is obtained; The robot drive module is used to send the torque control signal to the robot actuator to drive the robot to the desired feature position and ensure that the feature point trajectory does not exceed the camera field of view while ensuring the smoothness of the trajectory.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images