Fixed-time visual servo control method based on fractional obstacle Lyapunov

Abstract

The present invention discloses a fixed-time visual servo control method based on fractional obstacle Lyapunov, comprising the following steps: S1, obtaining the conversion relationship from a three-dimensional camera coordinate system to a two-dimensional image coordinate system; S2, obtaining the conversion matrix T2 between the end of the manipulator and the depth camera; S3, defining the visual servo image error vector; S4, designing a fixed-time controller with a field of view constraint by combining the obstacle Lyapunov function and the fixed-time stability theory to obtain a camera speed controller; S5, controlling the speed of the manipulator so that the manipulator visual servo moves to a desired position, ensuring that the image feature points do not leave the constrained camera field of view. The present invention introduces a fixed-time field of view constraint method in the manipulator visual servo control, so that the image features are always within the field of view of the camera, thereby improving the steady-state and transient performance of the visual servo system.

Description

Technical Field

[0001] The invention relates to the technical field of robot arm image visual servo control, and in particular to a fixed-time visual servo control method based on fractional obstacle Lyapunov. Background Art

[0002] With the development of visual sensors, high-precision visual servo technology has been widely used in various robotics fields. The visual servo system is a control system that uses visual information as feedback. It is composed of visual image information and camera-related parameters. Compared with traditional conventional sensors, its information has higher dimensions and greater information volume. In this system, due to the limitations of the processing speed and sampling frequency of the visual part, the control frequency of the system is generally low, and it is mainly used in some slower control processes. For tasks that require rapid response and execution in practice, it is necessary to improve the dynamic response performance of the robot to meet them, that is, it is necessary to improve the convergence speed of the robot image visual servo. Moreover, in the process of fast visual servoing, visual features are easily lost, which will cause the visual servo system that completely relies on vision to obtain target information to stop servo control. In more serious cases, the system will become unstable and lead to control failure. Therefore, keeping the image features within the camera field of view is a very important part of the robot image visual servoing.

[0003] The patent with publication number CN115122325A discloses a robust visual servo control method for an anthropomorphic manipulator with field of view constraints. It adopts a ln-type obstacle Lyapunov function to complete the design of the visual servo controller, which can improve the steady-state and transient performance of the system. However, it adopts a constant performance constraint, which has limited effect on the system performance and the constraint effect on the camera field of view. Summary of the invention

[0004] In order to solve the problems of slow servo speed and easy loss of camera field of view in the existing robotic arm image visual servo technology, the present invention proposes a fixed-time visual servo control method based on fractional obstacle Lyapunov, introduces a time-varying exponential performance function, and performs real-time constraints on the camera field of view to ensure that the image features in the camera field of view will not be lost during the servo process; uses fractional obstacle Lyapunov function to complete the performance constraint on the error of image feature points, and improves its steady-state and transient performance; combines the fixed-time Lyapunov stability theorem to make the system stable at a fixed time, and further improve the servo speed.

[0005] The present application discloses a fixed-time visual servo control method based on fractional obstacle Lyapunov, comprising the following steps:

[0006] S1. Establish a six-axis robot image visual servo system, and set the coordinate system of the visual servo system reference frame to obtain the conversion relationship from the three-dimensional camera coordinate system to the two-dimensional image coordinate system;

[0007] S2, calibrate the visual servo system by the chessboard method to obtain the transformation matrix T2 between the end of the robotic arm and the depth camera;

[0008] S3, using a depth camera to collect the image feature information of the QR code corner points including pixel coordinates in real time and send it to the computer host, and defining the visual servoing image error vector according to the pixel coordinates of the current image feature corner points and the pixel coordinates of the expected feature corner points;

[0009] S4. Preset performance design is performed according to the image error vector, the camera field of view is constrained, and a fixed-time controller with field of view constraint is designed by combining the obstacle Lyapunov function and the fixed-time stability theory to obtain the camera speed controller;

[0010] S5. Combine the camera speed controller with the transformation matrix T2 to control the speed of the robotic arm so that the robotic arm visual servo moves to the desired position and ensures that the image feature points do not leave the constrained camera field of view.

[0011] Preferably, the six-axis robotic arm visual servo system includes a six-degree-of-freedom robotic arm, a depth camera mounted on the robotic arm in an eye-on-hand manner, and a QR code.

[0012] Preferably, the S1 comprises the following steps:

[0013] The coordinates of the three-dimensional feature points in the camera coordinate system are expressed as:

[0014] P i =[x i ,y i ,z i ] T , i∈{1,...,n};

[0015] Among them, n is the number of image feature points, x i ,y i Represents the coordinate value of the feature point in the camera coordinate system, z i Represents the depth value of the image obtained from the depth camera;

[0016] The two-dimensional image features in the image coordinate system corresponding to the three-dimensional feature points are:

[0017] s i =[u i ,v i ] T ;

[0018] Among them, u i 、v i Represents the image coordinates in the image coordinate system;

[0019] The relationship between the 3D feature point position and the 2D image coordinates is as follows:

[0020]

[0021] Among them, s i represents the feature points of a two-dimensional image, and δ is the focal length of the depth camera;

[0022] The time derivative of the feature point of a two-dimensional image is:

[0023]

[0024] Among them, V c is the camera speed controller, and V c =[v,w] T =[v x ,v y ,v z ,w x ,w y ,w z ] T , v is the linear velocity of the depth camera, w is the rotational velocity of the depth camera, v x 、v y 、v z Respectively represent the linear velocity of the depth camera on the x-axis, y-axis, and z-axis, w x ,w y ,w z Respectively represent the rotation speed of the depth camera on the x-axis, y-axis, and z-axis;

[0025]

[0026] Among them, Γ i is the image Jacobian matrix, and

[0027] Preferably, S2 comprises the following steps:

[0028] S21, using a chessboard calibration plate with N points, and randomly placing the chessboard calibration plate in the field of view of the depth camera;

[0029] S22, using a depth camera to collect images of the chessboard calibration plate;

[0030] S23, calculating the center pixel coordinates of N dots in the image;

[0031] S24, recording the three-dimensional position and posture of the end effector of the robot arm in the form of a translation vector and a rotation vector;

[0032] S25, repeating steps S22-S24 multiple times to obtain two-dimensional data information and three-dimensional data information of the two-dimensional code;

[0033] S26, solve the obtained two-dimensional code information, find the coordinate transformation matrix T1 between the chessboard calibration plate and the camera, and calculate the transformation matrix T2 from the end of the robot arm to the depth camera based on T1.

[0034] Preferably, S3 comprises the following steps:

[0035] The goal of the visual servo control scheme is to minimize the error to zero, and the error of visual servo is defined as:

[0036]

[0037] in, Represents the error in the u direction in the camera coordinate system, Represents the error in the v direction in the camera coordinate system, Represents the pixel coordinate value of the desired image feature point, given as a constant;

[0038] All image feature errors E h The derivative of is:

[0039]

[0040] Among them, Γ h is the set of image Jacobian matrices of all image feature points,

[0041]

[0042] Among them, E h is the total visual servoing image error vector.

[0043] Preferably, S4 comprises the following steps:

[0044] S41. Due to the limited field of view of the depth camera, the constraints on the pixel coordinates of the image feature points are as follows:

[0045]

[0046] Among them, u min 、v min is the lower limit of the image plane coordinates, u max 、v max is the upper limit of the image plane coordinates; if the pixel coordinates of the image feature points satisfy the above constraints, then the constraint of always staying within the camera duration range is satisfied;

[0047] S42, constraining image errors:

[0048]

[0049] S43, introducing a specific time-varying performance function to constrain the image error so that the image feature point error varies within a predetermined area:

[0050]

[0051] in, is the asymmetry constant, and is the performance function;

[0052] Performance functions and The expression is:

[0053]

[0054] Among them, L ui,0 =u max -u i,d >0,L vi,0 =v max -v i,d >0,γ ui and γ vi is a positive parameter, representing the convergence speed of the image feature point error; L ui,∞ and L vi,∞ Also a positive parameter, L ui,∞ Indicates transient performance index, L vi,∞ Represents a steady-state performance index and satisfies 0 <L ui,∞ <<L ui,0 , 0 <L vi,∞ <<L vi,0 ;

[0055] Combined with the performance function, the image error vector is redefined as follows:

[0056]

[0057] The time-varying constraints are transformed into time-invariant constraints, and the expression is:

[0058]

[0059] S44. Combining the barrier Lyapunov function and the fixed-time stability theory, the fixed-time speed controller with field constraints is obtained as follows:

[0060]

[0061] in, is the pseudo-inverse of the interaction matrix, λ1 and λ2 are positive gain parameters, and when n ≥ 3, is a full rank matrix with a rank of 6.

[0062]

[0063] in:

[0064]

[0065] in:

[0066]

[0067] in:

[0068]

[0069] express The upper bound of express The lower bound of express The upper bound of express The lower bound of .

[0070] Preferably, S5 comprises the following steps:

[0071] The visual servo controller obtained by S4 is programmed in the open source software platform visp on the host computer. The robot arm, depth camera, and visp platform communicate through ROS to complete the servo camera verification; the speed controller V c Combined with the transformation matrix T2, the speed of the end of the robot arm is obtained, so as to control the speed of the robot arm and complete the visual servo process.

[0072] Beneficial effects of the present invention:

[0073] (1) Through the constraints of the performance function, the image features of the robot arm are always within the field of view of the camera during the visual servoing process, avoiding violation of the camera field of view constraints;

[0074] (2) By using performance function constraints and designing the preset performance control method, the fractional obstacle Lyapunov function is combined to further improve the steady-state and transient performance of the visual servo system, making it more versatile, simpler and easier to implement;

[0075] (3) Based on the preset performance controller and combined with the fixed-time stability theory, a fixed-time field of view constraint method is introduced for the first time in the visual servo control of robotic arms, which is different from other robotic arm control methods with fixed-time output constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1is a flow chart of a visual servo control method according to an embodiment of the present invention;

[0077] Figure 2 Schematic diagram of a robotic arm visual servo test platform according to an embodiment of the present invention;

[0078] Figure 3 A camera field of view diagram according to an embodiment of the present invention;

[0079] Figure 4 This is a comparison diagram of the total error convergence of the visual servo control method according to an embodiment of the present invention and other control methods. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0081] The present application embodiment discloses a fixed-time visual servo control method based on fractional obstacle Lyapunov, the process is as follows: Figure 1 As shown in the figure, E represents the error of visual servoing, L represents the designed performance function; Z represents the visual servoing error defined after the performance function is introduced, Indicates the speed of the robot end.

[0082] The fixed-time visual servo control method based on fractional obstacle Lyapunov disclosed in the embodiment of the present application includes the following steps:

[0083] S1. Establish a six-axis robotic arm image visual servo system, such as Figure 2 As shown, it includes a 6-DOF robotic arm UR5, an Intel RealSense D435i depth camera installed on the robotic arm in an "eye-on-hand" manner, and a 36h11 series QR code. Figure 3 This is the camera field of view during the servo test. Figure 3 (a) in the figure represents the camera field of view at the initial moment of visual servoing. Figure 3 (b) in the figure represents the camera field of view during visual servoing. Figure 3 (c) in the figure represents the camera field of view at the end of visual servoing. From the camera field of view diagrams of the three processes, we can see that the four corner points of the camera gradually coincide with the four corner points of the QR code, completing the visual servoing experiment.

[0084] The coordinate system of the visual servo system reference frame is set to obtain the conversion relationship from the three-dimensional camera coordinate system to the two-dimensional image coordinate system.

[0085] The coordinates of the three-dimensional feature points in the camera coordinate system are expressed as:

[0086] P i =[x i ,y i,z i ] T , i∈{1,...,n};

[0087] Among them, n is the number of image feature points, x i ,y i Represents the coordinate value of the feature point in the camera coordinate system, z i Represents the depth value of the image obtained from the depth camera;

[0088] The two-dimensional image features in the image coordinate system corresponding to the three-dimensional feature points are:

[0089] s i =[u i ,v i ] T ;

[0090] Among them, u i 、v i Represents the image coordinates in the image coordinate system;

[0091] The relationship between the 3D feature point position and the 2D image coordinates is as follows:

[0092]

[0093] Among them, s i represents the feature points of a two-dimensional image, and δ is the focal length of the depth camera;

[0094] The time derivative of the feature point of a two-dimensional image is:

[0095]

[0096] Among them, V c is the camera speed controller, and V c =[v,w] T =[v x ,v y ,v z ,w x ,w y ,w z ] T , v is the linear velocity of the depth camera, w is the rotational velocity of the depth camera, v x 、v y 、v z Respectively represent the linear velocity of the depth camera on the x-axis, y-axis, and z-axis, w x ,w y ,w z Respectively represent the rotation speed of the depth camera on the x-axis, y-axis, and z-axis;

[0097]

[0098] Among them, Γ i is the image Jacobian matrix, and

[0099] S2. Calibrate the visual servo system through the chessboard method to obtain the transformation matrix T2 between the end of the robotic arm and the depth camera.

[0100] S21, using a chessboard calibration plate with N points, and randomly placing the chessboard calibration plate in the field of view of the depth camera;

[0101] S22, using a depth camera to collect images of the chessboard calibration plate;

[0102] S23, calculating the center pixel coordinates of N dots in the image;

[0103] S24, recording the three-dimensional position and posture of the end effector of the robot arm in the form of a translation vector and a rotation vector;

[0104] S25, repeating steps S22-S24 multiple times to obtain two-dimensional data information and three-dimensional data information of the two-dimensional code;

[0105] S26, solve the obtained two-dimensional code information, find the coordinate transformation matrix T1 between the chessboard calibration plate and the camera, and calculate the transformation matrix T2 from the end of the robot arm to the depth camera based on T1.

[0106] S3. Use a depth camera to collect the image feature information of the QR code corner points including pixel coordinates in real time and send it to the computer host. Define the visual servoing image error vector according to the pixel coordinates of the current image feature corner points and the pixel coordinates of the expected feature corner points.

[0107] The goal of the visual servo control scheme is to minimize the error to zero, and the error of visual servo is defined as:

[0108]

[0109] in, Represents the error in the u direction in the camera coordinate system, Represents the error in the v direction in the camera coordinate system, Represents the pixel coordinate value of the desired image feature point, given as a constant;

[0110] All image feature errors E h The derivative of is:

[0111]

[0112] Among them, Γ h is the set of image Jacobian matrices of all image feature points,

[0113]

[0114] Among them, E h is the total visual servoing image error vector.

[0115] S4. Preset performance design is performed according to the image error vector, the camera field of view is constrained, and a fixed-time controller with field of view constraints is designed by combining the obstacle Lyapunov function and fixed-time stability theory to obtain the camera speed controller.

[0116] S41. Due to the limited field of view of the depth camera, the constraints on the pixel coordinates of the image feature points are as follows:

[0117]

[0118] Among them, u min 、v min is the lower limit of the image plane coordinates, u max 、v max is the upper limit of the image plane coordinates; u min 、v min 、u max 、v max The units are all in pixels, which are determined by the camera resolution. If the pixel coordinates of the image feature points satisfy the above constraints, then the constraint of always staying within the camera duration is satisfied.

[0119] S42, constraining image errors:

[0120]

[0121] S43, introducing a specific time-varying performance function to constrain the image error so that the image feature point error varies within a predetermined area:

[0122]

[0123] in, is the asymmetry constant, and is the performance function;

[0124] Performance functions and The expression is:

[0125]

[0126] Among them, L ui,0 =u max -u i,d >0,L vi,0=v max -v i,d >0,γ ui and γ vi is a positive parameter, representing the convergence speed of the image feature point error; L ui,∞ and L vi,∞ Also a positive parameter, L ui,∞ Indicates transient performance index, L vi,∞ Represents a steady-state performance index and satisfies 0 <L ui,∞ <<L ui,0 , 0 <L vi,∞ <<L vi,0 ;

[0127] Combined with the performance function, the image error vector is redefined as follows:

[0128]

[0129] The time-varying constraints are transformed into time-invariant constraints, and the expression is:

[0130]

[0131] S44. Combining the barrier Lyapunov function and the fixed-time stability theory, the fixed-time speed controller with field constraints is obtained as follows:

[0132]

[0133] in, is the pseudo-inverse of the interaction matrix, λ1 and λ2 are positive gain parameters, and when n ≥ 3, is a full rank matrix with a rank of 6.

[0134]

[0135] in:

[0136]

[0137] in:

[0138]

[0139] in:

[0140]

[0141] express The upper bound of is 1 in this embodiment. express The lower bound of express The upper bound of is 1 in this embodiment. express The lower bound of

[0142] For any non-negative real number x, the function x→sgn α (x) is defined as: Function sgn α (x)=|x| α sign(x).

[0143] sign(x) is the sign function, which is defined as follows:

[0144]

[0145] S5. Combine the camera speed controller with the transformation matrix T2 to control the speed of the robotic arm so that the robotic arm visual servo moves to the desired position and ensures that the image feature points do not leave the constrained camera field of view.

[0146] The visual servo controller obtained by S4 is programmed in the open source software platform visp on the host computer. The robot arm, depth camera, and visp platform communicate through ROS to complete the servo camera verification; the speed controller V c Combined with the transformation matrix T2, the speed of the end of the robot arm is obtained, so as to control the speed of the robot arm and complete the visual servo process.

[0147] In a specific embodiment, Figure 4 As shown, by comparing the fixed-time visual servo control method based on fractional obstacle Lyapunov of the embodiment of the present application with other control methods, it can be seen that the control method proposed in the embodiment of the present application shows better dynamic response and faster convergence of visual feature error. In the final stage, the embodiment of the present application first achieves convergence using the fixed-time stability theory. Figure 4 The results verify the effectiveness of the method proposed in the embodiment of the present application, and confirm that the fixed-time visual servo control method based on fractional obstacle Lyapunov has better steady-state and transient performance.

[0148] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A fixed-time visual servo control method based on fractional obstacle Lyapunov, characterized in that: The following steps are involved: S1. Establish a six-axis robot image visual servo system, and set the coordinate system of the reference frame of the visual servo system to obtain the conversion relationship from the three-dimensional camera coordinate system to the two-dimensional image coordinate system; The coordinates of the three-dimensional feature points in the camera coordinate system are expressed as: P i [x i ,y i ,z i ] T ,i∈{1,...,n}! Among them, n is the number of image feature points, x i ,y i Represents the coordinate value of the feature point in the camera coordinate system, z i Represents the depth value of the image obtained from the depth camera; The two-dimensional image features in the image coordinate system corresponding to the three-dimensional feature points are: s i =[u i ,v i ] T ; Among them, u i 、v i Represents the image coordinates in the image coordinate system; The relationship between the 3D feature point position and the 2D image coordinates is as follows: Among them, s i represents the feature points of a two-dimensional image, and δ is the focal length of the depth camera; The time derivative of the feature point of a two-dimensional image is: Among them, V c is the camera speed controller, and V c =[v,w] T =[v x ,v y ,v z ,w x ,w y ,w z ] T , v is the linear velocity of the depth camera, w is the rotational velocity of the depth camera, v x 、v y 、v z Respectively represent the linear velocity of the depth camera on the x-axis, y-axis, and z-axis, w x ,w y ,w z Respectively represent the rotation speed of the depth camera on the x-axis, y-axis, and z-axis; Among them, Γ i is the image Jacobian matrix, and S2, calibrate the visual servo system by the chessboard method to obtain the transformation matrix T2 between the end of the robotic arm and the depth camera; S3, using a depth camera to collect the QR code corner image feature information including pixel coordinates in real time and send it to the computer host, and defining the visual servo image error vector according to the pixel coordinates of the current image feature corner and the pixel coordinates of the expected feature corner; The error of visual servoing is defined as: in, Represents the error in the u direction in the camera coordinate system, Represents the error in the v direction in the camera coordinate system, Represents the pixel coordinates of the desired feature corner point, given as a constant; All image feature errors E h The derivative of is: Among them, Γ h is the set of image Jacobian matrices of all image feature points, Among them, E h is the error of all image features; S4. Preset performance design is performed according to the image error vector, the camera field of view is constrained, and a fixed-time controller with field of view constraint is designed by combining the obstacle Lyapunov function and the fixed-time stability theory to obtain the camera speed controller; S41. The constraints of the pixel coordinates of the image feature points are as follows: Among them, u min 、v min is the lower limit of the image plane coordinates, u max 、v max is the upper limit of the image plane coordinates; S42, constraining image errors: S43, introducing a specific time-varying performance function to constrain the image error so that the image feature point error varies within a predetermined area: in, is the asymmetry constant, and is the performance function; Performance functions and The expression is: Among them, L ui,0 =u max -u i,d >0,L vi,0 =v max -v i,d >0,γ ui and γ vi is a positive parameter, representing the convergence speed of the image feature point error; L ui,∞ and L vi,∞ Also a positive parameter, L ui,∞ Indicates transient performance index, L vi,∞ Represents a steady-state performance index and satisfies 0 <L ui,∞ <<L ui,0 , 0 <L vi,∞ <<L vi,0 ; Combined with the performance function, the image error vector is redefined as follows: The time-varying constraints are transformed into time-invariant constraints, and the expression is: S44. Combining the barrier Lyapunov function and the fixed-time stability theory, the fixed-time speed controller with field constraints is obtained as follows: in, is the pseudo-inverse of the interaction matrix, λ1 and λ2 are positive gain parameters, and when n ≥ 3, is a full rank matrix, rank 6, in: in: in: express The upper bound of express The lower bound of express The upper bound of express The lower bound of S5. Combine the camera speed controller with the transformation matrix T2 to control the speed of the robotic arm so that the robotic arm visual servo moves to the desired position and ensures that the image feature points do not leave the constrained camera field of view.

2. The fixed-time visual servo control method based on fractional obstacle Lyapunov according to claim 1 is characterized in that: The six-axis robotic arm image visual servo system includes a six-degree-of-freedom robotic arm, a depth camera installed on the robotic arm in an eye-on-hand manner, and a QR code.

3. The fixed-time visual servo control method based on fractional obstacle Lyapunov according to claim 2 is characterized in that: The S2 comprises the following steps: S21, using a chessboard calibration plate with N dots, and randomly placing the chessboard calibration plate in the field of view of the depth camera; S22, using a depth camera to collect images of the chessboard calibration plate; S23, calculating the center pixel coordinates of N dots in the image; S24, recording the three-dimensional position and posture of the end effector of the robot arm in the form of a translation vector and a rotation vector; S25, repeating steps S22-S24 multiple times to obtain two-dimensional data information and three-dimensional data information of the two-dimensional code; S26, solve the obtained two-dimensional code information, find the coordinate transformation matrix T1 between the chessboard calibration plate and the camera, and calculate the transformation matrix T2 from the end of the robot arm to the depth camera based on T1.

4. The fixed-time visual servo control method based on fractional obstacle Lyapunov according to claim 3 is characterized in that: The S5 comprises the following steps: The visual servo controller obtained by S4 is programmed in the open source software platform visp on the host computer. The robot arm, depth camera, and visp platform communicate through ROS. The speed controller V c Combined with the transformation matrix T2, the speed of the end of the robot arm is obtained, so as to control the speed of the robot arm and complete the visual servo process.

Images