A Robust Visual Servo Control Method for the Outline of Complex-Shaped Target Objects

Abstract

The present invention discloses a robust contour visual servo control method for complex-shaped objects. Under a hand-eye visual servo system, a quasi-uniform B-spline curve is used to fit the contour image of the object and construct an expected image feature; the contour of the object to be servoed in the currently acquired current image is extracted, and the preliminary image feature is also extracted; the optimal estimation is performed on the preliminary image feature to obtain the accurate image feature and the feature depth, the interaction matrix corresponding to the current image is calculated, the vector output speed for controlling the camera movement is obtained through processing, and the vector output speed is converted into the vector control speed of the robot end by combining with the hand-eye matrix to drive the robot to move. The present invention realizes the robot visual servo control based on the contour of a complex object by using the quasi-uniform B-spline curve feature, and at the same time designs a real-time estimation module for image features based on the extended Kalman filter and Lie algebra optimization, improving the robustness of the visual servo system to the dynamic occlusion interference environment.

Description

Technical Field

[0001] The present invention belongs to the field of robot vision servo control, and particularly relates to a robust contour vision servo control method for complex-shaped target objects. Background Art

[0002] Vision servo, as an important type of control method, greatly improves the flexibility and intelligence of robot systems. Among them, image-based visual servo (IBVS) uses feature errors from the image space to control the movement of the robot. An important issue in IBVS is to select appropriate image features. In many practical application scenarios, the target object usually has a complex shape, and in this case, the most commonly used point features and line features may not be accurately extracted from the image. Since the contour reflects the overall shape of the target object, the contour information of the target object can be used to construct image features and thus achieve vision servo control, which is called contour vision servo.

[0003] Image features describing contour information are mainly divided into two categories: image moments and parametric curve features. Although image moments are easy to extract and calculate, it is usually difficult to select appropriate high-order moments, and image moments are difficult to process non-closed contours. Parametric curves are very suitable for describing complex contour curves due to their powerful modeling ability, can handle non-closed contours, and are more intuitive than image moments. Therefore, performing vision servo control through parametric curve features is beneficial to improving the adaptability of the robot system to the environment. However, existing parametric curve features are difficult to achieve a good balance between the expression ability of complex shapes and the computational complexity.

[0004] In addition, due to environmental noises such as illumination changes and camera movements, the parametric curve features extracted from the image are often unstable. More seriously, due to some improper system configurations, the target object may be occluded during the vision servo process. These factors will all lead to a decline in the performance of vision servo, and especially when occlusion occurs, it may directly cause the failure of the vision servo task. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and propose a robust contour vision servo control method for complex-shaped target objects. This method introduces a quasi-uniform B-spline curve to describe the complex contour of the target object and uses the control vertices of the curve to construct the image features of the vision servo. At the same time, an image feature real-time estimation module based on the extended Kalman filter and Lie algebra optimization is designed to achieve robust contour vision servo control.

[0006] The specific technical solution of the present invention is as follows:

[0007] S1. Under the hand-eye vision servo system, given the camera internal parameters, the hand-eye matrix, and the contour image of the target object under the given desired pose of the robot, "given" means preset and known in advance. Use a quasi-uniform B-spline curve to process the contour image of the target object, extract the contour from the contour image of the target object, and construct the desired image features for controlling the robot motion and their image coordinates;

[0008] The complex-shaped target object mentioned above usually refers to an object whose cross-section cannot be described by a basic geometric figure and whose thickness is less than 5 mm.

[0009] The hand-eye vision servo system includes a camera, a robot, and a platform. The target object is placed on the platform, the camera is fixed at the end of the robot, and the robot drives the camera to move.

[0010] S2. Extract the contour of the target object in the current image obtained in real time from the current pose of the robot. Use a quasi-uniform B-spline curve to extract the preliminary image features of the current image and their image coordinates for the contour of the image to be servoed;

[0011] S3. Process the preliminary image features of the current image and their image coordinates according to the image feature real-time estimation module to achieve the optimal estimation of the accurate image features and feature depth of the current image;

[0012] S4. Calculate the interaction matrix corresponding to the current image according to the accurate image features and feature depth of the current image;

[0013] S5. According to the deviation between the image features and their image coordinates of the current image and the desired image features and their image coordinates, and the interaction matrix obtained in S4, process to obtain the vector output speed for controlling the camera motion, and convert the vector output speed into the vector control speed of the robot end in combination with the hand-eye matrix, and then drive the robot to move;

[0014] S6. Repeat steps S2 - S5 until the deviation in S5 converges to the preset range, then it is considered that the robot reaches the desired pose.

[0015] In step S1 mentioned above, use a quasi-uniform B-spline curve with n control vertices to fit the contour in the contour image of the target object, obtain n control vertices, use the n control vertices as the desired image features of the vision servo, call it the quasi-uniform B-spline feature, and use the image coordinates of the n control vertices as the image coordinates of the quasi-uniform B-spline feature.

[0016] In addition, the number of control vertices is restricted as follows to control the 6 degrees of freedom of the robot and avoid generating non-singular solutions:

[0017] For a non-closed contour, the number n of control vertices shall not be less than 4;

[0018] For a closed contour, since the first and last control vertices coincide, the number of control vertices n must be no less than 5.

[0019] In step S2, the image contour of the target object is obtained from the current image captured by the camera through the Canny edge detection algorithm and the Rosenfeld skeleton thinning algorithm, and then the quasi-uniform B-spline curve is used to fit the target object contour image to obtain the preliminary image features of the current image and their image coordinates.

[0020] In step S3, the image feature real-time estimation module is specifically processed as follows:

[0021] According to the integrity of the image contour of the target object in the current image obtained in S2, it is judged whether the target object is occluded, and the real-time optimal estimation of the accurate image features and the feature depth is correspondingly performed:

[0022] A. If the target object is not occluded, the extended Kalman filter (EKF) is directly used to perform optimal estimation filtering on the image coordinates of the preliminary image features to obtain the image coordinates of the accurate image features and their feature depth;

[0023] The feature depth mentioned above refers to the distance between the control vertex and the camera imaging plane.

[0024] B. If the target object is occluded, then:

[0025] First, according to the incomplete image contour corresponding to the preliminary image features in the current image, the camera pose M k of the current image is predicted, specifically obtained by solving the following objective function:

[0026]

[0027] ΔM = exp(Δξ^)

[0028] where ΔM is the relative camera pose between the previous moment and the current moment, and ΔM is represented in the form of Lie algebra as exp(Δξ^), K c is the camera internal parameter, M k-1 is the camera pose at the (k - 1)-th moment, is the sampling point of the incomplete image contour partially occluded in the current image at the k-th moment, P(u i ) is the corresponding point of the sampling point in the CAD model of the target object; Δξ represents the camera pose transformation vector represented by Lie algebra, and Δξ^ represents converting the pose transformation vector Δξ represented by Lie algebra into a homogeneous transformation matrix; M represents the camera pose, k represents the serial number of the moment, i represents the serial number of the control vertex, and m represents the total number of control vertices.

[0029] Optimize and solve the above objective function. After optimization, obtain the optimal camera pose transformation vector Δξ in the Lie algebra representation, and then through the formula M k-1 calculate the predicted camera pose M in the current image by exp(Δξ^) k ;

[0030] Then, use the predicted camera pose M k and predict the complete contour of the object in the current image according to the projection relationship of the preset camera and the CAD model of the known object. Fit the complete contour of the object with a quasi-uniform B-spline curve to obtain the preliminary image features and their image coordinates, and then perform optimal estimation filtering on the image coordinates of the preliminary image features using the Extended Kalman Filter (EKF) to obtain the image coordinates and feature depths of the accurate image features.

[0031] In step S4, first process according to the following formula to obtain the interaction matrix of a single control vertex in the quasi-uniform B-spline curve corresponding to the current image:

[0032]

[0033] where (x i , y i ) is the image coordinate of the i-th control vertex, Z i is the feature depth of the i-th control vertex, and L i represents the interaction matrix of the i-th control vertex;

[0034] Then, construct the interaction matrix of all control vertices in the image features from the interaction matrices of each control vertex:

[0035] L BF = [L1 L2...L n T

[0036] where L BF represents the interaction matrix of the image features, L n represents the interaction matrix of the n-th control vertex in the image features, and T represents matrix transpose.

[0037] For the visual servo task of objects with complex shapes, the present invention first uses a quasi-uniform B-spline curve to describe the complex contour of the object, then constructs the image features of the visual servo according to the control vertices of the curve and gives the corresponding interaction matrix form to achieve contour-based visual servo control. Finally, for the dynamic environment where the object may be occluded, an image feature real-time estimation module based on the Extended Kalman Filter and Lie algebra optimization is designed. This module can obtain the optimal estimation of the current image feature values and feature depths, improving the robustness of the robot visual servo system. ​

[0038] The beneficial effects of the present invention are as follows:

[0039] (1) The present invention uses a quasi-uniform B-spline curve to describe the contour information of an object with a complex shape. The quasi-uniform B-spline curve has both a powerful shape expression ability and a simple mathematical expression form, reducing the model complexity of contour visual servo while ensuring visual servo performance.

[0040] (2) Through the image feature real-time estimation module based on extended Kalman filter and Lie algebra optimization, the present invention realizes the optimal estimation of the current image feature value and feature depth in a dynamic environment where the object may be occluded, improving the robustness of the robot vision servo system.

[0041] (3) The contour visual servo control method provided by the present invention can be directly extended to the robot vision servo tasks of objects with complex shapes, and is used to guide the initial positioning and visual tracking of robots in actual scenarios such as robot-assisted operation, assembly, and welding, having good engineering practical value.

[0042] Generally speaking, the present invention realizes robot vision servo control based on the contour of a complex object through quasi-uniform B-spline curve features, and at the same time designs an image feature real-time estimation module based on extended Kalman filter and Lie algebra optimization, improving the robustness of the visual servo system to dynamic occlusion interference environments and having good engineering practical value. Brief Description of the Drawings

[0043] Figure 1 is a flowchart of the method of the present invention.

[0044] Figure 2 is a flowchart of the image feature real-time estimation module in the method of the present invention.

[0045] Figure 3 is a schematic diagram of the robot vision servo system in the embodiment of the present invention.

[0046] Figure 4 is a graph of the visual servo experimental results for a non-closed contour object in the embodiment of the present invention.

[0047] Figure 5 is a graph of the visual servo experimental results for a closed contour object in the embodiment of the present invention.

[0048] Figure 6 is a graph of the image feature prediction results when the object is partially occluded in the embodiment of the present invention.

[0049] In the figure: robot body 1, server 2, monocular camera 3, closed contour object 4, non-closed contour object 5. Detailed Embodiments

[0050] The following uses a 6-degree-of-freedom robotic arm visual servo system as an example to further illustrate the present invention.

[0051] This embodiment is described using a 6-degree-of-freedom robotic arm visual servo system: Figure 3 It is a general schematic diagram of a 6-degree-of-freedom robotic arm visual servo system, including a robot body 1, a server 2, a monocular camera 3, a closed contour target 4, and an open contour target 5. The robot body 1 is a 6-degree-of-freedom robotic arm, and the monocular camera 3 is fixed at the end of the robotic arm. The target has two forms, the open contour target 5 and the closed contour target 4. The goal of the visual servo task is: to control the movement of the robot according to the deviation between the current image contour and the desired contour, so that the current image contour coincides with the desired contour. The implementation process of this embodiment is described in detail using the open contour target 5 as an example, and the final experimental results are given. Since the implementation process of the closed contour target is basically the same, only its final experimental results are directly given.

[0052] As Figure 1 shown, the implementation process of the embodiment includes the following steps:

[0053] S1. In the hand-eye visual servo system, given the camera internal parameters, the hand-eye matrix, and the target contour image under the given desired pose of the robot, "given" means preset and known in advance. Process the target contour image using a quasi-uniform B-spline curve, extract the contour from the target contour image, and construct the desired image features and their image coordinates for controlling the movement of the robot;

[0054] The image contour of the target is represented by a quasi-uniform B-spline curve with n control vertices. The n control vertices are used as the image features of the visual servo and are called quasi-uniform B-spline features. In addition, in order to control the 6 degrees of freedom of the robot and avoid generating non-singular solutions, the number of control vertices is restricted as follows: for an open contour, n must be no less than 4; for a closed contour, since the first and last control vertices coincide, n must be no less than 5.

[0055] In this embodiment, the degree of the quasi-uniform B-spline curve is set to 3. The number of control vertices n of the open contour target is taken as 8, and the number of control vertices n of the closed contour target is taken as 20. The calibration result of the camera internal parameters is: f x = 379.89 pixels, f y = 379.60 pixels, u0 = 313.79 pixels, v0 = 242.24 pixels. The calibration result of the hand-eye matrix is:

[0056] S2. Extract the contour of the target object in the current image obtained in real time from the current pose of the robot, and use a quasi-uniform B-spline curve to fit the contour of the image to be servoed to extract the preliminary image features of the current image and their image coordinates;

[0057] Obtain the image contour of the target object from the original image captured by the camera through the Canny edge detection algorithm and the Rosenfeld skeleton thinning algorithm. The initial value of the quasi-uniform B-spline image feature is obtained through curve fitting.

[0058] S3. According to the image feature real-time estimation module, process the image coordinates of the preliminary image features of the current image to achieve the optimal estimation of the accurate image features and feature depth of the current image;

[0059] Figure 2 It is the flowchart of the quasi-uniform B-spline image feature real-time estimation module. On the basis of obtaining the initial value of the current image features, first determine whether the target object is occluded. If there is no occlusion, directly perform the optimal estimation of the image feature value and feature depth based on the extended Kalman filter. Otherwise, first perform the occlusion feature prediction based on the Lie algebra nonlinear optimization, and then perform the extended Kalman filter optimal estimation. Finally, obtain the optimal estimation value of the current quasi-uniform B-spline image feature and feature depth.

[0060] A. If the target object is not occluded, directly use the extended Kalman filter (EKF) to perform the optimal estimation filtering process on the image coordinates of the preliminary image features, and obtain the image coordinates of the accurate image features and their feature depth;

[0061] B. If the target object is occluded, then:

[0062] First, according to the incomplete image contour corresponding to the preliminary image features in the current image, predict the camera pose M of the current image k Specifically, it is obtained by solving the following objective function:

[0063]

[0064] ΔM = exp(Δξ^)

[0065] Use the nonlinear optimization algorithm to optimize and solve the above objective function. After the optimization is completed, obtain the optimal camera pose transformation vector Δξ represented by the Lie algebra, and then calculate the predicted camera pose M of the current image through the formula M k-1 exp(Δξ^) to obtain M k ;

[0066] In this embodiment, the number m of the image contour sampling points is 40, and the optimization process is solved through the Levenberg-Marquardt (LM) algorithm.

[0067] Then, the predicted camera pose M is utilized k and, based on the projection relationship of the preset camera and the CAD model of the known target object, the complete contour of the target object in the current image is predicted. The complete contour of the target object is fitted with a quasi-uniform B-spline curve to obtain the preliminary image features and their image coordinates. Then, the extended Kalman filter (EKF) is used to perform optimal estimation filtering on the image coordinates of the preliminary image features to obtain the image coordinates and feature depths of the accurate image features.

[0068] S4. Calculate the interaction matrix corresponding to the current image according to the accurate image features and feature depths of the current image;

[0069] First, the interaction matrix of a single control vertex in the quasi-uniform B-spline curve corresponding to the current image is obtained by processing according to the following formula:

[0070]

[0071] Then, the interaction matrix of all control vertices in the image features is constructed from the interaction matrices of each control vertex:

[0072] L BF =[L1 L2...L n T

[0073] S5. According to the deviation between the image coordinates of the image features of the current image and the image coordinates of the desired image features and the interaction matrix obtained in S4, process to obtain the vector output speed for controlling the camera movement. Combine the hand-eye matrix to convert the vector output speed into the vector control speed of the robot end, and then drive the robot to move;

[0074] In this embodiment, a proportional controller is used to calculate the output speed for controlling the camera movement, and the gain is set to 0.2.

[0075] S6. Repeat S2 to S5 until the deviation in S5 converges to the preset range, then it is considered that the robot reaches the desired pose.

[0076] The experimental results are shown in Figures 4 - 6.

[0077] Figure 4 shows the visual servo experimental results for non-closed contour target objects. Among them, Figure 4(a) shows the initial and desired positions of the target object in the image, Figure 4(b) shows the feature trajectory and the final convergence during the camera movement, Figure 4(c) shows the convergence of the image error, and Figure 4(d) shows the camera speed.

[0078] ​Figure 5 shows the visual servo experiment results for the closed contour target. Among them, Figure 5(a) shows the initial and desired positions of the target in the image, Figure 5(b) shows the feature trajectory and the final convergence during the camera movement, Figure 5(c) shows the convergence of the image error, and Figure 5(d) shows the camera speed.

[0079] From Figures 4 and 5, it can be seen that the image feature error converges to near zero, and the final image contour coincides with the desired contour, and the visual servo task is successfully completed.

[0080] In addition, during the visual servo process, the target is partially occluded by setting random artificial disturbances. Figure 6 shows the prediction results of the image features when the target is partially occluded. Among them, Figures 6(a) and 6(b) are the prediction results for the non-closed contour target, and Figures 6(c) and 6(d) are the prediction results for the closed contour target. It can be seen that when the target is occluded, the current image feature curve is accurately predicted and the visual servo is executed normally.

[0081] From the results of the above embodiments, it can be seen that the method provided by the present invention is effective for the contour visual servo control of complex-shaped targets and is robust to possible occlusions of the targets.

[0082] The present invention is not limited to the above embodiments. For those of ordinary skill in the art of this technology, without departing from the principle of the present invention, several improvements and retouches can be made, and these improvements and retouches are also considered within the protection scope of the present invention. The content not described in detail in this specification belongs to the prior art well-known to those of ordinary skill in the art.

Claims

1. A robust contour visual servo control method for complex-shaped objects, characterized in that It includes the following steps: S1. Under the hand-eye vision servo system, given the camera internal parameters, the hand-eye matrix, and the target object contour image under the given desired pose of the robot, use the quasi-uniform B-spline curve to process the target object contour image, extract the contour from the target object contour image, and construct the desired image features and their image coordinates for controlling the robot movement; S2. Extract the contour of the target object in the current image obtained in real time from the current pose of the robot. Use the quasi-uniform B-spline curve to extract the preliminary image features and their image coordinates of the current image for the contour of the image to be servoed; S3. According to the image feature real-time estimation module, process the preliminary image features and their image coordinates of the current image to achieve the optimal estimation of the accurate image features and the feature depth of the current image; S4. Calculate the interaction matrix corresponding to the current image according to the accurate image features and the feature depth of the current image; S5. According to the deviation between the image features and their image coordinates of the current image and the desired image features and their image coordinates, and the interaction matrix obtained in S4, process to obtain the vector output speed for controlling the camera movement, and combine the hand-eye matrix to convert the vector output speed into the vector control speed of the robot end, thereby driving the robot movement; S6. Repeat S2 to S5 until the deviation in S5 converges to the preset range, then it is considered that the robot reaches the desired pose; In the step S3, the image feature real-time estimation module is specifically processed as follows: According to the integrity of the image contour of the target object in the current image obtained in S2, judge whether the target object is occluded, and correspondingly perform the real-time optimal estimation of the accurate image features and the feature depth: A. If the target object is not occluded, directly use the extended Kalman filter (EKF) to perform the optimal estimation filtering process on the image coordinates of the preliminary image features to obtain the image coordinates of the accurate image features and their feature depth; B. If the target object is occluded, then: First, based on the incomplete image contour corresponding to the preliminary image features in the current image, the camera pose M of the current image k is predicted, specifically obtained by solving the following objective function: where, ΔM is the relative pose of the camera between the previous moment and the current moment, K c is the camera internal parameter, M k-1 is the camera pose at the (k-1)-th moment, is the sampling point of the incomplete image contour partially occluded in the current image at the k-th moment, P(u i ) is the corresponding point of the sampling point in the CAD model of the target object; Δξ represents the camera pose transformation vector represented by Lie algebra, and Δξ ^ represents converting the pose transformation vector Δξ represented by Lie algebra into a homogeneous transformation matrix; M represents the camera pose, k represents the serial number of the moment, i represents the serial number of the control vertex, and m represents the total number of control vertices; Optimize and solve the above objective function to obtain the optimal camera pose transformation vector Δξ in Lie algebra representation, and then through the formula M k-1 exp(Δξ ^ ) to calculate the predicted camera pose M k ; Then, the predicted camera pose M is utilized k and the complete contour of the target object in the current image is predicted according to the projection relationship of the preset camera and the CAD model of the target object. The complete contour of the target object is fitted with a quasi-uniform B-spline curve to obtain the preliminary image features and their image coordinates. Then, the image coordinates of the preliminary image features are processed by extended Kalman filter (EKF) for optimal estimation filtering to obtain the image coordinates of the accurate image features and their feature depths.

2. A robust contour visual servo control method for complex-shaped objects according to claim 1, characterized in that: In the step S1, use a quasi-uniform B-spline curve with n control vertices to fit the contour in the target object contour image to obtain n control vertices, and use the n control vertices as the desired image features for visual servo; 3. A robust contour visual servo control method for complex-shaped objects according to claim 1, characterized in that: In the step S2, obtain the image contour of the target object from the current image captured by the camera through the Canny edge detection algorithm and the Rosenfeld skeleton thinning algorithm, and then use the quasi-uniform B-spline curve to fit the target object contour image to obtain the preliminary image features and their image coordinates of the current image; 4. A robust contour visual servo control method for complex-shaped objects according to claim 1, characterized in that: In the step S4, first process according to the following formula to obtain the interaction matrix of a single control vertex in the quasi-uniform B-spline curve corresponding to the current image: Among them, (x i , y i ) is the image coordinate of the i-th control vertex, Z i is the characteristic depth of the i-th control vertex, and L i represents the interaction matrix of the i-th control vertex; Then construct the interaction matrix of all control vertices in the image features from the interaction matrices of each control vertex: L BF = [L1 L2...L n T ​ where, L BF represents the interaction matrix of image features, and L n represents the interaction matrix of the nth control vertex in the image features, and T represents matrix transpose.

Images