Camera calibration method and system

Abstract

The present application provides a camera calibration method and system, the method comprising: providing a calibration plate, the calibration plate including a first feature point and a second feature point; moving a robotic arm equipped with a camera so that the camera's field of view at least partially covers the calibration plate, and acquiring an image captured by the camera; processing the image captured by the camera to determine a first feature matrix of a camera coordinate system; moving the robotic arm so that when a tool point of the robotic arm is aligned with the second feature point, the coordinates of the second feature point in a base coordinate system are recorded; calculating the second coordinate of the first feature point in the base coordinate system; calculating the third coordinate of the first feature point in the tool coordinate system based on the rigid transformation matrix of the base coordinate system and the tool coordinate system when the camera captures the image, to determine the second feature matrix of the tool coordinate system; and calculating the first transformation matrix between the camera coordinate system and the tool coordinate system based on the first and second feature matrices. The present application provides a simple, efficient, and highly accurate camera calibration method.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a camera calibration method and system. Background Art

[0002] Machine vision applications are becoming increasingly widespread. Combining machine vision with robotic arms can be applied in a variety of industrial scenarios. Before use, cameras must be calibrated. Existing methods typically use a checkerboard calibration method, which requires the camera to take multiple photos and calculate feature vectors in multiple poses. This is very cumbersome. Furthermore, after camera calibration using existing methods, calibration correction is required to achieve the required calibration accuracy. Consequently, existing camera calibration methods are complex, computationally intensive, and inefficient. Summary of the Invention

[0003] In response to the problems in the prior art, the purpose of this application is to provide a simple and easy-to-implement camera calibration method and system.

[0004] The first aspect of the present application provides a camera calibration method, comprising the following steps:

[0005] Providing a calibration plate, the calibration plate comprising a first characteristic point and a second characteristic point;

[0006] Moving a robotic arm provided with a camera so that the field of view of the camera at least partially covers the calibration plate, and acquiring an image captured by the camera;

[0007] Processing the image captured by the camera, extracting the first coordinates of the first feature point in the camera coordinate system, and determining a first feature matrix of the camera coordinate system;

[0008] When moving the robotic arm so that the robotic arm tool point is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system are recorded;

[0009] Calculating a second coordinate of the first feature point in the base coordinate system according to a positional relationship between a preset second feature point and the first feature point;

[0010] Calculating the third coordinate of the first feature point in the tool coordinate system according to the base coordinate system and the rigid transformation matrix of the tool coordinate system when the camera captures the image, so as to determine a second feature matrix of the tool coordinate system;

[0011] A first transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the first characteristic matrix and the second characteristic matrix.

[0012] In some embodiments, the acquiring of the camera image further includes recording the position information of the tool point of the robot arm in the base coordinate system when the camera captures the image;

[0013] The rigid transformation matrix between the base coordinate system and the tool coordinate system when the camera captures the image is calculated based on the position information of the tool point of the manipulator in the base coordinate system when the camera captures the image.

[0014] In some embodiments, the upper surface of the calibration plate is provided with a plurality of characteristic patterns, and the first characteristic point is a designated position point in the characteristic pattern;

[0015] Processing the camera-captured image to extract the first coordinates of the first feature point in the camera coordinate system includes the following steps:

[0016] Performing edge image segmentation on the image captured by the camera to obtain characteristic graphic edge image features;

[0017] The first coordinates of the first feature point are calculated according to the edge image features of the feature pattern and the position information of the first feature point in the feature pattern.

[0018] In some embodiments, the camera is a 3D camera, and the image captured by the camera is a 3D point cloud image;

[0019] Performing edge image segmentation on the image captured by the camera comprises the following steps:

[0020] Performing plane fitting on the 3D point cloud image to extract the upper surface feature image of the calibration plate;

[0021] Perform edge feature image segmentation on the upper surface feature image to obtain feature pattern edge image features.

[0022] In some embodiments, performing edge feature image segmentation on the upper surface feature image to obtain feature pattern edge image features comprises the following steps:

[0023] Extracting edge point information from the upper surface feature image;

[0024] Segmenting the edge feature image according to the edge point information to obtain an edge image feature group;

[0025] Feature recognition is performed on the edge image feature group to distinguish outer contour edge image features and characteristic pattern edge image features among the edge image features.

[0026] In some embodiments, after distinguishing the outer contour edge image features and the characteristic pattern edge image features in the edge image features, the method further includes the following steps:

[0027] Using a straight line fitting method to calculate the coordinates of the intersection of each two intersecting straight lines in the outer contour edge image feature to obtain the coordinates of multiple corner points;

[0028] Calculating the verification coordinates of the first feature point based on the positional relationship between the corner point and the first feature point;

[0029] The verification coordinates of the first feature point are compared with the first coordinates of the first feature point, and the accuracy of the first coordinates is verified according to the comparison result.

[0030] In some embodiments, the calibration plate includes at least three first characteristic points, and the three first characteristic points are not located on the same straight line;

[0031] Determining the first characteristic matrix of the camera coordinate system includes: determining a first direction vector and a second direction vector based on the first coordinates of the first feature point; calculating a third direction vector perpendicular to the first direction vector and the second direction vector, to obtain a first characteristic matrix including the three direction vectors;

[0032] Among them, determining the second characteristic matrix of the tool coordinate system includes: determining the fourth direction vector and the fifth direction vector based on the third coordinate of the first feature point; calculating the sixth direction vector perpendicular to the fourth direction vector and the fifth direction vector, and obtaining the second characteristic matrix including the three direction vectors.

[0033] In some embodiments, the second feature point coincides with the first feature point, and the first feature point is a specified position point in a feature pattern provided on the surface of the calibration plate.

[0034] In some embodiments, the first feature point and the second feature point do not overlap, the first feature point is a specified position point in a feature pattern provided on the upper surface of the calibration plate, and the second feature point is a raised point provided on the upper surface of the calibration plate.

[0035] In some embodiments, after calculating the first transformation matrix of the camera coordinate system and the tool coordinate system, the following steps are further included:

[0036] Calculate a fourth coordinate of the first feature point in a tool coordinate system according to the first transformation matrix and the first coordinate of the first feature point;

[0037] Calculating an offset according to the third coordinate and the fourth coordinate of the first feature point;

[0038] A second transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the offset and the first transformation matrix.

[0039] In some embodiments, the calibration plate is further provided with check points, and the method further comprises the following steps:

[0040] Moving the robotic arm so that the robotic arm tool point is aligned with the check point, and recording the coordinates of the check point in the base coordinate system;

[0041] Calculating a fifth coordinate of the check point in the tool coordinate system based on the base coordinate system and a rigid transformation matrix of the tool coordinate system when the camera captures the image;

[0042] Calculating the coordinates of the check point in the camera coordinate system based on the positional relationship between the check point and the first feature point;

[0043] Calculating a sixth coordinate of the check point in the tool coordinate system based on the second transformation matrix;

[0044] The fifth coordinate and the sixth coordinate of the check point are compared, and the accuracy of the first transformation matrix is ​​verified according to the comparison result.

[0045] A second aspect of the present application further provides a camera calibration system, which is applied to the camera calibration method, and the system includes:

[0046] A calibration plate, comprising a first characteristic point and a second characteristic point;

[0047] A robotic arm control module, used to control the movement of the robotic arm and control the camera to capture images;

[0048] The camera calibration module is used to perform the following steps:

[0049] When the robotic arm is controlled to move by the robotic arm control module so that the field of view of the camera at least partially covers the calibration plate, acquiring an image captured by the camera;

[0050] Processing the image captured by the camera, extracting the first coordinates of the first feature point in the camera coordinate system, and determining a first feature matrix of the camera coordinate system;

[0051] When the manipulator arm control module moves the manipulator arm so that the manipulator arm tool point is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system are obtained from the manipulator arm control module and recorded;

[0052] Calculating a second coordinate of the first feature point in the base coordinate system according to a positional relationship between a preset second feature point and the first feature point;

[0053] Calculating the third coordinate of the first feature point in the tool coordinate system according to the base coordinate system and the rigid transformation matrix of the tool coordinate system when the camera captures the image, so as to determine a second feature matrix of the tool coordinate system;

[0054] A first transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the first characteristic matrix and the second characteristic matrix.

[0055] The camera calibration method and system provided in this application have the following advantages:

[0056] By adopting the camera calibration method of the present application, it is only necessary to take an image in one posture of the robotic arm and locate the second feature point through the robotic arm, and the camera calibration can be completed through corresponding calculations, and the implementation method is simple; this method has low requirements for the calibration plate, the operation method of the staff is very simple, the calculation amount is small, and the camera calibration based on the robotic arm can be completed quickly and efficiently; this method adopts a robotic arm linkage method, and obtains the first coordinate of the first feature point in the camera coordinate system through camera shooting, locates the second feature point through the robotic arm and calculates the second coordinate of the first feature point in the tool coordinate system based on this. Compared with the existing calibration method, the camera calibration accuracy is improved, the calibration correction step after the calibration is completed is saved, and the camera calibration efficiency is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0058] Figure 1 is a flowchart of a camera calibration method according to an embodiment of the present application;

[0059] Figure 2 This is a schematic structural diagram of a calibration plate according to an embodiment of the present application;

[0060] Figure 3 This is a flow chart of obtaining the first coordinates of the first feature point according to an embodiment of the present application;

[0061] Figure 4 is a schematic diagram of a characteristic image of the upper surface of a calibration plate according to an embodiment of the present application;

[0062] Figure 5 is a schematic diagram of determining a first direction vector and a second direction vector in a camera coordinate system according to an embodiment of the present application;

[0063] Figure 6 This is a flowchart of calculating offset and verifying calibration accuracy according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated descriptions will be omitted. "Or" and "or" in the specification may both mean "and" or "or". Although the terms "above", "below", "between", etc. may be used in this specification to describe different exemplary features and elements of the present application, these terms are used herein only for convenience, such as according to the directions of the examples described in the accompanying drawings. Nothing in this specification should be construed as requiring a specific three-dimensional orientation of the structure to fall within the scope of this application. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not intended to limit the number and importance of specific features.

[0065] like Figure 1 As shown, the first aspect of the present application provides a camera calibration method, comprising the following steps:

[0066] S100: Providing a calibration plate, wherein the calibration plate includes a first feature point and a second feature point;

[0067] S200: moving a robotic arm provided with a camera so that the field of view of the camera at least partially covers the calibration plate, and acquiring an image captured by the camera;

[0068] In this embodiment, the camera is provided at the end of the robotic arm, and the robotic arm is, for example, a six-axis robotic arm, but the present application is not limited thereto and can also be applied to robotic arms having other numbers of axes; obtaining the image captured by the camera here can be performed by directly communicating with the camera or by communicating with a robotic arm control module that controls the operation of the robotic arm; the robotic arm is, for example, a robotic arm of an intelligent robot, or can also be a separate robotic arm, and the robotic arm control module can control the movement of the robotic arm and the camera shooting action;

[0069] Here, the field of view of the camera is made to at least partially cover the calibration plate. It is necessary to make the field of view of the camera cover the portion of the calibration plate having the required first feature point. In order to subsequently identify the outer edge feature, the field of view of the camera can optionally be made to completely cover the calibration plate.

[0070] S300: Processing the image captured by the camera, extracting the first coordinates of the first feature point in the camera coordinate system, and determining a first feature matrix of the camera coordinate system;

[0071] S400: When moving the robotic arm so that the robotic arm tool point is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system are recorded;

[0072] The tool point of the robot arm here is the TCP (Tool Central Point) point at the end of the robot arm. When the tool point of the robot arm is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system can be read from the robot arm control module; it refers to the center position of the tool or tool, which is used to describe the position of the tool relative to the robot base coordinate system. In the initial state, the TCP point is the origin of the tool coordinate system. When the robot is manually operated or programmed to approach a point in space, the tool center point is actually moved to that point. Therefore, the trajectory motion of the robot can be regarded as the motion of the tool center point (TCP); the base coordinate system is a rectangular coordinate system based on the robot arm mounting base and used to describe the motion of the robot arm;

[0073] S500: Calculating a second coordinate of the first feature point in the base coordinate system according to a positional relationship between a preset second feature point and the first feature point;

[0074] S600: Calculating the third coordinate of the first feature point in the tool coordinate system according to the base coordinate system and the rigid transformation matrix of the tool coordinate system when the camera captures the image, so as to determine a second feature matrix of the tool coordinate system;

[0075] S700: Calculating a first transformation matrix between the camera coordinate system and the tool coordinate system according to the first characteristic matrix and the second characteristic matrix.

[0076] By adopting the camera calibration method of the present application, it is only necessary to capture an image in one posture of the robotic arm and locate the second feature point through the robotic arm. The camera calibration can be completed through the corresponding calculation, which is simple to implement. The method has low requirements for the calibration plate, the operator's operation method is very simple, the calculation amount is small, and the camera calibration based on the robotic arm can be completed quickly and efficiently. The method adopts a robotic arm linkage method. In steps S200 to S300, the first coordinate of the first feature point in the camera coordinate system is obtained by camera shooting. In steps S400 to S500, the second feature point is located by the robotic arm and the second coordinate of the first feature point in the tool coordinate system is calculated based on this. Then, in steps S600 to S700, the first transformation matrix between the camera coordinate system and the working coordinate system is calculated. Compared with the existing calibration method, this method improves the camera calibration accuracy, saves the calibration correction step after the calibration is completed, and further improves the camera calibration efficiency. After testing, when using a robotic arm with a repeatability of 0.001mm and a 3D camera with a 0.2mm pixel, the calibration accuracy is within ±0.25.

[0077] In this embodiment, the calibration plate has a plurality of feature patterns on its upper surface, and the first feature point is a designated location within the feature pattern. When processing a camera-captured image, the x-axis and y-axis directions of a camera coordinate system are determined based on the camera image (e.g., with the lower left corner of the image as the zero point, the first extension direction as the x-axis, and the second extension direction as the y-axis). The position and outline of the feature pattern are identified in the camera image, and then the first coordinate of the first feature point in the camera coordinate system is determined based on the position information of the first feature point in the feature pattern. The position information of the first feature point in the feature pattern may include, for example, the first feature point at the center point of the feature pattern, the first feature point at the intersection of the feature pattern, or the first feature point at a corner of the feature pattern. The first feature point and the second feature point do not overlap; the second feature point is a raised point on the upper surface of the calibration plate. Because the second feature point is raised from the upper surface of the calibration plate, it is easier to align the TCP point of the robot arm with the second feature point in step S400, and it is also easier to observe and check the alignment between the TCP point and the second feature point.

[0078] Figure 2 The structure of an exemplary calibration plate is shown. Three characteristic figures A, B, and C are provided on the upper surface of the calibration plate, and the first characteristic points correspond to the center points of each of the characteristic figures. The center point of characteristic figure A and the center point of characteristic figure B are located on a first straight line, and the center point of characteristic figure A and the center point of characteristic figure B are located on a second straight line. The first straight line and the second straight line intersect but are not collinear. There is an angle between the first straight line and the second straight line that is not 0°. In this embodiment, the first straight line and the second straight line are perpendicular to each other as an example, that is, the three first characteristic points form a right triangle, the center point of characteristic figure A serves as the right angle point of the right triangle, the center point of characteristic figure B serves as the endpoint of the long side of the right triangle, and the center point of characteristic figure C serves as the endpoint of the short side of the right triangle. Four slender needles a, b, c, and d are provided on the calibration plate, wherein the tips of needles a, b, and c are second characteristic points respectively. The three second characteristic points are not located on the same straight line, and the tip of needle d serves as a verification point. Figure 2The calibration plate structure in the figure is merely an example, and other variations are possible in different implementations. The number and position of the first and second feature points can be adjusted as needed. The feature pattern is not limited to a circle; it can also take the form of a square, triangle, pentagon, or other shapes that can be identified through image recognition. The second feature point can also be the vertex of a raised portion of another shape, and does not necessarily require the form of a slender needle. In an alternative embodiment, the second feature point can also take the form of a position point within a groove, or the groove can assist the robot arm in locating the TCP point and the second feature point. In another alternative embodiment, the second feature point can also coincide with the first feature point. For example, the center of circles A, B, and C is used as both the first and second feature points. In step S400, the robot arm is moved so that the TCP point is aligned with the center of the circle. After obtaining the coordinates of the second feature point in the tool coordinate system, it can be directly used as the second coordinates of the first feature point. In yet another alternative embodiment, the feature pattern can take the form of a cross-shaped groove or a cross-shaped protrusion, with the center point of the cross serving as both the first and second feature points, which also falls within the scope of protection of this application.

[0079] The following Figure 2 The implementation of the camera calibration method is described in detail using the calibration board shown in FIG. Figure 2 The calibration plate shown in and the following description are not intended to limit the scope of protection of this application.

[0080] In this embodiment, in step S200, when acquiring the camera image, the process also includes recording the position information of the robot arm tool point in the base coordinate system at the time the camera captured the image. The rigid transformation matrix between the base coordinate system and the tool coordinate system at the time the camera captured the image is calculated based on the position information of the robot arm tool point in the base coordinate system at the time the camera captured the image. The step of calculating the rigid transformation matrix is ​​performed after step S200 and must be completed before step S600. In other words, there is no order requirement for executing this step along with steps S300 to S500.

[0081] In this embodiment, the position information of the tool point of the manipulator in the base coordinate system when the camera captures the image can be read from the manipulator control module, including the coordinate values ​​and posture information of the x, y, and z axes of the base coordinate system, which is recorded here as P cap ={x cap ,y cap ,z cap ,w cap ,p cap ,r cap According to the rotation mode of the robot arm, based on the position information P capTo calculate the rigid transformation matrix from the origin of the base coordinate system to the tool point at the end of the robot arm. Here, the robot arm external rotation xyz sequence is taken as an example, and P cap Abbreviated as P cap = {X, Y, Z, W, P, R}, calculate the rigid transformation matrix R BtE as follows:

[0082]

[0083] In this embodiment, the camera is a 3D camera installed at the end of the robotic arm, and the image captured by the camera is a 3D point cloud image. Figure 3 As shown, in step S300, processing the image captured by the camera to extract the first coordinates of the first feature point in the camera coordinate system includes the following steps:

[0084] S310: performing plane fitting on the 3D point cloud image to extract a feature image of the upper surface of the calibration plate; Figure 4 The upper surface feature image of the calibration plate is schematically shown;

[0085] Specifically, after acquiring the 3D point cloud image, performing outlier noise filtering, straight-through filtering and other operations on the point cloud image, extracting the calibration plate point cloud feature image, performing plane geometry on the calibration plate point cloud feature image, extracting the upper surface feature image of the calibration plate, the upper surface feature image including the upper surface point cloud information of the calibration plate;

[0086] Performing edge feature image segmentation on the upper surface feature image to obtain feature pattern edge image features; specifically, the method comprises the following steps:

[0087] S320: Extracting edge point information from the upper surface feature image; for example, the Alpha Shapes algorithm may be used to quickly extract edge point information from the upper surface of the calibration plate;

[0088] S330: Segmenting the edge feature image according to the edge point information to obtain an edge image feature group;

[0089] S340: performing feature recognition on the edge image feature group to distinguish outer contour edge image features and characteristic pattern edge image features in the edge image features;

[0090] For example, a hu-moment pattern recognition method may be used to perform pattern recognition on the edge feature image feature group to distinguish between outer contour edge image features and characteristic pattern edge image features;

[0091] like Figure 2As shown, the calibration plate has a chamfer on the side opposite the characteristic pattern A, making it easier to distinguish between the outer contour edge image features and the characteristic pattern edge image features. After the outer contour edge image features are identified, different characteristic patterns can be distinguished based on the positional relationship between the outer contour edge image features and each characteristic pattern, which is used to subsequently determine the long side endpoints, right angle points, and short side endpoints of the right triangle. The chamfer is provided here for example only. In other alternative embodiments, the calibration plate can also have other edge shapes at other locations, which also fall within the scope of protection of this application.

[0092] S350: Calculate the coordinates of the intersection of each two intersecting straight lines in the outer contour edge image feature using a straight line fitting method, take the intersection as the corner point of the outer contour, and obtain the coordinates of n corner points in the camera coordinate system, recorded as Among them, P cornor_i ={x i ,y i ,z i}.

[0093] S360: Calculate the verification coordinates of the first feature point based on the positional relationship between the corner point and the first feature point. Specifically, the positional relationship between the corner point and the first feature point may include a distance between the corner point and the first feature point in the x-axis direction and a distance between the corner point and the first feature point in the y-axis direction. Adjust the coordinates of the corner point in the camera coordinate system based on the distance to obtain the verification coordinates of the first feature point in the camera coordinate system.

[0094] S370: Calculate the first coordinates of the first feature point according to the edge image features of the feature pattern and the position information of the first feature point in the feature pattern; take the feature pattern including three circles and the first feature point as the circle center as an example, that is, obtain the first coordinates of the circle center of each circular edge image feature in the camera coordinate system, recorded as P center_Set ={P center_1 ,P center_2 ,P center_3}, where P center_i ={x i ,y i ,z i}.

[0095] S380: Compare the check coordinates of the first feature point with the first coordinates of the first feature point, and verify the accuracy of the first coordinates based on the comparison result. Specifically, the difference between the check coordinates and the first coordinates of the first feature point is calculated, and a determination is made as to whether the difference is within a preset allowable difference range. If so, the accuracy verification of the first coordinates passes, and subsequent steps are performed. Otherwise, the accuracy verification of the first coordinates fails, and step S300 is repeated until the accuracy verification of the first coordinates passes. Therefore, in this embodiment, steps S350, S360, and S380 ensure the accuracy of the calculated first coordinates of the first feature point.

[0096] In step S300, determining the first characteristic matrix of the camera coordinate system includes:

[0097] determining a first direction vector and a second direction vector according to the first coordinates of the first feature point;

[0098] like Figure 4 As shown, the center point of the characteristic figure A is taken as the right angle point of the right triangle, and its first coordinate is marked as P center_cornorPnt The center point of the characteristic figure B is the endpoint of the long side of the right triangle, and its first coordinate is marked as P center_longPnt The center point of the characteristic figure C is the endpoint of the short side of the right triangle, and its first coordinate is marked as P center_shortPnt , and get a new set of circle center points:

[0099] P center_Set ={P center_longPnt ,P center_cornorPnt ,P center_shortPnt}, where P center_str ={x i ,y i ,z i};

[0100] Taking the right angle point of the right triangle as the starting point and the endpoint of the long side of the right triangle as the end point, calculate the first direction vector. Normalize the vector like Figure 5 As shown, the vector from the center point of characteristic pattern A to the center point of characteristic pattern B is the first direction vector;

[0101] Taking the right angle point of the right triangle as the starting point and the endpoint of the short side of the right triangle as the end point, calculate the second direction vector. Normalize the vector like Figure 5 As shown, the vector from the center point of the characteristic pattern A to the center point of the characteristic pattern C is the second direction vector;

[0102] Calculating a third direction vector perpendicular to the first direction vector and the second direction vector to obtain a first characteristic matrix including the three direction vectors;

[0103] Specifically, the first direction vector is cross-multiplied with the second direction vector to obtain the plane normal vector as the third direction vector. The calculated eigenvectors constitute the first eigenmatrix of the camera coordinate system

[0104] In this embodiment, in step S400, when the robot arm is moved to align the robot arm tool point with the second feature point, the coordinates of the second feature point in the base coordinate system are recorded and recorded as P teachSet0 ={P teach10 , P teach20 , P teach30 , P teach4}, where P teachi ={x i ,y i , z i};P teach10 , P teach20 , P teach30 are the coordinates of the second feature points corresponding to the needle tips of needles a, b, and c in the base coordinate system, P teach4 is the coordinate of the checkpoint corresponding to the needle tip of needle d in the base coordinate system;

[0105] In the step S500, the second coordinate of the first feature point in the base coordinate system is calculated based on the positional relationship between the preset second feature point and the first feature point, which is denoted as P. teachSet ={P teach1 , P teach2 , P teach3 , P teach4}, where P i ={x i ,y i , z i}, P teach1 , P teach2 , P teach3The second coordinates of each first feature point corresponding to the center of the characteristic figures B, A, and C in the base coordinate system are respectively; the positional relationship between the second feature points and the first feature points preset here may include, for each second feature point, the actual distance from the second feature point to each first feature point in two directions (the AB direction and the AC direction), and through this positional relationship, the coordinates of the second feature point in the base coordinate system can be converted into the second coordinates of the first feature point in the base coordinate system; in this embodiment, the three second feature points also form a right triangle, one second feature point is located on the straight line AB, one second feature point is located on the straight line AC, and the other second feature point serves as the right angle point of the right triangle, but the present application is not limited thereto;

[0106] In step S600, the rigid transformation matrix R of the tool coordinate system when the camera captures the image has been obtained. BtE , calculate the third coordinate set of the first feature point in the tool coordinate system as:

[0107] In step S600, determining the second characteristic matrix of the tool coordinate system includes the following steps:

[0108] determining a fourth direction vector and a fifth direction vector according to the third coordinate of the first feature point;

[0109] According to the arrangement order of the first feature points, the center point of the feature figure A is also used as the right angle point of the right triangle, and its third coordinate is P tool_cornorPnt , the center point of the characteristic figure B is the endpoint of the long side of the right triangle, and its third coordinate is P tool_longPnt The center point of the characteristic figure C is the endpoint of the short side of the right triangle, and its third coordinate is P tool_shortPnt , the third coordinate set of the first feature point in the tool coordinate system is:

[0110] P tool_Set ={P tool_longPnt , P tool_cornorPnt , P tool_shortPnt , P tool_checkPnt}, where P tool_str ={x i ,y i , z i};P tool_checkPnt Equal to the above R BtE *P teach4 , is the fifth coordinate of the check point in the tool coordinate system;

[0111] Taking the right angle point of the right triangle as the starting point and the endpoint of the long side of the right triangle as the end point, calculate the fourth direction vector. Normalize the vector

[0112] Step 8.2: Using the right angle point of the right triangle as the starting point and the endpoint of the short side of the right triangle as the end point, calculate the fifth direction vector. Normalize the vector

[0113] Calculating a sixth direction vector perpendicular to the fourth direction vector and the fifth direction vector to obtain a second characteristic matrix including the three direction vectors;

[0114] Specifically, the fourth direction vector is cross-multiplied with the fifth direction vector to obtain the plane normal vector as the sixth direction vector. The second characteristic matrix of the tool coordinate system is obtained, which is recorded as

[0115] In step S700, a first transformation matrix R between the camera coordinate system and the tool coordinate system is calculated based on the first characteristic matrix and the second characteristic matrix. CtE =R robot *R camera -1 , that is, R robot =R CtE *R camera Here, the first transformation matrix is ​​the rotation matrix from the camera coordinate system to the tool coordinate system.

[0116] like Figure 6 As shown, in this embodiment, after the step S700 of calculating the first transformation matrix of the camera coordinate system and the tool coordinate system, the step of calculating the offset and correcting the first transformation matrix based on the offset is further included, which specifically includes the following steps:

[0117] S810: Calculate the fourth coordinate of the first feature point in the tool coordinate system according to the first transformation matrix and the first coordinate of the first feature point; any one of the multiple first feature points can be selected for calculation. Here, the first feature point corresponding to the center point A is selected as an example. The calculation method of the fourth coordinate is: P center_cornorPnt_AfterTransform =R CtE *P center_cornorPnt If another first feature point is selected, its first coordinate is multiplied by the first transformation matrix to obtain the corresponding fourth coordinate;

[0118] S820: Calculate an offset based on the third coordinate and the fourth coordinate of the first feature point. Specifically, the offset is calculated as follows:

[0119]

[0120] S830: Calculate a second transformation matrix between the camera coordinate system and the tool coordinate system according to the offset and the first transformation matrix. Specifically, the second transformation matrix is ​​calculated as follows:

[0121]

[0122] The second conversion matrix is ​​a rigid body transformation matrix from the camera coordinate system to the tool coordinate system, and can be used in subsequent process operations based on the robotic arm equipped with a camera.

[0123] As described above, when executing step S400, the method further includes moving the robotic arm so that when the robotic arm tool point is aligned with the check point, the coordinates of the check point in the base coordinate system are recorded. In step S600, when calculating the third coordinate of the first feature point in the tool coordinate system, the method further includes calculating the fifth coordinate P of the check point in the tool coordinate system based on the rigid transformation matrix of the base coordinate system and the tool coordinate system when the camera captures the image. tool_checkPnt =R BtE *P teach4 .

[0124] like Figure 6 As shown, after step S830, the method further includes a step of verifying the accuracy of the second transformation matrix based on the check points, which specifically includes the following steps:

[0125] S910: Calculate the coordinates of the check point in the camera coordinate system based on the positional relationship between the check point and the first feature point; specifically, calculate the coordinates P of the check point in the camera coordinate system based on the distance between the check point and the first feature point and the first coordinate of the first feature point in the camera coordinate system. camera_checkPnt ={x, y, z, 1}; In this embodiment, the positional relationship between the check point and the first feature point may include the positional relationship between the check point and the three first feature points respectively, and the coordinates of the check point in the camera coordinate system can be calculated based on the first coordinates of the three first feature points and this positional relationship;

[0126] S920: Calculate a sixth coordinate of the check point in the tool coordinate system based on the second transformation matrix;

[0127] Specifically, based on the coordinates of the check point in the camera coordinate system and the second transformation matrix, the sixth coordinate of the check point in the tool coordinate system is equal to P′ tool_checkPnt =RT CtE *P camera_checkPnt ; Among them, P′ tool_checkPnt ={x′,y′,z′,1};

[0128] S930: Compare the fifth coordinate P of the check point tool_checkPnt and the sixth coordinate P′ tool_checkPnt , verifying the accuracy of the first conversion matrix according to the comparison result;

[0129] Specifically, the error between the fifth coordinate and the sixth coordinate of the checkpoint in the tool coordinate system is calculated as When Δ<ε (the set error range), it is considered that the accuracy of the calibration result meets the requirements, and the process of the camera calibration method is terminated. Otherwise, it is determined that the accuracy of the calibration result does not meet the requirements, and the process of the camera calibration method is re-executed until the error is less than the set error range.

[0130] The present application also provides a camera calibration system, which is applied to the camera calibration method. The system includes:

[0131] The calibration plate includes a first feature point and a second feature point; the first feature point and the second feature point may be overlapping feature points or non-overlapping feature points. The calibration plate may be, for example, Figure 2 The shapes shown, but the application is not limited thereto;

[0132] A robotic arm control module, used to control the movement of the robotic arm and control the camera to capture images;

[0133] The camera calibration module is used to perform the following steps:

[0134] When the robotic arm is controlled to move by the robotic arm control module so that the field of view of the camera at least partially covers the calibration plate, acquiring an image captured by the camera;

[0135] Processing the image captured by the camera, extracting the first coordinates of the first feature point in the camera coordinate system, and determining a first feature matrix of the camera coordinate system;

[0136] When the manipulator arm control module moves the manipulator arm so that the manipulator arm tool point is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system are obtained from the manipulator arm control module and recorded;

[0137] Calculating a second coordinate of the first feature point in the base coordinate system according to a positional relationship between a preset second feature point and the first feature point;

[0138] Calculating the third coordinate of the first feature point in the tool coordinate system according to the base coordinate system and the rigid transformation matrix of the tool coordinate system when the camera captures the image, so as to determine a second feature matrix of the tool coordinate system;

[0139] A first transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the first characteristic matrix and the second characteristic matrix.

[0140] By adopting the camera calibration system of the present application, it is only necessary to take an image in one posture of the robotic arm and locate the second feature point through the robotic arm, and the camera calibration can be completed through corresponding calculations, and the implementation method is simple; the system has low requirements for the calibration plate, the operation method of the staff is very simple, the calculation amount is small, and the camera calibration based on the robotic arm can be completed quickly and efficiently; the system adopts a robotic arm linkage method, and obtains the first coordinate of the first feature point in the camera coordinate system through camera shooting, locates the second feature point through the robotic arm and calculates the second coordinate of the first feature point in the tool coordinate system based on this. Compared with the existing calibration system, the camera calibration accuracy is improved, the calibration correction step after the calibration is completed is saved, and the camera calibration efficiency is further improved.

[0141] The implementation of each step in the system can adopt the specific implementation of the above-mentioned camera calibration method, which will not be repeated here.

[0142] The above content is a further detailed description of the present application in conjunction with specific preferred embodiments, and the specific implementation of the present application cannot be considered to be limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be considered to fall within the scope of protection of the present application.

Claims

1. A camera calibration method, characterized in that: The steps include: Providing a calibration plate, the calibration plate comprising a first characteristic point and a second characteristic point; Moving a robotic arm provided with a camera so that the field of view of the camera at least partially covers the calibration plate, and acquiring an image captured by the camera; Processing the image captured by the camera, extracting the first coordinates of the first feature point in the camera coordinate system, and determining a first feature matrix of the camera coordinate system; When moving the robotic arm so that the robotic arm tool point is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system are recorded; Calculating a second coordinate of the first feature point in the base coordinate system according to a positional relationship between a preset second feature point and the first feature point; Calculating the third coordinate of the first feature point in the tool coordinate system according to the base coordinate system and the rigid transformation matrix of the tool coordinate system when the camera captures the image, so as to determine a second feature matrix of the tool coordinate system; A first transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the first characteristic matrix and the second characteristic matrix.

2. The camera calibration method according to claim 1, wherein: The acquisition of the camera image also includes recording the position information of the tool point of the manipulator in the base coordinate system when the camera captures the image; The rigid transformation matrix between the base coordinate system and the tool coordinate system when the camera captures the image is calculated based on the position information of the tool point of the manipulator in the base coordinate system when the camera captures the image.

3. The camera calibration method according to claim 1, wherein: The upper surface of the calibration plate is provided with a plurality of characteristic patterns, and the first characteristic point is a designated position point in the characteristic pattern; Processing the camera-captured image to extract the first coordinates of the first feature point in the camera coordinate system includes the following steps: Performing edge image segmentation on the image captured by the camera to obtain characteristic graphic edge image features; The first coordinates of the first feature point are calculated according to the edge image features of the feature pattern and the position information of the first feature point in the feature pattern.

4. The camera calibration method according to claim 3, wherein: The camera is a 3D camera, and the image captured by the camera is a 3D point cloud image; Performing edge image segmentation on the camera-photographed image comprises the following steps: Performing plane fitting on the 3D point cloud image to extract the upper surface feature image of the calibration plate; Perform edge feature image segmentation on the upper surface feature image to obtain feature pattern edge image features.

5. The camera calibration method according to claim 4, wherein: Performing edge feature image segmentation on the upper surface feature image to obtain feature pattern edge image features includes the following steps: Extracting edge point information from the upper surface feature image; Segmenting the edge feature image according to the edge point information to obtain an edge image feature group; Feature recognition is performed on the edge image feature group to distinguish outer contour edge image features and characteristic pattern edge image features among the edge image features.

6. The camera calibration method according to claim 5, characterized in that: After distinguishing the outer contour edge image features and the characteristic pattern edge image features in the edge image features, the following steps are also included: Using a straight line fitting method to calculate the coordinates of the intersection of each two intersecting straight lines in the outer contour edge image feature to obtain the coordinates of multiple corner points; Calculating the verification coordinates of the first feature point based on the positional relationship between the corner point and the first feature point; The verification coordinates of the first feature point are compared with the first coordinates of the first feature point, and the accuracy of the first coordinates is verified according to the comparison result.

7. The camera calibration method according to claim 1, wherein: The calibration plate includes at least three first characteristic points, and the three first characteristic points are not located on the same straight line; Determining the first characteristic matrix of the camera coordinate system includes: determining a first direction vector and a second direction vector based on the first coordinates of the first feature point; calculating a third direction vector perpendicular to the first direction vector and the second direction vector, to obtain a first characteristic matrix including the three direction vectors; Among them, determining the second characteristic matrix of the tool coordinate system includes: determining the fourth direction vector and the fifth direction vector based on the third coordinate of the first feature point; calculating the sixth direction vector perpendicular to the fourth direction vector and the fifth direction vector, and obtaining the second characteristic matrix including the three direction vectors.

8. The camera calibration method according to claim 1, wherein: The second feature point coincides with the first feature point, and the first feature point is a designated position point in a feature pattern provided on a surface of the calibration plate.

9. The camera calibration method according to claim 1, wherein: The first feature point and the second feature point do not overlap, the first feature point is a specified position point in a feature pattern set on the upper surface of the calibration plate, and the second feature point is a raised point set on the upper surface of the calibration plate.

10. The camera calibration method according to claim 1, wherein: After calculating the first transformation matrix of the camera coordinate system and the tool coordinate system, the following steps are also included: Calculate a fourth coordinate of the first feature point in a tool coordinate system according to the first transformation matrix and the first coordinate of the first feature point; Calculating an offset according to the third coordinate and the fourth coordinate of the first feature point; A second transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the offset and the first transformation matrix.

11. The camera calibration method according to claim 10, wherein: The calibration plate is further provided with a check point, and the method further comprises the following steps: Moving the robotic arm so that the robotic arm tool point is aligned with the check point, and recording the coordinates of the check point in the base coordinate system; Calculating a fifth coordinate of the check point in the tool coordinate system based on the base coordinate system and a rigid transformation matrix of the tool coordinate system when the camera captures the image; Calculating the coordinates of the check point in the camera coordinate system based on the positional relationship between the check point and the first feature point; Calculating a sixth coordinate of the check point in the tool coordinate system based on the second transformation matrix; The fifth coordinate and the sixth coordinate of the check point are compared, and the accuracy of the first transformation matrix is ​​verified according to the comparison result.

12. A camera calibration system, characterized in that: The camera calibration method according to any one of claims 1 to 11, wherein the system comprises: A calibration plate, comprising a first characteristic point and a second characteristic point; A robotic arm control module, used to control the movement of the robotic arm and control the camera to capture images; The camera calibration module is used to perform the following steps: When the robotic arm is controlled to move by the robotic arm control module so that the field of view of the camera at least partially covers the calibration plate, acquiring an image captured by the camera; Processing the image captured by the camera, extracting the first coordinates of the first feature point in the camera coordinate system, and determining a first feature matrix of the camera coordinate system; When the manipulator arm control module moves the manipulator arm so that the manipulator arm tool point is aligned with the second feature point, the coordinates of the second feature point in the base coordinate system are obtained from the manipulator arm control module and recorded; Calculating a second coordinate of the first feature point in the base coordinate system according to a positional relationship between a preset second feature point and the first feature point; Calculating the third coordinate of the first feature point in the tool coordinate system according to the base coordinate system and the rigid transformation matrix of the tool coordinate system when the camera captures the image, so as to determine a second feature matrix of the tool coordinate system; A first transformation matrix between the camera coordinate system and the tool coordinate system is calculated according to the first characteristic matrix and the second characteristic matrix.

Images