Automatic compensation method and device for camera and manipulator coordinate system transformation matrix

Abstract

The invention discloses an automatic compensation method and device for a camera and manipulator coordinate system transformation matrix, and relates to the technical field of machine vision, and the method comprises the steps: arranging at least three calibration points on a flexible disc, and obtaining the coordinates of the calibration points before and after the camera is fixed again through the camera above the flexible disc, deriving a compensation conversion matrix of a camera image coordinate system; before the camera is fixed again, acquiring 9 point position coordinates recorded by the camera and 9 point position coordinates recorded by the manipulator when the camera and the manipulator perform the nine-point calibration method, and updating the 9 point position coordinates recorded by the camera by using the compensation conversion matrix; and on the basis of the nine point position new coordinates recorded by the camera and the nine point position coordinates recorded by the manipulator, a conversion matrix of a camera image coordinate system and a manipulator coordinate system is updated so as to recalculate the grabbing position of the manipulator. The method is easy to operate and high in precision, and compensation of the transformation matrix of the two coordinate systems can be achieved through coordinate transformation of the calibration points recorded by the camera.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to an automatic compensation method and device for the coordinate transformation matrix between a camera and a manipulator. Background Art

[0002] For the application scenario where a manipulator positions and grabs an object on a flexible disk, after obtaining the object image coordinates by a camera, they need to be converted into coordinates in the manipulator coordinate system through a transformation matrix, and then the manipulator grabs the object. After the device has been running for a long time, there will be looseness at the connection between the camera and the bracket, or at the connection between the camera and the lens, resulting in inaccurate grasping by the manipulator.

[0003] To solve the above problems, in the industry, it is usually necessary to re-fix the camera and then re-perform the nine-point calibration method between the camera and the manipulator to obtain a new transformation matrix. However, this situation is often time-consuming and affects production efficiency. Summary of the Invention

[0004] In view of the above problems and technical requirements, the inventor of the present invention proposes an automatic compensation method and device for the coordinate transformation matrix between a camera and a manipulator. The technical solution of the present invention is as follows:

[0005] In a first aspect, the present application provides an automatic compensation method for the coordinate transformation matrix between a camera and a manipulator, including the following steps:

[0006] Arrange at least three calibration points on the flexible disk, and obtain the coordinates of the calibration points before and after the camera is re-fixed through a camera above the flexible disk;

[0007] Based on the obtained coordinates of the calibration points before and after the camera is re-fixed, derive a compensation transformation matrix for the camera image coordinate system;

[0008] Before the camera is re-fixed, obtain the 9-point coordinates recorded by the camera and the 9-point coordinates recorded by the manipulator when the camera and the manipulator perform the nine-point calibration method, and update the 9-point coordinates recorded by the camera using the compensation transformation matrix;

[0009] Based on the new 9-point coordinates recorded by the camera and the 9-point coordinates recorded by the manipulator, update the transformation matrix between the camera image coordinate system and the manipulator coordinate system to recalculate the grasping position of the manipulator.

[0010] A further technical solution thereof is that, based on the obtained coordinates of the calibration points before and after the camera is re-fixed, deriving a compensation transformation matrix for the camera image coordinate system includes:

[0011] Obtain the height from the camera lens to the disk surface before and after the camera is re-fixed;

[0012] Update the coordinates of the calibration points recorded before the camera refixation based on the change in height before and after the camera refixation, denoted as the updated value of the calibration point coordinates;

[0013] Calculate the compensation transformation matrix of the camera image coordinate system based on the updated value of the calibration point coordinates and the calibration point coordinates recorded after the camera refixation.

[0014] A further technical solution thereof is that the coordinates of the i-th calibration point recorded before the camera refixation are denoted as The updated value of the i-th calibration point coordinates is denoted as

[0015] Then, update the coordinates of the calibration points recorded before the camera refixation based on the change in height before and after the camera refixation, expressed as:

[0016]

[0017] Among them, H1 is the height from the camera lens to the disk surface measured before the camera refixation, and H2 is the height from the camera lens to the disk surface measured after the camera refixation.

[0018] A further technical solution thereof is that the calculation formula for calculating the compensation transformation matrix T or the transformation matrix Tcr between the camera image coordinate system and the manipulator coordinate system based on the two sets of coordinates is the same, expressed as:

[0019]

[0020] When calculating , (x, y) represents the calibration point coordinates recorded after the camera refixation, and (x′, y′) represents the updated value of the calibration point coordinates recorded before the camera refixation;

[0021] When calculating , (x, y) represents the new coordinates of 9 points recorded by the camera, and (x′, y′) represents the coordinates of 9 points recorded by the manipulator;

[0022] Among them, a, b, c, d, e, f represent the coefficients to be solved in the corresponding matrix.

[0023] A further technical solution thereof is that the method further includes:

[0024] After the camera refixation, the relative position of the camera with respect to the set coordinate system on the disk surface is within a certain range of the relative position of the camera with respect to the set coordinate system on the disk surface before refixation, where the set coordinate system is established based on the calibration points.

[0025] A further technical solution is that the method for obtaining the relative position of the camera with respect to the set coordinate system on the disk is the same before and after the camera is refixed. The relative position of the camera with respect to the set coordinate system on the disk before refixing includes:

[0026] Obtain the coordinates of each calibration point in the set coordinate system. Combine the coordinates of the calibration points obtained before the camera is refixed. Through the open-source OPENCV function SolvePNP, obtain the relative position PA1(X, Y, Z, RX, RY, RZ) of the camera with respect to the set coordinate system on the disk before refixing;

[0027] Among them, (X, Y, Z) represents the spatial coordinates of the camera with respect to the set coordinate system on the disk, and (RX, RY, RZ) represents the three-axis rotation angles of the camera with respect to the set coordinate system on the disk.

[0028] A further technical solution is that the set coordinate system is established based on calibration points and includes:

[0029] Three calibration points are respectively set at the corresponding corners of the flexible disk. Taking the first calibration point as the origin of the set coordinate system, the horizontal axis from the first calibration point to the second calibration point is the X-axis of the set coordinate system, and the vertical axis from the first calibration point to the third calibration point is the Y-axis of the set coordinate system.

[0030] A further technical solution is that the calibration points are implemented using metal dots, and the coordinates of the calibration points are the center coordinates of the metal dots.

[0031] In a second aspect, the present application also provides an automatic compensation device for the conversion matrix between the camera and the manipulator coordinate system. At least three calibration points need to be arranged on the flexible disk. Then, the device includes:

[0032] A compensation conversion matrix calculation module, whose input is the coordinates of the calibration points before and after the camera is refixed obtained by the camera above the flexible disk, and based on the coordinates of the calibration points before and after the camera is refixed, derives the compensation conversion matrix of the camera image coordinate system;

[0033] A camera point coordinate update module, whose input is the 9 point coordinates recorded by the camera during the nine-point calibration method of the camera and the manipulator obtained before the camera is refixed, and uses the compensation conversion matrix to update the 9 point coordinates recorded by the camera;

[0034] A coordinate system conversion matrix update module, whose input is the 9 point coordinates recorded by the manipulator during the nine-point calibration method of the camera and the manipulator obtained before the camera is refixed, and based on the 9 new point coordinates recorded by the camera and the 9 point coordinates recorded by the manipulator, updates the conversion matrix between the camera image coordinate system and the manipulator coordinate system to recalculate the grasping position of the manipulator.

[0035] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the first aspect are implemented.

[0036] The beneficial technical effects of the present invention are as follows:

[0037] The present invention proposes a method for quickly compensating the conversion relationship between the camera image coordinate system and the robot coordinate system based on calibration points. When the camera becomes loose, it is not necessary to perform nine-point calibration again, which can reduce manual operations. The structure of the present invention is simple. It only needs to stick at least three calibration points on the flexible disk surface, and by measuring the change in the height from the camera lens to the disk surface, the compensation transformation matrix of the camera image coordinate system can be deduced through the coordinate transformation of the calibration points recorded by the camera, and then the compensation of the transformation matrix between the camera image coordinate system and the robot coordinate system can be realized.

[0038] To improve the accuracy of this method, it is also necessary to ensure that the error in the relative position of the camera with respect to the set coordinate system on the disk surface is within a certain range before and after the camera is refixed. After such an operation, in addition to compensating for the height direction after the camera adjustment, the skewness that occurs after the camera adjustment can also be corrected to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of the automatic compensation method for the conversion matrix between the camera and the robot coordinate system provided by the present application.

[0040] Figure 2 is a schematic diagram of the installation of calibration points and the positional relationship between the camera and the flexible disk provided by the present application.

[0041] Figure 3 is a schematic diagram of the change in the height from the camera lens to the disk surface before and after the camera is refixed provided by the present application.

[0042] Figure 4 is a schematic diagram of the automatic compensation device for the conversion matrix between the camera and the robot coordinate system provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The following further describes the specific embodiments of the present invention with reference to the drawings.

[0044] Please refer to Figure 1 As shown, an embodiment of the present application provides an automatic compensation method for the conversion matrix between the camera and the robot coordinate system, which specifically includes the following steps:

[0045] Step 1: The camera and the manipulator perform a nine-point calibration method, and obtain the nine-point coordinates P1(X1, Y1), P2(X2, Y2), ……, P9(X9, Y9) recorded by the camera and the nine-point coordinates R1(X1, Y1), R2(X2, Y2), ……, R9(X9, Y9) recorded by the manipulator.

[0046] Step 2: Arrange four MARK calibration points 2 on the flexible disk 1, and obtain the coordinates of each calibration point through the camera 3 above the flexible disk 1, as Figure 2 shown. Optionally, in order to simplify the calculation, the MARK points provided in this embodiment are implemented by metal dots with a thickness of 0.3 mm, and are pasted on the four corners of the flexible disk 1, and the coordinates of the four circle centers are determined through a circle-finding algorithm which are the coordinates of the MARK points.

[0047] Step 3: Based on the coordinates of the calibration points before and after the camera is refixed, derive the compensation transformation matrix of the camera image coordinate system. This step specifically includes the following contents:

[0048] Step 3.2: Measure the height from the current camera lens to the disk surface with a tape measure, denoted as H1.

[0049] Step 3.4: After the camera and the bracket, and the camera and the lens are refixed, measure the height from the camera lens to the disk surface again with a tape measure, denoted as H2, and obtain the coordinates of each calibration point again through the camera, that is, determine the coordinates of the four circle centers through a circle-finding algorithm

[0050] Step 3.6: Update the coordinates of the calibration points recorded before the camera is refixed based on the change in height before and after the camera is refixed. As Figure 3 shown, the updated values of the calibration point coordinates corresponding to C1, C2, C3, and C4 after calculating the height change are expressed as

[0051] where:

[0052] Step 3.8: Based on the updated values of the calibration point coordinates Cn1, Cn2, Cn3, Cn4 and the coordinates of the calibration points C1′, C2′, C3′, C4′ recorded after the camera is refixed, calculate the compensation transformation matrix T of the camera image coordinate system. The solution of T is to solve the following formula:

[0053]

[0054] In the above formula, (x, y) represents the corresponding coordinate values of C1′, C2′, C3′, C4′, and (x′, y′) represents the corresponding coordinate values of Cn1, Cn2, Cn3, Cn4; a, b, c, d, e, f represent the coefficients to be solved within the matrix.

[0055] Step 4: Update the coordinates of the 9 points recorded by the camera before the camera is refixed using the compensation transformation matrix T, that is, multiply P1(X1, Y1), P2(X2, Y2), ……, P9(X9, Y9) by the compensation transformation matrix T respectively to obtain the new coordinates of the 9 points P1′(X1, Y1), P2′(X2, Y2), ……, P9′(X9, Y9).

[0056] Step 5: Update the transformation matrix Tcr between the camera image coordinate system and the manipulator coordinate system based on the new coordinates of the 9 points recorded by the camera and the coordinates of the 9 points recorded by the manipulator. Since the position of the manipulator has not changed, the points of the manipulator can be represented by the originally recorded R1(X1, Y1), R2(X2, Y2), ……, R9(X9, Y9). The solution of Tcr is to solve the following formula:

[0057]

[0058] In the above formula, (x, y) represents the corresponding coordinate values of the new coordinates P1′~P9′ of the 9 points recorded by the camera, and (x′, y′) represents the corresponding coordinate values of the 9 points R1~R9 recorded by the manipulator; a, b, c, d, e, f represent the coefficients to be solved within the matrix.

[0059] Step 6: Recalculate the grasping position of the manipulator. Multiply the coordinates P of the image by the new transformation matrix Tcr to convert to the grasping position R of the manipulator, that is, P * Tcr = R.

[0060] To improve the accuracy of this method, the above method further includes: after the camera is refixed, the relative position of the camera with respect to the set coordinate system A on the disk is within a certain range of the relative position of the camera with respect to the set coordinate system on the disk before refixation. Specifically, it includes the following sub - steps:

[0061] Before step 3.4, obtain the relative position of the camera with respect to the set coordinate system on the disk before the camera is refixed.

[0062] First, obtain the coordinates of each calibration point in the set coordinate system A. The set coordinate system A is established based on the calibration points. As shown in Figure 2 Taking the first calibration point ① as the origin of the coordinate system A, the horizontal axis from the first calibration point ① to the second calibration point ② as the X - axis of the set coordinate system, and the vertical axis from the first calibration point ① to the third calibration point ④ as the Y - axis of the coordinate system A, establish the coordinate system A, and obtain the coordinates of each calibration point in this coordinate system PL1(X1, Y1), PL2(X2, Y2), PL3(X3, Y3), PL4(X4, Y4).

[0063] Then, combining the positions C1, C2, C3, and C4 of the 4 calibration points in the camera imaging, through the open-source OPENCV function SolvePNP, the relative position PA1 (X, Y, Z, RX, RY, RZ) of the camera relative to the set coordinate system on the disk before refixation is obtained, that is, C i and PL i are substituted into the function SolvePNP, and the rotation and translation vectors from the camera image coordinate system to coordinate system A can be calculated, so as to obtain the relative position PA1 of the camera relative to the set coordinate system on the disk. Among them, (X, Y, Z) represents the spatial coordinates of the camera relative to the set coordinate system on the disk, and (RX, RY, RZ) represents the three-axis rotation angles (i.e., orientations) of the camera relative to the set coordinate system on the disk.

[0064] Step 3.5: Obtain the relative position of the camera relative to the set coordinate system on the disk after the camera is refixed. This step is the same as the method for obtaining PA1 before the camera is refixed, that is, the coordinates C i ′ and PL i recorded after the camera is refixed are substituted into the function SolvePNP to obtain the relative position PA2 (X, Y, Z, RX, RY, RZ) of the camera relative to the set coordinate system on the disk after refixation. Then ensure that (RX, RY, RZ) of PA1 and PA2 are within the set error range, otherwise return to Step 3.4 to readjust the position and angle of the camera and the bracket, and / or the tightness between the camera and the lens until PA1 and PA2 meet the set conditions. After such operations, in addition to compensating for the height direction after the camera is adjusted, the skew situation that occurs after the camera is adjusted can also be corrected to a certain extent.

[0065] Based on the same inventive concept, another embodiment of the present application also provides an automatic compensation device for the coordinate system conversion matrix between the camera and the manipulator. Refer to Figure 4As shown, the device specifically includes a compensation transformation matrix calculation module, a camera position coordinate update module, and a coordinate system transformation matrix update module that are connected in sequence. Among them, the input of the compensation transformation matrix calculation module is the calibration point coordinates before and after the camera is refixed obtained by the camera above the flexible disk, and based on the obtained calibration point coordinates before and after the camera is refixed, the compensation transformation matrix T of the camera image coordinate system is derived. The input of the camera position coordinate update module is the 9 position coordinates P1(X1, Y1), P2(X2, Y2),..., P9(X9, Y9) recorded by the camera during the nine-point calibration method between the camera and the manipulator before the camera is refixed, and the 9 position coordinates P1' to P9' recorded by the camera are updated using the compensation transformation matrix T. The input of the coordinate system transformation matrix update module is the 9 position coordinates R1(X1, Y1), R2(X2, Y2),..., R9(X9, Y9) recorded by the manipulator during the nine-point calibration method between the camera and the manipulator before the camera is refixed, and based on the 9 new position coordinates recorded by the camera and the 9 position coordinates recorded by the manipulator, the transformation matrix Tcr between the camera image coordinate system and the manipulator coordinate system is updated to recalculate the grasping position of the manipulator.

[0066] Since the implementation solution provided by this device for solving the problem is similar to the implementation solution described in steps 1 to 6 of the above method, the specific limitations of each module in the embodiment of the automatic compensation device can refer to the limitations of the corresponding steps in the automatic compensation method in the above text, and will not be elaborated here. Each module in the above automatic compensation device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form to facilitate the processor to call and execute the operations corresponding to the above modules.

[0067] In one embodiment, a computer device is provided. The computer device can be a terminal or a server. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it realizes the steps in an automatic compensation method for the transformation matrix between a camera and a manipulator coordinate system.

[0068] The above is only a preferred embodiment of the present application, and the present invention is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included in the protection scope of the present invention.

Claims

1. An automatic compensation method for the coordinate system conversion matrix of a camera and a manipulator, characterized in that The method includes: Arranging at least three calibration points on a flexible disk, and acquiring the coordinates of the calibration points before and after the camera is refixed by a camera above the flexible disk; Deriving a compensation transformation matrix for the camera image coordinate system based on the acquired coordinates of the calibration points before and after the camera is refixed; Before the camera is refixed, acquiring the coordinates of 9 points recorded by the camera and the coordinates of 9 points recorded by the manipulator during the nine-point calibration method of the camera, and updating the coordinates of the 9 points recorded by the camera by using the compensation transformation matrix; Updating the transformation matrix between the camera image coordinate system and the manipulator coordinate system based on the new coordinates of the 9 points recorded by the camera and the coordinates of the 9 points recorded by the manipulator, so as to recalculate the grasping position of the manipulator.

2. The automatic compensation method for the coordinate system conversion matrix of the camera and the manipulator according to claim 1, characterized in that, The deriving of the compensation transformation matrix for the camera image coordinate system based on the acquired coordinates of the calibration points before and after the camera is refixed includes: Acquiring the height from the camera lens to the disk surface before and after the camera is refixed; Updating the coordinates of the calibration points recorded before the camera is refixed based on the change in height before and after the camera is refixed, denoted as the updated value of the calibration point coordinates; Calculating the compensation transformation matrix for the camera image coordinate system based on the updated value of the calibration point coordinates and the coordinates of the calibration points recorded after the camera is refixed.

3. The automatic compensation method for the coordinate system conversion matrix of the camera and the manipulator according to claim 2, characterized in that, Denote the coordinates of the $i$-th calibration point recorded before the camera is refixed as Denote the updated value of the coordinates of the $i$-th calibration point as Then the updating of the coordinates of the calibration points recorded before the camera is refixed based on the change in height before and after the camera is refixed is expressed as: where H1 is the height from the camera lens to the disk surface measured before the camera is refixed, and H2 is the height from the camera lens to the disk surface measured after the camera is refixed.

4. The automatic compensation method for the coordinate system conversion matrix of the camera and the manipulator according to claim 2, wherein, The calculation formulas for calculating the compensation transformation matrix T or the transformation matrix Tcr between the camera image coordinate system and the manipulator coordinate system based on two sets of coordinates are the same, and are expressed as: When calculating the , (x, y) represents the coordinates of the calibration points recorded after the camera is refixed, and (x′, y′) represents the updated values of the coordinates of the calibration points recorded before the camera is refixed; When calculating , (x, y) represents the new coordinates of 9 points recorded by the camera, and (x′, y′) represents the coordinates of 9 points recorded by the manipulator; where a, b, c, d, e, f represent the coefficients to be solved in the corresponding matrix.

5. The automatic compensation method for the camera and manipulator coordinate system conversion matrix according to claim 1, characterized in that, The method further includes: After the camera is refixed, the relative position of the camera with respect to the set coordinate system on the disk surface is within a certain range of the relative position of the camera with respect to the set coordinate system on the disk surface before refixing, where the set coordinate system is established based on the calibration points.

6. The automatic compensation method for the coordinate system transformation matrix of the camera and the manipulator according to claim 5, characterized in that, Before and after the camera is refixed, the method for acquiring the relative position of the camera with respect to the set coordinate system on the disk surface is the same, and the method for acquiring the relative position of the camera with respect to the set coordinate system on the disk surface before refixing includes: Acquiring the coordinates of each calibration point in the set coordinate system, and combining the acquired coordinates of the calibration points before the camera is refixed, and obtaining the relative position PA1(X, Y, Z, RX, RY, RZ) of the camera with respect to the set coordinate system on the disk surface before refixing through the open-source OPENCV function SolvePNP; where (X, Y, Z) represents the spatial coordinates of the camera with respect to the set coordinate system on the disk surface, and (RX, RY, RZ) represents the three-axis rotation angles of the camera with respect to the set coordinate system on the disk surface.

7. The automatic compensation method for the camera and manipulator coordinate system conversion matrix according to claim 5, characterized in that The establishment of the set coordinate system based on the calibration points includes: Three calibration points are respectively set at the corresponding corners of the flexible disk. Taking the first calibration point as the origin of the set coordinate system, the horizontal axis from the first calibration point to the second calibration point is the X axis of the set coordinate system, and the vertical axis from the first calibration point to the third calibration point is the Y axis of the set coordinate system.

8. The automatic compensation method for the coordinate system conversion matrix of the camera and the manipulator according to claim 1, characterized in that If the calibration points are implemented by metal dots, the coordinates of the calibration points are the center coordinates of the metal dots.

9. An automatic compensation device for the coordinate system transformation matrix of a camera and a manipulator, characterized in that, At least three calibration points are arranged on the flexible disk, and the device includes: A compensation transformation matrix calculation module, whose input is the coordinates of the calibration points before and after the camera is re-fixed, obtained by a camera above the flexible disk, and based on the obtained coordinates of the calibration points before and after the camera is re-fixed, a compensation transformation matrix of the camera image coordinate system is derived; A camera position coordinate update module, whose input is the 9 position coordinates recorded by the camera during the nine-point calibration method of the camera and the manipulator before the camera is re-fixed, and the 9 position coordinates recorded by the camera are updated using the compensation transformation matrix; A coordinate system transformation matrix update module, whose input is the 9 position coordinates recorded by the manipulator during the nine-point calibration method of the camera and the manipulator before the camera is re-fixed, and based on the 9 new position coordinates recorded by the camera and the 9 position coordinates recorded by the manipulator, the transformation matrix between the camera image coordinate system and the manipulator coordinate system is updated to recalculate the grasping position of the manipulator.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images