Three-dimensional camera and three-dimensional laser combined calibration method and system

Abstract

This invention discloses a method and system for joint calibration of a 3D camera and a 3D laser. By calibrating the pose transformation relationship between the 3D camera and the 3D laser, an industrial robot holds a workpiece, takes a picture under the 3D camera, and performs real-time 3D reconstruction. The point cloud coordinate data of the workpiece surface is converted into a 3D laser marking trajectory and sent to the 3D laser for marking operation. The advantages of this invention are: reduced application steps, adaptability to different workpiece shapes, and true automation and intelligence of workpiece marking.

Description

Technical Field

[0001] This invention relates to the field of laser calibration and cutting technology, and in particular to a method and system for joint calibration of a three-dimensional camera and a three-dimensional laser. Background Technology

[0002] 3D laser marking technology, with its advantages of low processing cost, high efficiency, and good processing quality, is now widely used in marking and cutting materials such as metals, plastics, and wood. 3D cameras, due to their non-contact, visualization, adaptability, and high precision, are gradually being adopted in fields such as product defect detection and material identification.

[0003] Industrial robots are multi-degree-of-freedom mechanical devices widely used in industrial fields. In practical production applications, they are mainly programmed and operated through two methods: online teaching and offline programming. Compared to absolute positioning accuracy, industrial robots have higher repeatability. Taking the ABB IRB4400L-10 / 2.53 industrial robot as an example, its absolute positioning accuracy is 0.16mm, while its repeatability is 0.05mm. Therefore, utilizing the repeatability of industrial robots for workpiece positioning can better ensure the accuracy of cameras and lasers.

[0004] Currently, there are few methods for simultaneously calibrating a camera, laser, and industrial robot as a processing unit. Chinese patent CN201910020824.8, "A Laser Marking Machine Based on 3D Scanning," proposes a laser marking machine based on 3D scanning. It combines a 3D scanner and a 3D laser marking machine by adding a 3D camera to the machine. However, this patent only addresses the camera and laser, does not integrate a robot module, and does not specifically describe the calibration method between the 3D camera and laser. Chinese patent CN201910604376.6, "Visual Matching Method, System, and Medium for Laser Processing," discloses a visual matching method, system, and medium for laser processing. By installing a calibration plate at the end of a flange, a closed kinematic chain equation is established between the camera coordinate system, flange coordinate system, calibration plate coordinate system, and robot base coordinate system. The homogeneous transformation matrix between the calibration plate coordinate system and the flange coordinate system is obtained, thereby achieving the calibration purpose between the camera and the robot. This patent only calibrates the camera and industrial robot, and has not yet integrated a three-dimensional laser module. Furthermore, the method mainly relies on the absolute positioning accuracy of the industrial robot to establish the closed kinematic chain equations between the four coordinate systems, so the calibration results cannot meet the high-precision requirements.

[0005] In response to the above situation, this invention proposes a joint calibration method and system for three-dimensional cameras and three-dimensional lasers, which can effectively improve the existing technology and overcome its shortcomings. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for joint calibration of a 3D camera and a 3D laser, thereby solving the aforementioned problems. The specific solution is as follows:

[0007] In a first aspect, the present invention provides a method for joint calibration of a three-dimensional camera and a three-dimensional laser, the method comprising:

[0008] Make the flat plate fixture coincide with the XY plane of the three-dimensional laser coordinate system;

[0009] Calibrate the pose transformation relationship between the 3D camera and the 3D laser;

[0010] Calibrate the transformation relationship between different poses of an industrial robot.

[0011] Preferably, the method for calibrating the pose transformation relationship between the 3D camera and the 3D laser includes:

[0012] The position of the industrial robot when the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system is marked as the first position, and the three-dimensional laser parameters are adjusted to mark the three-dimensional laser coordinate system on the flat plate fixture;

[0013] The industrial robot is moved sequentially to different positions within the imaging range of the 3D camera, and the positions of the industrial robot are marked as the second position and the third position, respectively. The coordinates of the 3D laser coordinate system in the 3D camera coordinate system are then determined by taking pictures of the 3D camera at the second and third positions.

[0014] The transformation relationship between the three-dimensional camera coordinate system and the three-dimensional laser coordinate system at the second position and the third position is calculated sequentially, and the transformation relationship between the three-dimensional camera coordinate system at the second position and the third position and the three-dimensional laser coordinate system at the first position is obtained respectively.

[0015] Preferably, the method involves sequentially calculating the transformation relationship between the three-dimensional camera coordinate system and the three-dimensional laser coordinate system at the second and third positions, respectively, to obtain the transformation relationship between the three-dimensional camera coordinate system at the second and third positions and the three-dimensional laser coordinate system at the first position.

[0016] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is:

[0017] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0018] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0019] z-direction vector:

[0020] Corrected y-direction vector:

[0021] The first transformation matrix T1 or the second transformation matrix T2 are both obtained form.

[0022] Preferably, the method for calibrating the transformation relationship between different poses of the industrial robot includes:

[0023] The third transformation matrix T3 of the industrial robot from the second position to the third position is calculated as follows:

[0024] T3 = T2 * T1 -1

[0025] Preferably, the method for aligning the flat plate fixture with the three-dimensional laser coordinate system XY plane includes:

[0026] The industrial robot is moved under a three-dimensional laser so that the flat fixture on the industrial robot coincides with the XY plane of the three-dimensional laser coordinate system.

[0027] Preferably, the method further includes:

[0028] For the workpiece installed at the end of the industrial robot, point clouds of the workpiece in two different poses are obtained by taking pictures at the second and third positions respectively.

[0029] By stitching together the point clouds of two different poses obtained through the transformation relationship between different poses, a three-dimensional reconstruction model of the workpiece is obtained, and the three-dimensional reconstruction model is visualized on the host computer.

[0030] Preferably, the method of stitching together point clouds obtained from two different poses through the transformation relationship between different poses to obtain a three-dimensional reconstruction model of the workpiece includes:

[0031] The point cloud obtained by the industrial robot at the second position is transformed by the calculated transformation matrix T3 from the second position to the third position, and then stitched with the point cloud obtained by the industrial robot at the third position to obtain a three-dimensional reconstruction model.

[0032] Preferably, the method further includes:

[0033] The three-dimensional trajectory is projected onto the three-dimensional reconstructed model of the workpiece;

[0034] The 3D trajectory is transformed into the 3D laser coordinate system through the pose transformation relationship between the 3D camera and the 3D laser.

[0035] The industrial robot is moved to its initial position in the three-dimensional laser coordinate system for three-dimensional marking.

[0036] Secondly, the present invention provides a joint calibration system for a three-dimensional camera and a three-dimensional laser, the system comprising:

[0037] The coordinate system matching module is used to make the flat plate fixture coincide with the three-dimensional laser coordinate system XY plane;

[0038] The 3D camera-laser calibration module is used to calibrate the pose transformation relationship between the 3D camera and the 3D laser.

[0039] The industrial robot pose transformation calibration module is used to calibrate the transformation relationship between different poses of an industrial robot.

[0040] Thirdly, the present invention provides a joint calibration system for a three-dimensional camera and a three-dimensional laser, the system comprising:

[0041] 3D camera;

[0042] 3D laser;

[0043] Industrial robots;

[0044] The communication bus is used to enable communication between the processor and the memory.

[0045] Memory, used to store computer programs;

[0046] A processor is configured to execute the computer program to perform the following steps:

[0047] Make the flat plate fixture coincide with the XY plane of the three-dimensional laser coordinate system;

[0048] Calibrate the pose transformation relationship between the 3D camera and the 3D laser;

[0049] Calibrate the transformation relationship between different poses of an industrial robot.

[0050] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0051] Beneficial effects: The 3D camera and 3D laser joint calibration method and system of the present invention, by calibrating the pose transformation relationship between the 3D camera and the 3D laser, allows an industrial robot to hold the workpiece, take pictures under the 3D camera, and perform real-time 3D reconstruction. The point cloud coordinate data of the workpiece surface is converted into a 3D laser marking trajectory and sent to the 3D laser for marking operation. This reduces the application steps, adapts to different workpiece shapes, and truly realizes the automation and intelligence of workpiece marking. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and the embodiments in the accompanying drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the overall structure of the three-dimensional camera and three-dimensional laser joint calibration system of the present invention.

[0054] Figure 2 This is a schematic flowchart of an embodiment of the three-dimensional camera and three-dimensional laser joint calibration method of the present invention.

[0055] Figure 3 This is a schematic diagram of the second embodiment of the three-dimensional camera and three-dimensional laser joint calibration method of the present invention.

[0056] Figure 4 This is a schematic diagram of an embodiment of the three-dimensional camera and three-dimensional laser joint calibration system of the present invention.

[0057] Figure 5 This is a schematic diagram of an embodiment of the three-dimensional camera and three-dimensional laser joint calibration system of the present invention.

[0058] In the diagram: 1-3D laser; 2-3D camera; 3-industrial robot; 4-host computer. Detailed Implementation

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments, which are preferred embodiments of the present invention. It should be understood that the described embodiments are merely some embodiments of the present invention, and not all embodiments; it should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The main idea of ​​the technical solution of this invention is to make the flat plate fixture coincide with the XY plane of the three-dimensional laser coordinate system; to calibrate the pose transformation relationship between the three-dimensional camera and the three-dimensional laser; and to calibrate the transformation relationship between different poses of the industrial robot.

[0061] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0062] Example 1

[0063] One embodiment of the present invention provides a joint calibration method for a 3D camera and a 3D laser, applicable to, for example... Figure 1 In the joint calibration system shown, such as Figure 2 As shown, the joint calibration method may specifically include the following steps:

[0064] S101, aligns the flat plate fixture with the XY plane of the three-dimensional laser coordinate system.

[0065] In the embodiments of the invention application, the joint calibration consists of a 3D laser, a 3D camera, an industrial robot, and a host computer, wherein the 3D laser, the 3D camera, and the industrial robot are electrically connected to the host computer in sequence.

[0066] Specifically, the industrial robot is moved under a three-dimensional laser so that the flat fixture on the industrial robot coincides with the XY plane of the three-dimensional laser coordinate system.

[0067] S102, calibrates the pose transformation relationship between the 3D camera and the 3D laser.

[0068] In an embodiment of this invention, the method for calibrating the first position is as follows:

[0069] The position of the industrial robot when the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system is marked as the first position, and the three-dimensional laser parameters are adjusted to mark the three-dimensional laser coordinate system on the flat plate fixture;

[0070] In an embodiment of this invention, the method for calibrating the second position is as follows:

[0071] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the second position, and the coordinates of the 3D laser coordinate system at the second position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0072] The transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0073] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0074] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0075] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0076] z-direction vector:

[0077] Corrected y-direction vector:

[0078] Obtain the first transformation matrix

[0079] In the embodiments of this invention, the method for calibrating the third position is as follows:

[0080] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the third position, and the coordinates of the 3D laser coordinate system at the third position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0081] The transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0082] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0083] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0084] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0085] z-direction vector:

[0086] Corrected y-direction vector:

[0087] Obtain the second transformation matrix

[0088] S103, calibrates the transformation relationship between different poses of an industrial robot.

[0089] In an embodiment of this invention, based on the coordinate projection relationship, the third transformation matrix T3 of the industrial robot from the second position to the third position can be calculated as follows:

[0090] T3 = T2 * T1 -1

[0091] Example 2

[0092] One embodiment of the present invention provides a joint calibration method for a 3D camera and a 3D laser, applicable to, for example... Figure 1 In the joint calibration system shown, such as Figure 3 As shown, the joint calibration method may specifically include the following steps:

[0093] S201, aligns the flat plate fixture with the XY plane of the three-dimensional laser coordinate system.

[0094] In the embodiments of the invention application, the joint calibration consists of a 3D laser, a 3D camera, an industrial robot, and a host computer, wherein the 3D laser, the 3D camera, and the industrial robot are electrically connected to the host computer in sequence.

[0095] Specifically, the industrial robot is moved under a three-dimensional laser so that the flat fixture on the industrial robot coincides with the XY plane of the three-dimensional laser coordinate system.

[0096] S202, calibrating the pose transformation relationship between the 3D camera and the 3D laser.

[0097] In an embodiment of this invention, the method for calibrating the first position is as follows:

[0098] The position of the industrial robot when the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system is marked as the first position, and the three-dimensional laser parameters are adjusted to mark the three-dimensional laser coordinate system on the flat plate fixture;

[0099] In an embodiment of this invention, the method for calibrating the second position is as follows:

[0100] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the second position, and the coordinates of the 3D laser coordinate system at the second position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0101] The transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0102] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0103] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0104] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0105] z-direction vector:

[0106] Corrected y-direction vector:

[0107] Obtain the first transformation matrix

[0108] In the embodiments of this invention, the method for calibrating the third position is as follows:

[0109] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the third position, and the coordinates of the 3D laser coordinate system at the third position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0110] The transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0111] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0112] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0113] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0114] z-direction vector:

[0115] Corrected y-direction vector:

[0116] Obtain the second transformation matrix

[0117] S203, calibrates the transformation relationship between different poses of an industrial robot.

[0118] In an embodiment of this invention, based on the coordinate projection relationship, the third transformation matrix T3 of the industrial robot from the third position to the second position can be calculated as follows:

[0119] T3 = T2 * T1 -1

[0120] S204, stitch together the point clouds obtained in different poses to obtain a three-dimensional reconstruction model, and then visualize the three-dimensional reconstruction model on the host computer.

[0121] Specifically, point clouds with two different poses are obtained by taking pictures of the workpiece installed at the end of the industrial robot at different positions.

[0122] Furthermore, the point clouds of two different poses are stitched together by the transformation relationship between different poses to obtain a three-dimensional reconstruction model of the workpiece, and the three-dimensional reconstruction model is visualized on the host computer.

[0123] Specifically, the point cloud obtained by the industrial robot at the second position is transformed by the calculated transformation matrix T3 from the second position to the third position, and then stitched with the point cloud obtained by the industrial robot at the third position to obtain a three-dimensional reconstruction model, thus realizing the stitching in the embodiment of this invention.

[0124] S205 marks the workpiece held by the end effector of the industrial robot.

[0125] Specifically, the three-dimensional trajectory is projected onto the three-dimensional reconstructed model of the workpiece; the three-dimensional trajectory is converted to the three-dimensional laser coordinate system through the pose transformation relationship between the three-dimensional camera and the three-dimensional laser; the industrial robot is moved to the first recorded position, and the three-dimensional laser is used to perform three-dimensional marking on the workpiece.

[0126] Example 3

[0127] One embodiment of the present invention provides a joint calibration system for a 3D camera and a 3D laser, applicable to, for example... Figure 1 In the joint calibration system shown, such as Figure 4 As shown, the joint calibration system may specifically include the following modules:

[0128] Platform coordinate matching ensures that the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system;

[0129] In the embodiments of the invention application, the joint calibration consists of a 3D laser, a 3D camera, an industrial robot, and a host computer, wherein the 3D laser, the 3D camera, and the industrial robot are electrically connected to the host computer in sequence.

[0130] Specifically, the industrial robot is moved under a three-dimensional laser so that the flat fixture on the industrial robot coincides with the XY plane of the three-dimensional laser coordinate system.

[0131] 3D camera-laser calibration module, calibrating the pose transformation relationship between the 3D camera and the 3D laser;

[0132] In an embodiment of this invention, the method for calibrating the first position is as follows:

[0133] The position of the industrial robot when the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system is marked as the first position, and the three-dimensional laser parameters are adjusted to mark the three-dimensional laser coordinate system on the flat plate fixture;

[0134] In an embodiment of this invention, the method for calibrating the second position is as follows:

[0135] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the second position, and the coordinates of the 3D laser coordinate system at the second position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0136] The transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0137] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0138] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0139] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0140] z-direction vector:

[0141] Corrected y-direction vector:

[0142] Obtain the first transformation matrix

[0143] In the embodiments of this invention, the method for calibrating the third position is as follows:

[0144] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the third position, and the coordinates of the 3D laser coordinate system at the third position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0145] The transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0146] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0147] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0148] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0149] z-direction vector:

[0150] Corrected y-direction vector:

[0151] Obtain the second transformation matrix

[0152] The industrial robot pose transformation calibration module calibrates the transformation relationship between different poses of the industrial robot.

[0153] In an embodiment of this invention, based on the coordinate projection relationship, the third transformation matrix T3 of the industrial robot from the third position to the second position can be calculated as follows:

[0154] T3 = T2 * T1 -1

[0155] Example 4

[0156] One embodiment of the present invention provides a joint calibration system for a 3D camera and a 3D laser, applicable to, for example... Figure 1 In the joint calibration system shown, such as Figure 5 As shown, the joint calibration system may specifically include the following modules:

[0157] The communication bus is used to enable communication between the processor and the memory.

[0158] Memory is used to store computer programs; memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. Optionally, memory may include at least one storage device.

[0159] A processor is configured to execute the computer program described above to perform the following steps:

[0160] Step 1: Align the flat plate fixture with the XY plane of the three-dimensional laser coordinate system;

[0161] In the embodiments of the invention application, the joint calibration consists of a 3D laser, a 3D camera, an industrial robot, and a host computer, wherein the 3D laser, the 3D camera, and the industrial robot are electrically connected to the host computer in sequence.

[0162] Specifically, the industrial robot is moved under a three-dimensional laser so that the flat fixture on the industrial robot coincides with the XY plane of the three-dimensional laser coordinate system.

[0163] Step 2: Calibrate the pose transformation relationship between the 3D camera and the 3D laser;

[0164] In an embodiment of this invention, the method for calibrating the first position is as follows:

[0165] The position of the industrial robot when the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system is marked as the first position, and the three-dimensional laser parameters are adjusted to mark the three-dimensional laser coordinate system on the flat plate fixture;

[0166] In an embodiment of this invention, the method for calibrating the second position is as follows:

[0167] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the second position, and the coordinates of the 3D laser coordinate system at the second position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0168] The transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the second position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0169] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0170] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0171] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0172] z-direction vector:

[0173] Corrected y-direction vector:

[0174] Obtain the first transformation matrix

[0175] In the embodiments of this invention, the method for calibrating the third position is as follows:

[0176] The industrial robot is moved into the imaging range of the 3D camera, the position of the industrial robot is marked as the third position, and the coordinates of the 3D laser coordinate system at the third position are determined by taking pictures with the 3D camera and calibrating the coordinates of the 3D laser coordinate system in the 3D camera coordinate system.

[0177] The transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system is calculated, thus obtaining the transformation relationship between the 3D camera coordinate system at the third position and the 3D laser coordinate system at the first position. The specific method is described in detail below:

[0178] Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), Py Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is as follows:

[0179] x-direction vector: n = (x1-x0, y1-y0, z1-z0);

[0180] y-direction vector: o = (x2-x0, y2-y0, z2-z0);

[0181] z-direction vector:

[0182] Corrected y-direction vector:

[0183] Obtain the second transformation matrix

[0184] Step 3: Calibrate the transformation relationship between different poses of the industrial robot.

[0185] In an embodiment of this invention, based on the coordinate projection relationship, the third transformation matrix T3 of the industrial robot from the third position to the second position can be calculated as follows:

[0186] T3 = T2 * T1 -1

[0187] Step four: stitch together the point clouds obtained from different poses to obtain a three-dimensional reconstruction model, and then visualize the three-dimensional reconstruction model on the host computer.

[0188] Specifically, point clouds with two different poses are obtained by taking pictures of the workpiece installed at the end of the industrial robot at different positions.

[0189] Furthermore, the point clouds of two different poses are stitched together by the transformation relationship between different poses to obtain a three-dimensional reconstruction model of the workpiece, and the three-dimensional reconstruction model is visualized on the host computer.

[0190] Specifically, the point cloud obtained by the industrial robot at the second position is transformed by the calculated transformation matrix T3 from the second position to the third position, and then stitched with the point cloud obtained by the industrial robot at the third position to obtain a three-dimensional reconstruction model, thus realizing the stitching in the embodiment of this invention.

[0191] Step 5: Mark the workpiece held by the end effector of the industrial robot.

[0192] Specifically, the three-dimensional trajectory is projected onto the three-dimensional reconstructed model of the workpiece; the three-dimensional trajectory is transformed into the three-dimensional laser coordinate system through the pose transformation relationship between the three-dimensional camera and the three-dimensional laser; and the industrial robot is moved to the initial position of the three-dimensional laser coordinate system for three-dimensional marking.

[0193] The processor in this embodiment may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0194] Example 5

[0195] One embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described data processing method.

[0196] In the above embodiments, the 3D camera model can be Quanzhou Shensos Sensor Technology Co., Ltd. DS-DA6002, the 3D laser model can be Quanzhou Bingdian Technology Co., Ltd. BD-L270, and the industrial robot model can be Yaskawa GP7.

[0197] In summary, the present invention provides a method and system for joint calibration of a 3D camera and a 3D laser. By calibrating the pose transformation relationship between the 3D camera and the 3D laser, an industrial robot holds a workpiece, takes a picture under the 3D camera, and performs real-time 3D reconstruction. The point cloud coordinate data of the workpiece surface is converted into a 3D laser marking trajectory and sent to the 3D laser for marking operation. The beneficial effects of this invention are: reducing application steps, adapting to different workpiece shapes, and truly realizing the automation and intelligence of workpiece marking.

[0198] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0199] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0200] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0201] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0202] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0203] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0206] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0207] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0210] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional camera and three-dimensional laser combined calibration method, characterized in that, The method includes: Make the flat plate fixture coincide with the XY plane of the three-dimensional laser coordinate system; Calibrate the pose transformation relationship between the 3D camera and the 3D laser; Calibrate the transformation relationships between different poses of an industrial robot; The method for calibrating the pose transformation relationship between the 3D camera and the 3D laser includes: The position of the industrial robot when the flat plate fixture coincides with the XY plane of the three-dimensional laser coordinate system is marked as the first position, and the three-dimensional laser parameters are adjusted to mark the three-dimensional laser coordinate system on the flat plate fixture; The industrial robot is moved sequentially to different positions within the imaging range of the 3D camera, and the positions of the industrial robot are marked as the second position and the third position, respectively. The coordinates of the 3D laser coordinate system in the 3D camera coordinate system are then determined by taking pictures of the 3D camera at the second and third positions. The transformation relationship between the three-dimensional camera coordinate system and the three-dimensional laser coordinate system at the second position and the third position is calculated sequentially, and the transformation relationship between the three-dimensional camera coordinate system at the second position and the third position and the three-dimensional laser coordinate system at the first position is obtained respectively.

2. The method of claim 1, wherein, The method involves sequentially calculating the transformation relationship between the three-dimensional camera coordinate system and the three-dimensional laser coordinate system at the second and third positions, respectively, to obtain the transformation relationship between the three-dimensional camera coordinate system at the second and third positions and the three-dimensional laser coordinate system at the first position. Let the coordinates of the origin of the three-dimensional laser coordinate system in the three-dimensional camera, a point on the positive X-axis, and any point on the plane formed by the X and Y axes on the positive Y-axis be P0(x0,y0,z0), respectively. x (x1,y1,z1), P y Given (x2, y2, z2), the transformation relationship from the three-dimensional camera coordinate system to the three-dimensional laser coordinate system is: x-direction vector: n = (x1-x0, y1-y0, z1-z0); y-direction vector: o = (x2-x0, y2-y0, z2-z0); z-direction vector: ; Corrected y-direction vector: ; The first transformation matrix T1 or the second transformation matrix T2 is obtained in the form form.

3. The method of claim 2, wherein, The method for calibrating the transformation relationship between different poses of an industrial robot includes: The third transformation matrix T3 of the industrial robot from the second position to the third position is calculated as follows: T3 = T2 T1 -1 .

4. The method of claim 1, wherein, The method for aligning the flat plate fixture with the three-dimensional laser coordinate system XY plane includes: The industrial robot is moved under a three-dimensional laser so that the flat fixture on the industrial robot coincides with the XY plane of the three-dimensional laser coordinate system.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: For the workpiece installed at the end of the industrial robot, point clouds of the workpiece in two different poses are obtained by taking pictures at the second and third positions respectively. By stitching together the point clouds of two different poses obtained through the transformation relationship between different poses, a three-dimensional reconstruction model of the workpiece is obtained, and the three-dimensional reconstruction model is visualized on the host computer.

6. The method of claim 5, wherein, The method involves stitching together point clouds obtained from two different poses based on the transformation relationship between different poses to obtain a three-dimensional reconstructed model of the workpiece. The method includes: The point cloud obtained by the industrial robot at the second position is transformed by the calculated transformation matrix T3 from the second position to the third position, and then stitched with the point cloud obtained by the industrial robot at the third position to obtain a three-dimensional reconstruction model.

7. The method of claim 6, wherein, The method further includes: The three-dimensional trajectory is projected onto the three-dimensional reconstructed model of the workpiece; The 3D trajectory is transformed into the 3D laser coordinate system through the pose transformation relationship between the 3D camera and the 3D laser. The industrial robot is moved to its initial position in the three-dimensional laser coordinate system to perform three-dimensional marking on the workpiece.

8. A combined three-dimensional camera and three-dimensional laser calibration system, comprising: The system employs the joint calibration method of a 3D camera and a 3D laser as described in any one of claims 1-7, and comprises: The coordinate system matching module is used to make the flat plate fixture coincide with the three-dimensional laser coordinate system XY plane; The 3D camera-laser calibration module is used to calibrate the pose transformation relationship between the 3D camera and the 3D laser. The industrial robot pose transformation calibration module is used to calibrate the transformation relationship between different poses of an industrial robot.

9. A combined three-dimensional camera and three-dimensional laser calibration system, characterized by, The system employs the joint calibration method of a 3D camera and a 3D laser as described in any one of claims 1-7, and comprises: 3D camera; 3D laser; Industrial robots; The communication bus is used to enable communication between the processor and 3D cameras, 3D lasers, and industrial robots. Memory, used to store computer programs; A processor is configured to execute the computer program to perform the following steps: Make the flat plate fixture coincide with the XY plane of the three-dimensional laser coordinate system; Calibrate the pose transformation relationship between the 3D camera and the 3D laser; Calibrate the transformation relationship between different poses of an industrial robot.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images