Calibration method and apparatus for robots

Abstract

A kind of calibration method for robot, comprising the following steps: capturing the image of environment and forming the three-dimensional virtual object of environment (301);The camera calibration process is executed, and the camera calibration process comprises the following steps: based on the image of the captured environment and the three-dimensional virtual object of environment, calibration object (403) is placed in target area in environment (302);Image of calibration object (403) is captured using first camera (102) by the relative movement between first camera (102) and calibration object (403) associated with robot (101) (303), wherein first camera (102) is 2D camera;Based on the image of calibration object (403) captured by first camera (102), determine the parameters of first camera (102) for calibration (304).The calibration method can guide user (107) to carry out camera calibration by AR technology, so that operation is intuitive and convenient, advantageously reduces the operation complexity and improves accuracy. Also relates to a kind of calibration device for robot.

Description

Technical Field

[0001] This disclosure relates to the field of machine vision, and more specifically, to calibration methods, apparatus, computing devices, computer-readable storage media, and program products for robots. Background Technology

[0002] With the development of robotics technology, an increasing number of robotic workstations are being used in industrial applications, such as automated loading and unloading, welding, stamping, painting, and various other processes. Robots can be flexibly combined with different equipment to meet the demands of demanding production processes. They can easily enable multi-machine automated production lines and digital factory layouts, maximizing manpower savings and improving enterprise efficiency.

[0003] Despite the numerous advantages of robots in industrial applications, their use is limited by the high technical and professional requirements for integration, operation, and maintenance. In particular, calibration, including vision system calibration (camera calibration), TCP (tool center point) calibration, and hand-eye calibration, has become a major challenge for users of robots. Summary of the Invention

[0004] Robot calibration is one of the key technologies in robot operation. However, existing calibration methods require users to possess considerable professional knowledge and spend a significant amount of time and effort to calibrate the robot and determine whether its operation is ideal, which increases the operational threshold for users. For example, existing calibration methods require a large amount of manual operation, resulting in low calibration efficiency, complex procedures, and a high dependence on the operator's subjective judgment and experience.

[0005] A first embodiment of this disclosure provides a calibration method for a robot, the calibration method including performing a camera calibration process, wherein performing the camera calibration process includes the following steps: capturing an image of an environment and forming a three-dimensional virtual object of the environment; placing a calibration object in a target area of ​​the environment based on the captured image of the environment and the three-dimensional virtual object of the environment; capturing an image of the calibration object using a first camera by means of relative movement between a first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera; and determining parameters of the first camera for calibration based on the image of the calibration object captured by the first camera.

[0006] In this embodiment, augmented reality (AR) technology can be used to guide users in camera calibration, making the operation intuitive and convenient. This reduces operational complexity and improves accuracy, thereby effectively lowering the operational threshold for users and enabling users without professional knowledge to easily perform camera calibration.

[0007] A second embodiment of this disclosure provides a calibration device for a robot, the calibration device including a camera calibration unit, the camera calibration unit comprising: an environment capture module configured to capture images of an environment and form a three-dimensional virtual object of the environment; a placement module configured to place a calibration object in a target area of ​​the environment based on the captured images of the environment and the three-dimensional virtual object of the environment; a first camera capture module configured to capture images of the calibration object using a first camera through relative movement between a first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera; and a parameter determination module configured to determine parameters of the first camera for calibration based on the images of the calibration object captured by the first camera.

[0008] A third embodiment of this disclosure provides a computing device, the computing device including: a processor; and a memory for storing computer-executable instructions that, when executed, cause the processor to perform the method described in the first embodiment.

[0009] A fourth embodiment of this disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon for performing the method described in the first embodiment.

[0010] A fifth embodiment of this disclosure provides a computer program product tangibly stored on a computer-readable storage medium and including computer-executable instructions that, when executed, cause at least one processor to perform the method described in the first embodiment. Attached Figure Description

[0011] Features, advantages, and other aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description, in which several embodiments of the disclosure are illustrated by way of example and not limitation, in the drawings:

[0012] Figure 1 Exemplary scenarios in which embodiments of this disclosure can be applied are shown.

[0013] Figure 2 Another exemplary scenario in which embodiments of this disclosure can be applied is shown.

[0014] Figure 3 A flowchart of a calibration method for a robot according to an embodiment of the present disclosure is shown.

[0015] Figure 4An exemplary arrangement for a camera calibration process for a robot, according to embodiments of the present disclosure, is shown.

[0016] Figure 5 Another flowchart of a calibration method for a robot according to an embodiment of the present disclosure is shown.

[0017] Figure 6 An exemplary calibration artifact for TCP calibration according to an embodiment of the present disclosure is shown.

[0018] Figure 7 An exemplary arrangement for a TCP calibration process for a robot, according to embodiments of the present disclosure, is shown.

[0019] Figure 8 Another flowchart of a calibration method for a robot according to an embodiment of the present disclosure is shown.

[0020] Figure 9 An exemplary arrangement for a robot's hand-eye calibration process, according to embodiments of the present disclosure, is shown.

[0021] Figure 10 A block diagram of an exemplary calibration apparatus for a robot according to an embodiment of the present disclosure is shown.

[0022] Figure 11 A block diagram of an exemplary computing device for robot calibration according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] Various exemplary embodiments of this disclosure are described in detail below with reference to the accompanying drawings. While the exemplary methods and apparatuses described below include software and / or firmware executed on hardware among other components, it should be noted that these examples are merely illustrative and should not be considered limiting. For example, it is conceivable that any or all hardware, software, and firmware components may be implemented exclusively in hardware, exclusively in software, or in any combination of hardware and software. Therefore, although exemplary methods and apparatuses have been described below, those skilled in the art will readily understand that the examples provided are not intended to limit the ways in which these methods and apparatuses may be implemented.

[0024] Furthermore, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the methods and systems according to various embodiments of this disclosure. It should be noted that the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. In different drawings, the same reference numerals denote the same or similar elements.

[0025] The terms “comprising,” “including,” and similar terms as used herein are open-ended, meaning “including / including but not limited to,” implying that other content may also be included. The term “based on” means “at least partially based on.” The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment,” and so on.

[0026] This article discusses various coordinate systems used in robot systems, such as the robot's base coordinate system, tool coordinate system, and camera coordinate system. The robot's base coordinate system, based on the robot's mounting base, describes the robot's body motion. The tool coordinate system (TCS) is established with the tool center point (TCP) as its origin. Before tool assembly, the TCP is the default center point of the robot's end effector flange; after tool assembly, the robot's TCP moves to the tool end effector. The camera coordinate system, based on the camera, describes the object's motion.

[0027] Figure 1An exemplary scenario 100 in which embodiments of the present disclosure can be applied is shown. Scenario 100 includes a robot 101 and an associated first camera 102, which is fixed to the robot 101. For example, the robot 101 may be a multi-jointed manipulator or a multi-degree-of-freedom machine device for industrial applications. The robot 101 includes a movable end effector 103, to which the first camera 102 may be fixed and move together with the end effector 103. That is, in this scenario 100, since the end effector of the robot 101 and the first camera 102 are fixed together, it is referred to as "eye in hand". The end effector 103 of the robot 101 may also have a tooling part 104 for assembling (e.g., adsorption, insertion, etc.) tools or components (e.g., welding torch, nozzle, bolt, etc.) for processing a target workpiece. Scenario 100 also includes a target object 105. The target object 105 may be an actual object to be processed (e.g., a target workpiece) or a calibration object (e.g., a calibration plate, etc.). The first camera 102 is a 2D camera, and a second camera 106 is also arranged in scene 100. The second camera 106 is a 3D camera, which can be a binocular camera, a structured light camera, or any camera capable of returning depth information. In scene 100, the second camera 106 can be arranged relatively above the robot 101 to capture and track the movement of the robot 101 (e.g., its end effector, etc.). In scene 100, the user 107 can carry (e.g., handheld, head-mounted, etc.) an AR device 108, which can include, but is not limited to, a smartphone, tablet, teach pendant, or head-mounted device. The AR device 108 can capture images of the real environment or objects in the environment from different positions and angles within its field of view 109 using multiple cameras disposed on or outside it, to form three-dimensional virtual objects. The AR device 108 can have a display to display the corresponding three-dimensional virtual objects of the real environment or objects in the environment in the virtual environment.

[0028] Figure 2 An exemplary scenario 200 in which embodiments of the present disclosure can be applied is shown. Scenario 200 is similar to scenario 100, except that the associated first camera 102 is fixed to the outside of robot 101. That is, in this scenario 200, the first camera 102 is located outside robot 101 and does not move with the end effector 103 of robot 101, referred to as eye to hand.

[0029] Figure 3 A flowchart of a calibration method 300 for a robot according to an embodiment of the present disclosure is shown. Figure 4An exemplary arrangement 400 for a camera calibration process for a robot, according to embodiments of the present disclosure, is shown. Method 300 can be applied to, for example... Figure 1 The exemplary scenario 100 shown (eye in hand) and as shown Figure 2 The exemplary scenario 200 shown is (eye outside hand). The following is combined with... Figure 3 and Figure 4 Let's describe method 300.

[0030] refer to Figure 3 Method 300 includes steps 301-304 for performing a camera calibration process. Additionally, method 300 may also include steps for performing a TCP calibration process (see...). Figure 5 ) and used to perform the hand-eye calibration process (see Figure 8 The steps are as follows.

[0031] In image measurement and machine vision applications, to determine the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters (e.g., intrinsic and extrinsic parameters, distortion parameters). These parameters are typically obtained through experimentation and calculation; this process of solving for these parameters is called camera calibration. By determining the camera parameters through calibration, lens distortion can be corrected, a corrected image can be generated, and a 3D scene can be reconstructed from the obtained image.

[0032] Method 300 begins with step 301, capturing an image of the environment and forming a three-dimensional virtual object of the environment.

[0033] In some embodiments, step 301 may include: taking images of the environment from different positions and at different angles to obtain at least two images for each scene; measuring the depth information of each pixel in the images using a triangulation method based on the stereo images; and forming a three-dimensional virtual object of the environment based on the depth information and the two-dimensional information contained in the images. For example, the AR device held by user 107 may be equipped with or connected to multiple cameras, which take images of the real environment from different positions and at different angles. Taking two cameras as an example, when two cameras take photos of the same scene from different positions in the real environment, due to the different shooting angles of the two cameras, the depth of each pixel in the two-dimensional image can be triangulated based on the different positions of pixels in the two images of the same scene. Thus, although the image captured by a single camera is two-dimensional, by supplementing it with the depth information obtained through triangulation, the three-dimensional information required to form a three-dimensional virtual object of the real environment can be obtained.

[0034] Next, method 300 proceeds to step 302. In step 302, based on the captured image of the environment and the three-dimensional virtual objects of the environment, a calibrated object is placed in a target area of ​​the environment. For example, the target area could be a specific area in a workbench.

[0035] In some embodiments, step 302 may include: detecting visual markers arranged in the environment from an image of the captured environment; determining a target area based on the detected visual markers; capturing an image of a calibrated object and forming a three-dimensional virtual object of the calibrated object; overlaying the three-dimensional virtual object of the calibrated object onto a three-dimensional virtual object of the environment for display as an augmented reality image; and moving the calibrated object such that the three-dimensional virtual object of the calibrated object is located within a three-dimensional virtual object of the target area. For example, the target area in the environment may be defined by multiple identifiable visual markers, and the target area can be quickly determined by detecting the visual markers. The multiple visual markers may be arranged to cover the field of view of a first camera so that they can be captured by the first camera. Similarly, an AR device 108 held by user 107 may capture images of the calibrated object through multiple cameras and form a three-dimensional virtual object of the calibrated object. After the three-dimensional virtual object of the calibrated object is overlaid onto a three-dimensional virtual object of the environment for display as an augmented reality image (e.g., real-time tracking display), the user 107 or the robot 101 may be guided to move the calibrated object by, for example, voice, text, or images. For example, when the 3D virtual object of the calibrated object is at least partially located outside the 3D virtual object of the target area, the AR device 108 can prompt the user 107 or guide the robot 101 to continue moving the calibrated object until the 3D virtual object of the calibrated object is located within the 3D virtual object of the target area.

[0036] refer to Figure 4 The exemplary arrangement 400 includes a plurality of visual markers 402 (e.g., four illustrated) defining a target area 401. The visual markers 402 may have characteristic colors (e.g., colors different from other parts of the environment) and / or characteristic patterns (e.g., specific graphic codes (barcodes, QR codes, etc.)). For example, the AR device 108 can capture an image of the environment using a camera and identify the visual markers 402 from the image to determine the target area 401 defined by the visual markers 402. The exemplary arrangement 400 also includes a calibration object 403 (e.g., a calibration board) placed in the target area 401. The calibration object 403 may have a reference pattern attached to its surface for camera calibration; this reference pattern includes, but is not limited to, checkerboard patterns, circular array patterns, non-circular array patterns, etc.

[0037] Next, method 300 proceeds to step 303. In step 303, an image of the calibration object is captured using the first camera, which is a 2D camera, by means of relative movement between the first camera associated with the robot and the calibration object.

[0038] In some embodiments, when the first camera is fixed to the outside of the robot, step 303 may include: obtaining a plurality of actual positions for placing a calibration object in the target area, the plurality of actual positions corresponding to a plurality of virtual positions in a three-dimensional virtual object of the environment; moving the calibration object so that the three-dimensional virtual object of the calibration object is respectively located at the plurality of virtual positions; and using the first camera to capture images of the calibration object at the plurality of actual positions respectively. For example, when the first camera 102 is fixed to the outside of the robot 101 (i.e., the eye is outside the hand), it is necessary to place the calibration object 403 at a suitable position in the target area 401 so that the corner points on the calibration object 403 cover the field of view of the first camera 102, and the correlation between camera parameters and different poses of the calibration object 403 will affect the determination of specific camera parameters. In one example, multiple actual positions for placing the calibration object 403 in the target area 401 can be calculated based on factors such as whether the corners of the calibration object 403 are covered by the field of view of the first camera 102, the correlation between camera parameters, and different poses. The AR device 108 then instructs the user 107 or robot 101 to place the calibration object 403 at these calculated actual positions. Similar to step 302, the calibration object 403 can be placed at multiple actual positions by determining whether the 3D virtual object of the calibration object 403 is placed at multiple virtual positions corresponding to these actual positions. The first camera 102 then captures images of the calibration object 403 at each of these actual positions.

[0039] In some embodiments, when the first camera is fixed to the robot, step 303 can be performed by moving the first camera around the calibration object to multiple actual positions, and using the first camera to capture images of the calibration object at the multiple actual positions respectively.

[0040] Next, method 300 proceeds to step 304. In step 304, parameters of the first camera used for calibration are determined based on the image of the calibration object captured by the first camera. As previously mentioned, the parameters of the first camera may include at least one of intrinsic parameters, extrinsic parameters, and distortion parameters. In one example, the classic Zhang Zhengyou calibration method (see “A Flexible New Technique for Camera Calibration”, IEEE TRANSACTIONS ANALYSIS AND MACHINE INTELLIGENCE, Volume 22, Issue 11, 2000) can be used to determine the parameters of the first camera. The number of checkerboard grid points used is pxq. Each time the calibration board is placed, the corresponding camera parameters will change. Feature points (e.g., corner points of the checkerboard pattern) in the captured image of the calibration board are detected to form pxq equations. After placing the calibration plate n times, n x px q equations are formed. The intrinsic and all extrinsic parameters are estimated under ideal, distortion-free conditions. Then, the least squares method is applied to estimate the distortion coefficients under actual radial distortion. Finally, the maximum likelihood method and optimization estimation are used to improve the estimation accuracy. In other examples, the parameters of the first camera can be determined based on any applicable existing camera calibration method, which will not be detailed further.

[0041] Compared with the traditional camera calibration process, the camera calibration process according to Method 300 can provide users with guidance based on AR technology during the camera calibration, making the operation intuitive and convenient, effectively reducing the complexity of operation and improving accuracy, thereby effectively lowering the operating threshold for users, so that users without professional knowledge can easily perform camera calibration.

[0042] Figure 5 A flowchart of a calibration method 500 for a robot according to an embodiment of the present disclosure is shown. Figure 6 An exemplary calibration artifact 600 for TCP calibration according to an embodiment of the present disclosure is shown. Figure 7 An exemplary arrangement 700 for a TCP calibration process for a robot, according to embodiments of the present disclosure, is shown. Method 500 can be applied to, for example... Figure 1 The exemplary scenario 100 shown (eye in hand) and as shown Figure 2 The exemplary scenario 200 shown is (eye outside hand). The following is combined with... Figure 5 , Figure 6 and Figure 7 Let's describe method 500.

[0043] refer to Figure 5Method 500 includes steps 501-504 for performing the TCP calibration procedure. Method 500 may be included in or performed independently of method 300.

[0044] As previously described, the tool section 104 on the end effector 103 of robot 101 can be equipped with a tool to process the target workpiece. To describe the tool's pose in space, a coordinate system, namely the tool coordinate system TCS, is bound (defined) to the tool, with the origin of the tool coordinate system TCS being TCP. Before the tool is assembled, the default TCP is located on the end effector 103 of robot 101, for example, at the center point of the flange of the tool section 104. For example, the coordinates of the robot end effector flange center point can be obtained from the robot controller. Alternatively, the coordinates of the robot end effector flange center point can be obtained from a teach pendant, which is a handheld device used for manual robot operation, programming, parameter configuration, and monitoring. After the tool is assembled, the robot TCP (e.g., located on the end effector) needs to be calibrated, and the origin of the tool coordinate system TCS is set from the default TCP to the robot TCP.

[0045] Method 500 begins at step 501, using a second camera to capture multiple images of a first workpiece positioned in multiple poses, wherein the multiple poses have different spatial tilt angles, the first workpiece is mounted on the end effector of a robot, and the second camera is a 3D camera and is fixed externally to the robot. For example, the first workpiece could be, for example, Figure 6 and Figure 7 The calibration workpiece 600 shown is used to simulate the tools in actual use. Figure 6 The diagram shows a side view 601 and a top view 602 of a calibration workpiece 600, which has a top end 611 and an end end 612. The calibration workpiece 600 can be mounted to the tool section 104 of the end end 103 of the robot 101 via the top end 612. (Reference) Figure 7 An exemplary arrangement 700 includes a second camera 106 and multiple different poses (e.g., Figure 7 The calibration workpiece 600 is placed in four poses (701, 702, 703, 704), each pose having a different spatial tilt angle.

[0046] Next, method 500 proceeds to step 502. In step 502, based on the captured multiple images, multiple workpiece end coordinates in the 3D camera coordinate system of the second camera are determined for the end of the first workpiece in multiple poses. As previously mentioned, before tooling assembly, the default TCP is the center point of the robot end flange, while after tooling / workpiece assembly, the robot TCP (e.g., located on the tool / workpiece end) needs to be calibrated. In this step, the position of the workpiece end in the 3D camera coordinate system is determined with the 3D camera as a reference. In some embodiments, the number of poses can be at least four.

[0047] In some embodiments, step 502 may include: detecting marker points arranged on a first workpiece from multiple captured images to obtain multiple marker point coordinates in a 3D camera coordinate system; and determining multiple workpiece end coordinates in the 3D camera coordinate system based on the multiple marker point coordinates. Figure 6 As shown, the calibration workpiece 600 may have one or more marking points 613 (e.g., Figure 6 (There are four markers in total). Marker points 613 can be used, for example, to ensure that the calibrated workpiece 600 or at least a portion thereof (e.g., including marker points 613 and end cap 612) is within the field of view of the second camera 106. After identifying marker points 613 in the captured image, multiple marker point coordinates in the 3D camera coordinate system can be obtained at different poses. By establishing a rigid relationship between marker points 613 and end cap 612 on the calibrated workpiece 600 (e.g., the marker points are rigidly connected to the robot TCP, so the distance between them is determined), multiple corresponding workpiece end cap coordinates of end cap 612 in the 3D camera coordinate system can be obtained.

[0048] In some embodiments, the marker point may have a characteristic color and / or a characteristic pattern. For example, marker point 613 may have a characteristic color (e.g., a color different from other parts of the calibrated workpiece 600) and / or a characteristic pattern (e.g., a specific graphic code (barcode, QR code, etc.)) for identification / detection.

[0049] Next, method 500 proceeds to step 503. In step 503, multiple end-effector coordinates in the robot's base coordinate system are obtained for the robot in multiple poses. For example, the multiple end-effector coordinates in the robot's base coordinate system for different poses can be obtained from the robot controller or teach pendant.

[0050] Next, method 500 proceeds to step 504. In step 504, based on multiple workpiece end-effector coordinates and multiple robot end-effector coordinates, the coordinates of the robot TCP to be calibrated in the robot's base coordinate system under a certain robot posture are determined, and the relative relationship between the robot TCP and the default TCP located on the robot's end-effector is determined. The following section combines... Figure 7 This describes an exemplary TCP calibration process.

[0051] refer to Figure 7The robot 101 assumes four different poses. The second camera 106 identifies the marker point 613 on the calibration workpiece 600, obtains the coordinates of marker point 613 in the 3D camera coordinate system, and derives the coordinates of the end face 612 of the calibration workpiece 600 in the 3D camera coordinate system. Simultaneously, the coordinates of the center point of the end flange of the robot 101 are obtained from, for example, the robot controller. The following coordinates can be obtained:

[0052] In the first pose 701: the robot end effector has coordinates (x1, y1, z1) in the robot base coordinate system, and the workpiece end effector has coordinates (mx1, my1, mz1) in the 3D camera coordinate system.

[0053] In the second pose 702: the robot end effector has coordinates (x2, y2, z2) in the robot base coordinate system, and the workpiece end effector has coordinates (mx2, my2, mz2) in the 3D camera coordinate system.

[0054] In the third pose 703: the robot end effector has coordinates (x3, y3, z3) in the robot base coordinate system, and the workpiece end effector has coordinates (mx3, my3, mz3) in the 3D camera coordinate system.

[0055] In pose 704: the robot end effector is located at coordinates (x4, y4, z4) in the robot base coordinate system, and the workpiece end effector is located at coordinates (mx3, my3, mz3) in the 3D camera coordinate system.

[0056] A mathematical translation transformation is performed on the above coordinates to ensure that the coordinates of the workpiece end in the 3D camera coordinate system are the same. The following coordinates can be obtained:

[0057] In the first pose 701: the robot end effector has coordinates (x1, y1, z1) in the robot base coordinate system, and the workpiece end effector has coordinates (mx1, my1, mz1) in the 3D camera coordinate system.

[0058] In the second pose 702: the robot end effector has coordinates (x2-mx2+mx1, y2-my2+my1, z2-mz2+mz1) in the robot base coordinate system, and the workpiece end effector has coordinates (mx1, my1, mz1) in the 3D camera coordinate system.

[0059] In the third pose 703: the robot end effector has coordinates (x3-mx3+mx1, y3-my3+my1, z3-mz3+mz1) in the robot base coordinate system, and the workpiece end effector has coordinates (mx1, my1, mz1) in the 3D camera coordinate system.

[0060] In pose 704: the robot end effector has coordinates (x4-mx4+mx1, y4-my4+my1, z4-mz4+mz1) in the robot base coordinate system, and the workpiece end effector has coordinates (mx1, my1, mz1) in the 3D camera coordinate system.

[0061] After translation transformation, the robot TCP coordinates (x0, y0, z0) in the robot base coordinate system are located at the center of the sphere. The default TCP point (the coordinates of the end flange center point are obtained from the teach pendant) is located on the sphere. The robot TCP coordinates (x0, y0, z0) can be solved according to the following equations (1)-(4):

[0062] (x1-x0) 2 +(y1-y0) 2 +(z1-z0) 2 =R 2 (1)

[0063] (x2-mx2+mx1-x0) 2 +(y2-my2+my1-y0) 2 (2)

[0064] +(z2-mz2+mz1-z0) 2 =R2

[0065] (x3-mx3+mx1-x0) 2 +(y3-my3+my1,-y0) 2 (3)

[0066] +(z3-mz3+mz1-z0) 2 =R 2

[0067] (x4-mx4+mx1-x0) 2 +(y4-my4+my1-y0) 2 (4)

[0068] +(z4-mz4+mz1-z0) 2 =R 2

[0069] Based on the solved values ​​(x0, y0, z0) and the robot state (x1, y1, z1, rx1, ry1, rz1) at the first robot pose, the TCP value (x) can be obtained. tcp ,y tcp ,z tcp ,rx tcp ,ry tcp ,rz tcpThis allows us to determine the relative relationship between the robot's TCP and the default TCP located at the end of the robot.

[0070] Traditional TCP calibration processes typically use a 3-, 4-, or 5-point method, requiring the user to move the TCP to a reference point (e.g., a fixed point placed within the robot's workspace) 3, 4, or 5 times using different postures until the TCP coincides with the reference point. However, such traditional TCP calibration methods require manual intervention from the user, demanding familiarity with the operation and specialized knowledge, and suffer from drawbacks such as slow calibration speed and insufficient calibration accuracy. Compared to the traditional TCP calibration process, the TCP calibration process according to Method 500 can automatically perform TCP calibration without manual intervention, effectively reducing operational complexity, enabling rapid TCP calibration, avoiding errors caused by manual intervention, and improving calibration accuracy.

[0071] Figure 8 Another flowchart of a calibration method 800 for a robot according to an embodiment of the present disclosure is shown. Figure 9 An exemplary arrangement 900 for a robot's hand-eye calibration process, according to embodiments of the present disclosure, is shown. Method 800 can be applied to, for example... Figure 1 The exemplary scenario 100 shown (eye in hand) and as shown Figure 2 The exemplary scenario 200 shown is (eye outside hand). The following is combined with... Figure 8 and Figure 9 Let's describe method 800.

[0072] refer to Figure 8 Method 800 includes steps 801-805 for performing the hand-eye calibration process. Method 800 may be included in or performed independently of method 300.

[0073] The purpose of hand-eye calibration is to obtain the relationship between the robot coordinate system and the camera coordinate system, and finally transfer the visual recognition results to the robot coordinate system. As mentioned earlier, depending on where the camera is fixed, hand-eye calibration can be divided into two forms: if the camera and the robot end effector are fixed together, it is called "eye in hand"; if the camera is fixed on a base outside the robot, it is called "eye outside hand".

[0074] Method 800 begins at step 801, using a first camera to capture a first target image of the second workpiece to be processed by the robot, and using a second camera to capture a second target image of the second workpiece. The second workpiece may be, for example, such as... Figure 9 The target workpiece 901 shown is to be processed by robot 101. A first target image of the target workpiece 901 can be captured by a first camera 102, and a second target image of the target workpiece 901 can be captured by a second camera 106.

[0075] Then, method 800 proceeds to step 802. In step 802, based on the first target image and the second target image, a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera is determined. In this step, the transformation relationship between different camera coordinate systems can be determined, that is, the mapping relationship between the coordinates of a point in the first target image and the coordinates of the corresponding point in the second target image can be determined.

[0076] In some embodiments, step 802 may include: detecting a plurality of feature points arranged in the second workpiece from the first target image and the second target image to obtain a first plurality of feature coordinates of the plurality of feature points in a 2D camera coordinate system and a second plurality of feature coordinates in a 3D camera coordinate system. In some embodiments, the number of feature points may be at least four. For example, the coordinates of the first camera 102 in the 2D camera coordinate system may be calculated using two-dimensional information in the target image based on camera parameters obtained after camera calibration of the first camera 102 (the camera calibration may be, for example, the camera calibration process described herein or any other applicable camera calibration process).

[0077] refer to Figure 9 Multiple feature points 902 (e.g., four in the figure) can be arranged on the target workpiece 901. The feature points 902 can have specific shapes, sizes, markings, etc., to facilitate feature point identification / detection from the captured image. For example, the target workpiece 901 can be a PCB board, and the feature points 902 can be screw holes marked with CAD on the PCB board. The first camera 102 and the second camera 106 can respectively identify the feature points 902 from the captured target image. The coordinates of the multiple feature points 902 in the 2D camera coordinate system of the first camera 102 and the 3D camera coordinate system of the second camera 106 can be obtained as follows:

[0078] The first feature point is 902, with coordinates (x1, y1, z1) in the 3D camera coordinate system and (X1, Y1, Z1) in the 2D camera coordinate system.

[0079] The second feature point 902 has coordinates (x2, y2, z2) in the 3D camera coordinate system and (X2, Y2, Z2) in the 2D camera coordinate system.

[0080] The third feature point is 903, with coordinates (x3, y3, z3) in the 3D camera coordinate system and (X3, Y3, Z3) in the 2D camera coordinate system.

[0081] The fourth feature point 902 has coordinates (x4, y4, z4) in the 3D camera coordinate system and (X4, Y4, Z4) in the 2D camera coordinate system.

[0082] The coordinate transformation relationship between the 2D camera and the 3D camera coordinate systems satisfies the following equation (5):

[0083]

[0084] According to P 3D =H1·P 2D , where P 3D and P 2D By determining the coordinates of the same point in different camera coordinate systems, the homography matrix H1 between the two camera coordinate systems can be obtained. In other words, the first transformation relationship between the 2D camera coordinate system of the first camera 102 and the 3D camera coordinate system of the second camera 106 can be determined.

[0085] Method 800 then proceeds to step 803. In step 803, the robot's end effector is moved to multiple locations to determine a first plurality of tool coordinates of the robot's TCP in the robot's base coordinate system and a second plurality of tool coordinates of the robot's TCP in the 3D camera coordinate system. In this step, the first plurality of tool coordinates of the robot's TCP in the robot's base coordinate system and the second plurality of tool coordinates in the 3D camera coordinate system can be determined, for example, by capturing an image of the robot's end effector using a second camera (e.g., using the TCP calibration procedure described herein or any other applicable TCP calibration procedure). In some embodiments, the number of locations can be at least four.

[0086] For example, by moving the robot's end effector 103 (e.g., translating or posing in different positions) to multiple positions (e.g., four), the robot's TCP coordinates in the base coordinate system of robot 101 and the camera coordinate system of second camera 106 are determined respectively.

[0087] In some embodiments, step 803 may include: translating the robot's end effector to at least one position in one of the plurality of positions, and determining at least one tool coordinate of the robot TCP in the robot's base coordinate system based on the relative relationship between the robot TCP and the default TCP. For example, for ease of calculation, the robot's end effector can be translated to at least one position in a certain robot posture based on the relative relationship between the robot TCP and the default TCP obtained from the robot controller, and the coordinates of the robot TCP in the robot's base coordinate system can be determined according to the coordinates of the default TCP in the robot's base coordinate system obtained from the robot controller.

[0088] Then method 800 proceeds to step 804. In step 804, a second transformation relationship between the robot's base coordinate system and the 3D camera coordinate system is determined using a first plurality of tool coordinates and a second plurality of tool coordinates.

[0089] The coordinate transformation relationship between the base coordinate system of robot 101 and the 3D camera coordinate system satisfies the following equation (6):

[0090] P 3D =H2·P robot (6)

[0091] P 3D and P robot Let H2 be the coordinates of the same point in the 3D camera coordinate system and the robot base coordinate system, respectively. From this, the homography matrix H2 between the 3D camera coordinate system and the robot base coordinate system can be obtained, that is, the second transformation relationship between the robot base coordinate system and the 3D camera coordinate system can be determined.

[0092] Then method 800 proceeds to step 805. In step 805, based on the first and second transformation relationships, a third transformation relationship between the robot's base coordinate system and the 2D camera coordinate system is determined.

[0093] Based on equations (5) and (6), the coordinate transformation relationship between the robot base coordinate system and the 3D camera coordinate system satisfies the following equation (7):

[0094] P robot =H2 -1 ·H1·P 2D (7)

[0095] P robot and P 2D Let H2 be the coordinates of the same point in the robot's base coordinate system and the 2D camera's coordinate system, respectively. From this, the homography matrix H2 between the robot's base coordinate system and the 2D camera's coordinate system can be obtained. -1 H1, that is, determining the third transformation relationship (hand-eye relationship) between the robot's base coordinate system and the 2D camera coordinate system.

[0096] In some embodiments, the at least one location is at or near the at least one feature point. For example, to reduce hand-eye calibration errors, the at least one location may be at or near an identifiable feature point. In some embodiments, AR techniques as previously described herein may be used to intuitively and conveniently guide the robot's end effector to at or near an identifiable feature point, for example, by inserting the end effector 612 of the calibration workpiece 600 into feature point 902, to further reduce hand-eye calibration errors.

[0097] Compared with the traditional hand-eye calibration process, the hand-eye calibration process according to Method 800 can automatically perform hand-eye calibration without human intervention, effectively reducing the complexity of operation. It is also simple to calculate and can quickly achieve hand-eye calibration, while avoiding errors caused by manual intervention and improving calibration accuracy.

[0098] Figure 10 A block diagram of an exemplary calibration device 1000 for a robot according to an embodiment of the present disclosure is shown. The device 1000 includes a camera calibration unit 1010, a TCP calibration unit 1020, and a hand-eye calibration unit 1030. The device 1000 also includes a communication unit (not shown) for communicating with other external devices (e.g., receiving / sending instructions and data from and / or to external devices).

[0099] The camera unit 1010 includes an environment capture module 1011, a placement module 1012, a first camera capture module 1013, and a parameter determination module 1014.

[0100] The environment capture module 1011 is configured to capture images of the environment and form a three-dimensional virtual object of the environment. In some embodiments, the environment capture module 1011 may be further configured to: take pictures of the environment from different positions and at different angles, thereby obtaining at least two images for each scene; measure the depth information of each pixel in the image using a triangulation method based on the stereo image; and form a three-dimensional virtual object of the environment based on the depth information and the two-dimensional information contained in the image.

[0101] The placement module 1012 is configured to place a calibrated object in a target area of ​​the environment based on an image of the captured environment and a three-dimensional virtual object of the environment. In some embodiments, the placement module 1012 may be further configured to: detect visual markers arranged in the environment from an image of the captured environment; determine a target area based on the detected visual markers; capture an image of the calibrated object and form a three-dimensional virtual object of the calibrated object; overlay the three-dimensional virtual object of the calibrated object onto a three-dimensional virtual object of the environment for display as an augmented reality image; and move the calibrated object such that the three-dimensional virtual object of the calibrated object is located within the three-dimensional virtual object of the target area. In some embodiments, the visual markers have characteristic colors and / or patterns.

[0102] The first camera capture module 1013 is configured to capture images of a calibration object using a first camera via relative movement between a first camera associated with the robot and a calibration object, wherein the first camera is a 2D camera. In some embodiments, when the first camera is fixed external to the robot, the first camera capture module 1013 may be further configured to: obtain a plurality of actual positions for placing the calibration object in a target area, the plurality of actual positions corresponding to a plurality of virtual positions in a three-dimensional virtual object of the environment; move the calibration object such that the three-dimensional virtual object of the calibration object is located at the plurality of virtual positions respectively; and capture images of the calibration object at the plurality of actual positions using the first camera respectively.

[0103] The parameter determination module 1014 is configured to determine the parameters of the first camera used for calibration based on the image of the calibration object captured by the first camera.

[0104] The TCP calibration unit 1020 includes a first workpiece capture module 1021, a workpiece coordinate determination module 1022, a robot coordinate acquisition module 1023, and a TCP coordinate determination module 1024.

[0105] The first workpiece capture module 1021 is configured to use a second camera to capture multiple images of a first workpiece placed in multiple poses, wherein the first workpiece is mounted on the end of a robot, and the second camera is a 3D camera and is fixed to the outside of the robot.

[0106] The workpiece coordinate determination module 1022 is configured to: determine multiple workpiece end coordinates in the 3D camera coordinate system of the second camera for the end of the first workpiece in multiple poses, based on multiple captured images. In some embodiments, the workpiece coordinate determination module 1022 may be configured to: detect marker points arranged on the first workpiece from the multiple captured images to obtain multiple marker point coordinates in the 3D camera coordinate system; and determine multiple workpiece end coordinates in the 3D camera coordinate system based on the multiple marker point coordinates. In some embodiments, the marker points have characteristic colors and / or characteristic patterns.

[0107] The robot coordinate acquisition module 1023 is configured to obtain multiple robot end coordinates in the robot's base coordinate system under multiple poses.

[0108] The TCP coordinate determination module 1024 is configured to: determine the coordinates of the robot TCP to be calibrated in the robot's base coordinate system in a robot posture based on multiple workpiece end coordinates and multiple robot end coordinates, and determine the relative relationship between the robot TCP and the default TCP located on the robot's end.

[0109] The hand-eye calibration unit 1030 includes a second workpiece capture module 1031, a first transformation determination module 1032, a tool coordinate determination module 1033, a second transformation determination module 1034, and a third transformation determination module 1035.

[0110] The second workpiece capture module 1031 is configured to: use a first camera to capture a first target image of the second workpiece to be processed by the robot, and use a second camera to capture a second target image of the second workpiece.

[0111] The first transformation determination module 1032 is configured to: determine a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera based on the first target image and the second target image. In some embodiments, the first transformation determination module 1032 may be further configured to: detect a plurality of feature points arranged in the second workpiece from the first target image and the second target image to obtain a first plurality of feature coordinates of the plurality of feature points in the 2D camera coordinate system and a second plurality of feature coordinates in the 3D camera coordinate system; and use the first plurality of feature coordinates and the second plurality of feature coordinates to determine the first transformation relationship.

[0112] The tool coordinate determination module 1033 is configured to: move the end effector of the robot to multiple positions, determine a first plurality of tool coordinates of the robot TCP in the robot's base coordinate system, and determine a second plurality of tool coordinates of the robot TCP in the 3D camera coordinate system. In some embodiments, the tool coordinate determination module 1033 may be further configured to: translate to at least one of the multiple positions in a robot pose, and determine at least one tool coordinate of the robot TCP in the robot's base coordinate system based on the relative relationship between the robot TCP and the default TCP. In some embodiments, the at least one position is at or near at least one feature point.

[0113] The second transformation determination module 1034 is configured to use a first plurality of tool coordinates and a second plurality of tool coordinates to determine a second transformation relationship between the robot's base coordinate system and the 3D camera coordinate system.

[0114] The third transformation determination module 1035 is configured to: determine the third transformation relationship between the robot's base coordinate system and the 2D camera coordinate system based on the first transformation relationship and the second transformation relationship.

[0115] Although Figure 10 In the example, the camera calibration unit 1010, TCP calibration unit 1020 and hand-eye calibration unit 1030 are shown as integrated in the device 1000, but it should be understood that the device 1000 may include at least one of these units to implement the corresponding calibration process separately.

[0116] Figure 11 A block diagram of an exemplary computing device 1100 for implementing focusing of an industrial camera according to an embodiment of the present disclosure is shown. The computing device 1100 includes a processor 1101 and a memory 1102 coupled to the processor 1101. The memory 1102 is used to store computer-executable instructions that, when executed, cause the processor 1101 to perform the methods described in the above embodiments (e.g., any one or more steps of the foregoing methods 300, 500, or 800).

[0117] Alternatively, the above methods can be implemented using a computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for executing the various embodiments of this disclosure. The computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combinations thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0118] Therefore, in another embodiment, this disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon for performing the methods of various embodiments of this disclosure.

[0119] In another embodiment, this disclosure provides a computer program product tangibly stored on a computer-readable storage medium and including computer-executable instructions that, when executed, cause at least one processor to perform the methods of various embodiments of this disclosure.

[0120] Generally, the various example embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of the embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0121] Computer-readable program instructions or computer program products for executing the various embodiments of this disclosure can also be stored in the cloud. When needed, users can access the computer-readable program instructions stored in the cloud for executing an embodiment of this disclosure via mobile internet, fixed network or other networks, thereby implementing the technical solutions disclosed in the various embodiments of this disclosure.

[0122] While embodiments of this disclosure have been described with reference to several specific examples, it should be understood that the embodiments of this disclosure are not limited to the specific embodiments disclosed. The embodiments of this disclosure are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A calibration method for a robot, the calibration method comprising performing a camera calibration process, wherein, The camera calibration process includes the following steps: A. Capture images of the environment and create a three-dimensional virtual object of that environment. B. Based on the captured image of the environment and the three-dimensional virtual objects of the environment, place the calibrated object in the target area of ​​the environment. C, using the first camera to capture images of the calibration object via relative movement between the first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera, and D. Based on the image of the calibration object captured by the first camera, determine the parameters of the first camera used for calibration. The calibration method further includes performing a hand-eye calibration process, wherein performing the hand-eye calibration process includes the following steps: I. The first camera is used to capture a first target image of the second workpiece to be processed by the robot, and the second camera is used to capture a second target image of the second workpiece. J. Based on the first target image and the second target image, determine a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera. K, move the robot's end effector to multiple positions, determine the robot's TCP in the robot's base coordinate system as a first plurality of tool coordinates, and determine the robot's TCP in the 3D camera coordinate system as a second plurality of tool coordinates. L, using the first plurality of tool coordinates and the second plurality of tool coordinates, determines a second transformation relationship between the robot's base coordinate system and the 3D camera coordinate system, and Based on the first transformation relationship and the second transformation relationship, M determines the third transformation relationship between the robot's base coordinate system and the 2D camera coordinate system.

2. The calibration method according to claim 1, wherein, Step A includes: A1, taking images of the environment from different positions and angles to obtain at least two images for each scene. A2, a triangulation method based on stereo images is used to measure the depth information of each pixel in the image, and A3, a three-dimensional virtual object of the environment is formed based on the depth information and the two-dimensional information contained in the image.

3. The calibration method according to claim 1, wherein, Step B includes: B1, Detecting visual markers arranged in the captured environment from images of the environment, B2, Based on the detected visual markers, determine the target region. B3, capture the image of the calibration object and form a three-dimensional virtual object of the calibration object. B4, superimposing the three-dimensional virtual object of the calibrated object onto the three-dimensional virtual object of the environment to display it as an augmented reality image, and B5, Move the calibration object so that the three-dimensional virtual object of the calibration object is located within the three-dimensional virtual object of the target area.

4. The calibration method according to claim 3, wherein, The visual markers have characteristic colors and / or characteristic patterns.

5. The calibration method according to claim 3, wherein, When the first camera is fixed to the outside of the robot, step C includes: C1, obtain multiple actual positions for placing the calibration object in the target area, the multiple actual positions corresponding to multiple virtual positions in a three-dimensional virtual object of the environment. C2, move the calibration object so that the three-dimensional virtual object of the calibration object is located at the plurality of virtual positions respectively, and C3, using the first camera to capture images of the calibrated object at the plurality of actual locations respectively.

6. The calibration method according to claim 1, wherein, The calibration method further includes performing a TCP calibration process, wherein performing the TCP calibration process includes the following steps: E. A second camera is used to capture multiple images of a first workpiece placed in multiple poses, wherein the multiple poses have different spatial tilt angles. The first workpiece is mounted on the end effector of the robot. The second camera is a 3D camera and is fixed to the outside of the robot. F, based on the captured multiple images, determines multiple workpiece end coordinates in the 3D camera coordinate system of the second camera for the end of the first workpiece in multiple poses. G, obtains multiple robot end-effector coordinates in the robot's base coordinate system under the multiple poses, and H, based on the multiple workpiece end coordinates and the multiple robot end coordinates, determines the coordinates of the robot TCP to be calibrated in the robot's base coordinate system under one of the robot's postures, and determines the relative relationship between the robot TCP and the default TCP located at the end of the robot.

7. The calibration method according to claim 6, wherein, Step F includes: F1 detects marker points arranged on the first workpiece from multiple captured images to obtain the coordinates of the marker points in the 3D camera coordinate system, and F2, based on the coordinates of the multiple marker points, determines the coordinates of the ends of the first workpiece in the 3D camera coordinate system.

8. The calibration method according to claim 7, wherein, The markers have characteristic colors and / or characteristic patterns.

9. The calibration method according to claim 8, wherein, Step J includes: J1 detects multiple feature points arranged in the second workpiece from the first target image and the second target image to obtain a first plurality of feature coordinates of the multiple feature points in the 2D camera coordinate system and a second plurality of feature coordinates in the 3D camera coordinate system, and J2 uses the first plurality of feature coordinates and the second plurality of feature coordinates to determine the first transformation relationship.

10. The calibration method according to claim 9, wherein, Step K includes: For at least one of the plurality of positions, the robot is translated to the at least one position in the one posture of the robot, and based on the relative relationship between the robot TCP and the default TCP, at least one tool coordinate of the robot TCP in the robot's base coordinate system is determined.

11. The calibration method according to claim 10, wherein, The at least one location is at or near at least one feature point.

12. A calibration device for a robot, the calibration device comprising a camera calibration unit, the camera calibration unit comprising: The environment capture module is configured to capture images of the environment and form a three-dimensional virtual object of the environment. The placement module is configured to place a calibrated object in a target area of ​​the environment based on a captured image of the environment and a 3D virtual object of the environment. A first camera capture module is configured to capture an image of the calibration object using the first camera via relative movement between a first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera. The parameter determination module is configured to determine the parameters of the first camera used for calibration based on the image of the calibration object captured by the first camera. The calibration device further includes a hand-eye calibration unit, which includes: The second workpiece capture module is configured to use the first camera to capture a first target image of the second workpiece to be processed by the robot, and to use the second camera to capture a second target image of the second workpiece. The first transformation determination module is configured to determine a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera based on the first target image and the second target image. The tool coordinate determination module is configured to move the robot's end effector to multiple positions, determine a first plurality of tool coordinates of the robot TCP in the robot's base coordinate system, and determine a second plurality of tool coordinates of the robot TCP in the 3D camera coordinate system. The second transformation determination module is configured to use the first plurality of tool coordinates and the second plurality of tool coordinates to determine a second transformation relationship between the robot's base coordinate system and the 3D camera coordinate system, and The third transformation determination module is configured to determine a third transformation relationship between the robot's base coordinate system and the 2D camera coordinate system based on the first transformation relationship and the second transformation relationship.

13. The calibration device according to claim 12, wherein, The environment capture module is further configured as follows: The environment is captured by cameras from different positions and angles, resulting in at least two images for each scene. The triangulation method based on stereo images measures the depth information of each pixel in the image, and A three-dimensional virtual object of the environment is formed based on the depth information and the two-dimensional information contained in the image.

14. The calibration device according to claim 12, wherein, The placement module is further configured as follows: Detecting visual markers arranged in the captured environment from images of the environment. Based on the detected visual markers, the target region is determined. Capture an image of the calibrated object and form a three-dimensional virtual object of the calibrated object. The three-dimensional virtual object of the calibrated object is superimposed on the three-dimensional virtual object of the environment to be displayed as an augmented reality image, and Move the calibration object so that the three-dimensional virtual object of the calibration object is located within the three-dimensional virtual object of the target area.

15. The calibration apparatus according to claim 14, wherein, The visual markers have characteristic colors and / or characteristic patterns.

16. The calibration apparatus according to claim 12, wherein, When the first camera is fixed to the outside of the robot, the first camera capture module is further configured as follows: Multiple actual positions for placing the calibrated object in the target area are obtained, the multiple actual positions corresponding to multiple virtual positions in a three-dimensional virtual object of the environment. Move the calibration object so that the three-dimensional virtual object of the calibration object is located at the plurality of virtual positions respectively, and The first camera is used to capture images of the calibrated object at the various actual locations.

17. The calibration apparatus according to claim 12, further comprising a TCP calibration unit, the TCP calibration unit comprising: A first workpiece capture module is configured to use a second camera to capture multiple images of the first workpiece placed in multiple poses, wherein the first workpiece is mounted on the end effector of the robot, and the second camera is a 3D camera and is fixed to the outside of the robot. The workpiece coordinate determination module is configured to determine multiple workpiece end coordinates in the 3D camera coordinate system of the second camera at multiple poses of the end of the first workpiece based on multiple captured images. The robot coordinate acquisition module is configured to obtain multiple robot end-effector coordinates in the robot's base coordinate system under the multiple poses, and The TCP coordinate determination module is configured to determine the coordinates of the robot TCP to be calibrated in the robot's base coordinate system in a certain posture of the robot, based on the multiple workpiece end coordinates and the multiple robot end coordinates, and to determine the relative relationship between the robot TCP and the default TCP located at the end of the robot.

18. The calibration apparatus according to claim 17, wherein, The workpiece coordinate determination module is further configured as follows: Marker points arranged on the first workpiece are detected from multiple captured images to obtain the coordinates of multiple marker points in the 3D camera coordinate system, and Based on the coordinates of the multiple marker points, the coordinates of the ends of the first workpiece in the 3D camera coordinate system are determined.

19. The calibration apparatus according to claim 18, wherein, The markers have characteristic colors and / or characteristic patterns.

20. The calibration apparatus according to claim 19, wherein, The first transformation determination module is further configured as follows: Multiple feature points arranged in the second workpiece are detected from the first target image and the second target image to obtain a first plurality of feature coordinates of the multiple feature points in the 2D camera coordinate system and a second plurality of feature coordinates in the 3D camera coordinate system. The first transformation relationship is determined using the first plurality of feature coordinates and the second plurality of feature coordinates.

21. The calibration apparatus according to claim 20, wherein, The tool coordinate determination module is further configured as follows: For at least one of the plurality of positions, the robot is translated to the at least one position in the one posture of the robot, and based on the relative relationship between the robot TCP and the default TCP, at least one tool coordinate of the robot TCP in the robot's base coordinate system is determined.

22. The calibration apparatus according to claim 21, wherein, The at least one location is at or near the at least one feature point.

23. A computing device, the computing device comprising: processor; as well as A memory for storing computer-executable instructions that, when executed, cause the processor to perform the method according to any one of claims 1-11.

24. A computer-readable storage medium having computer-executable instructions stored thereon for performing the method according to any one of claims 1-11.

25. A computer program product tangibly stored on a computer-readable storage medium and comprising computer-executable instructions that, when executed, cause at least one processor to perform the method according to any one of claims 1-11.

Images