Hand-eye calibration method and device

Abstract

The invention provides a hand-eye calibration method and device. The method comprises the steps that first image information, corresponding first point cloud data, second image information and corresponding second point cloud data of a target calibration plate of a mechanical arm under different poses are obtained; extracting a target calibration plate angular point in the first image information and the second image information, and determining a first three-dimensional coordinate and a second three-dimensional coordinate of the target calibration plate angular point based on the first point cloud data and the second point cloud data; determining a mapping relation equation; and a coefficient matrix is obtained through transformation, a linear equation is constructed based on pose data of the mechanical arm under different poses and corresponding image pixel data, the hand-eye matrix is corrected through the solving result, and hand-eye calibration of the mechanical arm and the depth camera is completed. The point cloud picture is obtained through the depth camera, the three-dimensional information of the target point is obtained through the point cloud picture, the linear algorithm with high stability is constructed, and the calibration precision is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics technology, and in particular to a hand-eye calibration method and device. Background Art

[0002] In the task of using a robotic arm to perform automated circuit board inspection, in order to ensure that the probe accurately touches the solder joints of components, it is necessary to obtain the transformation relationship between the visual coordinate system and the robotic arm coordinate system, that is, hand-eye calibration. With the help of the hand-eye matrix, the robotic arm can obtain the position information of a certain point in the world through the camera and transfer it to the robotic arm coordinate system. In related technologies, the hand-eye matrix calibration mainly includes the nine-point method and the Tsai-Lenz method to solve the AX=XB problem. However, these are all calculation methods designed for 2D camera hand-eye calibration.

[0003] In the industrial field, there are a large number of work scenarios that require three-dimensional information, but 2D cameras lack one dimension of information, resulting in insufficient work accuracy and unable to meet the needs of high-precision work applications. Summary of the invention

[0004] The present disclosure provides a hand-eye calibration method and device to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a hand-eye calibration method is provided, the method comprising:

[0006] After using the robotic arm to control the depth camera to identify the target calibration plate, first image information of the target calibration plate at different positions of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information are obtained;

[0007] Extracting corner points of a target calibration plate in the first image information and the second image information, and determining first three-dimensional coordinates and second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data, respectively;

[0008] Determine a mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates, and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate;

[0009] Transforming the mapping relationship equation based on the Lie algebra algorithm to obtain a coefficient matrix;

[0010] Based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data, a linear equation is constructed and solved to obtain a solution result;

[0011] The solution result is used to correct the hand-eye matrix and complete the hand-eye calibration of the robotic arm and the depth camera.

[0012] In one embodiment, the acquiring of first image information of the target calibration plate at different postures of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information includes:

[0013] Identify the target calibration plate area within the viewing range of the depth camera;

[0014] Controlling the robotic arm to collect first image information of the target calibration plate and first point cloud data corresponding to the first image information in a first posture;

[0015] Control the robot arm to obtain second image information at a second posture and second point cloud data corresponding to the second image information.

[0016] In one possible implementation manner, extracting the corner points of the target calibration plate in the first image information and the second image information includes:

[0017] According to the preset calibration plate corner point parameters, the target calibration plate corner points are extracted from the first image information and the target calibration plate corner points are extracted from the second image information.

[0018] In one possible implementation manner, the determining of the mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate includes:

[0019] Based on the first three-dimensional coordinates and the second three-dimensional coordinates, the coordinates are converted into a robot arm base coordinate system under corresponding positions and postures to obtain a first coordinate and a second coordinate;

[0020] Based on the first coordinate and the second coordinate of the corner point of the target calibration plate, an initial mapping relationship equation is obtained;

[0021] The initial mapping relationship equation is transformed based on the initial value and error value of the hand-eye matrix rotation term to obtain the mapping relationship equation.

[0022] In one possible implementation manner, the mapping relationship equation is transformed based on the Lie algebra algorithm to obtain a coefficient matrix, including:

[0023] Using a small angle assumption, the error value of the hand-eye matrix rotation term is transformed to obtain a first transformation error;

[0024] Transforming the transformation error based on a Lie algebra algorithm to obtain a second transformation error;

[0025] Substitute the second transformation error into the mapping relationship equation to obtain a coefficient matrix.

[0026] In one possible implementation, constructing a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data includes:

[0027] Using a camera to obtain images of the target calibration plate of the robotic arm at different positions and pixel data of the images;

[0028] The linear equation is constructed using the pose data and the corresponding pixel data.

[0029] In one possible implementation, using the solution result to correct the hand-eye matrix includes:

[0030] The hand-eye matrix is ​​corrected using the solution result to obtain the corrected hand-eye matrix rotation amount and hand-eye matrix translation amount.

[0031] In one possible implementation, it also includes:

[0032] Verify the hand-eye calibration accuracy of the corrected hand-eye matrix rotation and hand-eye matrix translation.

[0033] In one possible implementation, the depth camera is disposed on a robotic arm.

[0034] According to a second aspect of the present disclosure, a hand-eye calibration device is provided, the device comprising:

[0035] an acquisition module, configured to acquire, after using the robotic arm to control the depth camera to identify the target calibration plate, first image information of the target calibration plate of the robotic arm at different positions, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information;

[0036] an extraction module, configured to extract corner points of a target calibration plate in the first image information and the second image information, and determine first three-dimensional coordinates and second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data, respectively;

[0037] A determination module, used to determine a mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate;

[0038] A transformation module, used for transforming the mapping relationship equation based on a Lie algebra algorithm to obtain a coefficient matrix;

[0039] A construction module is used to construct a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data, and solve it to obtain a solution result;

[0040] The calibration module is used to correct the hand-eye matrix using the solution results to complete the hand-eye calibration of the robotic arm and the depth camera.

[0041] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0042] at least one processor; and

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0045] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0046] The hand-eye calibration method and device disclosed in the present invention obtains multiple image information and corresponding point cloud data of the target calibration plate in different postures through a depth camera, and then obtains different three-dimensional coordinates of the corner points of different target calibration plates. According to the equality of the three-dimensional coordinates of the corner points of the same target calibration plate, combined with the initial value and error value of the rotation term of the hand-eye matrix, a mapping relationship equation is obtained, and a coefficient matrix is ​​obtained after transformation, so as to obtain a linear equation based on the coefficient matrix and the posture data of the manipulator in different postures and the corresponding image pixel data, and obtain a solution result after solving the linear equation, and use the solution result to correct the hand-eye matrix, thereby completing the hand-eye calibration of the manipulator and the depth camera. The present application obtains the point cloud map of the target calibration plate in different postures in space through a 3D camera, extracts the three-dimensional information of the points in the point cloud map, and thus constructs a linear equation with strong stability, which effectively improves the calibration accuracy.

[0047] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0049] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0050] Figure 1 A schematic diagram of the implementation process of the hand-eye calibration method according to an embodiment of the present disclosure is shown;

[0051] Figure 2 A schematic diagram of the structure of a hand-eye calibration device according to an embodiment of the present disclosure is shown;

[0052] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0054] A hand-eye calibration method and device provided by the present application are described below in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the embodiment of the present application provides a hand-eye calibration method, including:

[0056] S101, after using a robotic arm to control a depth camera to identify a target calibration plate, obtain first image information of the target calibration plate at different positions of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information;

[0057] It can be understood that in the present application, a depth camera is set on the robotic arm. The depth camera is a camera device that can obtain depth information of an object. It measures the infrared light reflected by the object to achieve accurate measurement of the depth of the object.

[0058] The present application first sets a target calibration plate, and then uses a 3D camera to obtain an image of the target calibration plate. It should be noted that the first image information and its corresponding first point cloud data and the second image information and its corresponding second point cloud data are obtained by controlling the robotic arm to move to different positions. It should be noted that in the present application, the robotic arm can be controlled to enable the 3D camera to obtain a point cloud image of the target calibration plate in different positions in space through a pyramid shooting mode.

[0059] S102, extracting corner points of a target calibration plate in the first image information and the second image information, and determining first three-dimensional coordinates and second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data, respectively;

[0060] It can be understood that in the present application, the corner points of the target calibration plate are set in advance, and the corner points of the target calibration plate in the first image information and the second image information are extracted respectively to obtain the first three-dimensional coordinates of the corner points of the target calibration plate in the first image and the second three-dimensional coordinates in the second image.

[0061] S103, determining a mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates of the corner points of the target calibration plate and the initial value and error value of the hand-eye matrix rotation term;

[0062] It can be understood that because the corner point of the target calibration plate is a corner point, the first three-dimensional coordinate and the second three-dimensional coordinate are the same, and the first equation is obtained. Then, the first equation is transformed based on the initial value and error value of the hand-eye matrix rotation term to obtain the mapping relationship equation.

[0063] S104, transforming the mapping relationship equation based on a Lie algebra algorithm to obtain a coefficient matrix;

[0064] In this application, the Lie algebra algorithm is used to transform the mapping relationship equation to obtain a coefficient matrix.

[0065] S105, constructing a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data, and solving it to obtain a solution result;

[0066] The posture data of the robot arm in different postures and the corresponding image pixel data under the posture data are obtained to construct a linear equation, and the linear equation is solved to obtain the solution result.

[0067] S106, using the solution result to correct the hand-eye matrix, and complete the hand-eye calibration of the robotic arm and the depth camera.

[0068] The hand-eye matrix is ​​corrected using the solution obtained by solving the linear equation to obtain the rotation and translation after the hand-eye matrix correction.

[0069] The hand-eye calibration method provided by the present application obtains multiple image information and corresponding point cloud data of the target calibration plate in different postures through a depth camera, and then obtains different three-dimensional coordinates of the corner points of different target calibration plates. According to the equality of the three-dimensional coordinates of the corner points of the same target calibration plate, combined with the initial value and error value of the rotation term of the hand-eye matrix, a mapping relationship equation is obtained, and the coefficient matrix is ​​obtained after transformation, so as to obtain a linear equation based on the coefficient matrix and the posture data of the manipulator in different postures and the corresponding image pixel data, and obtain a solution result after solving the linear equation, and use the solution result to correct the hand-eye matrix, thereby completing the hand-eye calibration of the manipulator and the depth camera. The present application obtains the point cloud map of the target calibration plate in different postures in space through a 3D camera, extracts the three-dimensional information of the points in the point cloud map, and thus constructs a linear algorithm with strong stability, which effectively improves the calibration accuracy.

[0070] In some optional embodiments, the step of acquiring first image information of the target calibration plate at different postures of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information includes:

[0071] Identify the target calibration plate area within the viewing range of the depth camera;

[0072] Controlling the robotic arm to collect first image information of the target calibration plate and first point cloud data corresponding to the first image information in a first posture;

[0073] Control the robot arm to obtain second image information at a second posture and second point cloud data corresponding to the second image information.

[0074] Specifically, first determine whether the target calibration plate area is within the field of view of the depth camera, and determine that the depth camera can capture the target calibration plate. Then, the processor can control the robotic arm to control the depth camera to collect first image information of the target calibration plate in the first posture, and obtain first point cloud data corresponding to the first image information. Then, change the posture of the robotic arm, and control the depth camera to collect second image information of the target calibration plate when the robotic arm is in the second posture, and obtain second point cloud data corresponding to the second image information.

[0075] In some optional embodiments, extracting the corner points of the target calibration plate in the first image information and the second image information includes:

[0076] According to the preset calibration plate corner point parameters, the target calibration plate corner points are extracted from the first image information and the target calibration plate corner points are extracted from the second image information.

[0077] Specifically, the present application sets calibration plate corner point parameters in advance, so as to extract target calibration plate corner points from the first image information and the second image information according to the calibration plate corner point parameters. The present application may also use other methods to extract calibration plate corner points, which are not limited in the present application.

[0078] In some optional embodiments, the determining of the mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate includes:

[0079] Based on the first three-dimensional coordinates and the second three-dimensional coordinates, the coordinates are converted into a robot arm base coordinate system under corresponding positions and postures to obtain a first coordinate and a second coordinate;

[0080] Based on the first coordinate and the second coordinate of the corner point of the target calibration plate, an initial mapping relationship equation is obtained;

[0081] The initial mapping relationship equation is transformed based on the initial value and error value of the hand-eye matrix rotation term to obtain the mapping relationship equation.

[0082] Specifically, the 3D camera takes an image of the calibration plate and extracts the point cloud coordinates of the pre-selected corner points on the calibration plate. The coordinates represent the three-dimensional position information of the point in the camera coordinate system, which is recorded as P C1 ; Change the robot arm posture to obtain the three-dimensional position information of the same corner point in the camera coordinate system under the second posture, denoted as P C2 Then, P C1 Convert to the robot arm coordinate system to get

[0083]

[0084] in, is the rotation matrix from the base coordinate system of the robot to the end coordinate system in the first posture. is the rotation matrix from the base coordinate system of the robot to the end coordinate system in the first posture; is the rotation amount of the hand-eye matrix; is the translation of the hand-eye matrix;

[0085] P C2 Convert to the robot arm coordinate system to get

[0086]

[0087] in, is the rotation matrix from the base coordinate system of the robot to the end coordinate system in the second posture. is the rotation matrix from the base coordinate system of the robot to the end coordinate system in the second posture;

[0088] Because P B1 and P B2 is the mathematical expression of the same corner point in the robot base coordinate system, so P B1 =P B2 , that is, the initial mapping relationship equation,

[0089]

[0090] For any robot arm position, the rotation matrix Disassembly,

[0091]

[0092] in, is the initial value of the hand-eye matrix rotation term, and ΔR is the error term of the hand-eye matrix rotation term;

[0093] The initial mapping relationship equation is transformed to obtain a mapping relationship equation,

[0094]

[0095] Among them, ΔR is composed of the small angles of the three axes Rotation composition.

[0096] In some optional embodiments, the mapping relationship equation is transformed based on the Lie algebra algorithm to obtain a coefficient matrix, including:

[0097] Using a small angle assumption, the error value of the hand-eye matrix rotation term is transformed to obtain a first transformation error;

[0098] Transforming the transformation error based on a Lie algebra algorithm to obtain a second transformation error;

[0099] Substitute the second transformation error into the mapping relationship equation to obtain a coefficient matrix.

[0100] Specifically, based on the small angle assumption, we get

[0101]

[0102] Ignoring the second-order and higher-order small-angle quantities, we get

[0103]

[0104] in, yes The Lie algebra of .

[0105] According to Lie algebra, mathematical transformation is performed to obtain the second transformation error,

[0106]

[0107] Substituting the second transformation error into the mapping relationship equation, we get

[0108]

[0109] Right now

[0110]

[0111] So the coefficient matrix is ​​defined as,

[0112]

[0113] In some optional embodiments, constructing a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data includes:

[0114] Using a camera to obtain images of the target calibration plate of the robotic arm at different positions and pixel data of the images;

[0115] The linear equation is constructed using the pose data and the corresponding pixel data.

[0116] Specifically, in this application, the pixel data in multiple images and the corresponding robot arm posture data can be used to construct the following linear equation:

[0117]

[0118] Solved

[0119] X=(A T A) -1 ·A T ·B (16)

[0120] In some optional embodiments, using the solution result to correct the hand-eye matrix includes:

[0121] The hand-eye matrix is ​​corrected using the solution result to obtain the corrected hand-eye matrix rotation amount and hand-eye matrix translation amount.

[0122] Specifically, in this application, the hand-eye matrix is ​​corrected by the above solution results to obtain:

[0123]

[0124] In some embodiments, the hand-eye calibration method provided by the present application further includes:

[0125] Verify the hand-eye calibration accuracy of the corrected hand-eye matrix rotation and hand-eye matrix translation.

[0126] Specifically, the present application can further verify the hand-eye calibration accuracy through a point cloud stitching experiment, specifically verifying the hand-eye calibration accuracy of the corrected hand-eye matrix rotation and the corrected hand-eye matrix translation through a point cloud stitching experiment.

[0127] like Figure 2 As shown, an embodiment of the present application provides a hand-eye calibration device, the device comprising:

[0128] An acquisition module 201 is used to control the depth camera to identify the target calibration plate using the robotic arm, and then acquire first image information of the target calibration plate at different positions of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information;

[0129] An extraction module 202 is used to extract the corner points of the target calibration plate in the first image information and the second image information, and determine the first three-dimensional coordinates and the second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data respectively;

[0130] A determination module 203 is used to determine a mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates of the corner points of the target calibration plate and the initial value and error value of the hand-eye matrix rotation term;

[0131] A transformation module 204, configured to transform the mapping relationship equation based on a Lie algebra algorithm to obtain a coefficient matrix;

[0132] A construction module 205 is used to construct a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data, and solve it to obtain a solution result;

[0133] The calibration module 206 is used to correct the hand-eye matrix using the solution result to complete the hand-eye calibration of the robot arm and the depth camera.

[0134] The hand-eye calibration device provided in the present application uses the acquisition module 201 to control the depth camera to identify the target calibration plate, and then obtains the first image information of the target calibration plate in different postures of the robotic arm, the first point cloud data corresponding to the first image information, the second image information and the second point cloud data corresponding to the second image information; the extraction module 202 extracts the corner points of the target calibration plate in the first image information and the second image information, and determines the first three-dimensional coordinates and the second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data respectively; the determination module 203 determines the mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates of the corner points of the target calibration plate and the initial value and error value of the hand-eye matrix rotation term; the transformation module 204 transforms the mapping relationship equation based on the Lie algebra algorithm to obtain a coefficient matrix; the construction module 205 constructs a linear equation based on the coefficient matrix and the posture data of the robotic arm in different postures and the corresponding image pixel data, and solves it to obtain a solution result; the calibration module 206 uses the solution result to correct the hand-eye matrix to complete the hand-eye calibration of the robotic arm and the depth camera.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0136] Figure 3A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0137] like Figure 3 As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0138] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as the hand-eye calibration method. For example, in some embodiments, the hand-eye calibration method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the hand-eye calibration method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the hand-eye calibration method in any other appropriate manner (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0145] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0148] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A hand-eye calibration method, characterized in that: The method comprises: After using the robotic arm to control the depth camera to identify the target calibration plate, first image information of the target calibration plate at different positions of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information are obtained; Extracting corner points of a target calibration plate in the first image information and the second image information, and determining first three-dimensional coordinates and second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data, respectively; Determine a mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates, and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate; Transforming the mapping relationship equation based on the Lie algebra algorithm to obtain a coefficient matrix; Based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data, a linear equation is constructed and solved to obtain a solution result; The solution result is used to correct the hand-eye matrix and complete the hand-eye calibration of the robotic arm and the depth camera.

2. The method according to claim 1, characterized in that The step of acquiring first image information of the target calibration plate at different positions of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information includes: Identify the target calibration plate area within the viewing range of the depth camera; Controlling the robotic arm to collect first image information of the target calibration plate and first point cloud data corresponding to the first image information in a first posture; Control the robot arm to obtain second image information at a second posture and second point cloud data corresponding to the second image information.

3. The method according to claim 1, characterized in that: The extracting the target calibration plate corner points in the first image information and the second image information includes: According to the preset calibration plate corner point parameters, the target calibration plate corner points are extracted from the first image information and the target calibration plate corner points are extracted from the second image information.

4. The method according to claim 1, characterized in that: The mapping relationship equation is determined based on the first three-dimensional coordinates, the second three-dimensional coordinates and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate, including: Based on the first three-dimensional coordinates and the second three-dimensional coordinates, the coordinates are converted into a robot arm base coordinate system under corresponding positions and postures to obtain a first coordinate and a second coordinate; Based on the first coordinate and the second coordinate of the corner point of the target calibration plate, an initial mapping relationship equation is obtained; The initial mapping relationship equation is transformed based on the initial value and error value of the hand-eye matrix rotation term to obtain the mapping relationship equation.

5. The method according to claim 1, characterized in that The mapping relationship equation is transformed based on the Lie algebra algorithm to obtain a coefficient matrix, including: Using a small angle assumption, the error value of the hand-eye matrix rotation term is transformed to obtain a first transformation error; Transforming the transformation error based on a Lie algebra algorithm to obtain a second transformation error; Substitute the second transformation error into the mapping relationship equation to obtain a coefficient matrix.

6. The method according to claim 1, characterized in that The step of constructing a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data comprises: Using a camera to obtain images of the target calibration plate of the robotic arm at different positions and pixel data of the images; The linear equation is constructed using the pose data and the corresponding pixel data.

7. The method according to claim 1, characterized in that Correcting the hand-eye matrix using the solution result includes: The hand-eye matrix is ​​corrected using the solution result to obtain the corrected hand-eye matrix rotation amount and hand-eye matrix translation amount.

8. The method according to claim 7, characterized in that Also includes: Verify the hand-eye calibration accuracy of the corrected hand-eye matrix rotation and hand-eye matrix translation.

9. The method according to claim 1, characterized in that: The depth camera is arranged on the robotic arm.

10. A hand-eye calibration device, characterized in that: The device comprises: An acquisition module is used to control the depth camera to identify the target calibration plate using the robotic arm, and then acquire first image information of the target calibration plate at different positions of the robotic arm, first point cloud data corresponding to the first image information, second image information, and second point cloud data corresponding to the second image information; an extraction module, configured to extract corner points of a target calibration plate in the first image information and the second image information, and determine first three-dimensional coordinates and second three-dimensional coordinates of the corner points of the target calibration plate based on the first point cloud data and the second point cloud data, respectively; A determination module, used to determine a mapping relationship equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the initial value and error value of the hand-eye matrix rotation term of the corner point of the target calibration plate; A transformation module, used for transforming the mapping relationship equation based on a Lie algebra algorithm to obtain a coefficient matrix; A construction module is used to construct a linear equation based on the coefficient matrix and the posture data of the robot arm at different postures and the corresponding image pixel data, and solve it to obtain a solution result; The calibration module is used to correct the hand-eye matrix using the solution results to complete the hand-eye calibration of the robotic arm and the depth camera.

Images