Hand-eye calibration method and device

Abstract

The invention provides a hand-eye calibration method and device. The method comprises the following steps: acquiring first image information and first point cloud data of a target calibration board by using a depth camera; a calibration plate corner point in the first image information is extracted, and a first three-dimensional coordinate of the calibration plate corner point is determined based on the first point cloud data; acquiring second image information and second point cloud data of the target calibration plate in different poses; a calibration plate corner point in the second image information is extracted, and a second three-dimensional coordinate of the calibration plate corner point is determined based on the second point cloud data; constructing a hand-eye matrix equation based on the first three-dimensional coordinates and the second three-dimensional coordinates of the same calibration plate angular points and the hand-eye matrix; the hand-eye matrix equation is solved, and hand-eye calibration of the mechanical arm and the depth camera is completed. According to the method, the 3D camera is introduced to obtain the point cloud picture of the target calibration board, the three-dimensional information of the target point is obtained through the point cloud picture, the linear algorithm with high stability is constructed, and the calibration precision is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electronic equipment, and in particular to a hand-eye calibration method and device. Background Art

[0002] In electronic devices, circuit board inspection tasks are crucial to product quality and production efficiency. In the use of robotic arms for automated circuit board inspection tasks, visual inspection technology plays a key role in ensuring that the probe accurately touches the solder joints of components. In related technologies, 2D cameras are often used to obtain circuit board image data, but in this process, the data obtained using 2D cameras lacks height information, resulting in low operating accuracy and unable to meet high-precision application scenarios. Summary of the invention

[0003] The present disclosure provides a hand-eye calibration method and device to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a hand-eye calibration method is provided, the method comprising:

[0005] After the depth camera is controlled by the robotic arm to identify the target calibration plate, first image information and first point cloud data of the target calibration plate are acquired at different positions with the same posture;

[0006] Extracting corner points of a calibration plate from the first image information, and determining first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data;

[0007] Using a robotic arm to control a depth camera to collect second image information and second point cloud data of the target calibration plate in different postures;

[0008] Extracting the corner points of the calibration plate in the second image information, and determining the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data;

[0009] Constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner points of the same calibration plate;

[0010] Solve the hand-eye matrix equation to complete the hand-eye calibration of the robotic arm and the depth camera.

[0011] In one possible implementation manner, the acquiring the first image information and the first point cloud data of the target calibration plate at the same posture and different positions includes:

[0012] Identify the target calibration plate area within the viewing range of the depth camera;

[0013] Control the robotic arm to collect first image information and first point cloud data of a first target point on the target calibration plate in the same posture at different height planes;

[0014] Among them, the first target points include multiple ones.

[0015] In one possible implementation manner, extracting the calibration plate corner points in the first image information includes:

[0016] According to preset calibration plate corner point parameters, the calibration plate corner points are extracted from the first image information.

[0017] In one possible implementation manner, the method of using a robotic arm to control a depth camera to collect second image information and second point cloud data of the target calibration plate in different postures includes:

[0018] Identify the target calibration plate area within the viewing range of the depth camera;

[0019] Control the robotic arm to collect second image information and second point cloud data of a second target point on the target calibration plate at different postures;

[0020] Wherein, the second target point includes multiple ones.

[0021] In one possible implementation manner, the first image information and the second image information include the same calibration plate corner points.

[0022] In one possible implementation, the constructing of the hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate includes:

[0023] Determine the first three-dimensional coordinates and the second three-dimensional coordinates of the corner point of the same calibration plate;

[0024] Performing homogeneous transformation on the first three-dimensional coordinates and the second three-dimensional coordinates respectively to obtain transformation equations;

[0025] The three-dimensional coordinates in the transformation equation are converted into the robot base coordinate system to obtain the expansion equation;

[0026] Based on the expansion equation, an equivalent transformation is performed on the hand-eye matrix to obtain a column vector hand-eye matrix;

[0027] The second image information and the corresponding robot arm posture data are input into the column vector hand-eye matrix to obtain the hand-eye matrix equation.

[0028] In one possible implementation, a linear algorithm is used to solve the hand-eye matrix equation.

[0029] In one possible implementation manner, before extracting the calibration plate corner points in the first image information or extracting the calibration plate corner points in the second image information, the method further includes:

[0030] The depth camera is calibrated according to the first image information and the second image information of the calibration plate, and the first point cloud data and the second point cloud data are dedistorted to obtain corrected data.

[0031] In one possible implementation, the depth camera is disposed on a robotic arm.

[0032] According to a second aspect of the present disclosure, a hand-eye calibration device is provided, the device comprising:

[0033] A first acquisition module is used to control the depth camera to identify the target calibration plate using a mechanical arm, and then acquire first image information and first point cloud data of the target calibration plate at different positions with the same posture;

[0034] a first extraction module, configured to extract corner points of a calibration plate from the first image information, and determine first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data;

[0035] A second acquisition module is used to use a robotic arm to control the depth camera to acquire second image information and second point cloud data of the target calibration plate in different postures;

[0036] a second extraction module, configured to extract the corner points of the calibration plate in the second image information, and determine the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data;

[0037] A construction module, used for constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate;

[0038] The calibration module is used to solve the hand-eye matrix equation and complete the hand-eye calibration of the robotic arm and the depth camera.

[0039] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0044] The hand-eye calibration method and device disclosed in the present invention, the present application first sets the target calibration plate within the recognition range of the depth camera, and then uses the depth camera to shoot the target calibration plate in the same posture at different positions to obtain the first image information and the first point cloud data of the target calibration plate, and then the corner points of the calibration plate are mapped from the first image information, and the first three-dimensional coordinates of the corner points of the calibration plate are determined based on the first point cloud data; then the depth camera is controlled to shoot the target calibration plate in different postures to obtain the second image information and the second point cloud data, and similarly the corner points of the calibration plate are extracted from the second image information, and the second three-dimensional coordinates of the corner points of the calibration plate are determined, and the hand-eye matrix equation is constructed based on the equality of the three-dimensional coordinates of the same target calibration plate, and the hand-eye matrix equation is solved to complete the hand-eye calibration of the robot arm and the depth camera. The present application obtains the point cloud map of the target calibration plate at different postures in space by a 3D camera, extracts the three-dimensional information of the points in the point cloud map, and thus constructs a linear hand-eye matrix equation with strong stability, which effectively improves the calibration accuracy.

[0045] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0047] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0048] Figure 1 A schematic diagram of the implementation process of the hand-eye calibration method according to an embodiment of the present disclosure is shown;

[0049] Figure 2 A schematic diagram of the structure of a target calibration plate according to an embodiment of the present disclosure is shown;

[0050] Figure 3 A schematic diagram of the structure of the hand-eye calibration structure of an embodiment of the present disclosure is shown;

[0051] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] The hand-eye calibration method and device provided by the present application are described below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, an embodiment of the present application provides a hand-eye calibration method, the method comprising:

[0055] S101, after using a robotic arm to control a depth camera to identify a target calibration plate, first image information and first point cloud data of the target calibration plate are acquired at different positions with the same posture;

[0056] It can be understood that in the present application, a depth camera is set on the robotic arm. The depth camera is a camera device that can obtain depth information of an object. It measures the infrared light reflected by the object to achieve accurate measurement of the depth of the object.

[0057] The present application first sets up a target calibration plate, and then uses a 3D camera to obtain an image of the target calibration plate. It should be noted that the present application controls the robotic arm to shoot the target calibration plate in the same posture at different positions, and uses the image obtained in this manner as the first image information, and simultaneously obtains the first point cloud data corresponding to the first image information; it can be understood that the first image information includes multiple images.

[0058] like Figure 2 As shown, this application is approved by Figure 2 The image information and point cloud data are obtained in the manner described above. Specifically, the corner points for each shooting are set. For example, the first time, images are collected at the points P1, P2, P3, P4, P5, P6, P7, and P8 in the figure. Therefore, the first image information includes multiple images collected at the points P1, P2, P3, P4, P5, P6, P7, and P8.

[0059] S102, extracting corner points of a calibration plate in the first image information, and determining first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data;

[0060] In the present application, the calibration plate corner points may or may not be set in advance. If a certain calibration plate corner point is set, the pre-set calibration plate corner point is extracted from the first image information, and the first three-dimensional coordinates of the calibration plate corner point are determined based on the first point cloud data; if the calibration plate corner point is not set, all calibration plate corner points in the first image information are extracted, and the three-dimensional coordinates of all calibration plate corner points are determined.

[0061] S103, using a robotic arm to control a depth camera to collect second image information and second point cloud data of the target calibration plate in different postures;

[0062] This application can be Figure 2 P9-P16 in the figure are points for controlling the robot arm to collect images in different postures. It should be noted that the second image information contains the same calibration plate corner points as those in the first image information, wherein the same calibration plate corner points may be pre-set calibration plate corner points or may be the same calibration plate corner points in multiple images.

[0063] S104, extracting the corner points of the calibration plate in the second image information, and determining the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data;

[0064] If the calibration plate corner points are set in advance, the second image information is extracted according to the preset calibration plate corner points. If the calibration plate corner points are not set, all calibration plate corner points in the second image information can be extracted, and the second three-dimensional coordinates of the calibration plate corner points can be determined through the second point cloud data.

[0065] S105, constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate;

[0066] Because they are the same calibration plate corner points, the coordinates are the same under the base coordinates, and the hand-eye matrix equation is constructed based on the equation and the hand-eye matrix. It can be understood that the hand-eye matrix equation is a linear equation.

[0067] S106, solving the hand-eye matrix equation to complete the hand-eye calibration of the robotic arm and the depth camera.

[0068] After solving the hand-eye matrix equation, the rotation and translation of the hand-eye matrix are obtained, completing the hand-eye calibration of the robotic arm and the depth camera.

[0069] The hand-eye calibration method provided by the present application uses a robotic arm to control a depth camera to obtain the first image information and the first point cloud data of a target calibration plate at different positions in the same posture, and collects the second image information and the second point cloud data of the target calibration plate at different postures, extracts the first three-dimensional coordinates and the second three-dimensional coordinates of the same calibration plate corner points in the first image information and the second image information, and then uses the same calibration plate corner points and the hand-eye matrix to obtain the hand-eye matrix equation. After solving the equation, the hand-eye calibration of the robotic arm and the depth camera is completed. The present application obtains the point cloud map of the target calibration plate at different postures in space through a 3D camera, extracts the three-dimensional information of the points in the point cloud map, and thus constructs a linear algorithm with strong stability, which effectively improves the calibration accuracy.

[0070] In some optional embodiments, the step of acquiring the first image information and the first point cloud data of the target calibration plate at the same posture and different positions includes:

[0071] Identify the target calibration plate area within the viewing range of the depth camera;

[0072] Control the robotic arm to collect first image information and first point cloud data of a first target point on the target calibration plate in the same posture at different height planes;

[0073] Among them, the first target points include multiple ones.

[0074] Specifically, in this application, it is first determined whether the target calibration plate area is within the field of view of the depth camera, and it is determined that the depth camera can capture the target calibration plate. Then, the processor can be used to control the robotic arm to keep the posture of the end of the robotic arm unchanged, and move the robotic arm in different height planes. Figure 2 Images are collected at points P1-P8.

[0075] In some optional embodiments, extracting the corner points of the calibration plate in the first image information includes:

[0076] According to preset calibration plate corner point parameters, the calibration plate corner points are extracted from the first image information.

[0077] It is understandable that in this application, the calibration plate corner point parameters can be pre-set, and then the calibration plate corner points are extracted according to the calibration plate corner point parameters. The collected image must ensure that it contains the same calibration plate corner points p Ci , whose coordinates in the camera coordinate system are p Ci =[x i y i z i ] T .

[0078] In some embodiments, the method of using a robotic arm to control a depth camera to collect second image information and second point cloud data of the target calibration plate in different postures includes:

[0079] Identify the target calibration plate area within the viewing range of the depth camera;

[0080] Control the robotic arm to collect second image information and second point cloud data of a second target point on the target calibration plate at different postures;

[0081] Wherein, the second target point includes multiple ones.

[0082] In some optional embodiments, the first image information and the second image information contain the same calibration plate corner points.

[0083] Specifically, when acquiring the second image information, the robot arm is controlled to move at random points in space in different postures such as Figure 2 The collected images must contain the same calibration plate corner points p Cj , whose coordinates in the camera coordinate system are p Cj =[x j y j z j ] T .

[0084] In some optional embodiments, the constructing the hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate includes:

[0085] Determine the first three-dimensional coordinates and the second three-dimensional coordinates of the corner point of the same calibration plate;

[0086] Performing homogeneous transformation on the first three-dimensional coordinates and the second three-dimensional coordinates respectively to obtain transformation equations;

[0087] The three-dimensional coordinates in the transformation equation are converted into the robot base coordinate system to obtain the expansion equation;

[0088] Based on the expansion equation, an equivalent transformation is performed on the hand-eye matrix to obtain a column vector hand-eye matrix;

[0089] The second image information and the corresponding robot arm posture data are input into the column vector hand-eye matrix to obtain the hand-eye matrix equation.

[0090] Specifically, in this application, Cj With p Ci is the point cloud coordinates of the same corner point of the chessboard in different camera coordinate systems. According to the principle of homogeneous transformation, the transformation formula is obtained

[0091]

[0092] Because P Bi and P Bj is the coordinate of the same corner point of the chessboard in the base coordinate system, so P B1 =P B2 ;

[0093] Then P B1 and P B2 Convert to the robot arm coordinate system to get the expansion formula:

[0094]

[0095] in, is the rotation matrix from the base coordinate system of the robot to the end coordinate system in a certain posture. is the rotation matrix from the base coordinate system of the robot to the end coordinate system in a certain posture; is the rotation amount of the hand-eye matrix; is the translation of the hand-eye matrix; is the rotation matrix from the base coordinate system of the robot to the end coordinate system in another posture; is the rotation matrix from the base coordinate system of the robot to the end coordinate system in another posture.

[0096] Among them, the left-hand rotation is analyzed, and the hand-eye matrix

[0097] In this application, the hand-eye matrix is ​​changed into a column vector form. Specifically, the unknown term of the rotation amount is equivalently transformed into a column vector form, and the coefficient matrix dimension is modified to perform an equivalent transformation to obtain:

[0098]

[0099] For the translation amount, continue to do equivalent transformation to get

[0100]

[0101] Through the above transformation steps, all unknown quantities of the hand-eye matrix are written into column vector form. Therefore, the coordinates of the same chessboard corner point in different positions of the robot arm are:

[0102]

[0103] make

[0104] The equation is equivalent to AX=B. The pixel data in multiple images and the corresponding robot arm posture data can be used to construct the following linear equation:

[0105]

[0106] In some embodiments, a linear algorithm is used to solve the hand-eye matrix equation.

[0107] Solving the equation, we get

[0108] X=(A T A) -1 ·A T ·B (9)

[0109] In some optional embodiments, before extracting the calibration plate corner points in the first image information or extracting the calibration plate corner points in the second image information, the method further includes:

[0110] The depth camera is calibrated according to the first image information and the second image information of the calibration plate, and the first point cloud data and the second point cloud data are dedistorted to obtain corrected data.

[0111] Specifically, in the present application, after obtaining the first image information and the second image information, the depth camera can be calibrated using the same calibration plate corner points or other methods, thereby dedistorting the first point cloud data and the second point cloud data to obtain corrected data, thereby making the point cloud data of the camera more accurate.

[0112] like Figure 3 As shown, an embodiment of the present application provides a hand-eye calibration device, the device comprising:

[0113] A first acquisition module 301 is used to control the depth camera to identify the target calibration plate using a mechanical arm, and then acquire first image information and first point cloud data of the target calibration plate at the same posture and different positions;

[0114] A first extraction module 302, configured to extract corner points of a calibration plate in the first image information, and determine first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data;

[0115] The second acquisition module 303 is used to use the robot arm to control the depth camera to acquire the second image information and the second point cloud data of the target calibration plate in different postures;

[0116] A second extraction module 304 is used to extract the corner points of the calibration plate in the second image information, and determine the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data;

[0117] A construction module 305, for constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate;

[0118] The calibration module 306 is used to solve the hand-eye matrix equation to complete the hand-eye calibration of the robotic arm and the depth camera.

[0119] The hand-eye calibration device provided in the present application comprises a first acquisition module 301 which uses a robotic arm to control a depth camera to identify a target calibration plate, and then obtains first image information and first point cloud data of the target calibration plate at the same posture and different positions; a first extraction module 302 extracts the corner points of the calibration plate in the first image information, and determines the first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data; a second acquisition module 303 uses a robotic arm to control a depth camera to obtain second image information and second point cloud data of the target calibration plate at different postures; a second extraction module 304 extracts the corner points of the calibration plate in the second image information, and determines the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data; a construction module 305 constructs a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the same calibration plate corner points; a calibration module 306 solves the hand-eye matrix equation to complete the hand-eye calibration of the robotic arm and the depth camera.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0121] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0122] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0123] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the hand-eye calibration method. For example, in some embodiments, the hand-eye calibration method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the hand-eye calibration method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the hand-eye calibration method in any other appropriate manner (e.g., by means of firmware).

[0125] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0132] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0133] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A hand-eye calibration method, characterized in that: The method comprises: After the depth camera is controlled by the robotic arm to identify the target calibration plate, first image information and first point cloud data of the target calibration plate are acquired at different positions with the same posture; Extracting corner points of a calibration plate from the first image information, and determining first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data; Using a robotic arm to control a depth camera to collect second image information and second point cloud data of the target calibration plate in different postures; Extracting the corner points of the calibration plate in the second image information, and determining the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data; Constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner points of the same calibration plate; Solve the hand-eye matrix equation to complete the hand-eye calibration of the robotic arm and the depth camera.

2. The method according to claim 1, characterized in that The step of acquiring the first image information and the first point cloud data of the target calibration plate at different positions with the same posture includes: Identify the target calibration plate area within the viewing range of the depth camera; Control the robotic arm to collect first image information and first point cloud data of a first target point on the target calibration plate in the same posture at different height planes; Wherein, the first target point includes multiple ones.

3. The method according to claim 1, characterized in that The extracting the corner points of the calibration plate in the first image information includes: According to preset calibration plate corner point parameters, the calibration plate corner points are extracted from the first image information.

4. The method according to claim 1, characterized in that: The method of using a robotic arm to control a depth camera to collect second image information and second point cloud data of the target calibration plate in different postures includes: Identify the target calibration plate area within the viewing range of the depth camera; Control the robotic arm to collect second image information and second point cloud data of a second target point on the target calibration plate at different postures; Wherein, the second target point includes multiple ones.

5. The method according to claim 1, characterized in that The first image information and the second image information contain the same calibration plate corner points.

6. The method according to claim 1, characterized in that The method of constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate includes: Determine the first three-dimensional coordinates and the second three-dimensional coordinates of the corner point of the same calibration plate; Performing homogeneous transformation on the first three-dimensional coordinates and the second three-dimensional coordinates respectively to obtain transformation equations; The three-dimensional coordinates in the transformation equation are converted into the robot base coordinate system to obtain the expansion equation; Based on the expansion equation, an equivalent transformation is performed on the hand-eye matrix to obtain a column vector hand-eye matrix; The second image information and the corresponding robot arm posture data are input into the column vector hand-eye matrix to obtain the hand-eye matrix equation.

7. The method according to claim 6, characterized in that A linear algorithm is used to solve the hand-eye matrix equation.

8. The method according to claim 1, characterized in that Before extracting the calibration plate corner points in the first image information or extracting the calibration plate corner points in the second image information, the method further includes: The depth camera is calibrated according to the first image information and the second image information of the calibration plate, and the first point cloud data and the second point cloud data are dedistorted to obtain corrected data.

9. The method according to claim 1, characterized in that: The depth camera is arranged on the robotic arm.

10. A hand-eye calibration device, characterized in that: The device comprises: A first acquisition module is used to control the depth camera to identify the target calibration plate using a mechanical arm, and then acquire first image information and first point cloud data of the target calibration plate at different positions with the same posture; a first extraction module, configured to extract corner points of a calibration plate from the first image information, and determine first three-dimensional coordinates of the corner points of the calibration plate based on the first point cloud data; A second acquisition module is used to use a robotic arm to control the depth camera to acquire second image information and second point cloud data of the target calibration plate in different postures; a second extraction module, configured to extract the corner points of the calibration plate in the second image information, and determine the second three-dimensional coordinates of the corner points of the calibration plate based on the second point cloud data; A construction module, used for constructing a hand-eye matrix equation based on the first three-dimensional coordinates, the second three-dimensional coordinates and the hand-eye matrix of the corner point of the same calibration plate; The calibration module is used to solve the hand-eye matrix equation and complete the hand-eye calibration of the robotic arm and the depth camera.

Images