Mechanical arm hand-eye calibration method and system based on 3D camera

Abstract

The invention relates to the technical field of machine vision, aims at the field of robot system application, and provides a mechanical arm hand-eye calibration method and system based on a 3D camera. A calibration plate with a single-feature pattern is installed at the tail end of a mechanical arm, the calibration plate can synchronously move along with the tail end of the mechanical arm, a 3D camera is installed under a base coordinate system of the mechanical arm, after the moving range of the mechanical arm is determined, the mechanical arm is automatically controlled to move m times, calibration plate image information under different postures is collected m times, and the calibration plate image information is obtained. And selecting n times of image information to calculate a calibration matrix X so as to complete the calibration of the mechanical arm and the 3D camera. According to the invention, calibration is carried out based on the calibration board of the single feature pattern, the design is simple and the cost is low in hardware structure, the attitude information of the calibration board does not need to be calculated in algorithm, the internal reference of the camera does not need to be calibrated, the calculation process is simplified, the calculation complexity is reduced, and the automatic calibration process is realized.

Description

[0001] This application is a divisional application of the patent application with the original application number CN202210970848.1, the application date of August 13, 2022, and the invention name of "Calibration method, system, electronic device and storage medium for robotic arm and 3D camera". Technical Field

[0002] The present application relates to machine vision technology and is aimed at the application field of robot systems. It proposes a 3D camera-based robotic arm hand-eye calibration technology solution for calibration between the robotic arm and the 3D camera, including a calibration method, a calibration system, an electronic device and a storage medium. Background Art

[0003] The robot system is generally composed of a visual sensor, a robotic arm system, and a main control computer. If the robotic arm system and the visual sensor are to achieve precise coordination, hand-eye calibration must be done. The calibration technology of the robotic arm and the 3D camera plays an important role in the field of industrial automation, and is widely used in guiding positioning, disordered grasping, etc. Hand-eye calibration is the matching relationship between the position information obtained by the calibration camera and the position information of the robotic arm, that is, the conversion relationship between the position in the image (2D) or point cloud (3D) and the robotic arm calibration position (such as tool coordinates or global coordinates). Existing calibration technologies usually use more complex calibration plates for calibration. In this type of calibration process, there are not only problems with the complex structure of the calibration plate and the large amount of calibration calculation, but also problems with the inability to achieve automatic calibration, which makes the robotic arm's automation capability insufficient. Summary of the invention

[0004] In order to overcome at least one problem or shortcoming in the prior art, the present application proposes a 3D camera-based robotic arm hand-eye calibration technology solution for calibration between a robotic arm and a 3D camera, including a calibration method, a calibration system, an electronic device and a storage medium.

[0005] 1. Calibration method

[0006] One of the purposes of the present application is to provide a calibration method, which is applied between a robot arm and a 3D camera, and mainly includes the following steps: calibrating the end tool coordinate system of the robot arm, calibrating the end tool coordinates of the robot arm to the center position of the feature point of the calibration plate, wherein the feature point includes a single feature pattern; the calibration plate is installed at the end of the robot arm and can move synchronously with the end of the robot arm; moving the robot arm so that the center position of the feature point is at or near the center position of the field of view of the 3D camera; the 3D camera is fixed at a fixed position in the base coordinate system of the robot arm; determining the moving range of the robot arm, and controlling the robot arm to move within the moving range; controlling the 3D camera to collect image information of the calibration plate in different postures m times; selecting the 2D image of the calibration plate collected n times and the 3D point cloud image corresponding to the 2D image from the m times of the image information collected, wherein m≥n≥4; extracting the spatial coordinates of the feature points in each of the 2D images in the base coordinate system , and its spatial coordinates in the corresponding 3D point cloud image ; , Indicates Second image acquisition; wherein, in calculating the spatial coordinates in the 3D point cloud image When processing is performed based on the depth map structure separated by the xyz components, the center point coordinates of the marker point in the 2D grayscale image are used as the reference, and the point cloud structure separated by the X, Y and Z direction components is used to take the X, Y and Z direction values ​​corresponding to the feature point in the 3D point cloud image as the spatial coordinates The value of; according to the spatial coordinates of the feature points under different postures and its corresponding spatial coordinates , calculate the calibration matrix .

[0007] The calibration method proposed in the present invention performs hand-eye calibration based on a calibration plate with a single feature point. The design is simple and the cost is lower. The specification requirements for the calibration plate are reduced. Only the position of a single feature point needs to be calculated, and the posture information of the calibration plate does not need to be calculated. There is no need to calibrate the internal parameters of the 3D camera, which simplifies the operation process and reduces the calculation complexity.

[0008] Based on the above calibration method, preferably, the calibration matrix can also be calculated After that, according to the calibration matrix Calculate the residual values ​​of the feature points collected m times, and regard the feature points whose residual values ​​are within the preset residual threshold range as valid points, otherwise as invalid points; count the number of valid points, and if the number of valid points exceeds the preset proportion, determine the calibration matrix obtained by this calculation Valid, otherwise the calibration matrix is ​​determined is invalid.

[0009] Based on the above calibration method, it is preferred to determine the calibration matrix After being invalid, all invalid points of the feature points collected m times are deleted to form a new valid point set. According to the spatial coordinates of each feature point in the valid point set and its corresponding spatial coordinates , recalculate the calibration matrix .

[0010] Based on any of the above calibration methods, it is preferred that the movement range of the robotic arm in the X-axis direction and the Y-axis direction can be automatically calculated according to the center position of the feature point and the distance between the center position and the boundary of the field of view of the 3D camera; the movement range of the robotic arm in the Z-axis direction can be automatically calculated according to the depth of field of the 3D camera; the robotic arm is automatically controlled to move m times within the movement range in the X-axis direction, the Y-axis direction and the Z-axis direction; and the 3D camera is controlled to collect image information of the calibration plate in the m different postures.

[0011] Based on any of the above calibration methods, preferably, when controlling the 3D camera to collect image information of the calibration plate in m different postures, it is also possible to automatically determine whether the collected image information is qualified. If it is unqualified, the 3D camera is controlled to re-collect the image information or the robotic arm is controlled to move and collect again. The method of automatically determining whether the collected image information is qualified includes but is not limited to: A. checking the clarity of the feature points in the collected image information; B. checking whether the feature points are beyond the field of view of the 3D camera; C. checking whether the feature points are beyond the depth of field of the 3D camera.

[0012] 2. Calibration system

[0013] One of the purposes of the present application is to provide a calibration system, which includes a robotic arm, a 3D camera, a calibration plate and a control unit; the calibration plate is installed at the end of the robotic arm and can move synchronously with the end of the robotic arm, and a single feature pattern is provided on the calibration plate as a feature point; the 3D camera is fixed at a fixed position in the base coordinate system of the robotic arm; the control unit is connected to the robotic arm and the 3D camera signal, and is used to execute and store a computer program. When the computer program is executed by the control unit, the robotic arm is controlled to move within an automatically determined moving range, the 3D camera is controlled to collect image information, and the operation corresponding to any one of the calibration methods described in the first part is implemented.

[0014] Based on the above calibration system, preferably, the calibration plate can be mainly composed of a calibration substrate and a calibration template; the front of the calibration substrate is in a rectangular shape, a calibration area is provided in the upper part of the front, and an installation area is provided in the lower part of the front; the installation area includes one or more mounting holes matching the flange at the end of the robotic arm; the calibration plate is installed at the end of the robotic arm through the mounting holes; the characteristic point of the calibration template is a single characteristic pattern, which is fixed in the calibration area.

[0015] 3. Electronic equipment

[0016] One of the purposes of the present application is to provide an electronic device. The electronic device may mainly include a processor and a memory; the memory is used to store a computer program; the processor is used to communicate interactively with a robotic arm and a 3D camera, and when executing the computer program, control the robotic arm to move within an automatically determined movement range, control the 3D camera to collect image information, and implement operations corresponding to any calibration method described in the first part.

[0017] 4. Computer Program Storage Media

[0018] One of the purposes of the present application is to provide a computer-readable storage medium for storing a computer program, and when a computer reads the computer program in the storage medium, the computer executes the operation corresponding to any one of the calibration methods described in the first part.

[0019] In summary, due to the adoption of the above technical scheme, the beneficial effects of the present invention include: the 3D camera-based robotic arm calibration method, system, electronic device and storage medium proposed in the present invention are calibrated based on a calibration plate with a single feature pattern, the hardware structure is simple in design and lower in cost, and the specification requirements for the calibration plate are reduced. In terms of algorithm, only the position of a single feature point needs to be calculated, and there is no need to calculate the posture information of the calibration plate, and there is no need to calibrate the internal parameters of the 3D camera, which simplifies the calculation process, reduces the calculation complexity, and realizes the automatic calibration operation. The robotic arm can automatically determine its movement range and automatically collect the image information required m times. It should be noted that different embodiments of the present application may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of several mentioned in the present application, or any other beneficial effects that may be obtained but are not exhaustively described. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, without paying creative labor, the solutions shown in these drawings can be replaced, adjusted, combined, etc. to create different technical solutions; the present application can also be applied to other similar scenarios based on these drawings to obtain application solutions for other scenarios.

[0021] in:

[0022] Figure 1 A schematic diagram of a calibration process according to some embodiments of the present application;

[0023] Figure 2 A simplified schematic diagram of a robotic arm workbench according to some embodiments of the present application;

[0024] Figure 3 A schematic diagram of an image acquisition process according to some embodiments of the present application;

[0025] Figure 4 A three-dimensional schematic diagram of a calibration plate structure according to some embodiments of the present application;

[0026] Figure 5 A front schematic diagram of a calibration plate structure according to some embodiments of the present application;

[0027] Figure 6 A side schematic diagram of a calibration plate structure according to some embodiments of the present application.

[0028] In the figure: 100-calibration plate, 101-calibration area, 102-installation area, 103-calibration template, 104-single feature pattern, 105-center position mark, 106-installation hole, 107-calibration substrate, 200-3D camera, 300-mechanical arm;

[0029] It should be noted that the same reference numerals in the figures represent the same structure or operation. Similar reference numerals and letters represent similar items in the drawings of this application. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. Because this application has many drawings and reference numerals, if there is a discrepancy between the description of the drawings and the illustrations of the drawings in the specification, those skilled in the art should understand it based on the logic of the technical principles recorded in this application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, and it is not possible to exhaustively describe all the embodiments.

[0031] 1. Calibration method of robotic arm and 3D camera

[0032] This application proposes a robotic arm calibration method based on a 3D camera, such as Figure 2 The calibration method shown is applied between the robot arm and the 3D camera, such as Figure 1 The calibration method shown mainly includes the following steps:

[0033] First, before starting the calibration, a calibration plate with a single feature pattern of the feature point designed in this application is installed on the end axis of the robot arm. The calibration plate can move synchronously with the end of the robot arm and remain relatively static during the movement. The single feature pattern generally refers to a pattern with only a single feature point relative to array patterns such as a checkerboard, such as a single circular feature, square feature or other shape feature, but it can also be a single feature pattern formed by a small area pattern relative to a conventional large-area array pattern. As long as the feature point accounts for a relatively small proportion of the entire 3D camera's field of view, it is not easy to move beyond the field of view when the robot arm is moved automatically, for example, the effective area range of the single feature pattern does not exceed the width size of the robot arm, for example, the area of ​​the feature pattern does not exceed 1% / 5%10% of the 3D camera's field of view, etc., it can also be regarded as a single feature pattern. In some embodiments, a positioning mark can also be set at the center of the single feature pattern to locate the single feature pattern faster and more accurately. This application is based on a single feature pattern calibration plate, which can also make the entire calibration process simple and fast.

[0034] During the calibration process, the end tool coordinate system of the robot is first calibrated, and the end tool coordinates of the robot are calibrated to the center position of the feature point of the calibration plate; then the 3D camera is fixed in a fixed position in the base coordinate system of the robot, and it is ensured that the field of view of the 3D camera covers the arm span of the robot.

[0035] Then the robotic arm can be moved manually or automatically so that the center position of the feature point is at or near the center position of the field of view of the 3D camera. In actual work, it is not necessary to move the robotic arm so that the center position of the feature point is exactly at the center of the field of view of the 3D camera. A deviation value range can be set. As long as the center position of the feature point falls within the deviation value range of the center of the field of view, that is, it falls near the center of the field of view, the next step of calibration can be continued.

[0036] Next, the moving range of the robot arm in the field of view of the 3D camera is automatically determined, and the robot arm is controlled to move within the moving range; and the 3D camera is controlled to collect image information of the calibration plate in different postures m times; from the collected image information m times, the 2D image of the calibration plate collected n times and the 3D point cloud image corresponding to the 2D image are selected, where m≥n≥4; and the spatial coordinates of the feature points in each of the 2D grayscale images in the base coordinate system are further extracted. , and its spatial coordinates in the corresponding 3D point cloud image ; , Indicates The image is collected; finally, the spatial coordinates of the feature points under different postures are and its corresponding spatial coordinates , calculate the calibration matrix , complete the hand-eye calibration.

[0037] In some embodiments, the calibration matrix is ​​calculated After that, according to the calibration matrix Calculate the residual value of the feature points collected m times. The residual value in mathematical statistics refers to the difference between the actual observation value and the fitted calculation value. The feature points whose residual values ​​are within the preset residual threshold range are regarded as valid points, otherwise they are regarded as invalid points. Count the number of valid points. If the number of valid points exceeds the preset proportion, determine the calibration matrix obtained by this calculation. Valid, otherwise the calibration matrix is ​​determined is invalid.

[0038] In some embodiments, further, when the calibration matrix After being judged as invalid, all invalid points in the feature points collected m times are deleted to form a new valid point set. and its corresponding spatial coordinates , recalculate the calibration matrix Of course, it should be noted that the number of new valid point sets must also satisfy the requirement of being greater than or equal to n and greater than or equal to 4. Therefore, in actual operation, the value of m can be slightly larger, and as many collection times as possible can be performed, such as 16 times, 20 times, 24 times, 30 times, 36 times, 40 times, 50 times, etc., and can be collected multiple times at different heights, such as collecting 6 times at 3 different heights to obtain 18 sets of valid data, collecting 9 times at 4 different heights to obtain 36 sets of valid data, collecting 5 times at 6 different heights to obtain 30 sets of valid data, collecting 12 times at 3 different heights to obtain 36 sets of valid data, and so on.

[0039] In some embodiments, further, the movement range of the robotic arm in the X-axis and Y-axis directions can be automatically calculated based on the center position of the feature point and the distance between the center position and the boundary of the field of view of the 3D camera; and the movement range of the robotic arm in the Z-axis direction can be automatically calculated based on the depth of field of the 3D camera. At the same time, in some embodiments, the rotation angle range of the robotic arm on the X-axis and Y-axis can be further determined, for example, the rotation angle is between 0-45°, and there is no rotation on the Z axis, etc.

[0040] Then, the robotic arm is automatically controlled to move m times within the moving range in the X-axis direction, the Y-axis direction, and the Z-axis direction; in some embodiments, the operation of controlling the robotic arm to move m times may be a random control, that is, after the 3D camera determines that the feature point of the calibration plate is located at or near the center of the field of view, the next position point is randomly generated within the moving range of the robotic arm, and the moving operation may be performed during the gaps in interactive communication between the robotic arm, the 3D camera, and other units to improve the self-calibration efficiency.

[0041] Of course, in some embodiments, in addition to randomly controlling the movement of the robotic arm, the robotic arm can also be automatically controlled to move within the range of movement according to a certain rule. For example, in the process of controlling the 3D camera to collect the image information of the calibration plate under the m different postures, further restrictions can be imposed on the different postures, such as requiring that: the calibration plate follows the end of the robotic arm to move synchronously m times, and the feature points of the calibration plate are not collinear in space every 3 times. Or it can be further required that: the calibration plate follows the end of the robotic arm to move synchronously m times at different horizontal heights, and if the number of images collected at the same horizontal height is greater than or equal to 3, the feature points of each calibration plate in each image collected at the same horizontal height are not collinear. In some embodiments, n data are selected from the m image acquisitions to complete the calibration process, and n data can be randomly selected from the m data, or it can be required that each feature point in the selected n data is not collinear at the horizontal height and / or vertical height.

[0042] In some embodiments, it is also necessary to automatically determine whether the collected image information is qualified. If it is unqualified, the 3D camera is controlled to re-collect the image information or the robotic arm is controlled to move and collect again. The method of automatically determining whether the collected image information is qualified includes but is not limited to: A. Checking the clarity of the feature points in the collected image information; B. Checking whether the feature points are beyond the field of view of the 3D camera; C. Checking whether the feature points are beyond the depth of field of the 3D camera.

[0043] like Figure 3 As shown, Figure 3The flowchart of a specific embodiment of the robot arm automatically collecting data m times is shown. First, the 3D camera calculates the center coordinates of the feature point through the first shooting; then, according to the center coordinates of the feature point and the distance of the center coordinates of the feature point from the boundary of the field of view image of the 3D camera, the movement range of the robot arm in the X-axis and Y-axis directions is calculated; then, according to the depth of field of the 3D camera, the movement range of the robot arm in the Z-axis direction is determined, and at the same time, the rotation angle range of the robot arm in the X-axis and Y-axis is determined, and it does not rotate around the Z-axis; further, the next position point is randomly generated according to the first shooting point or the last shooting point, and the robot arm is automatically moved to the next position point during communication; then, the image information collected by the 3D camera is analyzed and processed to determine whether the collected image information is qualified: first, feature point detection is performed. If the feature point detection is successful, it is determined that the clarity of the collected image meets the requirements, otherwise the image information is collected again after the robot arm moves to the next position point. If no feature point is identified in the image, or the detected feature pattern area is smaller than the feature threshold, that is, the feature point exceeds the camera's field of view, the image needs to be re-collected. If there is a situation where the field of view is exceeded, the robot arm must be required to move to the next position to re-collect the image, otherwise it will always collect the image at the same wrong point, that is, it will enter an infinite loop; after the feature point detection is successful, continue to determine whether the feature point exceeds the depth of field of the 3D camera. The previous step is to detect whether the feature point is qualified in the X-axis and Y-axis directions. This step is to detect in the Z-axis direction, detect the center position of the feature point, map it to the corresponding area in the point cloud image according to the size of the single feature pattern, perform plane fitting in the point cloud image, calculate the distance from all point clouds to the plane, and calculate the thickness of the point cloud. If the thickness is greater than the depth of field threshold, it is determined that the feature point exceeds the depth of field, otherwise it is deemed not to exceed. Finally, after passing the qualification test, calculate the point cloud coordinate XYZ value of the feature point, and add 1 to the acquisition times m. When the m value reaches the preset acquisition times, stop moving the robot arm.

[0044] After ensuring that m sets of image information are collected, record the current position information of the feature points of the corresponding calibration plate at the end of the robotic arm , and save the corresponding 2D image and 3D point cloud image, and calculate the 3D spatial coordinates of the current feature point based on these two images In some embodiments, the spatial coordinates of the feature points in the 3D point cloud image are The calculation method can be based on the depth map structure separated by xyz components. The center point coordinates of the marker point in the 2D grayscale image can be used as the reference to obtain the corresponding X, Y, and Z values ​​in the depth image separated by xyz components captured by the 3D camera as the corresponding spatial coordinates of the feature point in the 3D point cloud image. For example, based on the two-dimensional coordinates (u, v) of the center position of the feature point in the 2D image, find the values ​​of the positions corresponding to the points (u, v), (u, v+h), and (h, w+2h) in the 3D point cloud image to form the spatial coordinates. The (x, y, z) values ​​of the 2D image are represented by h, w, and h represents the height of the 2D image. In general, the depth map structure separated by xyz components is as wide as the 2D image, and its height can be three times that of the 2D image.

[0045] In calculating the calibration matrix In the process, a centroid calculation program can be set up according to the point cloud conversion formula To calculate the calibration matrix , such as using Principle to solve the calibration matrix .

[0046] In some embodiments, the spatial coordinates of each feature point in the camera coordinate system are calculated under different postures. The centroid coordinate position , , and calculate the spatial coordinates of each feature point in the base coordinate system under different postures The centroid coordinate position , , so as to obtain the centroid coordinate position of each feature point in the 2D image And the centroid coordinate position of each feature point in the depth map .

[0047] After the centroid calculation procedure, a decentralization step may also be performed. In some embodiments, the calculation calibration matrix The process also includes a decentralization step: according to the centroid coordinate position , calculate the point cloud coordinate position of each feature point after decentralization in the camera coordinate system , ; According to the centroid coordinate position , calculate the point cloud coordinate position of each feature point after decentralization in the base coordinate system , .

[0048] Next, after decentralization, singular value decomposition can be performed. Singular value decomposition (SVD) is a matrix decomposition method. Unlike eigendecomposition, SVD does not require that the matrix to be decomposed must be a square matrix. In some embodiments, the calculation calibration matrix The process also includes the singular value decomposition step: all the decentralized point cloud coordinate positions Convert to matrix , ; All decentralized point cloud coordinate positions Convert to matrix , ; Then, through the matrix and matrix Calculate the matrix , ; for the matrix Perform singular value decomposition to obtain the left singular value component and the right singular value components , ;in, is the transpose operation; is a matrix with these multiple eigenvalues ​​as the main diagonal, that is, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; and are all unitary matrices.

[0049] Furthermore, the calculation calibration matrix The process may also include a rotation and translation matrix calculation step: according to the left singular value component and the right singular value components , calculate the rotation transformation matrix ;

[0050]

[0051] in, Indicates the value of the determinant to be solved.

[0052] Then according to the rotation transformation matrix , center of mass coordinate position and the centroid coordinate position , the translation matrix can be calculated , .

[0053] Finally, the rotation transformation matrix is and translation matrix Composition of calibration matrix .

[0054] It should be noted that the order of description of the above-mentioned procedures and steps is only used to illustrate several aspects of the calibration process, and does not mean that this application limits the order of the processing flow. Some of these steps do have a sequence due to logical relationships, but some processes can be processed sequentially or simultaneously. This is something that people in this field can understand or adjust according to actual needs.

[0055] 2. Calibration system of robotic arm and 3D camera

[0056] The calibration system proposed in this application, such as Figure 2 As shown, it is mainly composed of a mechanical arm 300, a 3D camera 200, a calibration board 100 and a control unit. The calibration system can implement operations corresponding to the calibration method disclosed in the first part, and has a system architecture and system functions corresponding to any embodiment of the calibration method described in the first part.

[0057] Wherein, the calibration plate is mounted at the end of the manipulator and can move synchronously with the end of the manipulator, and a single characteristic pattern is provided on the calibration plate as a characteristic point; the 3D camera is fixed at a fixed position in the base coordinate system of the manipulator; the control unit is connected to the manipulator and the 3D camera signal, and is used to execute and store a computer program. When the computer program is executed by the control unit, the calibration system implements the operation corresponding to the calibration method described in any one of the first parts. In some embodiments, the 3D camera can be a 3D array camera.

[0058] In some embodiments, Figure 4-6 As shown, the calibration plate 100 proposed in the present application can be mainly composed of a calibration substrate 107 and a calibration template 103. The front of the calibration substrate 107 is flat and can be rectangular in shape. The upper part of the front is provided with a calibration area 101, and the lower part of the front is provided with a mounting area 102; wherein the mounting area 102 includes one or more mounting holes 106 that match the flange at the end of the mechanical arm 300; the calibration template 103 is fixed in the calibration area 101, and the characteristic point on the calibration template 103 can be a single characteristic pattern 104. The calibration plate designed in the present application is rectangular in shape, with one end being the calibration area and the other end being the mounting area. Compared with conventional calibration plates, it has a simple structure, is easy to install, and has low cost.

[0059] In some embodiments, the center of the single feature image may be provided with a center position mark 105 for quick positioning. For example, the single feature image may include a circular feature point, and a cross-marked center position mark 105 may be provided at the center of the circular feature point.

[0060] In some embodiments, the mounting holes in the mounting area can be flange mounting holes that match the flange at the end of the robotic arm, so as to be mounted at the end of the robotic arm. Furthermore, when the mounting area includes multiple mounting holes, the multiple mounting holes can be arranged in a concentric circle distribution, and the concentric circles match the flange at the end of the robotic arm, so that all the mounting holes of the calibration plate can be matched and fixedly connected with the flange at the end of the robotic arm, or only one or more mounting holes can be matched and connected with the flange according to certain angles and lengths. In addition, if Figure 6 The side of the calibration substrate shown may also be provided with one or more mounting holes for easy side mounting. Of course, one or more mounting holes may be provided on both the left and right sides of the calibration substrate.

[0061] In some embodiments, the calibration substrate of the present application may be made of metal, such as an aluminum substrate, an alloy substrate, etc. Further, the calibration template may be fixed in the calibration area by pasting, painting, adsorption, snapping, clamping or screwing.

[0062] The present application installs a calibration plate with a single feature pattern at the end of the robotic arm so that the calibration plate can move synchronously with the end of the robotic arm, and after installing a 3D camera in the base coordinate system of the robotic arm, obtains n 2D grayscale images of the calibration plate in different postures captured by the 3D camera, and calculates the spatial coordinates of each feature point in the depth image based on the depth map structure separated by the xyz components, and calculates the calibration matrix X through the coordinates of each feature point in the 2D grayscale image and the depth map. The present invention performs hand-eye calibration based on a calibration plate with a single feature point, has a simple design and low cost in hardware structure, only needs to calculate the position of a single feature point in the algorithm, does not need to calculate the posture information of the calibration plate, does not need to calibrate the internal parameters of the 3D camera, simplifies the operation process, and reduces the calculation complexity.

[0063] 3. Electronic equipment

[0064] The electronic device proposed in this application mainly includes a processor and a memory.

[0065] The memory is used to store computer programs; the memory can be but is not limited to: random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0066] The processor is used to communicate interactively with the robot arm and the 3D camera, and when executing the computer program, implements the operation corresponding to the calibration method described in any one of the first parts. The processor can be an integrated circuit chip with signal processing capabilities. It can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0067] It should be noted again that the electronic device described in this section corresponds to the calibration method described in the first section of this application, has corresponding software function modules and computer programs, can complete the operations corresponding to all the embodiments included in the winding detection method, and also has various corresponding embodiments, which will not be described in detail in this section.

[0068] 4. Computer Program Storage Media

[0069] If the functions described in this application are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0070] The present application proposes a computer-readable storage medium, wherein the storage medium is used to store a computer program, and when a computer reads the computer program in the storage medium, the computer runs operations corresponding to the calibration method described in the first part.

[0071] A computer-readable storage medium may include a propagated data signal containing computer program code, such as on a baseband or as part of a carrier wave. The propagated signal may have a variety of forms, including electromagnetic, optical, etc., or a suitable combination of forms. The storage medium may be, but is not limited to: a floppy disk, an optical disk, a hard disk, a USB flash disk (flash memory, USB flash disk), a TF card (T-Flash Card, also known as MicroSD card), an SD card (Secure Digital Memory Card), an MMC card (Multi Media Card), an SM card (Smart Media Card), a memory stick (Memory Stick Card), an XD card, a CF card (Compact Flash Card), etc.

[0072] The medium can be connected to an instruction execution system, device or apparatus to enable communication, propagation or transmission of the program for use. The program code on the computer storage medium can be transmitted through any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0073] The computer program codes required for the operation of each part of the present application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages, etc. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0074] The basic concepts and working principles have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only used as an example and does not constitute a limitation of the present application. Although it is not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to the present application. Such modifications, improvements and corrections are suggested in the present application, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present application. At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment" or "some embodiments" means a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "some embodiments" or "one embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0075] As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "a", "a", "a", "a", and / or "the" do not refer to the singular in quantity, but are a description used to distinguish and classify. Generally speaking, the terms "comprises" and "includes" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0076] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A 3D camera-based robotic arm hand-eye calibration method, characterized in that: The method comprises the following steps: Calibrate the end tool coordinate system of the robot arm, calibrate the end tool coordinate of the robot arm to the center position of the feature point of the calibration plate, the feature point includes a single feature pattern; the calibration plate is installed at the end of the robot arm and can move synchronously with the end of the robot arm; The mechanical arm is moved so that the center position of the feature point is at or near the center position of the field of view of the 3D camera; the 3D camera is fixed at a fixed position in the base coordinate system of the mechanical arm; Determine the moving range of the mechanical arm, and control the mechanical arm to move within the moving range; Control the 3D camera to collect image information of the calibration plate in different postures m times; From the image information collected m times, select the 2D image of the calibration plate collected n times and the 3D point cloud image corresponding to the 2D image, where m≥n≥4; Extract the spatial coordinates of the feature points in each of the 2D images in the base coordinate system , and its spatial coordinates in the corresponding 3D point cloud image ; , Indicates Second image acquisition; Wherein, in calculating the spatial coordinates in the 3D point cloud image When processing is performed based on the depth map structure separated by the xyz components, the center point coordinates of the marker point in the 2D grayscale image are used as the reference, and the point cloud structure separated by the X, Y and Z direction components is used to take the X, Y and Z direction values ​​corresponding to the feature point in the 3D point cloud image as the spatial coordinates The value of According to the spatial coordinates of feature points in different postures and its corresponding spatial coordinates , calculate the calibration matrix .

2. The calibration method according to claim 1, characterized in that: According to the spatial coordinates of each feature point under different postures and its corresponding centroid coordinate position , calculate the point cloud coordinate position of each feature point after decentralization in the base coordinate system ; According to the spatial coordinates of each feature point under different postures and its corresponding centroid coordinate position , calculate the point cloud coordinate position of each feature point after decentralization in the camera coordinate system ; After decentralization, the point cloud coordinate position and the point cloud coordinate position Perform singular value decomposition to obtain the left singular value component and the right singular value components ; According to the left singular value component and the right singular value components , calculate the rotation transformation matrix ; According to the rotation transformation matrix , the centroid coordinate position and the centroid coordinate position , calculate the translation matrix ; By the rotation transformation matrix and translation matrix Composition of calibration matrix .

3. The calibration method according to claim 1, characterized in that: The calibration matrix is ​​calculated Then, according to the calibration matrix Calculate the residual values ​​of the feature points collected m times, and regard the feature points whose residual values ​​are within a preset residual threshold range as valid points, otherwise as invalid points; Count the number of valid points. If the number of valid points exceeds the preset ratio, the calibration matrix calculated this time is determined. Valid, otherwise the calibration matrix is ​​determined is invalid.

4. The calibration method according to claim 2, characterized in that: When determining the calibration matrix After it is invalid, all invalid points of the feature points collected m times will be deleted to form a new valid point set. and its corresponding spatial coordinates , recalculate the calibration matrix .

5. The calibration method according to claim 1, characterized in that: Automatically calculate the movement range of the robot arm in the X-axis direction and the Y-axis direction according to the center position of the feature point and the distance between the center position and the boundary of the field of view of the 3D camera; Automatically calculating the movement range of the robotic arm in the Z-axis direction according to the depth of field of the 3D camera; Automatically controlling the robot arm to move m times within a moving range in the X-axis direction, the Y-axis direction, and the Z-axis direction; The 3D camera is controlled to collect image information of the calibration plate in the m different postures.

6. The calibration method according to claim 1, characterized in that: When controlling the 3D camera to collect image information of the calibration plate in m different postures, automatically judging whether the collected image information is qualified, and if unqualified, controlling the 3D camera to re-collect the image information or controlling the mechanical arm to move and collect again; The method of automatically judging whether the collected image information is qualified includes: A. Check the clarity of the feature points in the collected image information; B. Check whether the feature point is beyond the field of view of the 3D camera; C. Check whether the feature point exceeds the depth of field of the 3D camera.

7. A 3D camera-based robotic arm hand-eye calibration system, characterized by: The system comprises a mechanical arm (300), a 3D camera (200), a calibration plate (100) and a control unit; The calibration plate (100) is installed at the end of the mechanical arm (300) and can move synchronously with the end of the mechanical arm (300), and a single characteristic pattern is provided on the calibration plate (100) as a characteristic point; The 3D camera (200) is fixed at a fixed position in the base coordinate system of the mechanical arm (300); The control unit is connected to the robot arm (300) and the 3D camera (200) by signals, and is used to execute and store a computer program. When the computer program is executed by the control unit, the robot arm (300) is controlled to move within an automatically determined movement range, the 3D camera (200) is controlled to collect image information, and operations corresponding to the calibration method according to any one of claims 1 to 6 are implemented.

8. The calibration system according to claim 7, characterized in that: The calibration plate (100) mainly consists of a calibration substrate (107) and a calibration template (103); The front face of the calibration substrate (107) is in the shape of a rectangle, the upper part of the front face is provided with a calibration area (101), and the lower part of the front face is provided with a mounting area (102); The mounting area (102) comprises one or more mounting holes (106) matching the flange at the end of the mechanical arm (300); the calibration plate is mounted on the end of the mechanical arm (300) through the mounting holes; The characteristic point of the calibration template (103) is a single characteristic pattern (104) which is fixed in the calibration area (101).

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is used to interactively communicate with the robotic arm and the 3D camera, and when executing the computer program, controls the robotic arm to move within an automatically determined moving range, controls the 3D camera to collect image information, and implements operations corresponding to the calibration method as described in any one of claims 1-6.

10. A computer program storage medium, characterized in that: The storage medium is used to store a computer program. When a computer reads the computer program in the storage medium, the computer runs the calibration method as described in any one of claims 1 to 6.

Images