Hand-eye calibration method, visual robot, hand-eye calibration device and storage medium

Abstract

Embodiments of the present application disclose a hand-eye calibration method, which comprises: obtaining any reference calibration point in a reference point array determined in a calibration object, and a three-dimensional physical coordinate under a calibration object coordinate system; controlling movement of a mechanical arm of a vision robot to obtain light field images of the reference point array obtained by a light field camera of the vision robot at at least two poses; obtaining a target pose transformation relationship of the calibration object relative to the light field camera based on the at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array; and performing hand-eye calibration on the vision robot based on the target pose transformation relationship. Embodiments of the present application also disclose a vision robot, a hand-eye calibration device and a storage medium.

Description

Technical Field

[0001] This application relates to the fields of electronics and information technology, and in particular to a hand-eye calibration method, a visual robot, a hand-eye calibration device, and a storage medium. Background Technology

[0002] As industrial processes demand increasingly higher precision, the robotic arms of vision robots require high-precision vision sensors, such as cameras, as their "eyes" to coordinate and complete these processes. To ensure the grasping accuracy of the robotic arm, hand-eye calibration is necessary after installing the camera to obtain the camera's pose relative to the robotic arm. This establishes information conversion between the vision sensor and the vision robot, allowing the robot to convert visual information (such as visual images) of the calibrated object into positional information acceptable to the robot, thereby controlling the robotic arm to accurately grasp the calibrated object. Currently, hand-eye calibration methods require further research. Summary of the Invention

[0003] This application aims to provide a hand-eye calibration method, a visual robot, a hand-eye calibration device, and a storage medium.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a hand-eye calibration method, the method comprising:

[0006] Obtain the three-dimensional physical coordinates of any reference calibration point in the reference lattice determined in the calibration object, in the calibration object coordinate system;

[0007] Control the movement of the manipulator of the vision robot to obtain light field images of the reference dot array captured by the light field camera of the vision robot in at least two poses;

[0008] Based on at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array, the target pose transformation relationship of the calibration object relative to the light field camera is obtained.

[0009] Hand-eye calibration of the visual robot is performed based on the target pose transformation relationship.

[0010] This application provides a vision robot, the vision robot comprising:

[0011] A light field camera is used to capture images of light fields.

[0012] The robot body has a base and a robotic arm mounted on the base, the robotic arm being used to fix the object to be photographed;

[0013] A control device is used to obtain the three-dimensional physical coordinates of any reference calibration point in a defined reference point array of the calibration object in the calibration object coordinate system; control the movement of the manipulator of the vision robot to obtain light field images of the reference point array captured by the manipulator through the light field camera of the vision robot in at least two poses; obtain the target pose transformation relationship of the calibration object relative to the light field camera based on the at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array; and perform hand-eye calibration of the vision robot based on the target pose transformation relationship.

[0014] This application provides a hand-eye calibration device, the hand-eye calibration device comprising:

[0015] The memory is used to store the hand-eye calibration program;

[0016] The processor is used to execute the hand-eye calibration program stored in the memory to implement the hand-eye calibration method described above.

[0017] This application provides a storage medium, characterized in that the storage medium stores one or more programs, which can be executed by one or more processors to implement the hand-eye calibration method described above.

[0018] The hand-eye calibration method, visual robot, hand-eye calibration device, and storage medium provided in this application embodiment obtain the three-dimensional physical coordinates of any reference calibration point in a determined reference point array of the calibration object in the calibration object coordinate system; control the movement of the manipulator of the visual robot to obtain light field images of the reference point array captured by the light field camera of the visual robot in at least two poses; based on the at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array, obtain the target pose transformation relationship of the calibration object relative to the light field camera; perform hand-eye calibration on the visual robot based on the target pose transformation relationship; thus, based on the acquired light field images in different poses and the three-dimensional physical coordinates of any reference calibration point in the calibration object, the target pose transformation relationship of the calibration object relative to the light field camera is determined, thereby quickly completing the hand-eye calibration of the visual robot and improving the accuracy and efficiency of hand-eye calibration. Attached Figure Description

[0019] Figure 1 A schematic diagram of the device used in a hand-eye calibration device provided in an embodiment of this application;

[0020] Figure 2 A schematic diagram showing the end effector of the vision robot provided in this application moving to different poses;

[0021] Figure 3A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0022] Figure 4 A schematic diagram illustrating the selection of a reference dot matrix as provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram illustrating the principle of light field imaging provided in the embodiments of this application;

[0024] Figure 6 A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0025] Figure 7 A schematic diagram illustrating the light field information contained in light during propagation, provided in an embodiment of this application;

[0026] Figure 8 A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0027] Figure 9 A schematic diagram of the center view image and parallax image provided in the embodiments of this application;

[0028] Figure 10 A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0029] Figure 11 The embodiments of this application provide three-dimensional point cloud coordinates of all target calibration points in the camera coordinate system;

[0030] Figure 12 A schematic diagram showing the comparison between the calibration plate reconstruction results and the actual values, as well as the absolute distance error, provided in the embodiments of this application;

[0031] Figure 13 A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0032] Figure 14 A schematic diagram of the optical configuration and light propagation process of a combined light field camera provided for embodiments of this application;

[0033] Figure 15 A schematic diagram illustrating the projection center calibration of a microlens provided in an embodiment of this application;

[0034] Figure 16 A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0035] Figure 17 A schematic diagram of the point feature detection process provided in the embodiments of this application;

[0036] Figure 18 A schematic diagram illustrating the extraction process of the circle of confusion feature provided in an embodiment of this application;

[0037] Figure 19 A flowchart illustrating an optional hand-eye calibration method provided in this application embodiment;

[0038] Figure 20 A schematic diagram illustrating the point feature reprojection process provided in an embodiment of this application;

[0039] Figure 21 This is a schematic diagram of the structure of a vision robot provided in an embodiment of this application;

[0040] Figure 22 This is a schematic diagram of the structure of a hand-eye calibration device provided in an embodiment of this application. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0042] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0043] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0044] In related technologies, there are two types of hand-eye calibration: hand-eye calibration with the eye on the hand and hand-eye calibration with the eye outside the hand. The hand-eye calibration method provided in this application is applicable to both situations. Here, this application uses the case of "eye outside the hand" as an example to explain the basic principle of hand-eye calibration. (Refer to...) Figure 1 and Figure 2 As shown, the hand-eye calibration device uses the following equipment: a visual robot, a light field camera, and a calibration object. The calibration object is fixed to the robotic arm of the visual robot. The light field camera is fixed in the world coordinate system, and the light field image is captured on the calibration object fixed to the robotic arm of the visual robot. While ensuring that the light field camera can observe the calibration object, the robotic arm of the visual robot is controlled to move, thereby moving the calibration object to several positions. The pose of the visual robot at each position is recorded. Here, the camera coordinate system of the light field camera is denoted as {C}, the base coordinate system of the robot is denoted as {B}, and the coordinate system of the end effector (griper) at the i-th position is denoted as {G}. i The calibration coordinate system (world) at the i-th position is denoted as {W}. i It should be noted that the base coordinate system {B} of the vision robot and the end effector coordinate system {G} are different. i}, camera coordinate system {C} and calibration object coordinate system {W} i This forms a loop, through which transformations between different coordinate systems can be achieved. Here, the calibration object is fixed in the manipulator of the vision robot. When the end effector of the manipulator moves to two different positions G1 and G2, the calibration object moves to positions W1 and W2 along with the manipulator. Throughout the calibration process, since the pose transformation relationship bHc of the light field camera relative to the base and the pose transformation relationship gHw of the calibration object relative to the manipulator are fixed, the pose transformation relationship of the calibration object relative to the manipulator can be expressed as follows (Equation 1).

[0045] gHw=g1Hb·bHc·cHw1=g2Hb·bHc·cHw2 (Formula 1)

[0046] Where gHw represents the pose transformation relationship between the calibration object and the robot arm, gHb represents the pose transformation relationship between the base and the end effector of the robot arm, bHc represents the pose transformation relationship between the light field camera and the base, and cHw represents the pose transformation relationship between the calibration object and the light field camera.

[0047] Furthermore, by transforming (Formula 1), we obtain (Formula 2) as follows.

[0048] (g2Hb) -1 ·g1Hb·bHc=bHc·cHw2·(cHw1) -1(Formula 2)

[0049] Let A = (g2Hb) -1 ·g1Hb,B=cHw2·(cHw1) -1 X = bHc, (Formula 2) can be simplified to the form A·X = X·B. The problem of solving the hand-eye calibration problem is transformed into solving the problem of X in A·X = X·B. The calibration object is photographed in any two poses, and the pose transformation relationship gHb of the base relative to the end of the robot hand under different poses is obtained by reading the joint angles on the robot teach pendant and calculating the pose transformation relationship cHw of the calibration object relative to the light field camera. Then, based on the pose transformation relationship gHb of the base relative to the end of the robot hand and the pose transformation relationship cHw of the calibration object relative to the light field camera, X is solved, that is, the pose transformation relationship bHc of the light field camera relative to the base is solved.

[0050] Based on the aforementioned hand-eye calibration device and the principle of hand-eye calibration, embodiments of this application provide a hand-eye calibration method applied to visual robots, with reference to... Figure 3 As shown, the method includes the following steps:

[0051] Step 101: Obtain the three-dimensional physical coordinates of any reference calibration point in the reference lattice determined in the calibration object, in the coordinate system of the calibration object.

[0052] In this embodiment, the calibration object is a calibration board with a specific grid or pattern, such as a checkerboard calibration board. Calibration boards are widely used in machine vision, image measurement, photogrammetry, 3D reconstruction, and other fields. By photographing a flat plate with a fixed-spacing pattern array using a camera and performing calculations using a calibration algorithm, the geometric model of the camera can be obtained, thus achieving high-precision measurement and reconstruction results. The flat plate with the fixed-spacing pattern array is the calibration board.

[0053] In this embodiment of the application, the calibration object includes multiple calibration points, and the distance between any two adjacent calibration points is equal. For example, the distance between two adjacent calibration points can be 0.5 millimeters (mm).

[0054] In this embodiment, the reference point array is an array composed of a certain number of calibration points selected from all calibration points included in the calibration object, and the reference point array is used for calculation. A reference calibration point is any calibration point in the reference point array.

[0055] In practical applications, the calibration object coordinate system is a spatial rectangular coordinate system established with any calibration point on the calibration object as the origin, the left-right direction of the calibration object as the X-axis direction, the front-back direction of the calibration object as the Y-axis direction, and the up-down direction of the calibration object as the Z-axis direction. For example, refer to... Figure 4As shown, taking the calibration point located in the lower left corner of all calibration points of the calibration object as the origin, 9×12 calibration points are selected as the reference point matrix, and the three-dimensional physical coordinates of each reference calibration point in the reference point matrix in the calibration object coordinate system are determined, that is, the position coordinates of each reference calibration point in the calibration object coordinate system.

[0056] Step 102: Control the movement of the vision robot's manipulator to obtain light field images of the reference dot matrix captured by the vision robot's light field camera in at least two poses.

[0057] In this embodiment, the light field image is the original image obtained by the light field camera capturing the calibration object in at least two poses while the robot arm is fixed to the robot arm. Here, the light field image can be the original image obtained by the light field camera capturing the reference matrix in the calibration object fixed to the robot arm in at least two poses; the light field image can also be the original image obtained by the light field camera capturing the reference matrix in the calibration object in at least two poses while the light field camera is fixed to the robot arm. This application does not impose specific limitations on this.

[0058] Here, we explain the optical configuration of the light field camera, which consists of three parts: a front optical system, a microlens array, and a photoelectric sensor. The front optical system is a lens group that transmits light; the microlens array focuses and defocuses the collected light information; finally, the photoelectric sensor collects all the light information. For example... Figure 5 As shown in the schematic diagram of the light field imaging principle, the main lens group is replaced by a single main lens, with the microlens array placed at the focal plane of the main lens. The microlens array contains m×m independent microlenses, all arranged in a hexagonal structure, with each microlens covering n×n pixels. Here, the microlens array acts as a defocusing element, its purpose being to scatter the light emitted from the target onto several detector units after being focused by the main lens group, allowing the detectors to record light information from different positions and angles. Another function of the microlens array is to image the light from the main lens group onto the detector, covering several pixels on the detector; thus, the entire light from the main lens group can be divided into several sub-apertures by the detector units.

[0059] Step 103: Based on at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array, obtain the target pose transformation relationship of the calibration object relative to the light field camera.

[0060] In this embodiment, the target pose transformation relationship is used to calibrate the pose transformation of the calibration object relative to the light field camera.

[0061] In this embodiment, the visual robot controls the movement of its own manipulator to obtain light field images of the reference point array captured by the light field camera of the visual robot in at least two poses. Based on the light field images of the manipulator in at least two different poses and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array, the target pose transformation relationship of the calibration object relative to the light field camera is obtained.

[0062] Step 104: Perform hand-eye calibration on the visual robot based on the target pose transformation relationship.

[0063] In this embodiment, the visual robot obtains the target pose transformation relationship between the calibration object and the light field camera based on the three-dimensional physical coordinates corresponding to all reference calibration points in at least two light field images and a reference point array. Then, based on this target pose transformation relationship, the visual robot performs hand-eye calibration using the Tasi two-step method, thereby obtaining the pose transformation relationship between the light field camera and the base (eye to hand), or the pose transformation relationship between the light field camera and the robotic end effector (eye at hand). Here, the Tasi two-step method for hand-eye calibration of the visual robot is a commonly used technique, and this application will not elaborate on it.

[0064] The hand-eye calibration method provided in this application obtains the three-dimensional physical coordinates of any reference calibration point in a determined reference point array of the calibration object in the calibration object coordinate system; controls the movement of the manipulator of a vision robot to obtain light field images of the reference point array captured by the manipulator's light field camera in at least two poses; based on the at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array, obtains the target pose transformation relationship of the calibration object relative to the light field camera; performs hand-eye calibration on the vision robot based on the target pose transformation relationship; thus, based on the acquired light field images in different poses and the three-dimensional physical coordinates of any reference calibration point in the calibration object, the target pose transformation relationship of the calibration object relative to the light field camera is determined, thereby quickly completing the hand-eye calibration of the vision robot and improving the accuracy and efficiency of hand-eye calibration.

[0065] Embodiments of this application provide a hand-eye calibration method applied to visual robots, with reference to... Figure 6 As shown, the method includes the following steps:

[0066] Step 201: Obtain the three-dimensional physical coordinates of any reference calibration point in the reference lattice determined in the calibration object, in the coordinate system of the calibration object.

[0067] Step 202: Control the movement of the vision robot's manipulator to obtain light field images of the reference dot matrix captured by the vision robot's light field camera in at least two poses.

[0068] Step 203: Render each of the at least two light field images using a light field rendering algorithm to obtain a multi-view image corresponding to each light field image.

[0069] Among them, multi-view images include central view images.

[0070] In this embodiment of the application, multi-view images refer to images obtained when each light field image is illuminated by light at different incident angles. Here, refer to Figure 7 As shown, the light field is the light field information contained in a beam of light during its propagation. Therefore, the light field image includes all light field information. The light field information can be represented by a four-dimensional parameterized L(x,y,u,v). The light field information includes information such as light intensity, light position, and light direction. L represents the light intensity, (x,y) represents the coordinate information in the light field, i.e., the arrangement order of microlenses, and (u,v) represents the angle information in the light field, i.e., the macro-pixel arrangement.

[0071] In this embodiment, since the light field camera is equivalent to a small camera array, the light field image captured by the light field camera contains not only the spatial position information of the light in the shooting scene but also the incident angle information. Furthermore, the visual robot uses a light field rendering algorithm to decouple the information of the uv plane and xy plane of the light field camera, thereby obtaining multi-view images of each light field image at different incident angles in at least two light field images at different poses. It should be noted that the essence of the light field rendering algorithm is to rearrange the pixels at each position in the area (macropixel) covered by each microlens in the light field image, i.e., the macropixel pixel arrangement (u,v), according to the microlens arrangement order (x,y).

[0072] Step 204: Process the multi-view image corresponding to each light field image using the epipolar image algorithm to obtain the parallax image corresponding to each light field image.

[0073] In this embodiment, the parallax image is a two-dimensional image, which is used for the positioning of calibration points.

[0074] In this embodiment, the visual robot renders each of at least two light field images using a light field rendering algorithm to obtain a multi-view image corresponding to each light field image. Then, it processes the multi-view image corresponding to each light field image using an epipolar plane image algorithm to obtain a disparity image corresponding to each light field image. The basic idea of ​​the epipolar plane image (EPI) algorithm is that in four-dimensional light field data L(x,y,u,v), if the coordinates u and x are fixed, and the pixels at the (y,v) coordinates are rearranged according to their correspondence, an EPI image at the (y,v) coordinate can be generated. Similarly, if v and y in the four-dimensional light field data L(x,y,u,v) are fixed, and the pixels at the (x,u) coordinates are rearranged, an EPI image at the (x,u) coordinate can be generated. Then, based on the epipolar slope, an algorithm is used to calculate the corresponding disparity to obtain the estimated disparity value for each calibration point.

[0075] Step 205: Based on all reference calibration points in the central view image, locate the calibration points in the parallax image to obtain the target calibration points in the parallax image.

[0076] In this embodiment of the application, the target calibration point in the parallax image corresponds to the reference calibration point.

[0077] In this embodiment of the application, all target calibration points form a target dot matrix.

[0078] In this embodiment, step 205, based on all reference calibration points in the central view image, locates the calibration points in the disparity image to obtain the target calibration point in the disparity image. Figure 8 To provide further explanation,

[0079] Step 2051: Obtain the second pixel position of each reference calibration point in the image pixel coordinate system.

[0080] In this embodiment, the image pixel coordinate system is a corresponding coordinate system established on the central view image with a fixed pixel or calibration point in the central view image as the origin.

[0081] In this embodiment, the second pixel position refers to the position of the reference calibration point in the center view image in the image pixel coordinate system, not the actual distance position, but only indicates the row and column of the reference calibration point in the image pixel coordinate system.

[0082] Step 2052: Determine the calibration point corresponding to the second pixel position in the image pixel coordinate system of the disparity image as the target calibration point.

[0083] Here, refer to Figure 9 As shown, Figure 9 A in the image represents the center-view image. Figure 9 B in the diagram represents the disparity image. As can be seen, the positions of the calibration points in the central view image are the same as those in the disparity image. Therefore, the visual robot obtains the second pixel position of each reference calibration point in the central view image within the image pixel coordinate system. An image pixel coordinate system is then established in the disparity image with the corresponding calibration point in the central view image as its origin. Within this coordinate system, the calibration point in the disparity image corresponding to the second pixel position is determined as the target calibration point. This establishes a one-to-one correspondence between the target calibration point in the disparity image and the selected reference calibration point in the calibration object.

[0084] Step 206: Based on the first pixel position of the target calibration point in the image pixel coordinate system, obtain the three-dimensional point cloud coordinates of the target calibration point in the camera coordinate system.

[0085] In this embodiment of the application, step 206, which obtains the three-dimensional point cloud coordinates of the target calibration point in the camera coordinate system based on the first pixel position of the target calibration point in the image pixel coordinate system, is combined with... Figure 10 To provide further explanation,

[0086] Step 2061: Obtain the disparity value of each target calibration point in the disparity image.

[0087] Step 2062: Based on the first pixel position and disparity value, obtain the three-dimensional point cloud coordinates.

[0088] In this embodiment, the disparity image is a two-dimensional grayscale image. The grayscale value of each pixel (calibration point) in the disparity image represents depth information, and the disparity value is related to the depth information. The visual robot obtains the depth information of each target calibration point in the disparity image through light field calibration, and then obtains the disparity value of each target calibration point in the disparity image. Further, based on the first pixel position and disparity value of the target calibration point in the image pixel coordinate system, it is converted into the three-dimensional point cloud coordinates of the target calibration point in the camera coordinate system, thereby obtaining the three-dimensional point cloud coordinates of all target calibration points in the camera coordinate system. Figure 11 As shown, Figure 11 The diagram shows the 3D point cloud coordinates of all target calibration points in the camera coordinate system.

[0089] Step 207: Based on the three-dimensional point cloud coordinates corresponding to all target calibration points and the three-dimensional physical coordinates corresponding to the reference calibration points at the corresponding positions, obtain the target pose transformation relationship between the calibration object and the light field camera.

[0090] In this embodiment, the visual robot matches the three-dimensional point cloud coordinates of all target calibration points in the target point array in the camera coordinate system with the three-dimensional physical coordinates of the corresponding reference calibration points in the calibration object coordinate system, calculates the rigid body transformation between all the three-dimensional point cloud coordinates in the camera coordinate system and the three-dimensional physical coordinates in the calibration object coordinate system, and obtains the target pose transformation relationship of the calibration object relative to the light field camera.

[0091] Step 208: Perform hand-eye calibration on the visual robot based on the target pose transformation relationship.

[0092] In one feasible application scenario, a light field camera is fixed in the external environment, and a calibration plate with a high-precision dot matrix is ​​fixed to the end effector of a vision robot's manipulator. The spacing between adjacent calibration points on the calibration plate is 0.5 mm. (Refer to...) Figure 4 As shown, taking the point located in the lower left corner as the origin, 9×12 calibration points are selected as the reference point array. The end effector of the robot is moved to 13 different pose states, and the light field camera can capture the reference point array at each position, and the joint angle of the end effector of the robot is recorded.

[0093] Furthermore, after data acquisition is complete, the visual robot teach pendant can calculate the coordinate system {B} of the base relative to the coordinate system {G} of the robot end effector at each pose of the visual robot. i The pose transformation relationship of bHg i The object coordinate system {W} is calculated for each pose by point cloud matching. i Pose transformation relationship cHw relative to camera coordinate system {C} i Therefore, 12 sets of homogeneous transformation matrices A can be obtained for solving hand-eye calibration. N and B N (N = 1, 2, ..., 11). The calculated pose transformation relationship bHc of the light field camera relative to the base is:

[0094]

[0095] For example, the first set of pose data is used to verify the accuracy of the pose transformation relationship cHw between the calibration object and the camera calculated by point cloud matching. The target point matrix's corresponding 3D point cloud coordinates are reprojected onto the calibration object coordinate system using cHw, and compared with the reference point matrix's 3D physical coordinates in the calibration object coordinate system. The absolute distance between the reconstructed result and the actual value (3D physical coordinates) is used as the error metric. Figure 12 As shown, Figure 12 A in the diagram shows a comparison between the calibration plate reconstruction results and the actual values; Figure 12Figure B shows a schematic diagram of the absolute distance error between the reconstructed result and the actual value of the calibration plate. As can be seen from the figure, in the first set of pose transformation relationships, the maximum reconstruction error is 0.041 mm.

[0096] As described above, in this embodiment, a reference point array of the calibration object is captured by a light field camera, and the three-dimensional point cloud coordinates of the reference point array in the camera coordinate system are calculated. Given the point array spacing between adjacent calibration points in the calibration object, the origin can be determined to establish a calibration board coordinate system (z-direction coordinate is 0). Since the central view image and the parallax image correspond one-to-one, and the pixel values ​​on the parallax image correspond one-to-one with the three-dimensional point cloud, the central view image can be used for positioning. The three-dimensional point cloud coordinates corresponding to all target calibration points in the target point array in the camera coordinate system are matched with the three-dimensional physical coordinates of the corresponding reference calibration points in the calibration object coordinate system. The rigid body transformation between all the corresponding three-dimensional point cloud coordinates in the camera coordinate system and the corresponding three-dimensional physical coordinates in the calibration object coordinate system is calculated, obtaining the target pose transformation relationship of the calibration object relative to the light field camera, thereby achieving hand-eye calibration of the robot and improving the efficiency of hand-eye calibration.

[0097] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0098] Embodiments of this application provide a hand-eye calibration method applied to visual robots, with reference to... Figure 13 As shown, the method includes the following steps:

[0099] Step 301: Obtain the three-dimensional physical coordinates of any reference calibration point in the reference lattice determined in the calibration object, in the calibration object coordinate system.

[0100] Step 302: Control the movement of the vision robot's manipulator to obtain light field images of the reference dot matrix captured by the vision robot's light field camera in at least two poses.

[0101] Here, refer to Figure 14 As shown in Table 1, Figure 14 This diagram illustrates the optical configuration and light propagation process of a light field camera. Table 1 lists the meanings of the symbols used in the light field projection model. According to... Figure 14 The optical configuration of the non-focusing all-light camera shown involves light rays from a single point in three-dimensional physical space first passing through the main lens and then reaching a single convergence point, i.e., the image point. The light rays are then separated by a set of microlenses according to their propagation angles, forming a light field image on the sensor plane.

[0102] <![CDATA[P C (P C x ,P C y ,P C z )]]> Coordinates (mm) of the reference calibration point in the camera coordinate system {C} <![CDATA[l ci ]]> The position of the center of the i-th microlens <![CDATA[l pi ]]> The position of the i-th microlens at the center of its projection on the sensor plane <![CDATA[p ci ]]> The position of the dot-like feature corresponding to the i-th microlens <![CDATA[Q p (Q p u ,Q p v )]]> <![CDATA[The position of the projection of image point Q C on the sensor plane]]> <![CDATA[O c (The c u ,The c v )]]> The optical center of the camera in image coordinates <![CDATA[D p ]]> Diameter of the circle of confusion (pixels) <![CDATA[C p (C p u ,C p v )]]> The center of the diffusion circle (pixel) <![CDATA[S i ]]> Image distance (mm), which is the distance between the primary lens and the focal plane. <![CDATA[f l ]]> Microlens focal length (mm) <![CDATA[p p ]]> Actual physical size of a pixel (mm) <![CDATA[p m ]]> Effective aperture diameter of the main lens (mm) <![CDATA[f m ]]> Focal length of the main lens (mm) <![CDATA[p l ]]> True physical dimensions of the microlens (mm) <![CDATA[f #，m ]]> F-number of the primary lens <![CDATA[f #，l ]]> <![CDATA[F - number of the microlens, f #，l = f l / p l > M Magnification of the main lens PMR <![CDATA[PMR=p l / p p ]]>

[0103] Table 1

[0104] Step 303: Obtain a solid color image of the solid color calibration object by taking a picture with a light field camera.

[0105] In this embodiment, before photographing the solid-color calibration object, the aperture of the main lens of the light field camera needs to be adjusted to its minimum, thereby enabling the microlens to form small apertures. At this point, the solid-color image obtained by photographing the solid-color calibration object with the adjusted light field camera represents the tangency of adjacent small apertures (macropixels) formed by the microlens. In practical applications, refer to... Figure 15 As shown, Figure 15 The diagram shown is a schematic of the projection center calibration of the microlens. Figure 15 A in the diagram shows the minimum aperture obtained when the aperture of the main lens is adjusted to its minimum. Figure 15 B in the diagram shows the macropixel tangency map obtained by adjusting the light field camera to the macropixel tangency state.

[0106] In this embodiment, the solid color calibration object can be a calibration object with uniform brightness, such as a whiteboard, white wall or other solid color object.

[0107] Step 304: Perform calibration processing on the solid color image to obtain the third pixel position of the projection center of each microlens in the microlens array of the light field camera in the image coordinate system when the center of each microlens is projected onto the sensor plane of the light field camera.

[0108] In the embodiments of this application, reference is made to Figure 14 As shown, after the visual robot obtains a solid-color image of the solid-color calibration object captured by a light field camera, it performs a brightness-weighted calculation on the solid-color image using a microlens array calibration algorithm to obtain the microlens center l of each microlens in the microlens array (MLA) of the light field camera. ci and the center of the microlens l ci When projecting onto the sensor plane of the light field camera, the third pixel position l at the projection center in the image coordinate system. pi。

[0109] In practical applications, refer to Figure 14 As shown, since the main lens is an ideal thin lens, the coordinates P of any reference calibration point in the light field image in the camera coordinate system are... C (P C x P C y P C z All rays of light in P in three-dimensional space C Image point Q will be formed by the main lens. C (Q Cx Q C y Q C z Therefore, based on the thin lens imaging principle, any reference calibration point P in the light field image is given. C And like point Q C The relationship between them is shown in Formula 3.

[0110]

[0111] Furthermore, due to Therefore, the reference calibration point P in the light field image can be derived according to Formula 3. C And like point Q C The relationship between them is shown in Formula 4.

[0112]

[0113] Among them, -Q C z It's like a Q. C The distance from the primary lens in the Z direction. It is the reference calibration point P in the light field image. C In the camera coordinate system {C}, the distance f from the primary lens along the Z-axis is... m It is the focal length of the primary lens. (From image point Q) C The light rays will reach the MLA plane (also known as the microlens plane or microlens array plane) and encounter a set of microlenses. Since the microlenses are modeled as a pinhole array, the center of the projected image below the MLA will be offset from the center of the microlenses. Here, O c (O c u O c v () represents the optical center of the camera in image coordinates. The visual robot obtains the center of each microlens in the microlens array based on the solid color image. Here, the coordinates of the center of the i-th lens are represented as: Assuming ideal physical alignment within the light field camera, the MLA plane is parallel to the primary lens. That is, the distance between the microlens and the primary lens plane is constant. Here, the projection center l of the microlens is used. pi and light center O c Projected coordinates O on the sensor plane p Indicates the center of the microlens l ci Thus, the center of the microlens l is determined. ci and the microlens projection center l pi The relationship here is that the microlens projection center l pi and light center O c Projected coordinates O on the sensor plane pIndicates the center of the microlens l ci This can be represented by the following formula 5.

[0114]

[0115] It should be noted that the position l of the center of the i-th microlens here and thereafter ci It can be described as the center of the microlens l ci The position of the projection center of the i-th microlens on the sensor plane can be described as the microlens projection center l. pi or projection center l pi .

[0116] Step 305: For all projection centers, based on the obtained first preset window, determine the fourth pixel position of the point feature corresponding to the third pixel position in the sensor plane for any reference calibration point in each light field image, and obtain the fourth pixel position of all point features corresponding to all projection centers in the image coordinate system.

[0117] The first preset window is used to include the projection center corresponding to the reference calibration point.

[0118] In this embodiment of the application, the visual robot obtains the center l of each microlens in the microlens array of the light field camera. ci and the center of the microlens l ci When projecting onto the sensor plane of the light field camera, the third pixel position l at the projection center in the image coordinate system. pi Then, for all projection centers, based on the obtained first preset window, the position of the third pixel in the sensor plane for any reference calibration point in each light field image is determined. pi The fourth pixel position p of the dot-like feature ci This yields the fourth pixel position p of all point features corresponding to all projection centers in the image coordinate system. ci It should be noted that the fourth pixel position p in the image coordinate system refers to this point and all subsequent point features. ci It can be described as all point features p ci .

[0119] In this embodiment of the application, step 305, for all projection centers, based on the obtained first preset window, determines the position l of the third pixel in each light field image corresponding to any reference calibration point in the sensor plane. pi The fourth pixel position p of the dot-like feature ci process combination Figure 16 To provide further explanation,

[0120] Step 3051: Define a first region with a first preset window centered on the third pixel position in the light field image.

[0121] Step 3052: From all pixels in the first region, determine the pixels whose pixel features satisfy the feature selection conditions as the first target points.

[0122] In this embodiment of the application, the pixel features satisfying the feature selection conditions include: the brightness value of the pixel in the first region is greater than the brightness threshold and the brightness value of the pixel is greater than or equal to the brightness value of the adjacent pixel, and the distance between the pixel and the projection center corresponding to the position of the third pixel is the shortest.

[0123] Step 3053: Define a second region with a second preset window centered on the first target point in the light field image.

[0124] The size of the second preset window is smaller than the size of the first preset window.

[0125] Step 3054: Based on the gray-level weighted average of the pixel positions of all pixels in the second region, the fourth pixel position of the dot feature is obtained.

[0126] In a feasible application scenario, refer to Figure 14 As shown, from reference calibration point P C The light rays will encounter a set of microlenses. For each encountered microlens, the corresponding microimage contains at most one point feature. However, the actual white point on the calibration object is not infinitely small, and aberrations exist during actual imaging. Note that each point feature will cover several pixels. Therefore, the location of the point feature is calculated from the light field image using the following method. The point feature detection process is as follows: Figure 17 As shown. First, the corresponding microlens center is located spatially. Then, it is framed in the light field image at the third pixel position l. pi Centered on the first preset window W, determine the location of the window. l For example, in a 13×13 first region, a first target point is selected within the first region whose pixel features satisfy the feature selection conditions. These conditions include: the brightness value of the pixel within the first region is greater than a brightness threshold, the pixel's brightness value is greater than the brightness values ​​of its adjacent pixels, and the distance between the pixel and the projection center corresponding to the third pixel position is minimized. Further, a second preset window W is defined in the light field image, centered on the first target point. s For example, in the second 5×5 region, a gray-level weighted average is performed on the pixel positions of all pixels in the second region to obtain the fourth pixel position p of the dot-like feature. ciIn this way, by selecting calibration points whose pixel features meet the feature selection criteria, the problem of identical pixel values ​​is avoided. Setting the closest point can solve the problem of identical pixels and also eliminate the problem of finding other microlens areas. This allows for the accurate finding of calibration points that meet the requirements, so as to accurately calibrate the target pose transformation relationship of the object relative to the light field camera. This enables the rapid completion of hand-eye calibration for visual robots, improving the accuracy and efficiency of hand-eye calibration.

[0127] Step 306: Based on the third pixel position of all projection centers and the fourth pixel position of all point features, determine the blur circle feature corresponding to the reference calibration point.

[0128] In this embodiment, the dispersion circle feature includes the position of the center of the dispersion circle and the diameter of the dispersion circle.

[0129] In this embodiment, the visual robot determines the center C of the blur circle corresponding to the reference calibration point based on the third pixel position of all projection centers and the fourth pixel position of all point features using the following formula 6. p (C p u C p v ) and the diameter D of the circle of confusion p .

[0130]

[0131] Among them, l pi (l pi u , l pi v ) represents the pixel position of the i-th microlens center projected onto the sensor surface in the image coordinate system, p ci (p ci u p ci v (D) represents the pixel position of the point feature corresponding to the i-th microlens in the image coordinate system. p C p u C p v () indicates the characteristic of the circle of confusion. See reference. Figure 18 As shown, Figure 18 The process of extracting the feature of the circle of confusion is shown.

[0132] Here, refer to Figure 14 As shown, the derivation of Formula 6 can be obtained from Formulas 7-20. Specifically, based on the similarity of triangles... Then the fourth pixel position p of the point feature in the image coordinate system ci Through the center of the microlensci And like point Q C Projected coordinates Q on the sensor plane p This can be expressed as Formula 7 below.

[0133]

[0134] Substituting Equation 5 into Equation 7, we can obtain the position p of the fourth pixel of the point feature in the image coordinate system. ci Through the microlens projection center l pi The camera's optical center is projected onto the sensor plane at coordinate O. p The projected coordinates Qc of image point Qc on the sensor plane p (Q p u Q p v This is represented by the following formula 8.

[0135]

[0136] Where, the projected coordinates of point Q on the sensor plane are Q' ... p (Q p u Q p v The coordinates of Q in the camera coordinate system C (Q C x Q C y Q C z The relationship between them can be expressed by formula 9.

[0137]

[0138] From ΔO c AQ~ΔC l l ci Q gives the coordinates A(A) of the intersection point of ray PA and the principal lens in the camera coordinate system. x A y ,0), and the image point Q C The center of the i-th lens The relationship between them can be expressed by Formula 10.

[0139]

[0140] Furthermore, from ΔOAl ci ~Δl pi p ci l ci The coordinates A(A) of the intersection point of ray PA and the principal lens in the camera coordinate system can be obtained. xA y ,0), and the projection center of the i-th microlens The corresponding point feature is located at the fourth pixel position p in the image coordinate system. ci The relationship between them can be represented by Formula 11.

[0141]

[0142] Among them, A(A x A y (0) is the coordinate of the intersection point of ray PA and the principal lens in the camera coordinate system. pi (l pi u , l pi v ) represents the pixel position of the i-th microlens center projected onto the sensor surface in the image coordinate system, p ci (p ci u p ci v ) represents the pixel position of the point feature corresponding to the i-th microlens in the image coordinate system, l pi (l pi u , l pi v ) is the pixel position of the i-th microlens center projected onto the sensor plane in the image coordinate system.

[0143] The diameter D of the circle of confusion feature formed on the microlens plane can be obtained from the projection similarity. l The diameter D of the dispersion circle feature formed on the sensor plane p The relationship between them can be expressed by Formula 12.

[0144]

[0145] Combining Equation 10 with Equation 12, we obtain the diameter D of the blur circle feature formed on the MLA plane. l and center coordinates C l (C l u C l v ), center of microlens l ci The coordinates of point A in the camera coordinate system (A x A y The relationship between and can be expressed by formula 13.

[0146]

[0147] Transforming Formula 11, we obtain the projection center l using the microlens.pi and point features p ci This represents the coordinates of point A in the camera coordinate system (A x A y ,0), specifically represented by the following formula 14,

[0148]

[0149] Substituting Formula 14 into Formula 13, we obtain the diameter D of the blur circle feature formed on the microlens plane. l The pixel position l of the i-th microlens center projected onto the sensor surface in the image coordinate system pi (l pi u , l pi v The pixel position p of the point feature corresponding to the i-th microlens in the image coordinate system. ci (p ci u p ci v The pixel position l of the i-th microlens center projected onto the sensor plane in the image coordinate system. p i(l pi u , l pi v The relationship between the two can be expressed by formula 15.

[0150]

[0151] Right now

[0152]

[0153] By transforming Formula 12, the diameter D of the blur circle feature formed in the microlens plane is obtained. l The diameter D of the dispersion circle feature formed on the sensor plane p The relationship between them can be expressed by Equation 16; by transforming Equation 12, the feature center C of the circle of confusion formed in the microlens plane is obtained. l and the center C of the dispersion circle formed on the sensor plane p The relationship between them can be expressed by Formula 17.

[0154]

[0155]

[0156] Substituting Formula 16 into Formula 15, we get

[0157]

[0158] From (Formula 17) we have

[0159]

[0160] Substituting formula 19 into formula 18 yields the following result:

[0161]

[0162] Right now

[0163]

[0164] Transforming Formula 20 yields Formula 6, which determines the diameter D of the circle of confusion formed by the reference calibration point in the light field image on the sensor plane. p and the center C of the dispersion circle formed on the sensor plane p .

[0165] Step 307: Based on the blur circle features corresponding to all reference calibration points and the three-dimensional physical coordinates corresponding to all reference calibration points, obtain the target pose transformation relationship.

[0166] In this embodiment, step 307, based on the circle of confusion features corresponding to all reference calibration points and the three-dimensional physical coordinates corresponding to all reference calibration points, obtains the target pose transformation relationship. Figure 19 To provide further explanation,

[0167] Step 3071: Based on the features of the circle of confusion and the three-dimensional physical coordinates corresponding to all reference calibration points, the least squares method of the nonlinear equation system is used to determine the initial pose transformation relationship between the calibration object coordinate system and the camera coordinate system and the initial values ​​of the internal parameters of the light field camera.

[0168] In this embodiment of the application, the visual robot obtains the circle of confusion feature (D). p C p u C p v The coordinates of the light field image at any reference calibration point P in the camera coordinate system. C (P C x P C y P C z The first transformation relationship between them; here, the first transformation relationship can be expressed by the following formula 21,

[0169]

[0170] Here, refer to Figure 14The derivation of Formula 21 can be obtained through Formulas 22-28. Specifically, Formula 12 is transformed to obtain the diameter D of the blur circle feature formed on the microlens plane. l The diameter D of the dispersion circle feature formed on the sensor plane p The relationship between them can be expressed by formula 22.

[0171]

[0172] By transforming Equation 10, the diameter D of the blur circle feature formed on the microlens plane is obtained. l And like point Q C The relationship between them can be expressed by formula 23.

[0173]

[0174] Substituting Equation 23 into Equation 22, we obtain the diameter D of the dispersion circle feature formed on the sensor plane. p And like point Q C The relationship between them can be expressed by formula 24.

[0175]

[0176] Depend on achievable Substituting into Equation 24, we obtain the diameter D of the blur circle feature formed on the sensor plane. p And the coordinates P of any reference calibration point in the light field image in the camera coordinate system C The relationship between them can be expressed by formula 26.

[0177]

[0178] Right now

[0179]

[0180] Depend on achievable And because Then there is a circle of confusion C formed on the sensor plane. p C in p u And the coordinates P of any reference calibration point in the light field image in the camera coordinate system C The relationship between them can be expressed by formula 27; at the center C of the dispersion circle formed on the sensor plane p C in p v And the coordinates P of any reference calibration point in the light field image in the camera coordinate system C The relationship between them can be expressed by formula 28.

[0181]

[0182]

[0183] Furthermore, based on formulas 26 to 28, the feature of the circle of confusion (D) is obtained. p C p u C p v The coordinates of the light field image at any reference calibration point P in the camera coordinate system. C (P C x P C y P C z The first transformation relationship between ).

[0184] Furthermore, the image distance S of the primary lens, an internal parameter of the camera, is obtained. i Microlens focal length f l Pixel actual physical size p p The effective aperture diameter p of the main lens m The focal length f of the primary lens m To simplify, let Formula 21 can then be simplified to Formula 29.

[0185]

[0186] Furthermore, the coordinates P of any reference calibration point in the light field image in the camera coordinate system are introduced. C (P C x P C y P C z ) and the corresponding reference calibration point in the calibration object coordinate system in three-dimensional physical coordinates (P) w x P w y The pose transformation relationship between (0, 0) can be expressed by the following formula 30.

[0187]

[0188] Wherein, rot represents the rotational transformation relationship between the coordinates of the reference calibration point in the camera coordinate system and the corresponding coordinates of the reference calibration point in the calibration object coordinate system in the light field image; rot also represents the translational transformation relationship between the coordinates of the reference calibration point in the camera coordinate system and the corresponding coordinates of the reference calibration point in the calibration object coordinate system in the light field image.

[0189] Substituting formula 30 into formula 21, we can rearrange it into the following formula 31.

[0190]

[0191] For the calibration object at the nth pose, due to rot n and pos T n The blur circle at the m-th position in the reference lattice is constant and satisfies Formula 32, i.e.:

[0192]

[0193] Here, let

[0194]

[0195] Then, for all reference points on the nth pose calibration object, we have: Furthermore, the least squares solution is obtained through Singular Value Decomposition (SVD). Then obtain through constraints linear solutions, The linear solution includes the initial values ​​of the intrinsic parameters of the light field camera. As shown in Equation 33,

[0196]

[0197] Among them, v i Representing vectors The i-th element of the least squares solution v, in the above simplification, is let

[0198] Furthermore, based on The linear solution yields the rotation transformation relationship rot under different poses. n Translation transformation relationship pos T n Based on the initial values ​​of the internal parameters of the light field camera, and the rotation and translation transformation relationships, the initial pose transformation relationship between the calibration object coordinate system and the camera coordinate system is obtained.

[0199] Here, based on the rotation and translation transformation relationships, the initial pose transformation relationship between the calibration object coordinate system and the camera coordinate system under different poses is obtained, which can be expressed by the following formula 34.

[0200]

[0201] Step 3072: Construct a nonlinear optimization model. Input the initial pose transformation relationship and the initial values ​​of the internal parameters into the nonlinear optimization model to obtain the target pose transformation relationship output by the nonlinear optimization model.

[0202] Here, based on the ideal thin lens model, the theoretical internal parameters of the camera can be calculated from:

[0203] S i =f m (1-M)

[0204] f #，m =f #，l / (1-M)

[0205] p m =f m / f #，m

[0206] M = -1

[0207] Where M = -1, and Q is the pixel coordinate position of the camera's optical center projection center. p (Q p u Q p v The initial value can be the pixel coordinates of the center of the camera view, i.e., Q. p u =3960, Q p v =2718. Table 2 shows the camera's internal parameters.

[0208] Camera parameters numerical values PMR It can be known <![CDATA[f l ]]> It can be known <![CDATA[p l ]]> It can be known <![CDATA[p p ]]> It can be known <![CDATA[f m ]]> 200 (mm) M -1 <![CDATA[f #，l ]]> 8.1349 <![CDATA[f #，m ]]> 4.0675 <![CDATA[S i ]]> 400 (mm) <![CDATA[p m ]]> 49.1707 <![CDATA[Q p u ]]> 3960 <![CDATA[Q p v ]]> 2718

[0209] Table 2

[0210] Using the data in Table 2 as the initial input values ​​for the camera's intrinsic parameters, the initial pose transformation relationship cHw is solved. n As initial input values ​​for camera extrinsic parameters, nonlinear optimization is performed to minimize the circle of confusion reprojection error. The circle of confusion reprojection error function is determined to be a nonlinear optimization model:

[0211] agmin{||[D p C p u C p v ] compute -[D p C p u C p v ] estimate || 2}

[0212] Among them, [D p C p u C p v ] compute It is the coordinates of a known reference lattice in the calibration object coordinate system. By using the initial pose transformation relationship cHw n The characteristic of the circle of confusion calculated by formula 21, [D p C p u C p v ] estimate The feature of the circle of confusion is extracted through fitting using Formula 6. (Refer to...) Figure 20 As shown, Figure 20 The diagram illustrates the reprojection process of point features. The light gray point features are the calculated features, and the dark gray point features are the searched features.

[0213] Step 308: Perform hand-eye calibration on the visual robot based on the target pose transformation relationship.

[0214] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0215] Embodiments of this application provide a visual robot that can be used to implement... Figure 3 , Figure 6 , Figure 8 , Figure 10 , Figure 13 , Figure 16 and Figure 19 A corresponding embodiment provides a hand-eye calibration method, referring to... Figure 21 As shown, the visual robot 21 includes:

[0216] Light field camera 401, used to capture light field images;

[0217] The robot body 402 has a base and a robotic arm mounted on the base. The robotic arm is used to fix the object to be photographed.

[0218] The control device 403 is used to obtain the three-dimensional physical coordinates of any reference calibration point in the reference point array determined in the calibration object in the calibration object coordinate system; control the movement of the manipulator of the vision robot to obtain light field images of the reference point array captured by the light field camera of the vision robot in at least two poses; obtain the target pose transformation relationship of the calibration object relative to the light field camera based on the at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array; and perform hand-eye calibration of the vision robot based on the target pose transformation relationship.

[0219] In some embodiments of this application, the control device 403 is further configured to: render each of at least two light field images using a light field rendering algorithm to obtain a multi-view image corresponding to each light field image, wherein the multi-view image includes a central view image; process the multi-view image corresponding to each light field image using an epipolar image algorithm to obtain a disparity image corresponding to each light field image; locate the calibration points in the disparity image based on all reference calibration points in the central view image to obtain a target calibration point in the disparity image; obtain the three-dimensional point cloud coordinates of the target calibration point in the camera coordinate system based on the first pixel position of the target calibration point in the image pixel coordinate system; and obtain the target pose transformation relationship based on the three-dimensional point cloud coordinates corresponding to all target calibration points and the three-dimensional physical coordinates corresponding to the reference calibration points at the corresponding positions.

[0220] In some embodiments of this application, the control device 403 is further configured to obtain the second pixel position of each of the reference calibration points in the image pixel coordinate system; and determine the calibration point corresponding to the second pixel position of the disparity image in the image pixel coordinate system as the target calibration point.

[0221] In some embodiments of this application, the control device 403 is further configured to obtain the disparity value of each target calibration point in the disparity image; and to obtain the three-dimensional point cloud coordinates based on the first pixel position and the disparity value.

[0222] In some embodiments of this application, the control device 403 is further configured to acquire a solid color image obtained by capturing a solid color calibration object through a light field camera; perform calibration processing on the solid color image to obtain the third pixel position of the projection center in the image coordinate system when the center of each microlens in the microlens array of the light field camera is projected onto the sensor plane of the light field camera; for all projection centers, based on the obtained first preset window, determine the fourth pixel position of the point feature corresponding to the third pixel position of any reference calibration point in each light field image in the sensor plane, thereby obtaining the fourth pixel positions of all point features corresponding to all projection centers in the image coordinate system; wherein, the first preset window is used to include all projection centers corresponding to the reference calibration point; based on the third pixel positions of all projection centers and the fourth pixel positions of all point features, determine the circle of confusion feature corresponding to the reference calibration point; based on the circle of confusion feature corresponding to all reference calibration points and the three-dimensional physical coordinates corresponding to all reference calibration points, obtain the target pose transformation relationship.

[0223] In some embodiments of this application, the control device 403 is further configured to: define a first region in the light field image with the third pixel position as the center and determine a first preset window; determine a pixel whose pixel features satisfy the feature selection conditions from all pixels in the first region as a first target point; define a second region in the light field image with the first target point as the center and determine a second preset window; wherein the size of the second preset window is smaller than the size of the first preset window; and obtain the fourth pixel position of the point feature based on the gray-level weighted average of the pixel positions of all pixels in the second region.

[0224] In some embodiments of this application, the pixel features satisfy the feature selection conditions, including: the brightness value of the pixel in the first region is greater than the brightness threshold and the brightness value of the pixel is greater than or equal to the brightness value of the adjacent pixel, and the distance between the pixel and the projection center corresponding to the position of the third pixel is the shortest.

[0225] In some embodiments of this application, the control device 403 is further configured to determine the initial pose transformation relationship between the calibration object coordinate system and the camera coordinate system and the initial values ​​of the internal parameters of the light field camera based on the features of the circle of confusion and the three-dimensional physical coordinates corresponding to all reference calibration points, using the least squares method of nonlinear equations; construct a nonlinear optimization model, input the initial pose transformation relationship and the initial values ​​of the internal parameters into the nonlinear optimization model, and obtain the target pose transformation relationship output by the nonlinear optimization model.

[0226] Based on the foregoing embodiments, this hand-eye calibration device can be used for Figure 3 , Figure 6 , Figure 8 , Figure 10 , Figure 13 , Figure 16 and Figure 19A corresponding embodiment provides a hand-eye calibration method, referring to... Figure 22 As shown, the hand-eye calibration device 22 ( Figure 22 Hand-eye calibration device 22 and Figure 21 The visual robot 21 in the image includes: a processor 501 and a memory 502, wherein:

[0227] Memory 502 is used to store the hand-eye calibration program;

[0228] Processor 501 is used to execute the hand-eye calibration program stored in memory 502, and to perform the following steps:

[0229] Obtain the three-dimensional physical coordinates of any reference calibration point in the reference lattice determined in the calibration object, in the calibration object coordinate system;

[0230] Control the movement of the robotic arm of a vision robot to obtain light field images of a reference dot matrix captured by the light field camera of the vision robot in at least two poses;

[0231] Based on at least two light field images and the three-dimensional physical coordinates of all reference calibration points in the reference point array, the target pose transformation relationship of the calibration object relative to the light field camera is obtained.

[0232] Hand-eye calibration of a visual robot is performed based on the target pose transformation relationship.

[0233] In other embodiments of this application, the processor 501 is used to execute the hand-eye calibration program stored in the memory 502 to perform the following steps:

[0234] At least two light field images are rendered using a light field rendering algorithm to obtain a multi-view image corresponding to each light field image, including a central view image. The multi-view image is then processed using an epipolar image algorithm to obtain a disparity image corresponding to each light field image. Based on all reference calibration points in the central view image, calibration points in the disparity image are located to obtain target calibration points in the disparity image. Based on the first pixel position of the target calibration point in the image pixel coordinate system, the 3D point cloud coordinates of the target calibration point in the camera coordinate system are obtained. Based on the 3D point cloud coordinates corresponding to all target calibration points and the 3D physical coordinates corresponding to the reference calibration points at the corresponding positions, the target pose transformation relationship is obtained.

[0235] In other embodiments of this application, the processor 501 is used to execute the hand-eye calibration program stored in the memory 502 to perform the following steps:

[0236] Obtain the second pixel position of each reference calibration point in the image pixel coordinate system; determine the calibration point corresponding to the second pixel position of the disparity image in the image pixel coordinate system as the target calibration point.

[0237] In other embodiments of this application, the processor 501 is used to execute the hand-eye calibration program stored in the memory 502 to perform the following steps:

[0238] Obtain the disparity value of each target calibration point in the disparity image; based on the first pixel position and the disparity value, obtain the three-dimensional point cloud coordinates.

[0239] In other embodiments of this application, the processor 501 is used to execute the hand-eye calibration program stored in the memory 502 to perform the following steps:

[0240] Acquire a solid-color image of a solid-color calibration object captured by a light field camera; perform calibration processing on the solid-color image to obtain the third pixel position of the projection center of each microlens in the microlens array of the light field camera when projected onto the sensor plane of the light field camera in the image coordinate system; for all projection centers, based on the obtained first preset window, determine the fourth pixel position of the point feature corresponding to the third pixel position of any reference calibration point in each light field image in the sensor plane, thus obtaining the fourth pixel positions of all point features corresponding to all projection centers in the image coordinate system; wherein, the first preset window is used to include all projection centers corresponding to the reference calibration points; based on the third pixel positions of all projection centers and the fourth pixel positions of all point features, determine the circle of confusion feature corresponding to the reference calibration point; based on the circle of confusion feature corresponding to all reference calibration points and the three-dimensional physical coordinates corresponding to all reference calibration points, obtain the target pose transformation relationship.

[0241] In other embodiments of this application, the processor 501 is used to execute the hand-eye calibration program stored in the memory 502 to perform the following steps:

[0242] In the light field image, a first region with a first preset window is defined centered on the third pixel position. From all pixels in the first region, pixels whose pixel features satisfy the feature selection conditions are identified as the first target point. In the light field image, a second region with a second preset window is defined centered on the first target point. The size of the second preset window is smaller than the size of the first preset window. Based on the gray-level weighted average of the pixel positions of all pixels in the second region, the fourth pixel position of the point feature is obtained.

[0243] In other embodiments of this application, the pixel features satisfy the feature selection conditions, including: the brightness value of the pixel in the first region is greater than the brightness threshold and the brightness value of the pixel is greater than or equal to the brightness value of the adjacent pixel, and the distance between the pixel and the projection center corresponding to the position of the third pixel is the shortest.

[0244] In other embodiments of this application, the processor 501 is used to execute the hand-eye calibration program stored in the memory 502 to perform the following steps:

[0245] Based on the features of the circle of confusion and the three-dimensional physical coordinates corresponding to all reference calibration points, the least squares method of nonlinear equations is used to determine the initial pose transformation relationship between the calibration object coordinate system and the camera coordinate system, as well as the initial values ​​of the internal parameters of the light field camera. A nonlinear optimization model is constructed, and the initial pose transformation relationship and the initial values ​​of the internal parameters are input into the nonlinear optimization model to obtain the target pose transformation relationship output by the nonlinear optimization model.

[0246] Embodiments of this application provide a computer storage medium storing one or more programs, which can be executed by one or more processors to achieve... Figure 3 , Figure 6 , Figure 8 , Figure 10 , Figure 13 , Figure 16 and Figure 19 A corresponding embodiment provides a hand-eye calibration method.

[0247] It should be noted that the aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; it can also be various terminals that include one or any combination of the above-mentioned memory, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0248] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0249] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0250] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0251] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0252] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0253] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0254] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hand-eye calibration method, characterized in that, The method includes: Obtain the three-dimensional physical coordinates of any reference calibration point in the reference lattice determined in the calibration object, in the calibration object coordinate system; Control the movement of the manipulator of the vision robot to obtain light field images of the reference dot array captured by the light field camera of the vision robot in at least two poses; Based on at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array, the target pose transformation relationship of the calibration object relative to the light field camera is obtained. Hand-eye calibration of the visual robot is performed based on the target pose transformation relationship; The step of obtaining the target pose transformation relationship of the calibration object relative to the light field camera based on at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array includes: Positioning is achieved using the central view image. The 3D point cloud coordinates of all target calibration points in the target point array in the camera coordinate system are matched with the 3D physical coordinates of the corresponding reference calibration points in the calibration object coordinate system. The rigid body transformation between all the corresponding 3D point cloud coordinates in the camera coordinate system and the corresponding 3D physical coordinates in the calibration object coordinate system is calculated to obtain the target pose transformation relationship of the calibration object relative to the light field camera. The central view image and the parallax image are in one-to-one correspondence, and the pixel values ​​on the parallax image are in one-to-one correspondence with the 3D point cloud.

2. The hand-eye calibration method according to claim 1, characterized in that, The step of obtaining the target pose transformation relationship of the calibration object relative to the light field camera based on at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array includes: The light field rendering algorithm is used to render each of the at least two light field images to obtain a multi-view image corresponding to each light field image, wherein the multi-view image includes a center view image. The multi-view image corresponding to each light field image is processed by the epipolar image algorithm to obtain the parallax image corresponding to each light field image; Based on all the reference calibration points in the central view image, the calibration points in the disparity image are located to obtain the target calibration point in the disparity image; Based on the first pixel position of the target calibration point in the image pixel coordinate system, the three-dimensional point cloud coordinates of the target calibration point in the camera coordinate system are obtained; Based on the three-dimensional point cloud coordinates corresponding to all target calibration points and the three-dimensional physical coordinates corresponding to the reference calibration points at the corresponding positions, the target pose transformation relationship is obtained.

3. The hand-eye calibration method according to claim 2, characterized in that, The step of locating the calibration points in the disparity image based on all reference calibration points in the central view image to obtain the target calibration point in the disparity image includes: Obtain the second pixel position of each of the aforementioned reference calibration points in the image pixel coordinate system; The calibration point corresponding to the second pixel position in the image pixel coordinate system of the disparity image is determined as the target calibration point.

4. The hand-eye calibration method according to claim 2, characterized in that, The step of obtaining the second three-dimensional point cloud coordinates of the target calibration point in the camera coordinate system based on the first pixel position of the target calibration point in the image pixel coordinate system includes: Obtain the disparity value of each target calibration point in the disparity image; The coordinates of the three-dimensional point cloud are obtained based on the position of the first pixel and the disparity value.

5. The hand-eye calibration method according to claim 1, characterized in that, The step of obtaining the target pose transformation relationship of the calibration object relative to the light field camera based on at least two light field images and the three-dimensional physical coordinates corresponding to all reference calibration points in the reference point array includes: Acquire a solid color image of the solid color calibration object by shooting it with the light field camera; The solid color image is calibrated to obtain the third pixel position of the projection center in the image coordinate system when the center of each microlens in the microlens array of the light field camera is projected onto the sensor plane of the light field camera. For all projection centers, based on the obtained first preset window, the fourth pixel position of the point feature corresponding to the third pixel position of any reference calibration point in each light field image is determined in the sensor plane, thereby obtaining the fourth pixel positions of all point features corresponding to all projection centers in the image coordinate system; wherein, the first preset window is used to include all projection centers corresponding to the reference calibration point. Based on the third pixel position of all projection centers and the fourth pixel position of all point features, the blur circle feature corresponding to the reference calibration point is determined; Based on the dispersion circle features corresponding to all the reference calibration points and the three-dimensional physical coordinates corresponding to all the reference calibration points, the target pose transformation relationship is obtained.

6. The hand-eye calibration method according to claim 5, characterized in that, The light field image is a grayscale processed image. The step of determining the fourth pixel position of the point feature corresponding to the third pixel position in the sensor plane for any reference calibration point in each light field image, based on the obtained first preset window, includes: In the light field image, a first region with the first preset window is defined with the third pixel position as the center. From all pixels in the first region, determine the pixels whose pixel features satisfy the feature selection conditions as the first target points; In the light field image, a second region with a second preset window is defined, centered on the first target point; wherein the size of the second preset window is smaller than the size of the first preset window; The fourth pixel position of the dotted feature is obtained by performing a gray-level weighted average on the pixel positions of all pixels in the second region.

7. The hand-eye calibration method according to claim 6, characterized in that, The pixel features satisfy the feature selection conditions, including: the brightness value of the pixel in the first region is greater than the brightness threshold and the brightness value of the pixel is greater than or equal to the brightness value of the adjacent pixel, and the distance between the pixel and the projection center corresponding to the position of the third pixel is the shortest.

8. The hand-eye calibration method according to claim 5, characterized in that, The process of obtaining the target pose transformation relationship based on the dispersion circle features corresponding to all reference calibration points and the three-dimensional physical coordinates corresponding to all reference calibration points includes: Based on the blur circle feature and the three-dimensional physical coordinates corresponding to all the reference calibration points, the initial pose transformation relationship between the calibration object coordinate system and the camera coordinate system and the initial values ​​of the internal parameters of the light field camera are determined by using the least squares method of nonlinear equation system. A nonlinear optimization model is constructed, and the initial pose transformation relationship and the initial values ​​of the internal parameters are input into the nonlinear optimization model to obtain the target pose transformation relationship output by the nonlinear optimization model.

9. A visual robot, characterized in that, The visual robot includes: A light field camera is used to capture images of light fields. The robot body has a base and a robotic arm mounted on the base, the robotic arm being used to fix the object to be photographed; A control device is used to obtain the three-dimensional physical coordinates of any reference calibration point in a defined reference point array within the calibration object, in the calibration object coordinate system; control the movement of the manipulator of a vision robot to obtain light field images of the reference point array captured by the manipulator's light field camera in at least two poses; use the central view image for positioning, match the three-dimensional point cloud coordinates of all target calibration points in the target point array in the camera coordinate system with the three-dimensional physical coordinates of the corresponding reference calibration points in the calibration object coordinate system, calculate the rigid body transformation between all corresponding three-dimensional point cloud coordinates in the camera coordinate system and the corresponding three-dimensional physical coordinates in the calibration object coordinate system, and obtain the target pose transformation relationship of the calibration object relative to the light field camera; the central view image and the parallax image are in one-to-one correspondence, and the pixel values ​​on the parallax image are in one-to-one correspondence with the three-dimensional point cloud; perform hand-eye calibration on the vision robot based on the target pose transformation relationship.

10. A hand-eye calibration device, characterized in that, The hand-eye calibration device includes: The memory is used to store the hand-eye calibration program; A processor is configured to execute a hand-eye calibration program stored in the memory to implement the hand-eye calibration method as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the hand-eye calibration method as described in any one of claims 1 to 8.

Images