A hand-eye calibration method, device, electronic equipment and storage medium

Abstract

The application provides a hand-eye calibration method and device, electronic equipment and storage medium. The method comprises the following steps: determining image coordinates of a plurality of feature points in a calibration board in an image captured by a camera; for each feature point, after aligning a tool center point of a mechanical arm to the feature point through a sensor, acquiring a mechanical coordinate of the tool center point; and then determining a coordinate conversion matrix according to the image coordinates and the mechanical coordinates of the plurality of feature points, so as to complete hand-eye calibration of a robot. The hand-eye calibration method provided by the application aligns the feature points and the tool center point through the sensor in the calibration process, and compared with the manual alignment of the tool center point and the feature point, the error in the calibration process can be significantly reduced, and the accuracy of the calibration result is improved.

Description

Technical Field

[0001] This application relates to the field of robotics, and more specifically, to a hand-eye calibration method, apparatus, electronic device, and storage medium. Background Technology

[0002] In industrial robots, cameras are used to sense the position of parts. Based on the positions sensed by the cameras, the robotic arm can perform operations such as grasping, handling, assembling, welding, and painting on the parts. Before the robot is put into operation, hand-eye calibration is required to establish the transformation relationship from the camera's coordinate system to the robotic arm's coordinate system. The current hand-eye calibration process has a large error, resulting in low accuracy of the final calibration results. Summary of the Invention

[0003] This application provides a hand-eye calibration method, apparatus, electronic device, and storage medium, which can improve the accuracy of the hand-eye calibration process.

[0004] Firstly, a hand-eye alignment method is provided, the method comprising:

[0005] Determine the image coordinates of multiple feature points in the calibration board in the images captured by the camera;

[0006] For each feature point, after aligning the tool center point of the robotic arm to the feature point using a sensor, the mechanical coordinates of the tool center point are obtained.

[0007] A coordinate transformation matrix is ​​determined based on the image coordinates and machine coordinates corresponding to the multiple feature points, respectively.

[0008] In this embodiment, during the hand-eye calibration process, the image coordinates of multiple feature points on the calibration board in the images captured by the camera are determined. For each feature point, after aligning the tool center point of the robotic arm to the feature point using a sensor, the mechanical coordinates of the tool center point are obtained. Then, a coordinate transformation matrix is ​​determined based on the image coordinates and mechanical coordinates corresponding to the multiple feature points to complete the robot's hand-eye calibration. During the calibration process, aligning the feature points with the tool center point using a sensor significantly reduces errors and improves the accuracy of the calibration results compared to manually aligning the tool center point and feature points. Furthermore, when aligning the tool center point and feature points using a sensor, the robotic arm can be controlled to automatically align the tool center point and feature points, thereby shortening the time required for the hand-eye calibration process and improving calibration efficiency.

[0009] Optionally, before obtaining the mechanical coordinates of the tool center point after aligning the tool center point of the robotic arm to the feature point using a sensor, the method further includes:

[0010] The tool center point is controlled to move to a preset position corresponding to the feature point; wherein, the preset position is the predicted position where the tool center point is aligned with the feature point.

[0011] In this embodiment of the application, before obtaining the mechanical coordinates of the tool center point aligned with the feature point, the tool center point is positioned at a preset position corresponding to the feature point in advance. This can speed up the alignment of the tool center point and the feature point, thereby shortening the time required for the entire hand-eye calibration process and improving calibration efficiency.

[0012] Optionally, the plurality of feature points are respectively located within a plurality of feature regions of the calibration plate; the step of obtaining the mechanical coordinates of the tool center point after aligning the tool center point of the robotic arm to the feature points using a sensor includes:

[0013] If the sensor detects that the tool center point is not aligned with the feature point, then the tool center point is controlled to move within the feature area where the feature point is located;

[0014] During the process of controlling the movement of the tool's center point, when the sensor detects that the tool's center point is aligned with the feature point, the coordinates of the tool's center point are used as the mechanical coordinates.

[0015] In this embodiment, after moving the tool center point to a preset position, if the tool center point is not aligned with the preset position, the tool center point is moved within the feature area where the feature point is located. This enables a rapid search for the feature point, allowing for quick alignment of the tool center point and the feature point. This improves the efficiency of hand-eye calibration and shortens the time required for hand-eye calibration.

[0016] Optionally, controlling the movement of the tool's center point within the feature region where the feature point is located includes:

[0017] The tool's center point is controlled to move along a preset path with a preset step size within the feature region where the feature point is located;

[0018] Alternatively, the tool's center point can be controlled to move sequentially to each of a plurality of target locations; wherein the plurality of target locations are located within the feature region where the feature point is located, and the target locations are spaced apart from the preset locations by a preset step length.

[0019] In this embodiment of the application, during the process of controlling the center point of the tool to move within the feature area, the center point of the tool can move along a preset path with a preset step size or move sequentially to each target position. This can improve the alignment efficiency between the center point of the tool and the feature point, thereby shortening the time required for hand-eye calibration and improving the efficiency of hand-eye calibration.

[0020] Optionally, the sensor includes a transmitter disposed at each of the feature points and a receiver disposed at the center point of the tool;

[0021] When the sensor detects that the tool center point is aligned with the feature point, the coordinates of the tool center point are used as the machine coordinates, including:

[0022] Control the transmitter at the feature point to emit a detection signal;

[0023] When the receiver receives the detection signal, the coordinates of the tool's center point are used as the machine coordinates.

[0024] Optionally, a reflector is provided on the feature point, and the sensor is mounted at the tool center point. When the sensor detects that the tool center point is aligned with the feature point, the coordinates of the tool center point are used as the machine coordinates, including:

[0025] The sensor sends a detection signal in the direction of the feature point.

[0026] Upon receiving the detection signal reflected by the reflector at the feature point, the coordinates of the tool center point are used as the mechanical coordinates.

[0027] Optionally, the sensor is mounted on the feature point, and determining the image coordinates of the multiple feature points in the calibration plate in the image captured by the camera includes:

[0028] The image is obtained by taking a picture of the calibration board using the camera.

[0029] The image coordinates corresponding to each feature point are determined by using the coordinates of the sensor in the image.

[0030] In this embodiment, when a sensor is installed on the feature point, the image coordinates of the feature point can be determined by determining the coordinates of the sensor in the image, thus identifying and determining more accurate image coordinates.

[0031] Secondly, a hand-eye calibration device is provided, the device comprising:

[0032] The determination module is used to determine the image coordinates of multiple feature points in the calibration board in the images captured by the camera.

[0033] The acquisition module is used to acquire the mechanical coordinates of the tool center point for each feature point after aligning the tool center point of the robotic arm to the feature point using a sensor;

[0034] The determining module is further configured to determine a coordinate transformation matrix based on the image coordinates and the machine coordinates corresponding to the plurality of feature points, respectively.

[0035] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory is used to store executable program code; and the processor is used to call and run the executable program code from the memory, causing the electronic device to execute the hand-eye calibration method in any possible implementation of the first aspect.

[0036] Fourthly, a storage medium is provided that stores executable program code, which, when run on an electronic device, causes the electronic device to execute the hand-eye calibration method in any possible implementation of the first aspect.

[0037] Fifthly, an executable program code product is provided, comprising: executable program code that, when executed on an electronic device, causes the electronic device to execute the hand-eye calibration method in any possible implementation of the first aspect. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating an application scenario of a hand-eye calibration method provided in an embodiment of this application;

[0039] Figure 2 This is a flowchart illustrating the steps of a hand-eye calibration method provided in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram illustrating the alignment process between feature points and tool center points provided in an embodiment of this application;

[0041] Figure 4 This is a schematic flowchart of a hand-eye calibration method provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a hand-eye calibration device provided in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0045] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0046] Currently, the 9-point calibration method is generally used for robot hand-eye calibration. During calibration, a calibration board with multiple feature points (usually 9, which can be corner points) is selected. The calibration board is placed in the robot's working area, and then an image is captured by the robot's camera. Image recognition is then used to determine the coordinates of the 9 feature points in the image. These coordinates are the coordinates of the feature points in the camera's coordinate system (called the image coordinate system or camera coordinate system), and can be referred to as image coordinates or pixel coordinates.

[0047] For each feature point, after aligning the tool central point (TCP) of the robot arm with the feature point, the coordinates of the tool central point are determined. These coordinates are the coordinates of the tool point center in the robot arm's coordinate system (called the tool coordinate system, robot coordinate system, or machine coordinate system), and are referred to as machine coordinates or robot coordinates. Then, using the image coordinates and machine coordinates corresponding to multiple feature points respectively, a coordinate transformation matrix can be determined. After the robot is put into operation, the coordinate transformation matrix can convert the image coordinates of the parts in the camera coordinate system to the machine coordinates in the machine coordinate system. The robot arm uses the machine coordinates to determine the position of the parts and manipulate them, thus providing accurate position information for the robot arm's manipulation of the parts.

[0048] Aligning the tool center point with the feature point of the robotic arm refers to positioning the tool center point to coincide with the feature point. During this alignment process, the robotic arm is typically moved manually, and the alignment of the tool center point and feature point is observed. Once alignment is confirmed, the coordinates of the tool center point are used as the corresponding mechanical coordinates of the feature point. However, manual observation of the alignment can lead to significant errors, resulting in a large inaccuracy in the final coordinate transformation matrix and consequently, lower calibration accuracy. Furthermore, manual alignment of the tool center point and feature point is time-consuming and inefficient.

[0049] To address the aforementioned problems, this application provides a hand-eye calibration method. During the calibration process, the tool's center point and feature points are aligned using sensors. Compared to manually aligning these points, the error is smaller, thus reducing the error in the coordinate transformation matrix and improving the accuracy of the calibration results. Furthermore, when aligning the tool's center point and feature points using sensors, a robotic arm can be controlled to automatically align them, thereby reducing manual operation and shortening the calibration time, thus improving calibration efficiency.

[0050] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of a hand-eye calibration method provided in this application embodiment. The scenario includes a robotic arm 11 and a camera 12 in a robot. A calibration head 13 is mounted at the end of the robotic arm 11, and the end of the calibration head 13 is the tool center point of the robotic arm 11. The camera 12 is located outside the robotic arm 11. A calibration plate 14 is placed in the robot's working area. The surface of the calibration plate 14 has multiple circular feature regions 15, and the center point of each feature region 15 is a feature point 16.

[0051] The tool center point is equipped with a sensor 17. During the calibration process, the end of the calibration head 13 and the feature point 16 can be aligned through the sensor 17, that is, the tool center point is positioned to the feature point so that the tool center point coincides with the feature point.

[0052] See Figure 2 , Figure 2 This is a flowchart illustrating the steps of a hand-eye alignment method provided in an embodiment of this application. The method can be executed by an electronic device, such as a computer, and may include the following steps:

[0053] Step 201: Determine the image coordinates of multiple feature points in the calibration board in the images captured by the camera.

[0054] In this embodiment, during the hand-eye calibration process for the robot, a calibration board is first placed in the working area of ​​the robotic arm. An image is captured by a camera within the robot, and then image recognition is performed based on the captured image to determine the image coordinates of each feature point in the image. Figure 1 Taking the calibration board shown as an example, the electronic device is equipped with a neural network model for image recognition. After the image is captured, it can be input into the neural network model for image recognition to identify the image coordinates of the center point of each circular feature area in the calibration board, thereby obtaining the image coordinates corresponding to the nine feature points.

[0055] It should be noted that the number of feature points in the calibration board is not limited to nine. The specific number of feature points can be set according to requirements, and this embodiment does not impose any restrictions on this.

[0056] Step 202: For each feature point, after aligning the tool center point of the robotic arm to the feature point using the sensor, obtain the mechanical coordinates of the tool center point.

[0057] The sensor can be mounted at the center point of the tool, and the robotic arm corresponds to the camera. Aligning the center point of the robotic arm with the feature point means positioning the center point of the tool to the feature point.

[0058] For example, the sensor can be a voltage sensor. A voltage sensor can be installed at the end of the calibration head (i.e., the tool center point), and an electrode can be installed at each feature point. The electrode can output the target voltage. During hand-eye calibration, for each feature point, the electrode installed at the feature point can be activated first, causing the electrode to output the target voltage. Then, the robotic arm can be controlled to move, causing the end of the calibration head to abut against the calibration plate and move within the feature area where the feature point is located. During the movement, after confirming that the voltage sensor has detected the target voltage, the feature point is aligned with the tool center point. The coordinates of the tool center point at this time can be used as a mechanical coordinate corresponding to the feature point. This process can be repeated to obtain a mechanical coordinate corresponding to each feature point.

[0059] For example, the sensor can be a temperature sensor. A temperature sensor can be installed at the end of the calibration head (i.e., the tool center point), and a heating resistor can be installed at each feature point. When the heating resistor heats up, it raises the temperature of the feature point, making it higher than the ambient temperature. During hand-eye calibration, for each feature point, the heating resistor installed at the feature point can first be activated to make the temperature at the feature point higher than the ambient temperature. Then, the robotic arm can be controlled to move, causing the end of the calibration head to contact the calibration plate and move within the feature area where the feature point is located. During the movement, when the temperature detected by the temperature sensor is higher than the ambient temperature, it is determined that the feature point is aligned with the tool center point. The coordinates of the tool center point at this time can be used as a mechanical coordinate corresponding to the feature point. This process can be repeated to obtain a mechanical coordinate corresponding to each feature point.

[0060] It should be understood that the above are merely illustrative examples, and the specific types of sensors may include, but are not limited to, photoelectric sensors.

[0061] Step 203: Determine the coordinate transformation matrix based on the image coordinates and machine coordinates corresponding to multiple feature points.

[0062] like Figure 1 As shown, when the calibration plate has 9 feature points, 9 sets of coordinates can be obtained through steps 201 and 202. These 9 sets of coordinates correspond one-to-one with the 9 feature points. Each set of coordinates for a feature point includes one mechanical coordinate and one image coordinate. After obtaining the 9 sets of coordinates, the PnP (Perspective-n-Point) method can be used to determine the coordinate transformation matrix. The coordinate transformation matrix is ​​the calibration result, describing the transformation relationship between the camera coordinate system and the tool coordinate system.

[0063] After obtaining the coordinate transformation matrix, it can be stored. After the robot is put into operation, the coordinate transformation matrix can be used to convert the image coordinates of the parts sensed by the camera into the mechanical coordinates of the parts in the tool coordinate system. Then, the robot arm can be controlled based on the transformed mechanical coordinates, so that the actuator installed at the end of the robot arm can operate on the parts.

[0064] In practical applications, other methods besides the PnP method can also be used to determine the coordinate transformation matrix based on multiple sets of coordinates. This embodiment does not impose any restrictions on this.

[0065] In this embodiment, during the hand-eye calibration process, the image coordinates of multiple feature points on the calibration board in the images captured by the camera are determined. For each feature point, after aligning the tool center point of the robotic arm to the feature point using a sensor, the mechanical coordinates of the tool center point are obtained. Then, a coordinate transformation matrix is ​​determined based on the image coordinates and mechanical coordinates corresponding to the multiple feature points to complete the robot's hand-eye calibration. During the calibration process, aligning the feature points with the tool center point using a sensor significantly reduces errors and improves the accuracy of the calibration results compared to manually aligning the tool center point and feature points. Furthermore, when aligning the tool center point and feature points using a sensor, the robotic arm can be controlled to automatically align the tool center point and feature points, thereby shortening the time required for the hand-eye calibration process and improving calibration efficiency.

[0066] Optionally, after aligning the tool center point of the robotic arm to the feature point using sensors and before obtaining the mechanical coordinates of the tool center point, the method may further include:

[0067] The control tool's center point is moved to a preset position corresponding to the feature point; where the preset position is the position where the predicted tool's center point is aligned with the feature point.

[0068] In one implementation, after the calibration plate is placed in the working area of ​​the robotic arm, for each feature point, the position of the tool center point when it is aligned with the feature point can be predicted. This position is the preset position corresponding to the feature point. The preset position corresponding to each feature point can be predicted separately. The preset position is specifically the mechanical coordinate when the tool center point is in the preset position.

[0069] For each feature point, before aligning the tool center point with the feature point, the electronic device can control the movement of the robotic arm to move the tool center point to a preset position. Then, the sensor can align the tool center point with the feature point and obtain the mechanical coordinates of the tool center point when the feature point is aligned with the tool center point.

[0070] The preset position is the mechanical coordinate when the predicted tool center point is aligned with the feature point. After the tool center point is moved to the preset position, the tool center point may already be aligned with the feature point or be located near the feature point, which can greatly speed up the efficiency of aligning the tool center point with the feature point.

[0071] In this embodiment of the application, before obtaining the mechanical coordinates of the tool center point aligned with the feature point, the tool center point is positioned at a preset position corresponding to the feature point in advance. This can speed up the alignment of the tool center point and the feature point, thereby shortening the time required for the entire hand-eye calibration process and improving calibration efficiency.

[0072] Optionally, multiple feature points are located within multiple feature regions of the calibration plate; after aligning the tool center point of the robotic arm to the feature points using sensors, the mechanical coordinates of the tool center point are obtained, including:

[0073] If the sensor detects that the tool center point is not aligned with the feature point, then the tool center point is controlled to move within the feature area where the feature point is located.

[0074] During the movement of the control tool's center point, when the sensor detects that the tool's center point is aligned with the feature point, the coordinates of the tool's center point are used as the mechanical coordinates.

[0075] For example, when the sensor is a voltage sensor, for each feature point, after moving the tool center point to the preset position corresponding to the feature point, the electrode installed on the feature point can be controlled to output the target voltage. If it is determined that the voltage sensor has detected the target voltage, it is determined that the feature point and the tool center point are aligned, and the coordinates of the tool center point at this time can be obtained to obtain the mechanical coordinates corresponding to the feature point.

[0076] Conversely, if the voltage sensor does not detect the target voltage, it indicates that the feature point and the tool center point are not aligned, and the tool center point is located near the feature point. In this case, the robotic arm can be controlled to move the tool center point within the feature area where the feature point is located. During the movement, if the voltage sensor detects the target voltage, it indicates that the feature point and the tool center point are aligned, and then the mechanical coordinates of the tool center point can be obtained.

[0077] Optionally, during the movement of the tool's center point, the tool's center point can be controlled to move within a preset range at a preset position with a preset step size. Optionally, the preset range can be a circular area centered on the preset position, located within the feature area where the feature point is located, and smaller than the feature area. Figure 1 For example, the preset range can be a circular area with a radius of R centered at a preset position, where the radius R is smaller than the radius of the feature area. The preset step size can be, for example, Δx and Δy, meaning that each time the tool's center point moves, it can move Δx in the x-direction or Δy in the y-direction.

[0078] See Figure 3 , Figure 3 This is a schematic diagram illustrating the alignment process between feature points and tool center points provided in an embodiment of this application. For example... Figure 3 As shown, the convection process may include the following steps:

[0079] Step 301: Move the tool center point to the preset position corresponding to the feature point.

[0080] Step 302: Determine whether the tool center point and feature point are aligned using the sensor.

[0081] For example, when the sensor is a voltage sensor, for each feature point, after moving the tool center point to the corresponding preset position, the voltage detected by the voltage sensor is first determined. If the voltage sensor does not detect the target voltage, it is determined that the feature point and the tool center point are not aligned, and step 303 can be executed. If the voltage sensor detects the target voltage, it is determined that the feature point and the tool center point are aligned, and step 306 can be executed.

[0082] Step 303: Determine whether the number of times the tool's center point has moved has reached the preset number.

[0083] For example, for each feature point, after moving the tool's center point to the corresponding preset position, the corresponding number of moves can first be set to 0. During step 303, it is determined whether the number of moves has reached the preset number, which is the maximum number of moves. If the number of moves has not reached the preset number, step 304 is executed; if the maximum number of moves has been reached, step 305 is executed.

[0084] Step 304: Move the center point of the control tool once.

[0085] For example, during step 304, the electronic device controls the robotic arm to keep the coordinates of the tool center point in the z-direction constant (so that the tool center point is always in contact with the calibration plate), and moves the tool center point Δx in the x-direction or Δy in the y-direction to complete one movement of the tool center point. After each movement of the tool center point is completed, the process can return to step 302, and the movement count can be incremented by 1.

[0086] Step 305: Output a prompt message.

[0087] For example, when the number of moves reaches a preset number, it indicates that after multiple moves, the tool center point and the feature point are still not aligned. At this time, step 305 can be executed to output a prompt message, prompting the user to manually operate the robotic arm to manually align the tool center point and the feature point.

[0088] Step 306: Obtain machine coordinates.

[0089] For example, after the sensor detects that the tool center point is aligned with the feature point, step 306 can be executed to obtain the coordinates of the tool center point (these coordinates are the mechanical coordinates of the tool center point) and use these coordinates as the mechanical coordinates corresponding to the feature point.

[0090] In this embodiment, after moving the tool center point to a preset position, if the tool center point is not aligned with the preset position, the tool center point is moved within the feature area where the feature point is located. This enables a rapid search for the feature point, allowing for quick alignment of the tool center point and the feature point. This improves the efficiency of hand-eye calibration and shortens the time required for hand-eye calibration.

[0091] Optionally, the control tool's center point moves within the feature region where the feature point is located, including:

[0092] The control tool's center point moves along a preset path with a preset step size within the feature region where the feature point is located;

[0093] Alternatively, the control tool's center point can be moved sequentially to each of multiple target locations; wherein, all multiple target locations are located within the feature region where the feature point is located, and the target locations are spaced apart from the preset locations by a preset step length.

[0094] For example, for each feature point, the movement path of the tool's center point can be pre-planned. This movement path is the preset path corresponding to the feature point, and the preset path is located within the feature region where the feature point is located. The preset step size is, for example, Δ, which is the distance that the tool's center point moves each time it moves within the feature region.

[0095] For each feature point, after moving the tool center point to its corresponding preset position, if it is determined that the tool center point and the feature point are not aligned, the robotic arm can be controlled to move. Each time the robotic arm is controlled to move, the tool center point moves along a preset path by a preset step length Δx or Δy. After each movement of the tool center point, a sensor determines whether the tool center point and the feature point are aligned. When the tool center point and the feature point are aligned, the mechanical coordinates of the tool center point are acquired. When the tool center point and the feature point are not aligned, the tool center point continues to move along the preset path by a preset step length.

[0096] In practical applications, if the tool center point and feature point are not aligned after the preset path is completed, the electronic device can output a prompt message to prompt the user to manually operate the robotic arm to move the tool center point within the feature area until the sensor indicates that the tool center point and feature point are aligned.

[0097] For example, for each feature point, a preset position corresponding to the feature point is determined, and multiple target positions corresponding to the feature point are also determined. The target positions can be positions within the feature region where the feature point is located, separated from the preset position by a preset step size Δx or Δy. After moving the tool center point to the preset position corresponding to the feature point, if it is determined that the tool center point and the feature point are not aligned, the robotic arm can be controlled to move. Each time the robotic arm is controlled to move, the tool center point moves to one of the target positions.

[0098] Each time the tool center point moves to a target position, the sensor determines whether the tool center point is aligned with the feature point. When the tool center point is aligned with the feature point, the mechanical coordinates of the tool center point are acquired. When the tool center point is not aligned with the feature point, the tool center point continues to move to the next target position, and so on, until the sensor detects that the tool center point is aligned with the feature point.

[0099] In practical applications, if the tool center point is not aligned with the feature point at each target position during the movement of the tool center point to each target position, the electronic device can output a prompt message to prompt the user to manually operate the robotic arm to move the tool center point within the feature area until the sensor indicates that the tool center point is aligned with the feature point.

[0100] In this embodiment of the application, during the process of controlling the center point of the tool to move within the feature area, the center point of the tool can move along a preset path with a preset step size or move sequentially to each target position. This can improve the alignment efficiency between the center point of the tool and the feature point, thereby shortening the time required for hand-eye calibration and improving the efficiency of hand-eye calibration.

[0101] Optionally, the sensor includes a transmitter disposed at each feature point and a receiver disposed at the tool center point; when the sensor detects that the tool center point is aligned with the feature point, the coordinates of the tool center point are used as mechanical coordinates, including:

[0102] The transmitter at the control feature point emits a detection signal;

[0103] When the receiver receives the detection signal, the coordinates of the tool's center point are used as the machine coordinates.

[0104] For example, the sensor can be a photoelectric sensor, which includes a transmitter and a receiver. The transmitter emits a light signal (i.e., a detection signal), and the receiver receives the light signal. The receiver can be installed at the end of the calibration head (i.e., the tool center point), and a transmitter can be installed at each feature point on the calibration plate. During hand-eye calibration, for each feature point, the transmitter installed at the feature point is first activated, and the receiver installed on the calibration head is also activated. Then, the robotic arm can be controlled to move, causing the end of the calibration head to contact the calibration plate and move within the feature area where the feature point is located. During this movement, when the receiver receives the light signal, it can output a notification signal to the electronic device. Upon receiving the notification signal, the electronic device determines that the feature point is aligned with the tool center point and uses the coordinates of the tool center point at this time as a mechanical coordinate corresponding to the feature point. This process is repeated to obtain a mechanical coordinate corresponding to each feature point.

[0105] Optionally, a reflector is provided on the feature point, and a sensor is mounted at the tool center point. When the sensor detects that the tool center point is aligned with the feature point, the coordinates of the tool center point are used as the machine coordinates, including:

[0106] The sensor sends a detection signal in the direction of the feature point.

[0107] Upon receiving the detection signal reflected by the reflector at the feature point, the coordinates of the tool center point are used as the machine coordinates.

[0108] Optionally, when the sensor is a photoelectric sensor, a transmitter and receiver can be installed at the end of the calibration head (i.e., the tool center point), and a reflector for reflecting light signals can be installed on each feature point of the calibration plate. During hand-eye calibration, for each feature point, while controlling the end of the calibration head to abut against the calibration plate and move within the feature area of ​​the feature point, the transmitter is controlled to emit a light signal. When the receiver receives the light signal reflected by the reflector installed on the feature point, it can be determined that the feature point is aligned with the tool center point, and the coordinates of the tool center point at this time can be used as a mechanical coordinate corresponding to the feature point. This process can be repeated to obtain a mechanical coordinate corresponding to each feature point.

[0109] It should be understood that the above are merely illustrative examples, and the specific types of sensors may include, but are not limited to, photoelectric sensors. The methods for aligning feature points with the tool center point using sensors may include, but are not limited to, the examples described above.

[0110] Optionally, the sensor is mounted on the feature points to determine the image coordinates of multiple feature points in the calibration plate in the image captured by the camera, including:

[0111] The calibration board is photographed with a camera to obtain an image;

[0112] The image coordinates corresponding to each feature point are determined by the sensor's coordinates in the image.

[0113] In one implementation, when a sensor is mounted on a feature point, the image coordinates of the feature point can be determined by detecting the sensor's coordinates in the image. For example... Figure 1 As shown, when the sensor is a photoelectric sensor including a transmitter and a receiver, and a transmitter is installed at each feature point, in the process of determining the image coordinates of the feature points, after the image is captured by the camera, image recognition can be performed based on the captured image through a neural network model to identify the coordinates of each transmitter in the image, and the coordinates of the transmitter in the image are used as the image coordinates corresponding to the feature points.

[0114] In this embodiment, when a sensor is installed on the feature point, the image coordinates of the feature point can be determined by determining the coordinates of the sensor in the image, thus identifying and determining more accurate image coordinates.

[0115] See Figure 4 , Figure 4 This is a flowchart illustrating a hand-eye calibration method provided in an embodiment of this application. Figure 4 As shown, the method may include the following steps:

[0116] Step 401: Take a picture of the calibration board to obtain an image.

[0117] Step 402: Determine the image coordinates of each feature point in the image.

[0118] In this embodiment, during the hand-eye calibration process, the operator can first place the calibration board in the working area of ​​the robotic arm, and then start the electronic device. After the electronic device is started, it first takes an image of the calibration board through the camera, and then determines each feature area in the calibration board through image recognition, and determines the image coordinates corresponding to each feature point in the feature area.

[0119] Step 403: For each feature point, align the feature point with the tool center point and obtain the machine coordinates.

[0120] In this embodiment, after the electronic device determines each feature region in the calibration board through image recognition, for each feature point, it can sequentially align the feature point with the tool center point and obtain the mechanical coordinates of the tool center point when the feature point is aligned with the tool center point, thereby obtaining the mechanical coordinates corresponding to the feature point. In this way, a mechanical coordinate and image coordinate corresponding to each feature point can be obtained.

[0121] Step 404: Determine the coordinate transformation matrix based on multiple sets of machine coordinates and image coordinates.

[0122] Step 405: Save the coordinate transformation matrix.

[0123] In this embodiment, after obtaining an image coordinate and a mechanical coordinate corresponding to each feature point, the electronic device can determine the coordinate transformation matrix based on the mechanical coordinates and image coordinates corresponding to multiple feature points respectively, and then store the coordinate transformation matrix, thereby completing the hand-eye calibration of the robot.

[0124] The above text combined Figures 1 to 4 The hand-eye calibration method provided in the embodiments of this application has been described in detail; the following will be combined with Figure 5 and Figure 6The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0125] Figure 5 This is a schematic diagram of the structure of a hand-eye calibration device provided in an embodiment of this application. For example, as shown... Figure 5 As shown, the hand-eye calibration device 500 includes:

[0126] The determination module 501 is used to determine the image coordinates of multiple feature points in the calibration board in the images captured by the camera.

[0127] The acquisition module 502 is used to acquire the mechanical coordinates of the tool center point for each feature point after aligning the tool center point of the robotic arm to the feature point through the sensor;

[0128] The determining module 501 is further configured to determine a coordinate transformation matrix based on the image coordinates and the machine coordinates corresponding to the plurality of feature points respectively.

[0129] Optionally, the acquisition module 502 is further configured to, after aligning the tool center point of the robotic arm to the feature point via the sensor and before acquiring the mechanical coordinates of the tool center point, control the tool center point to move to a preset position corresponding to the feature point; wherein the preset position is the predicted position where the tool center point will be when aligned with the feature point.

[0130] Optionally, the acquisition module 502 is specifically used to control the tool center point to move within the feature area where the feature point is located if the sensor detects that the tool center point is not aligned with the feature point; during the process of controlling the tool center point to move, when the sensor detects that the tool center point is aligned with the feature point, the coordinates of the tool center point are used as the mechanical coordinates.

[0131] Optionally, the acquisition module 502 is specifically used to control the tool center point to move along a preset path with a preset step size within the feature area where the feature point is located; or, to control the tool center point to move sequentially to each of a plurality of target positions; wherein the plurality of target positions are located within the feature area where the feature point is located, and the target positions are spaced apart from the preset positions by a preset step size.

[0132] Optionally, the sensor includes a transmitter disposed at each of the feature points and a receiver disposed at the center point of the tool; the acquisition module 502 is specifically used to control the transmitter at the feature point to emit a detection signal; when the receiver receives the detection signal, the coordinates of the center point of the tool are used as the machine coordinates.

[0133] Optionally, a reflector is provided on the feature point, and the acquisition module 502 is specifically used to send a detection signal in the direction of the feature point through the sensor; when the detection signal reflected by the reflector on the feature point is received, the coordinates of the tool center point are used as the machine coordinates.

[0134] Optionally, the sensor is installed at the feature point, and the determining module 501 is specifically used to take a picture of the calibration board with the camera to obtain the image; and to determine the image coordinates corresponding to each feature point by using the coordinates of the sensor in the image.

[0135] It should be noted that the aforementioned hand-eye calibration device 500 is embodied in the form of a functional unit. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0136] For example, a "module" can be a software program, hardware circuit, or a combination of both that implements the above functions. Hardware circuits may include application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.

[0137] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of electronic device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0139] It should be understood that the device provided in this embodiment is used to perform the above-described hand-eye calibration method, and therefore can achieve the same effect as the above-described implementation method.

[0140] When using integrated units, the device may include a processing module and a storage module. The processing module may be a processor or a controller that implements or executes the various exemplary logic blocks, modules, and circuits shown in connection with the disclosure of this application. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0141] In addition, the device provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a hand-eye calibration method provided in the above embodiments.

[0142] This application also provides a storage medium storing executable program code. When the executable program code is run on an electronic device, the electronic device performs the above-described related method steps to implement the hand-eye calibration method provided in the above embodiments. The electronic device readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, Digital Video Discs (DVDs), Compact Disc Read-Only Memory (CD-ROM), microdrives, and magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), video random access memory (VRAM), flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and / or data.

[0143] This application also provides an executable program code product that, when run on an electronic device, causes the electronic device to perform the aforementioned related steps to implement the hand-eye calibration method provided in the above embodiments.

[0144] The storage medium, executable program code product or chip provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0145] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device is, for example, the push server described above.

[0146] For example, such as Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620, wherein the memory 610 stores executable program code 630, and the processor 620 is used to call and execute the executable program code 630 to perform a hand-eye calibration method.

[0147] For example, the memory 610 can be used to store the relevant program of the hand-eye calibration method provided in the embodiments of this application; the processor 620 can call the relevant program of the hand-eye calibration method stored in the memory 610 to execute the hand-eye calibration method of the embodiments of this application.

[0148] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0149] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0150] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hand-eye calibration method, characterized in that, The method includes: Determine the image coordinates of multiple feature points in the calibration board in the images captured by the camera; For each feature point, after aligning the tool center point of the robotic arm to the feature point using a sensor, the mechanical coordinates of the tool center point are obtained. A coordinate transformation matrix is ​​determined based on the image coordinates and machine coordinates corresponding to the multiple feature points, respectively.

2. The method as described in claim 1, characterized in that, After aligning the tool center point of the robotic arm to the feature point using a sensor and before obtaining the mechanical coordinates of the tool center point, the method further includes: The tool center point is controlled to move to a preset position corresponding to the feature point; wherein, the preset position is the predicted position where the tool center point is aligned with the feature point.

3. The method as described in claim 2, characterized in that, The plurality of feature points are respectively located within a plurality of feature regions of the calibration plate; the step of obtaining the mechanical coordinates of the tool center point after aligning the tool center point of the robotic arm to the feature points using a sensor includes: If the sensor detects that the tool center point is not aligned with the feature point, then the tool center point is controlled to move within the feature area where the feature point is located; During the process of controlling the movement of the tool's center point, when the sensor detects that the tool's center point is aligned with the feature point, the coordinates of the tool's center point are used as the mechanical coordinates.

4. The method as described in claim 3, characterized in that, The control of the tool's center point to move within the feature region where the feature point is located includes: The tool's center point is controlled to move along a preset path with a preset step size within the feature region where the feature point is located; Alternatively, the tool's center point can be controlled to move sequentially to each of a plurality of target locations; wherein the plurality of target locations are located within the feature region where the feature point is located, and the target locations are spaced apart from the preset locations by a preset step length.

5. The method as described in claim 3, characterized in that, The sensor includes a transmitter disposed at each of the feature points and a receiver disposed at the center point of the tool; When the sensor detects that the tool center point is aligned with the feature point, the coordinates of the tool center point are used as the machine coordinates, including: Control the transmitter at the feature point to emit a detection signal; When the receiver receives the detection signal, the coordinates of the tool's center point are used as the machine coordinates.

6. The method as described in claim 3, characterized in that, A reflector is provided on the feature point, and the sensor is mounted at the center point of the tool. When the sensor detects that the center point of the tool is aligned with the feature point, the coordinates of the center point of the tool are used as the machine coordinates, including: The sensor sends a detection signal in the direction of the feature point. Upon receiving the detection signal reflected by the reflector at the feature point, the coordinates of the tool center point are used as the mechanical coordinates.

7. The method according to any one of claims 1 to 6, characterized in that, The sensor is mounted on the feature point, and determining the image coordinates of the multiple feature points in the calibration plate in the image captured by the camera includes: The image is obtained by taking a picture of the calibration board using the camera. The image coordinates corresponding to each feature point are determined by using the coordinates of the sensor in the image.

8. A hand-eye calibration device, characterized in that, The device includes: The determination module is used to determine the image coordinates of multiple feature points in the calibration board in the images captured by the camera. The acquisition module is used to acquire the mechanical coordinates of the tool center point for each feature point after aligning the tool center point of the robotic arm to the feature point using a sensor; The determining module is further configured to determine a coordinate transformation matrix based on the image coordinates and the machine coordinates corresponding to the plurality of feature points, respectively.

9. An electronic device, characterized in that, include: Memory, used to store executable program code; A processor is configured to call and run the executable program code from the memory, causing the electronic device to perform the hand-eye calibration method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores executable program code, which, when executed by an electronic device, causes the electronic device to implement the hand-eye calibration method as described in any one of claims 1 to 7.

Images