Robot and camera calibration method, device, equipment, medium and product

Abstract

The invention provides a robot and camera calibration method, device, equipment, medium and product, which are applied to the technical field of computer vision, and the method comprises the following steps: in response to a calibration request, controlling a robot to move to a plurality of point locations; in the moving process of the robot, the compensation amount is determined according to the sensor data collected by the multi-mode sensor, and the compensation amount is adopted to carry out dynamic compensation on the robot and / or the camera; when the robot moves to each point location, acquiring image data shot by the camera, and determining second coordinate information of each point location according to the image data; and determining a calibration result of the robot and the camera based on the compensation amount, the first coordinate information and the second coordinate information. The compensation amount is determined according to the multi-mode sensor, the mechanical error of the robot and the image error of the camera are dynamically corrected, dynamic interference is eliminated, the calibration result is automatically determined in combination with coordinate information and the compensation amount, manual errors are avoided, and the production efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, medium, and product for robot and camera calibration. Background Technology

[0002] With the development of intelligent manufacturing automation, workshops typically use a combination of cameras and robots for automated production. During the application of intelligent manufacturing automation, when robots and cameras operate in combination, hand-eye calibration of the camera and robot is required.

[0003] In existing technologies, manual hand-eye calibration is generally performed, which results in long calibration times and the accuracy of manual operation depends on the operator's skill level, thus affecting the production efficiency of automated production lines. Summary of the Invention

[0004] In view of the above problems, a method, apparatus, device, medium, and product for robot and camera calibration are proposed to overcome or at least partially solve the above problems, including: A method for calibrating a robot with a camera, wherein the robot has an auxiliary calibration hole, a calibration rod is disposed in the auxiliary calibration hole, and a multimodal sensor is disposed on the calibration rod, including: In response to a calibration request, the robot is controlled to move to multiple points; wherein the multiple points have first coordinate information, which is coordinate information based on the robot coordinate system; During the robot's movement, a compensation amount is determined based on the sensor data collected by the multimodal sensor, and the compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, it acquires image data captured by the camera and determines the second coordinate information of each point based on the image data; wherein, the second coordinate information is coordinate information based on the camera coordinate system; Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration results of the robot and the camera are determined.

[0005] Optionally, controlling the robot to move to multiple points includes: In response to a calibration request, the robot is controlled to move to the first point. Based on the first coordinate information of the first point, the first coordinate information of other points is determined, and based on the first coordinate information of the other points, the robot is controlled to move to the other points; wherein, the first point is the intermediate point of the other points.

[0006] Optionally, the calibration rod is provided with an auxiliary calibration pattern, and based on the image data, the second coordinate information of each point is determined, including: Detect the target pixel region corresponding to the auxiliary calibration pattern in the image data, and determine the second coordinate information of each point based on the target pixel region.

[0007] Optionally, it also includes: When the target pixel region cannot be detected in the image data, the first coordinate information of the first point is updated, and the robot is re-controlled to move to multiple points.

[0008] Optionally, the multimodal sensor includes a force sensor, and the compensation amount includes a force compensation amount; Based on the sensor data collected by the multimodal sensor, a compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the force sensor, the force deviation of the robot's end effector is determined, and the force compensation amount is determined based on the force deviation of the end effector. The force compensation amount is used to dynamically compensate the end effector of the robot.

[0009] Optionally, the multimodal sensor includes a laser sensor, and the compensation amount includes a vibration compensation amount; Based on the sensor data collected by the multimodal sensor, a compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the laser sensor, the joint vibration amplitude deviation of the robot is determined, and the vibration compensation amount is determined based on the joint vibration amplitude deviation. The vibration compensation amount is used to dynamically compensate the joints of the robot.

[0010] Optionally, the multimodal sensor includes an ambient light sensor, and the compensation amount includes an ambient light compensation amount; Based on the sensor data collected by the multimodal sensor, a compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the ambient light sensor, the illumination intensity deviation of the camera is determined, and based on the illumination intensity deviation, the ambient light compensation amount is determined; The ambient light compensation amount is used to dynamically compensate the image exposure parameters of the camera.

[0011] Optionally, it also includes: During the robot's movement, when an abnormal load is detected at the robot's end effector, the coordinate compensation amount is determined based on the real-time force value at the robot's end effector. The planned target coordinate values ​​are corrected based on the coordinate compensation amount.

[0012] Optionally, based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result between the robot and the camera is determined, including: Based on the first coordinate information and the second coordinate information, determine the coordinate mapping relationship; Based on the compensation amount and coordinate mapping relationship, the calibration results of the robot and the camera are determined.

[0013] A device for calibrating a robot with a camera, the robot having an auxiliary calibration hole, a calibration rod disposed in the auxiliary calibration hole, and a multimodal sensor disposed on the calibration rod, comprising: A motion control module is used to control the robot to move to multiple points in response to a calibration request; wherein the multiple points have first coordinate information, which is coordinate information based on the robot coordinate system; The dynamic compensation module is used to determine the compensation amount based on the sensor data collected by the multimodal sensor during the movement of the robot, and to use the compensation amount to dynamically compensate the robot and / or the camera. The point calibration module is used to acquire image data captured by the camera when the robot moves to each point, and determine the second coordinate information of each point based on the image data; wherein, the second coordinate information is coordinate information based on the camera coordinate system; The calibration result determination module is used to determine the calibration result of the robot and the camera based on the compensation amount, the first coordinate information, and the second coordinate information.

[0014] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0015] A computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method described above.

[0016] A computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0017] The embodiments of the present invention have the following advantages: In this embodiment of the invention, in response to a calibration request, the robot is controlled to move to multiple points. Each point has first coordinate information, which is coordinate information based on the robot's coordinate system. During the robot's movement, a compensation amount is determined based on sensor data collected by a multimodal sensor, and this compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, image data captured by the camera is acquired, and second coordinate information for each point is determined based on the image data. This second coordinate information is coordinate information based on the camera's coordinate system. Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result of the robot and the camera is determined. This achieves the determination of the compensation amount based on the multimodal sensor, dynamically correcting robot mechanical errors and camera image errors, eliminating dynamic interference, and automatically determining the calibration result by combining coordinate information and the compensation amount, thus avoiding human error and improving production efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the steps of a method for robot and camera calibration provided in some embodiments of the present invention; Figure 2 This is a structural diagram of a robot and camera calibration device provided in some embodiments of the present invention; Figure 3 This is a structural diagram of a calibration rod provided in some embodiments of the present invention; Figure 4 This is a schematic diagram of an auxiliary calibration hole provided in some embodiments of the present invention; Figure 5 This is a flowchart of a second method for robot and camera calibration provided in some embodiments of the present invention; Figure 6 This is a flowchart of the steps of a method for robot and camera calibration provided in some embodiments of the present invention; Figure 7 This is a structural block diagram of a robot and camera calibration device provided in some embodiments of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] In practical applications, robots or cameras may experience certain pose deviations, which can lead to errors compared to the initially calibrated hand-eye coordinates, thus affecting the stability and accuracy of production. Therefore, after long-term operation on automated production lines, it is necessary to frequently perform hand-eye calibration operations on robots and cameras to ensure the accuracy of coordinate information between them.

[0022] In related technologies, hand-eye calibration operations are generally performed manually. However, manual operation is time-consuming, and the accuracy of manual operation is highly dependent on the operator's skill level, which can affect the production efficiency of automated production lines.

[0023] Based on this, this invention proposes adding auxiliary calibration holes and calibration rods to the robot to enable automatic hand-eye calibration, thereby improving production efficiency in automated production lines. Then, by adding multimodal sensors (such as laser sensors, force sensors, and ambient light sensors) to the calibration rods, the calibration results are dynamically corrected by collecting the robot's end force deviation, joint vibration amplitude deviation, and camera illumination intensity deviation in real time, in order to solve the problem of accuracy attenuation under complex working conditions.

[0024] Reference Figure 1 The diagram illustrates a flowchart of a robot-camera calibration method according to some embodiments of the present invention. Figure 2 The robot 1 is connected to the camera 2 and the calibration rod 3 for hand-eye calibration. For example... Figure 3 The calibration rod 3 has an auxiliary calibration pattern 4. For example... Figure 4 The robot 1 is connected to an auxiliary calibration hole 5 for mounting a calibration rod 3, and a multimodal sensor can be mounted in the calibration rod 3.

[0025] For example, multimodal sensors include laser sensors, force sensors, and ambient light sensors.

[0026] Among them, the laser sensor can detect vibrations of the robot's joints and minute changes in the end-effector's pose; the force sensor can monitor changes in the load and deformation of the robot's end-effector; and the ambient light sensor can collect the light intensity of the working environment, thereby optimizing the camera image processing parameters. In practical applications, sensors are installed in the calibration rod and auxiliary calibration hole. When they are fixed together, the sensors can output signals, which can then activate the multimodal sensor, enabling the robot to automatically begin hand-eye calibration.

[0027] Specifically, it may include the following steps: Step 101: In response to the calibration request, control the robot to move to multiple points; wherein the multiple points have first coordinate information, which is coordinate information based on the robot coordinate system.

[0028] In practical applications, calibration requests can be sent by operators through a control terminal. After receiving the calibration request, the robot performs automated hand-eye calibration. The robot can control itself to move to these points based on the first coordinate information of multiple points.

[0029] In some embodiments of the present invention, controlling the robot to move to multiple points includes: responding to a calibration request, controlling the robot to move to a first point; determining first coordinate information of other points based on first coordinate information of the first point, and controlling the robot to move to the other points based on the first coordinate information of the other points; wherein the first point is an intermediate point among the other points.

[0030] In practical applications, the first point can be a safety point, that is, a point that has a safe distance from other devices. When the robot moves to the first point, it can be used as an intermediate point with other points. Based on the first coordinate information of the first point and a preset algorithm, the first coordinate information of other points relative to the first point is calculated. The robot can then move to the other points in sequence according to the first coordinate information of the other points.

[0031] As examples, when an operator uses the automated hand-eye calibration function for the first time, the calibration rod can be placed in the robot's auxiliary calibration hole, and then the robot can be moved to the first point. The first coordinate information (X5, Y5) of the first point can be recorded. This first point can be used as the 5th point of the nine-point calibration in the hand-eye calibration, that is, the middle point.

[0032] After determining the first coordinate information of the first point, the coordinate information of the remaining 8 points in the nine-point calibration can be calculated (i.e., the first coordinate information of the other points relative to the first point). If the default interval between two adjacent points is 50mm (the operator can set the interval distance according to the actual situation), the first coordinate information of the other points relative to the first point is calculated as follows: (X1, Y1): X1=X5+50, Y1=Y5+50; (X2, Y2): X2=X5, Y2=Y5+50; (X3, Y3): X3=X5-50, Y3=Y5+50; (X4, Y4): X4=X5+50, Y4=Y5; (X6, Y6): X6=X5-50, Y6=Y5; (X7, Y7): X7=X5+50, Y7=Y5-50; (X8, Y8): X8=X5, Y8=Y5-50; (X9, Y9): X9=X5-50, Y9=Y5-50.

[0033] Step 102: During the movement of the robot, a compensation amount is determined based on the sensor data collected by the multimodal sensor, and the compensation amount is used to dynamically compensate the robot and / or the camera.

[0034] In practical applications, compensation amounts can be calculated based on sensor data collected by multimodal sensors as the robot moves to other locations. The calculated compensation amounts can then be used to dynamically compensate the robot and / or camera, enabling real-time correction of the robot and / or camera.

[0035] In the above embodiments, the sensor data collected by the multimodal sensor can dynamically correct errors caused by robot end-effector deformation, changes in ambient lighting, and joint vibration, thereby effectively improving calibration accuracy under complex working conditions.

[0036] In some embodiments of the present invention, the multimodal sensor includes a force sensor, and the compensation amount includes a force compensation amount; determining the compensation amount based on sensor data collected by the multimodal sensor, and using the compensation amount to perform dynamic compensation on the robot and / or the camera, includes: Based on the sensor data collected by the force sensor, the force deviation of the robot's end effector is determined, and the force compensation amount is determined based on the force deviation; the force compensation amount is then used to dynamically compensate the robot's end effector.

[0037] In practical applications, the force deviation at the robot's end effector during movement can be determined based on sensor data collected by the force sensor. For example, if the force sensor detects a sudden increase in force at the robot's end effector when it moves to a certain point, it indicates that the robot's joints are stuck or there is additional external resistance. Based on the force deviation data, the corresponding force compensation amount can be calculated using a preset algorithm.

[0038] Among them, the force compensation amount can be a parameter of the magnitude and direction of the force. The robot can dynamically adjust the end effector of the robot according to this force compensation amount, such as increasing or decreasing the corresponding driving force, so that the robot can move accurately to the next point according to the predetermined trajectory and posture, ensuring the accuracy of hand-eye calibration.

[0039] As some examples, force compensation can be calculated using the following formula:

[0040] Where K refers to the matrix, This refers to the force compensation amount; F error This refers to the force deviation at the robot's end effector, F. error = F actual -F safe F actual This refers to the actual value collected by the force sensor, F safe This refers to a pre-set safety value; This refers to the force compensation coefficient. This refers to the sensitivity matrix of the calibration matrix to the applied force.

[0041] In some embodiments of the present invention, the multimodal sensor includes a laser sensor, and the compensation amount includes a vibration compensation amount; based on the sensor data collected by the multimodal sensor, the compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the laser sensor, the joint vibration amplitude deviation of the robot is determined, and the vibration compensation amount is determined based on the joint vibration amplitude deviation; the vibration compensation amount is then used to dynamically compensate the joints of the robot.

[0042] In practical applications, the vibration amplitude deviation of robot joints during movement can be determined based on sensor data collected by laser sensors. For example, laser sensors monitor the movement trajectory of robot joints in real time. When the vibration amplitude of a joint exceeds the normal range, it indicates that the joint may have problems such as loosening or wear. Based on the joint vibration amplitude deviation data, the corresponding vibration compensation amount can be calculated using a preset algorithm.

[0043] The vibration compensation amount can be a parameter related to joint angle adjustment. The robot can dynamically adjust the joints based on this vibration compensation amount, such as fine-tuning the rotation angle of the joints, so that the vibration amplitude of the robot joints returns to a normal level, ensuring the stability and accuracy of the robot during movement, and thus ensuring the accuracy of hand-eye calibration.

[0044] As some examples, vibration compensation can be calculated in the following ways:

[0045] Where K refers to the matrix, This refers to the vibration compensation amount; This refers to the vibration compensation coefficient; 'i' refers to the joint. This refers to the vibration amplitude of the i-th joint; This refers to the vibration threshold. This refers to the sensitivity matrix of the calibration matrix to vibration.

[0046] In some embodiments of the present invention, the multimodal sensor includes an ambient light sensor, and the compensation amount includes an ambient light compensation amount; determining the compensation amount based on sensor data collected by the multimodal sensor, and using the compensation amount to perform dynamic compensation on the robot and / or the camera, includes: Based on the sensor data collected by the ambient light sensor, the illumination intensity deviation of the camera is determined, and the ambient light compensation amount is determined based on the illumination intensity deviation; the ambient light compensation amount is then used to dynamically compensate the image exposure parameters of the camera.

[0047] In practical applications, the ambient light sensor can be used to determine the light intensity deviation of the camera's environment based on sensor data collected by the ambient light sensor. For example, the ambient light sensor collects real-time data on the current ambient light intensity. When a deviation is detected between the light intensity and a preset light intensity value, it indicates that the current lighting conditions may affect the image quality captured by the camera. Based on the light intensity deviation data, a corresponding ambient light compensation amount can be calculated using a preset algorithm.

[0048] Among them, the ambient light compensation amount can be a parameter related to the adjustment of image exposure parameters. The camera can dynamically adjust the image exposure parameters according to this ambient light compensation amount, such as adjusting the exposure time and aperture size, so that the camera can take good quality images under different lighting conditions, ensuring the camera's accurate recognition of auxiliary calibration patterns in hand-eye calibration, thereby ensuring the accuracy of hand-eye calibration.

[0049] As examples, ambient light compensation can be calculated as follows:

[0050] Where K refers to the matrix, This refers to the amount of ambient light compensation; This refers to the ambient light compensation coefficient; I target This refers to the preset light intensity value; I current This refers to the current ambient light intensity; This refers to the sensitivity matrix of the calibration matrix to light intensity.

[0051] In some embodiments of the present invention, the method further includes: during the movement of the robot, when an abnormal load is detected at the end of the robot, determining a coordinate compensation amount based on the real-time force value at the end of the robot; and correcting the planned target coordinate value based on the coordinate compensation amount.

[0052] As examples, target coordinates refer to the expected coordinates of the initial coordinate information determined before the robot needs to move to each of the other points.

[0053] In practical applications, force sensors continuously monitor the force on the robot's end effector during its movement. When abnormal fluctuations in the force detected at the end effector, exceeding the preset normal load range, it indicates an abnormal load on the robot's end effector.

[0054] After detecting an abnormal load at the end of the robot, the coordinate compensation amount can be determined based on the real-time force value of the robot's end, the maximum allowable load of the end, and the compensation coefficient. The target coordinate value of the next point can be corrected by combining the coordinate compensation amount and the first coordinate information.

[0055] As examples, after calculating the first coordinate information of other points, the robot can move step by step to each point and make corrections during the movement. That is, before the robot moves to each point, it uses force sensors to detect whether the end effector has a sudden change in load due to the path trajectory (i.e., the end effector of the robot has an abnormal load).

[0056] For example, when the robot moves from point P1 to point P2, assuming the calibration rod at point P1 is stable at 2KG, during the calibration process, the robot moves to a critical point of a joint angle, such as a shoulder joint rotation angle greater than 120°. At this time, the direction of the robot's lever arm changes from horizontal to tilted, so the gravitational component of the calibration rod will increase. This is a sudden change in load.

[0057] If an anomaly is detected, a correction algorithm can be used to adjust the planned target coordinate values. The correction formula is as follows:

[0058] Among them, X new The X-coordinate value of the robot after calculation using a modified formula; Xoriginal This refers to the X-coordinate value of the robot's original plan (i.e., the X-coordinate value in the target coordinate value); This refers to the dynamic compensation amount in the X direction, the coordinate compensation amount that needs to be corrected due to sudden load changes. Y new The Y-coordinate value of the robot after calculation using a modified formula; Y original This refers to the Y-coordinate value of the robot's original plan (i.e., the Y-coordinate value in the target coordinate value); This refers to the dynamic compensation amount in the Y direction, which is the coordinate compensation amount that needs to be corrected due to sudden load changes.

[0059] in, and It can be determined using the following formula:

[0060] in, and This refers to the compensation coefficient; F max This refers to the maximum allowable load at the terminal; F error This refers to the real-time force value.

[0061] In the above embodiments, by automatically detecting anomalies and triggering a compensation mechanism, the time required for manual debugging can be reduced.

[0062] Step 103: When the robot moves to each point, it acquires image data captured by the camera and determines the second coordinate information of each point based on the image data; wherein the second coordinate information is coordinate information based on the camera coordinate system.

[0063] In practical applications, image data can contain auxiliary calibration patterns. As the robot moves to each preset point along the planned path, the camera can activate its shooting function to capture image data of the auxiliary calibration pattern at the current point. Then, using a preset pattern recognition algorithm, the center point or key feature points of the auxiliary calibration pattern are accurately located from the image. Based on the camera imaging principle, and combining the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters, the pixel coordinates in the image are converted into three-dimensional spatial coordinates in the camera coordinate system, i.e., the second coordinate information of each point.

[0064] In some embodiments of the present invention, the calibration rod is provided with an auxiliary calibration pattern, and the second coordinate information of each point is determined according to the image data, including: detecting a target pixel region corresponding to the auxiliary calibration pattern in the image data, and determining the second coordinate information of each point according to the target pixel region.

[0065] In practical applications, the calibration rod has an auxiliary calibration pattern. The camera can recognize the auxiliary calibration pattern for hand-eye calibration. When the camera detects a target pixel area corresponding to the auxiliary calibration pattern in the image data, it can calculate the second coordinate information of the point of the auxiliary calibration pattern in the camera's coordinate system based on the information in the target pixel area.

[0066] For example, when the robot performs hand-eye calibration at the first point (X5, Y5), it can send a signal to the camera and trigger the camera to take a picture. After the camera recognizes the calibration auxiliary pattern, it can record the second coordinate information (A5, B5) of the first point.

[0067] While the camera triggers the capture, multimodal sensors can be simultaneously activated to collect data. For example, a laser sensor can scan the robot's joint motion trajectory in real time to detect abnormal vibrations or displacements; a force sensor can record the force data of the robot's end effector when it is fixed on the calibration rod; and an ambient light sensor can collect the current ambient light intensity and transmit the data to the camera's image processing module to dynamically adjust the image exposure parameters.

[0068] In some examples, after calculating the first coordinate information of other points, the robot can move to other points in sequence, and the camera will take pictures in sequence and look for auxiliary calibration patterns to record the second coordinate information (A, B) of the camera at other points.

[0069] In some embodiments of the present invention, the method further includes: when the target pixel region cannot be detected in the image data, updating the first coordinate information of the first point and re-executing the control to move the robot to multiple points.

[0070] In practical applications, factors such as robot movement errors, camera shooting angle deviations, and obstruction or damage to the auxiliary calibration pattern may prevent the camera from detecting the target pixel area corresponding to the auxiliary calibration pattern in the captured image data. In such cases, to ensure the accuracy of hand-eye calibration, the first coordinate information of the first point can be updated, and the robot can be re-controlled to move to multiple points.

[0071] As some examples, if the default spacing between two adjacent points is 50mm, the first coordinate information of other points relative to the first point is calculated as follows: (X1, Y1): X1=X5+50, Y1=Y5+50; (X2, Y2): X2=X5, Y2=Y5+50; (X3, Y3): X3=X5-50, Y3=Y5+50; (X4, Y4): X4=X5+50, Y4=Y5; (X6, Y6): X6=X5-50, Y6=Y5; (X7, Y7): X7=X5+50, Y7=Y5-50; (X8, Y8): X8=X5, Y8=Y5-50; (X9, Y9): X9=X5-50, Y9=Y5-50.

[0072] When the robot moves to the point (X6, Y6) and the target pixel area cannot be detected in its image data, it may be because the distance between two adjacent points is too large or too small, resulting in the inability to recognize the pattern. Since the first coordinate information of the first point is determined (i.e., X5, Y5), the distance between two adjacent points can be reset based on the first point (e.g., increase the X5 of the first point by 50mm), and the first coordinate information of other points can be recalculated. Then, the robot moves to other points in sequence to perform hand-eye calibration based on the recalculated first coordinate information of other points.

[0073] In some examples, when the robot reaches other points based on the recalculated first coordinates, but the target pixel area still cannot be detected in its image data, the following approach can be used to gradually adjust the first coordinates of the first point, calculate the first coordinates of other points based on the adjusted first coordinates, and then move the robot to the other points sequentially for hand-eye calibration based on the recalculated first coordinates: 1. Subtract 50mm interval from the initial value of X5.

[0074] 2. Increase the spacing of Y5 by 50mm.

[0075] 3. Reduce the Y5 spacing by 50mm.

[0076] 4. At the same time, add a 50mm interval to X5 and Y5.

[0077] 5. At the same time, reduce the spacing of X5 and Y5 by 50mm.

[0078] 6. Increase the spacing of X5 by 50mm and decrease the spacing of Y5 by 50mm.

[0079] 7. Decrease the X5 spacing by 50mm and increase the Y5 spacing by 50mm.

[0080] Step 104: Based on the compensation amount, the first coordinate information, and the second coordinate information, determine the calibration results of the robot and the camera.

[0081] In practical applications, after obtaining the first coordinate information of the robot at each point, the second coordinate information obtained from the image data captured by the camera, and various compensation quantities (such as force compensation, vibration compensation, and ambient light compensation) calculated by various sensors, the calibration results of the robot and the camera can be determined based on these data.

[0082] For example, by combining the first and second coordinate information, a correspondence is established between the robot coordinate system and the camera coordinate system to determine the coordinate mapping relationship. Then, by combining the total compensation amount (summed from various compensation amounts) with the coordinate mapping relationship, the calibration results of the robot and the camera are determined.

[0083] In some embodiments of the present invention, determining the calibration result of the robot and the camera based on the compensation amount, the first coordinate information, and the second coordinate information includes: determining a coordinate mapping relationship based on the first coordinate information and the second coordinate information; and determining the calibration result of the robot and the camera based on the compensation amount and the coordinate mapping relationship.

[0084] As examples, the correspondence between the robot coordinate system and the camera coordinate system for all points can be generated based on the first coordinate information and the second coordinate information: (X1, Y1) = K1 (A1, B1), (X2, Y2) = K2 (A2, B2); (X3, Y3) = K3 (A3, B3), (X4, Y4) = K4 (A4, B4); (X5, Y5) = K5 (A5, B5), (X6, Y6) = K6 (A6, B6); (X7, Y7) = K7 (A7, B7), (X8, Y8) = K8 (A8, B8); (X9, Y9) = K9 (A9, B9).

[0085] Based on the correspondence of all points, the coordinate mapping relationship is determined by averaging, i.e., K. 相机 = (K1+K2+K3+K4+K5+K6+K7+K8+K9) / 9.

[0086] Summing up the various compensation amounts (force compensation, vibration compensation, and ambient light compensation) determines the total compensation amount:

[0087] in, This refers to the total compensation amount of the multimodal sensor; This refers to the force compensation amount; This refers to the amount of ambient light compensation; This refers to the vibration compensation amount.

[0088] By combining the total compensation amount and coordinate mapping relationship of the multimodal sensors, the calibration results of the robot and the camera are determined. ):

[0089] After determining the calibration results for the robot and camera, the calibration results can be output to the robot system and stored. During robot movement, motion data can be collected periodically to verify whether the robot's motion data conforms to K... 总 If an anomaly is detected in the calibration results, an anomaly warning can be output to the robot, causing the robot to start the automatic calibration program.

[0090] In this embodiment of the invention, in response to a calibration request, the robot is controlled to move to multiple points. Each point has first coordinate information, which is coordinate information based on the robot's coordinate system. During the robot's movement, a compensation amount is determined based on sensor data collected by a multimodal sensor, and this compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, image data captured by the camera is acquired, and second coordinate information for each point is determined based on the image data. This second coordinate information is coordinate information based on the camera's coordinate system. Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result of the robot and the camera is determined. This achieves the determination of the compensation amount based on the multimodal sensor, dynamically correcting robot mechanical errors and camera image errors, eliminating dynamic interference, and automatically determining the calibration result by combining coordinate information and the compensation amount, thus avoiding human error and improving production efficiency.

[0091] The following combination Figure 5 The present invention will be described by way of example: Step 501: The robot performs hand-eye calibration with the camera.

[0092] Step 502: The operator inserts the calibration rod into the auxiliary calibration hole, turns on the automatic hand-eye calibration function, and starts the dynamic compensation algorithm.

[0093] Step 503: The robot moves to the first point (X5, Y5) and triggers the camera to take a picture and record the second coordinate information (A5, B5) of the first point.

[0094] Step 504: Determine the first coordinate information of other points, move to other points in sequence, and trigger the camera to take pictures and record the second coordinate information of the points during the movement, and perform dynamic compensation.

[0095] When the robot moves to other locations, if the camera cannot detect the target pixel area in the image data, the first coordinate information of the first location is updated, and the robot is controlled to move to multiple locations again, recording the second coordinate information.

[0096] Step 505: Determine the coordinate mapping relationship between the first coordinate information and the second coordinate information, as well as the total compensation amount.

[0097] Step 506: Based on the coordinate mapping relationship and the total compensation amount, determine the calibration results of the robot and the camera.

[0098] In the above embodiments, by performing dynamic compensation during the robot's movement to the designated location, the positional accuracy of the robot and the camera can be effectively improved, thereby increasing the production precision and operational stability of the automated production line, reducing errors caused by human operation, and improving the accuracy of the automated production line.

[0099] In this embodiment of the invention, in response to a calibration request, the robot is controlled to move to multiple points. Each point has first coordinate information, which is coordinate information based on the robot's coordinate system. During the robot's movement, a compensation amount is determined based on sensor data collected by a multimodal sensor, and this compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, image data captured by the camera is acquired, and second coordinate information for each point is determined based on the image data. This second coordinate information is coordinate information based on the camera's coordinate system. Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result of the robot and the camera is determined. This achieves the determination of the compensation amount based on the multimodal sensor, dynamically correcting robot mechanical errors and camera image errors, eliminating dynamic interference, and automatically determining the calibration result by combining coordinate information and the compensation amount, thus avoiding human error and improving production efficiency.

[0100] Reference Figure 6 The diagram shows a flowchart of another method for calibrating a robot and a camera according to some embodiments of the present invention. The robot has an auxiliary calibration hole, a calibration rod is provided in the auxiliary calibration hole, and a multimodal sensor is provided on the calibration rod.

[0101] Specifically, it may include the following steps: Step 601: In response to the calibration request, control the robot to move to multiple points; wherein the multiple points have first coordinate information, which is coordinate information based on the robot coordinate system.

[0102] Step 602: During the movement of the robot, the force deviation of the robot's end effector is determined based on the sensor data collected by the force sensor, and the force compensation amount is determined based on the force deviation of the end effector.

[0103] Step 603: Use the force compensation amount to perform dynamic compensation on the end effector of the robot.

[0104] Step 604: During the movement of the robot, the joint vibration amplitude deviation of the robot is determined based on the sensor data collected by the laser sensor, and the vibration compensation amount is determined based on the joint vibration amplitude deviation.

[0105] Step 605: Use the vibration compensation amount to dynamically compensate the joints of the robot.

[0106] Step 606: During the robot's movement, the camera's illumination intensity deviation is determined based on sensor data collected by the ambient light sensor, and the ambient light compensation amount is determined based on the illumination intensity deviation.

[0107] Step 607: Use the ambient light compensation amount to dynamically compensate the image exposure parameters of the camera.

[0108] Step 608: When the robot moves to each point, it acquires image data captured by the camera and determines the second coordinate information of each point based on the image data; wherein the second coordinate information is coordinate information based on the camera coordinate system.

[0109] Step 609: Based on the compensation amount, the first coordinate information, and the second coordinate information, determine the calibration results of the robot and the camera.

[0110] In this embodiment of the invention, in response to a calibration request, the robot is controlled to move to multiple points. Each point has first coordinate information, which is coordinate information based on the robot's coordinate system. During the robot's movement, a compensation amount is determined based on sensor data collected by a multimodal sensor, and this compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, image data captured by the camera is acquired, and second coordinate information for each point is determined based on the image data. This second coordinate information is coordinate information based on the camera's coordinate system. Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result of the robot and the camera is determined. This achieves the determination of the compensation amount based on the multimodal sensor, dynamically correcting robot mechanical errors and camera image errors, eliminating dynamic interference, and automatically determining the calibration result by combining coordinate information and the compensation amount, thus avoiding human error and improving production efficiency.

[0111] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0112] Reference Figure 7 The diagram shows a schematic of a robot and camera calibration device according to some embodiments of the present invention. The robot has an auxiliary calibration hole, in which a calibration rod is provided, and the calibration rod is provided with a multimodal sensor.

[0113] Specifically, it can include the following modules: The mobile control module 701 is used to control the robot to move to multiple points in response to a calibration request; wherein the multiple points have first coordinate information, and the first coordinate information is coordinate information based on the robot coordinate system; The dynamic compensation module 702 is used to determine the compensation amount based on the sensor data collected by the multimodal sensor during the movement of the robot, and to use the compensation amount to perform dynamic compensation on the robot and / or the camera. The point calibration module 703 is used to acquire image data captured by the camera when the robot moves to each point, and determine the second coordinate information of each point based on the image data; wherein the second coordinate information is coordinate information based on the camera coordinate system; The calibration result determination module 704 is used to determine the calibration result of the robot and the camera based on the compensation amount, the first coordinate information, and the second coordinate information.

[0114] In some embodiments of the present invention, the motion control module 701 includes: The first point control submodule is used to control the robot to move to the first point in response to a calibration request; Other point control submodules are used to determine the first coordinate information of other points based on the first coordinate information of the first point, and control the robot to move to the other points based on the first coordinate information of the other points; wherein, the first point is the middle point of the other points.

[0115] In some embodiments of the present invention, the calibration rod is provided with an auxiliary calibration pattern, and the point calibration module 703 includes: The second coordinate information determination submodule is used to detect the target pixel region corresponding to the auxiliary calibration pattern in the image data, and determine the second coordinate information of each point based on the target pixel region.

[0116] In some embodiments of the present invention, the apparatus further includes: The coordinate information update module is used to update the first coordinate information of the first point when the target pixel area cannot be detected in the image data, and to re-execute the control to move the robot to multiple points.

[0117] In some embodiments of the present invention, the multimodal sensor includes a force sensor, and the compensation amount includes a force compensation amount; the dynamic compensation module 702 includes: The force compensation amount determination submodule is used to determine the end force deviation of the robot based on the sensor data collected by the force sensor, and to determine the force compensation amount based on the end force deviation. The end-effector compensation submodule is used to dynamically compensate the end of the robot using the force compensation amount.

[0118] In some embodiments of the present invention, the multimodal sensor includes a laser sensor, and the compensation amount includes a vibration compensation amount; the dynamic compensation module 702 includes: The vibration compensation amount determination submodule is used to determine the joint vibration amplitude deviation of the robot based on the sensor data collected by the laser sensor, and to determine the vibration compensation amount based on the joint vibration amplitude deviation. The joint compensation submodule is used to dynamically compensate the joints of the robot using the vibration compensation amount.

[0119] In some embodiments of the present invention, the multimodal sensor includes an ambient light sensor, and the compensation amount includes an ambient light compensation amount; the dynamic compensation module 702 includes: An ambient light compensation amount determination submodule is used to determine the light intensity deviation of the camera based on the sensor data collected by the ambient light sensor, and to determine the ambient light compensation amount based on the light intensity deviation. The image exposure parameter compensation submodule is used to dynamically compensate the image exposure parameters of the camera using the ambient light compensation amount.

[0120] In some embodiments of the present invention, the apparatus further includes: The coordinate compensation amount determination module is used to determine the coordinate compensation amount based on the real-time force value of the robot's end effector when an abnormal load is detected during the robot's movement. The target coordinate value correction module is used to correct the planned target coordinate values ​​based on the coordinate compensation amount.

[0121] In some embodiments of the present invention, the calibration result determination module 704 is used for: Based on the first coordinate information and the second coordinate information, determine the coordinate mapping relationship; Based on the compensation amount and coordinate mapping relationship, the calibration results of the robot and the camera are determined.

[0122] In this embodiment of the invention, in response to a calibration request, the robot is controlled to move to multiple points. Each point has first coordinate information, which is coordinate information based on the robot's coordinate system. During the robot's movement, a compensation amount is determined based on sensor data collected by a multimodal sensor, and this compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, image data captured by the camera is acquired, and second coordinate information for each point is determined based on the image data. This second coordinate information is coordinate information based on the camera's coordinate system. Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result of the robot and the camera is determined. This achieves the determination of the compensation amount based on the multimodal sensor, dynamically correcting robot mechanical errors and camera image errors, eliminating dynamic interference, and automatically determining the calibration result by combining coordinate information and the compensation amount, thus avoiding human error and improving production efficiency.

[0123] Some embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0124] Some embodiments of the present invention also provide a computer-readable storage medium on which a computer program is stored, and which, when executed by a processor, implements the method described above.

[0125] Some embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0126] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0128] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0129] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0134] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0135] The above provides a detailed description of the method, apparatus, device, medium, and product for robot and camera calibration. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for calibrating a robot with a camera, characterized in that, The robot has an auxiliary calibration hole, in which a calibration rod is disposed. The calibration rod is equipped with a multimodal sensor, including: In response to a calibration request, the robot is controlled to move to multiple points; wherein the multiple points have first coordinate information, which is coordinate information based on the robot coordinate system; During the robot's movement, a compensation amount is determined based on the sensor data collected by the multimodal sensor, and the compensation amount is used to dynamically compensate the robot and / or the camera. When the robot moves to each point, it acquires image data captured by the camera and determines the second coordinate information of each point based on the image data; wherein, the second coordinate information is coordinate information based on the camera coordinate system; Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration results of the robot and the camera are determined.

2. The method according to claim 1, characterized in that, Controlling the robot to move to multiple locations includes: In response to a calibration request, the robot is controlled to move to the first point. Based on the first coordinate information of the first point, the first coordinate information of other points is determined, and based on the first coordinate information of the other points, the robot is controlled to move to the other points; wherein, the first point is the intermediate point of the other points.

3. The method according to claim 2, characterized in that, The calibration rod is equipped with an auxiliary calibration pattern. Based on the image data, the second coordinate information of each point is determined, including: Detect the target pixel region corresponding to the auxiliary calibration pattern in the image data, and determine the second coordinate information of each point based on the target pixel region.

4. The method according to claim 3, characterized in that, Also includes: When the target pixel region cannot be detected in the image data, the first coordinate information of the first point is updated, and the robot is re-controlled to move to multiple points.

5. The method according to any one of claims 1-4, characterized in that, The multimodal sensor includes a force sensor, and the compensation amount includes a force compensation amount; Based on the sensor data collected by the multimodal sensor, a compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the force sensor, the force deviation of the robot's end effector is determined, and the force compensation amount is determined based on the force deviation of the end effector. The force compensation amount is used to dynamically compensate the end effector of the robot.

6. The method according to any one of claims 1-4, characterized in that, The multimodal sensor includes a laser sensor, and the compensation amount includes a vibration compensation amount; Based on the sensor data collected by the multimodal sensor, a compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the laser sensor, the joint vibration amplitude deviation of the robot is determined, and the vibration compensation amount is determined based on the joint vibration amplitude deviation. The vibration compensation amount is used to dynamically compensate the joints of the robot.

7. The method according to any one of claims 1-4, characterized in that, The multimodal sensor includes an ambient light sensor, and the compensation amount includes an ambient light compensation amount; Based on the sensor data collected by the multimodal sensor, a compensation amount is determined, and the compensation amount is used to dynamically compensate the robot and / or the camera, including: Based on the sensor data collected by the ambient light sensor, the illumination intensity deviation of the camera is determined, and based on the illumination intensity deviation, the ambient light compensation amount is determined; The ambient light compensation amount is used to dynamically compensate the image exposure parameters of the camera.

8. The method according to any one of claims 1-4, characterized in that, Also includes: During the robot's movement, when an abnormal load is detected at the robot's end effector, the coordinate compensation amount is determined based on the real-time force value at the robot's end effector. The planned target coordinate values ​​are corrected based on the coordinate compensation amount.

9. The method according to any one of claims 1-4, characterized in that, Based on the compensation amount, the first coordinate information, and the second coordinate information, the calibration result between the robot and the camera is determined, including: Based on the first coordinate information and the second coordinate information, determine the coordinate mapping relationship; Based on the compensation amount and coordinate mapping relationship, the calibration results of the robot and the camera are determined.

10. A device for calibrating a robot and a camera, characterized in that, The robot has an auxiliary calibration hole, in which a calibration rod is disposed. The calibration rod is equipped with a multimodal sensor, including: A motion control module is used to control the robot to move to multiple points in response to a calibration request; wherein the multiple points have first coordinate information, which is coordinate information based on the robot coordinate system; The dynamic compensation module is used to determine the compensation amount based on the sensor data collected by the multimodal sensor during the movement of the robot, and to use the compensation amount to dynamically compensate the robot and / or the camera. The point calibration module is used to acquire image data captured by the camera when the robot moves to each point, and determine the second coordinate information of each point based on the image data; wherein, the second coordinate information is coordinate information based on the camera coordinate system; The calibration result determination module is used to determine the calibration result of the robot and the camera based on the compensation amount, the first coordinate information, and the second coordinate information.

11. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 9.

Images