Camera posture calibration method, device and electronic equipment for robot system

Abstract

The present application provides a camera pose calibration method, device and electronic device for a robot system, wherein a specific implementation of the method includes: obtaining a first rough pose of a calibration object in a first coordinate system and a second rough pose of a camera in a second coordinate system; setting a distance parameter from the camera to the calibration object, and setting a motion waypoint sequence of a manipulator; selecting multiple target waypoints from the motion waypoint sequence, and determining multiple pose groups according to the multiple target waypoints, the first rough pose and the second rough pose, each pose group corresponding to a target waypoint, each pose group including a third pose of the end of the manipulator in a base coordinate system obtained based on the corresponding target waypoint and a fourth pose of the calibration object in a camera coordinate system; performing an iterative calibration operation according to the multiple pose groups, and obtaining a calibration value of the second rough pose when the cross-validation error is less than a first error threshold. The method can improve the accuracy of the calibration result.

Description

Technical Field

[0001] The present application relates to the field of computer network technology, and in particular to a camera posture calibration method, device and electronic equipment for a robot system. Background Art

[0002] In a robot system that includes a vision module, it is necessary to determine the relative position relationship between the camera and the robot in order to correctly apply the image information obtained by the vision module to the robot system. The process of determining the above relative position relationship is called hand-eye calibration.

[0003] The hand-eye calibration process is usually divided into two cases: "eye on hand" (that is, the camera is located on the robot's mechanical arm) and "eye outside hand" (that is, the camera is located outside the robot's mechanical arm, for example, the camera can be fixed to a coordinate system that is fixed relative to the base of the mechanical arm). When in the "eye outside hand" case, the above relative position relationship can be regarded as the conversion relationship between the camera coordinate system and the robot base coordinate system; when in the "eye on hand" case, the above relative position relationship can be regarded as the conversion relationship between the camera coordinate system and the robot's mechanical arm end coordinate system.

[0004] In the related art, in order to determine the above conversion relationship, there are the following manual calibration method and automatic calibration method.

[0005] The above manual calibration method obtains the robot's posture information and the posture information of the calibration object in the camera coordinate system by manually teaching the robot, so as to calculate the coordinate transformation relationship to be required. Here, the coordinate transformation relationship to be required can usually be solved in the following way, taking the case of "eyes outside the hand" as an example: by moving the robot arm with the calibration object fixed at the end to multiple positions, collecting the posture of the end of the robot arm and the posture of the calibration object at these multiple positions, and using the transformation relationship between each coordinate system, the following equation can be constructed: in, represents the end position of the robot arm at the i-th motion position; The hand-eye calibration result to be solved; Represents the calibration object pose at the i-th position. Through the form AX=XB (where, ) equation, the calibration result can be obtained.

[0006] The above automatic calibration method can generally adopt a method of setting motion waypoints for the robot, automatically collecting the robot's posture information and image information, and determining the calibration result based on the robot's posture information and image information.

[0007] Although the above two methods can obtain calibration results, the accuracy of the calibration results is low. Summary of the invention

[0008] The purpose of the embodiments of the present application is to provide a camera pose calibration method, device and electronic equipment for a robot system.

[0009] In the first aspect, the present application provides a camera pose calibration method for a robot system. The method includes: obtaining a first rough pose of a calibration object in a first coordinate system and a second rough pose of a camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are respectively a robot end coordinate system and a base coordinate system; setting a distance parameter from the camera to the calibration object, and setting a motion waypoint sequence of the robot based on the distance parameter; selecting multiple target waypoints from the motion waypoint sequence, and determining multiple pose groups according to the multiple target waypoints, the first rough pose and the second rough pose, each pose group corresponds to a target waypoint, and each pose group includes a third pose of the robot end in the base coordinate obtained based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system; performing an iterative calibration operation according to the multiple pose groups, and obtaining a calibration value of the second rough pose when the cross-validation error is less than the first error threshold. In this way, the cross-validation error can be gradually reduced by determining multiple pose groups for iterative calibration operations, and the first error threshold is used to provide a quantitative index for the calibration result, thereby improving the accuracy of the calibration result.

[0010] Preferably, the step of selecting a plurality of target waypoints from the moving waypoint sequence and determining a plurality of posture groups according to the plurality of target waypoints, the first rough posture and the second rough posture comprises: randomly selecting a target waypoint from the moving waypoint sequence; obtaining the corresponding third posture of the end of the manipulator according to the target waypoint, the first rough posture and the second rough posture; judging whether the fourth posture of the calibration object in the camera coordinate system can be read from the image captured by the camera when the end of the manipulator is in the third posture; if not, returning to the step of randomly selecting a target waypoint from the moving waypoint sequence; if so, storing the third posture and the corresponding fourth posture as a posture group until a plurality of posture groups are determined. In this way, a plurality of posture groups with high credibility can be determined, laying a foundation for improving the accuracy of the calibration results.

[0011] Preferably, the step of obtaining the corresponding third posture of the end of the manipulator according to the target waypoint, the first rough posture and the second rough posture comprises: calculating the corresponding third preliminary posture of the end of the manipulator according to the target waypoint, the first rough posture and the second rough posture; judging whether the end of the manipulator can reach the position corresponding to the third preliminary posture; if not, returning to the step of randomly selecting a target waypoint from the motion waypoint sequence; if yes, setting the third preliminary posture as the third posture. In this way, the end of the manipulator can accurately reach the third posture, thereby improving the accuracy of the calibration result.

[0012] Preferably, the iterative calibration operation is performed according to the multiple posture groups to obtain the calibration value of the second coarse posture when the cross-validation error is less than the first error threshold, including: calculating the first calibration value of the second coarse posture according to the multiple posture groups; calculating the cross-validation error corresponding to the first calibration value; if the cross-validation error is less than the first error threshold, using the first calibration value as the calibration value of the second coarse posture; if the cross-validation error is greater than the first error threshold, continuing to calibrate the second coarse posture until the cross-validation error is less than the first error threshold, so as to obtain the target calibration result.

[0013] Preferably, if the cross-validation error is greater than the first error threshold, the second coarse pose is continuously calibrated until the cross-validation error is less than the first error threshold, including: if the cross-validation error is greater than the first error threshold and less than the second error threshold, the first coarse pose and the second coarse pose are updated according to the first calibration value, and the step of randomly selecting a target waypoint from the moving waypoint sequence is returned to realize an iterative calibration process.

[0014] Preferably, updating the first coarse pose and the second coarse pose according to the first calibration value includes: taking the first calibration value as the new second coarse pose, and determining a new first coarse pose based on the new second coarse pose.

[0015] Preferably, if the camera is disposed on the robotic arm, the first coordinate system is a base coordinate system; if the camera is disposed outside the robotic arm, the first coordinate system is a robotic arm end coordinate system.

[0016] In a second aspect, the present application provides a camera pose calibration device for a robot system, comprising: a first acquisition module, used to acquire a first rough pose of a calibration object in a first coordinate system and a second rough pose of a camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are a robot arm end coordinate system and a base coordinate system, respectively; a setting module, used to set a distance parameter from the camera to the calibration object, and to set a motion waypoint sequence of the robot arm based on the distance parameter; a second acquisition module, used to select a plurality of target waypoints from the motion waypoint sequence, and to determine a plurality of pose groups according to the plurality of target waypoints, the first rough pose and the second rough pose, each pose group corresponding to a target waypoint, and each pose group including a third pose of the robot arm end in the base coordinate system acquired based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system; a calibration module, used to perform an iterative calibration operation according to the plurality of pose groups to obtain a calibration value of the second coarse pose when the cross-validation error is less than a first error threshold.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method provided in the first aspect are performed.

[0019] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flow chart of a camera pose calibration method for a robot system provided in an embodiment of the present application;

[0022] Figure 2 A flow chart of a method for determining multiple posture groups involved in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a camera posture calibration device for a robot system provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of an electronic device for executing a camera pose calibration method for a robot system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0027] With the development of intelligent technologies such as the Internet of Things, artificial intelligence, and big data, the demand for using these intelligent technologies to transform and upgrade the traditional logistics industry has become increasingly strong, and intelligent logistics (Intelligent Logistics System) has become a research hotspot in the logistics field. Intelligent logistics uses artificial intelligence, big data, and various information sensors, radio frequency identification technology, global positioning system (GPS) and other Internet of Things devices and technologies, which are widely used in basic activities such as material transportation, warehousing, distribution, packaging, loading and unloading, and information services, to achieve intelligent analysis and decision-making, automated operation, and efficient optimization management of the material management process. Internet of Things technologies include sensor equipment, RFID technology, laser infrared scanning, infrared sensing recognition, etc. The Internet of Things can effectively connect materials in logistics with the network, monitor materials in real time, and sense environmental data such as humidity and temperature in the warehouse to ensure the storage environment of materials. Through big data technology, all data in logistics can be sensed and collected, uploaded to the data layer of the information platform, and filtered, mined, and analyzed, and finally provide accurate data support for business processes (such as transportation, warehousing, storage and retrieval, picking, packaging, sorting, outbound, inventory, distribution, etc.). The application of artificial intelligence in logistics can be roughly divided into two directions: 1) AI-enabled intelligent devices such as unmanned trucks, AGVs, AMRs, forklifts, shuttles, stackers, unmanned delivery vehicles, drones, service robots, robotic arms, and smart terminals to replace some manual labor; 2) Software systems such as transportation equipment management systems, warehouse management, equipment scheduling systems, and order distribution systems driven by technologies or algorithms such as computer vision, machine learning, and operations optimization to improve labor efficiency. With the research and progress of smart logistics, this technology has been applied in many fields, such as retail and e-commerce, electronic products, tobacco, medicine, industrial manufacturing, footwear, textiles, and food.

[0028] In the related art, there is a problem of low accuracy of camera pose calibration results; in order to solve this problem, the present application provides a camera pose calibration method, device and electronic device for a robot system; further, by first obtaining a first rough pose of a calibration object in a first coordinate system and a second rough pose of a camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are a robot end coordinate system and a base coordinate system, respectively; then, setting a distance parameter from the camera to the calibration object, and setting a motion waypoint sequence of the robot arm based on the distance parameter; secondly, selecting a plurality of target waypoints from the motion waypoint sequence, and determining a plurality of pose groups according to the plurality of target waypoints, the first rough pose and the second rough pose, each pose group corresponding to a target waypoint, and each pose group including a third pose of the robot end in the base coordinate system obtained based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system; finally, performing an iterative calibration operation according to the plurality of pose groups, and obtaining a calibration value of the second rough pose when the cross-validation error is less than a first error threshold. In this way, by determining multiple posture groups for iterative calibration operations, the cross-validation error can be gradually reduced, and the first error threshold is used to provide a quantitative indicator for the calibration result, thereby improving the accuracy of the calibration result.

[0029] The defects existing in the above-mentioned related technologies are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed in the embodiments of the present invention for the above-mentioned problems below should be the contributions made by the inventor to the present invention during the process of the present invention.

[0030] For the convenience of writing, this application uses a unified notation method to mark the position and posture of each object when giving examples. Furthermore, the notation method uses "H" as the symbol to represent the position and posture, the subscript of "H" as the target object, and the superscript of "H" as the target coordinate system where the target object is located. "Marker" is used to represent the calibration object; cam is used to represent the camera or camera coordinate system; "base" is used to represent the base coordinate system; endlink is used to represent the end coordinate system of the robot arm, etc. The object is represented by its English name or English abbreviation. For example, It can be regarded as the position and posture of the calibration object in the base coordinate system; It can be regarded as the position and posture of the camera in the coordinate system of the end of the robotic arm.

[0031] Please refer to Figure 1 , Figure 1This is a flowchart of a camera pose calibration method for a robot system in some embodiments of the present application. The camera pose calibration method for the robot system can be used to calculate the relative position relationship between the camera and the robot, and can be applied to a device such as a server or a cloud platform that can perform information exchange and data calculation. The method may include the following steps:

[0032] Step 101, obtaining a first rough pose of the calibration object in a first coordinate system and a second rough pose of the camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are a robot end coordinate system and a base coordinate system respectively;

[0033] The calibration object may be a calibration plate with marks. The first rough posture and the second rough posture may be approximate postures detected by simple measuring tools such as a tape measure and a ruler. The base coordinate system may be a coordinate system established with a reference point selected on the robot body (e.g., a robot base) as the coordinate origin. The end-of-arm coordinate system may be a coordinate system established with the end-of-arm of the robot as the coordinate origin.

[0034] In the hand-eye calibration process, there are two situations: the camera is on the robotic arm and the camera is outside the robotic arm. Specifically, if the camera is set on the robotic arm, the first coordinate system is the base coordinate system; if the camera is set outside the robotic arm, the first coordinate system is the robotic arm end coordinate system.

[0035] That is, if the camera is set on the robot arm, at this time, the first coordinate system is the base coordinate system, and the second coordinate system is the robot arm end coordinate system. Further, the first rough pose obtained at this time is the pose of the calibration object in the base coordinate system (for example, it can be recorded as ); The second rough pose obtained is the pose of the camera in the coordinate system of the end of the robot arm (for example, it can be recorded as ).

[0036] If the camera is set outside the robot, at this time, the first coordinate system is the robot end coordinate system, and the second coordinate system is the base coordinate system. Further, the first rough pose obtained at this time is the pose of the calibration object in the robot end coordinate system (for example, it can be recorded as ); The second rough pose obtained is the pose of the camera in the base coordinate system (for example, it can be recorded as ).

[0037] Step 102, setting a distance parameter from the camera to the calibration object, and setting a motion waypoint sequence of the robotic arm based on the distance parameter;

[0038] In some application scenarios, the distance parameter between the camera and the calibration object can be set. In these application scenarios, the above distance parameter can be set based on the fact that the calibration object can be observed in the camera. For example, when the calibration object can be observed in the camera, the above distance parameter is set to 50 cm, 1 meter, etc. It should be noted that the above distance parameter can represent the height between the origin of the coordinate system and the calibration object in the camera coordinate system established with the camera shooting angle of view as the origin of the coordinate system, that is, the coordinate of the calibration object in the camera coordinate system is (0,0,h). For example, the coordinate of the camera shooting angle of view in the camera coordinate system is (0,0,0). When the above distance parameter is 1 meter, the coordinate of the calibration object in the camera coordinate system can be (0.0.1).

[0039] After setting the above distance parameters, the motion waypoint sequence of the robot arm can be set based on the distance parameters.

[0040] The above-mentioned motion waypoint sequence can be regarded as a set of position information of various positions (ie, motion waypoints) to which the robot arm needs to move.

[0041] In some application scenarios, for example, the homogeneous transformation matrix corresponding to each moving waypoint can be calculated first, so as to obtain the above moving waypoint sequence through the homogeneous transformation matrix. Specifically, for example, the moving waypoint sequence with coordinates (x w ,y w ,z w ) is set as the basic waypoint A. If the coordinates of the basic waypoint A are marked as p base-waypoint , then p base-waypoint =(x w ,y w ,z w ), then the position information corresponding to the basic waypoint A can be represented in the form of a vector, and the obtained vector can be normalized to process the waypoint information more conveniently. For example, the result obtained after normalization can be regarded as the vertical axis coordinate information, and then the vector outer product corresponding to the vertical axis coordinate information can be calculated to obtain the horizontal axis coordinate information or the vertical axis coordinate information corresponding to the vertical axis coordinate information, so as to obtain the above-mentioned homogeneous transformation matrix. For example, the normalized result can be recorded as p norm , then the vertical axis coordinate information z corresponding to the basic waypoint A is axis =p norm , then the horizontal axis coordinate information x can be calculated by, for example, [1,1,0] axis , that is, x axis =[1,1,0]×z axis Then the above vertical axis coordinate information y can be calculated axis , that is, y axis =z axis × axis. Further, we can get the rotation matrix R = And the translation matrix T = p base-waypoint T Then we can get the homogeneous transformation matrix H cam-waypoint , that is

[0042] Get the above homogeneous transformation matrix H cam-waypoint Afterwards, the homogeneous transformation matrix H can be determined cam-waypoint The multiple results obtained after rotating the horizontal axis, the vertical axis, and the vertical axis of the coordinate system by preset angles respectively. The multiple results obtained here can be regarded as a motion waypoint sequence.

[0043] In these application scenarios, perturbation data can be added to the basic waypoint A so that the data obtained in the subsequent iterative calibration operation can have a smaller fluctuation, which is conducive to obtaining the corresponding calibration result through the basic waypoint A calculation. For example, the perturbation data can be set to δ cm, that is, if the coordinates of the basic waypoint A are (0,0,h), the coordinates after perturbation can be (x w ,y w ,z w ), where x w ∈[-δ,+δ],y w ∈[-δ,+δ],z w ∈[h-δ, h+δ]. Here, δ may include 4, 5, etc.

[0044] Step 103, selecting a plurality of target waypoints from the motion waypoint sequence, and determining a plurality of pose groups according to the plurality of target waypoints, the first rough pose, and the second rough pose, each pose group corresponding to a target waypoint, each pose group including a third pose of the end of the manipulator in the base coordinate system obtained based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system;

[0045] After obtaining the motion waypoint sequence, multiple target waypoints can be selected from the motion waypoint sequence to determine multiple posture groups using the multiple target waypoints.

[0046] In some application scenarios, multiple target waypoints can be selected from a moving waypoint sequence by a preset rule. The preset rule here can, for example, include selecting multiple target waypoints in sequence according to the time sequence of obtaining the waypoints. The preset rule here can, for example, also include selecting according to the position of the waypoints in the moving waypoint sequence (for example, the time sequence of the waypoints can be obtained to arrange the waypoints and obtain the position of the waypoints in the moving waypoint sequence). Further, for example, the waypoints located in the odd or even positions of the moving waypoint sequence can be determined as target waypoints, thereby obtaining multiple target waypoints.

[0047] In other application scenarios, a target waypoint B can be randomly selected from the motion waypoint sequence, and the corresponding pose group can be calculated according to the target waypoint B, the first rough pose and the second rough pose. The pose group here can include the third pose of the end of the manipulator in the base coordinate system obtained at the target waypoint B and the fourth pose of the calibration object in the camera coordinate system. Then, by analogy, multiple target waypoints can be selected from the motion waypoint sequence to determine multiple pose groups. Further, after a target waypoint is randomly selected, the target waypoint can be deleted from the motion waypoint sequence to avoid the situation where the same calibration result is obtained by randomly selecting the same target waypoint next time. For example, the motion waypoint sequence includes waypoint A, waypoint B and waypoint C. If waypoint A is randomly selected as the target waypoint for the first time, after obtaining the first pose group. The second time, one of the motion waypoint sequences with only waypoint B and waypoint C can be randomly selected as the current target waypoint.

[0048] Step 104 , performing an iterative calibration operation according to the plurality of posture groups to obtain a calibration value of the second coarse posture when the cross-validation error is less than a first error threshold.

[0049] The above-mentioned iterative calibration operation can be regarded as multiple calibration operations based on different pose groups. The calibration operation here can be regarded as an operation to obtain the calibration result based on the last third pose and the last fourth pose. That is, the pose of the camera in the second coordinate system can be obtained based on the last third pose and the last fourth pose, and then the pose of the calibration object in the first coordinate system can be determined based on the pose, and then the corresponding calibration result of the current time can be re-determined.

[0050] In some application scenarios, the calibration value of the second coarse pose when the cross-validation error is less than the first error threshold can be regarded as the target calibration result.

[0051] The cross-validation error can be determined, for example, by means of K-fold cross-validation. For example, after determining n pose groups, the cross-validation error can be determined based on these n pose groups. Specifically, for example, for the ith pose group in the data set using the n pose groups mentioned above, an equation of the form AX=XB can be constructed and solved to obtain the pose a of the camera in the second coordinate system determined by the (n-1) pose groups after removing the ith pose group mentioned above; then, the pose b' of the calibration object in the first coordinate system corresponding to the ith pose group can be calculated based on the pose a, and then the error e between the pose b' and the actual pose b of the calibration object in the first coordinate system actually corresponding to the ith pose group can be calculated. i Finally, the error e corresponding to each pose group can be calculated separately i, to accumulate the n error data, and then find the error average e corresponding to the n data errors, so that the error average e can be determined as the cross-validation error. Here, if the camera is set outside the robot arm, in the above equation of the form AX=XB, If the camera is mounted on a robotic arm, in the above equation of the form AX=XB,

[0052] Furthermore, taking the case where the camera is set outside the robot arm as an example, the process of obtaining the cross-validation error is further explained. By constructing the above equation of the form AX=XB, the pose of the camera in the base coordinate system determined by the remaining (n-1) pose groups after removing the i-th pose group is obtained. (that is, the above-mentioned posture a), we can calculate Calculated (That is, the above-mentioned posture b'). Then we can calculate Calculated (i.e., the above-mentioned posture b), which can then be obtained by, for example, making a difference and The error between i Finally, we can calculate Get the cross validation error e.

[0053] After obtaining the cross-validation error, it can be determined whether the cross-validation error is less than the first error threshold, and the calibration value of the second coarse posture corresponding to the cross-validation error being less than the first error threshold (e.g., the above-mentioned posture a) can be determined. In some application scenarios, the calibrator can be regarded as the current calibration result. Here, the above-mentioned first error threshold can be set within a reasonable range, such as 2%, 5%, etc., and can be set specifically according to the actual application scenario.

[0054] In this embodiment, through the above steps 101 to 104, the robot arm can automatically perform hand-eye calibration through simple initial parameter settings, eliminating the cumbersome process of manually operating the robot arm movement. Iterative calibration operations can be performed by determining multiple posture groups to gradually reduce the cross-validation error, and the first error threshold is used to provide a quantitative indicator for the calibration result, thereby improving the accuracy of the calibration result.

[0055] Please refer to Figure 2 , Figure 2 This is a flow chart of a method for determining multiple pose groups involved in this application, such as Figure 2As shown, the step 103 of selecting a plurality of target waypoints from the moving waypoint sequence and determining a plurality of posture groups according to the plurality of target waypoints, the first rough posture and the second rough posture may include the following sub-steps:

[0056] Sub-step 1031, randomly selecting a target waypoint from the moving waypoint sequence;

[0057] After obtaining the moving waypoint sequence, a target waypoint can be randomly selected from the moving waypoint sequence to improve the credibility of the calibration result.

[0058] Sub-step 1032, acquiring a corresponding third posture of the end of the manipulator according to the target waypoint, the first rough posture, and the second rough posture;

[0059] After the target waypoint is selected, the third pose of the end of the robot arm can be determined according to the target waypoint and the first rough pose and the second rough pose obtained previously.

[0060] In some optional implementations, the step 1032 may determine the third posture by the following steps:

[0061] Step 1, calculating a third preliminary pose of the corresponding robotic arm end according to the target waypoint, the first rough pose and the second rough pose;

[0062] For example, when the camera is set outside the robot arm, after obtaining the position H of the target waypoint waypoint , first rough pose Second rough pose After that, you can use the calculation formula Calculate the third preliminary pose

[0063] Step 2, determining whether the end of the robotic arm can reach the position corresponding to the third preliminary posture;

[0064] After obtaining the third preliminary posture, it can be determined whether the end of the robot arm can reach the position corresponding to the third preliminary posture. In some application scenarios, for example, the judgment can be made based on the feedback information sent by the robot based on the current posture of the end of the robot arm. That is, the robot can feedback the posture of the end of the robot arm, and then determine whether the feedback posture is the same as the third preliminary posture, and then determine whether the end of the robot arm has reached the position corresponding to the third preliminary posture.

[0065] Step 3, if not, return to the step of randomly selecting a target waypoint from the motion waypoint sequence;

[0066] If it is determined that the end of the robot arm has not reached the position corresponding to the third preliminary posture, it is possible to return to the step of randomly selecting a target waypoint from the motion waypoint sequence, that is, it is possible to return to the above step 1031 to reselect the next target waypoint, and the new third preliminary posture can be re-determined through the next target waypoint.

[0067] Step 4: If possible, setting the third preliminary pose as the third pose.

[0068] If it is determined that the end of the robot arm has reached the position corresponding to the third preliminary posture, the current third preliminary posture can be determined as the third posture.

[0069] Through the above steps 1 to 4, the end of the robotic arm can accurately reach the third posture, thereby improving the accuracy of the calibration result.

[0070] Sub-step 1033, determining whether the fourth posture of the calibration object in the camera coordinate system can be read from the image captured by the camera when the end of the robotic arm is in the third posture;

[0071] In some application scenarios, when the end of the robotic arm moves to the third posture, the camera can be instructed to capture the current image. Then it can be determined whether the calibration object can be detected in the image sent back by the camera. If the calibration object can be detected, the fourth posture of the calibration object in the camera coordinate system can be read. In these application scenarios, if the calibration object cannot be detected, the posture of the end of the robotic arm can be readjusted so that the fourth posture of the calibration object in the camera coordinate system can be detected from the image.

[0072] Sub-step 1034, if not, return to the step of randomly selecting a target waypoint from the motion waypoint sequence;

[0073] If it is determined that the fourth pose of the calibration object in the camera coordinate system cannot be read in the captured image, the process can return to the above step 1031, reselect the next target waypoint, and re-determine a new third preliminary pose through the next target waypoint to re-capture the image for judgment.

[0074] Sub-step 1035, if possible, storing the third posture and the corresponding fourth posture as a posture group until multiple posture groups are determined.

[0075] If it is determined that the fourth posture of the calibration object in the camera coordinate system can be read in the acquired image, the third posture and the fourth posture can be determined as a posture group. And so on, multiple posture groups can be determined.

[0076] Through the above steps 1031 to 1035, multiple posture groups with high credibility can be determined, laying a foundation for improving the accuracy of the calibration results.

[0077] In some optional implementations, the above step 103 may include the following sub-steps:

[0078] Sub-step 1, calculating a first calibration value of the second rough posture according to the multiple posture groups;

[0079] In some application scenarios, the first calibration value of the second rough posture can be obtained according to multiple posture groups. In these application scenarios, for example, the first calibration value can be determined by referring to the process of calculating posture a in step 103 above, which will not be described in detail here.

[0080] Sub-step 2: calculating the cross-validation error corresponding to the first calibration value;

[0081] The process of determining the cross-validation error corresponding to the first calibration value in the above sub-step 2 may refer to the process of determining the cross-validation error in the above step 103, for example, and will not be described in detail here.

[0082] Sub-step three: if the cross-validation error is less than a first error threshold, using the first calibration value as a calibration value of the second coarse pose;

[0083] When the cross-validation error is less than the first error threshold, the first calibration value may be determined as the calibration value of the second rough pose, that is, the first calibration value is determined as the target calibration result.

[0084] Sub-step four: if the cross-validation error is greater than the first error threshold, continue to calibrate the second coarse pose until the cross-validation error is less than the first error threshold.

[0085] If it is determined that the cross-validation error is greater than the first error threshold, the second rough pose may be further calibrated. In some application scenarios, the second rough pose may be further calibrated by repeating the process of obtaining the first calibration value until the cross-validation error corresponding to a certain first calibration value is less than the first error threshold, so as to obtain a target calibration result.

[0086] In some optional implementations, the above sub-step four may also include: if the cross-validation error is greater than the first error threshold and less than the second error threshold, updating the first coarse pose and the second coarse pose according to the first calibration value, and returning to the step of randomly selecting a target waypoint from the motion waypoint sequence.

[0087] In some application scenarios, the second error threshold can be regarded as an error threshold that can meet the current calibration application scenario; the first error threshold can be regarded as an error threshold that can make the target calibration result more accurate. That is, the second error threshold is greater than the first error threshold. In these application scenarios, if the cross-validation error is between the first error threshold and the second error threshold, the first calibration value can be used to update the second coarse pose and the first coarse pose.

[0088] In some optional implementations, the updating of the first coarse pose and the second coarse pose according to the first calibration value may include: using the first calibration value as the new second coarse pose, and determining a new first coarse pose based on the new second coarse pose.

[0089] For example, when it is determined that the cross-validation error corresponding to the first calibration value A is between the first error threshold and the second error threshold, the second rough pose needs to be further calibrated. At this time, for example, the first calibration value A can be determined as the new second rough pose. Then, the first rough pose can be determined based on an equation of the form AX=XB. For example, when the camera is set outside the robotic arm, the second rough pose can be determined based on the calculation formula Get the first rough pose

[0090] In a new round of calibration, a new target waypoint can be randomly selected again, and then the corresponding cross-validation error can be determined based on the new first coarse pose and the new second coarse pose, and the target calibration result can be obtained when the cross-validation error is less than the first error threshold.

[0091] In some application scenarios, during the verification process, the cross-validation error with the largest value can be deleted when two sets of calibration data are added each time, so as to alleviate the inaccurate calibration results caused by the cross-validation error.

[0092] Please refer to Figure 3 , Figure 3 A schematic diagram of a camera position calibration device for a robot system provided in an embodiment of the present application. Figure 3As shown, the camera posture calibration device of the robot system includes a first acquisition module 301, a setting module 302, a second acquisition module 303 and a calibration module 304. Among them, the first acquisition module 301 is used to obtain a first rough pose of the calibration object in a first coordinate system and a second rough pose of the camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are the robot end coordinate system and the base coordinate system respectively; the setting module 302 is used to set the distance parameter from the camera to the calibration object, and set the motion waypoint sequence of the robot based on the distance parameter; the second acquisition module 303 is used to select multiple target waypoints from the motion waypoint sequence, and determine multiple pose groups according to the multiple target waypoints, the first rough pose and the second rough pose, each pose group corresponds to a target waypoint, and each pose group includes a third pose of the robot end in the base coordinate system obtained based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system; the calibration module 304 is used to perform iterative calibration operations according to the multiple pose groups to obtain a calibration value of the second coarse pose when the cross-validation error is less than the first error threshold.

[0093] Optionally, the second acquisition module 303 is further used to: randomly select a target waypoint from the moving waypoint sequence; obtain the corresponding third posture of the robotic arm end according to the target waypoint, the first rough posture and the second rough posture; determine whether the fourth posture of the calibration object in the camera coordinate system can be read from the image captured by the camera when the robotic arm end is in the third posture; if not, return to the step of randomly selecting a target waypoint from the moving waypoint sequence; if so, store the third posture and the corresponding fourth posture as a posture group until multiple posture groups are determined.

[0094] Optionally, the second acquisition module 303 is further used to: calculate a third preliminary pose of the corresponding robotic arm end according to the target waypoint, the first rough pose and the second rough pose; determine whether the robotic arm end can reach the position corresponding to the third preliminary pose; if not, return to the step of randomly selecting a target waypoint from the motion waypoint sequence; if so, set the third preliminary pose as the third pose.

[0095] Optionally, the calibration module 304 is further used to: calculate a first calibration value of the second coarse posture based on the multiple posture groups; calculate a cross-validation error corresponding to the first calibration value; if the cross-validation error is less than a first error threshold, use the first calibration value as the calibration value of the second coarse posture; if the cross-validation error is greater than the first error threshold, continue to calibrate the second coarse posture until the cross-validation error is less than the first error threshold.

[0096] Optionally, the calibration module 304 is further used to: if the cross-validation error is greater than the first error threshold and less than a second error threshold, update the first coarse pose and the second coarse pose according to the first calibration value, and return to the step of randomly selecting a target waypoint from the moving waypoint sequence.

[0097] Optionally, the calibration module 304 is further configured to: use the first calibration value as the new second coarse pose, and determine a new first coarse pose based on the new second coarse pose.

[0098] Optionally, if the camera is set on the robotic arm, the first coordinate system is a base coordinate system; if the camera is set outside the robotic arm, the first coordinate system is a robotic arm end coordinate system.

[0099] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0100] Please refer to Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device for executing a camera pose calibration method for a robot system provided in an embodiment of the present application, the electronic device comprising: a processor 401 and a memory 402, the processor 401 and the memory 402 are interconnected and communicate with each other via a communication bus 403 and / or other forms of connection mechanisms (not shown), the memory 402 stores a computer program executable by the processor 401, and when the computing device is running, the processor 401 executes the computer program to execute a method in any optional implementation of the above embodiments.

[0101] Understandably, Figure 4 The structure shown is for illustration only, and the electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0102] The embodiment of the present application provides a storage medium, and when the computer program is executed by a processor, the method in any optional implementation of the above embodiment can be executed. Wherein, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk.

[0103] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0104] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0105] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0106] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0107] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A camera pose calibration method for a robot system, characterized in that: include: Obtain a first rough pose of the calibration object in a first coordinate system and a second rough pose of the camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are respectively a robot end coordinate system and a base coordinate system; wherein, if the camera is set on the robot, the first coordinate system is the base coordinate system, and the second coordinate system is the robot end coordinate system; if the camera is set outside the robot, the first coordinate system is the robot end coordinate system, and the second coordinate system is the base coordinate system; Setting a distance parameter from the camera to the calibration object, and setting a motion waypoint sequence of the robotic arm based on the distance parameter; Selecting a plurality of target waypoints from the motion waypoint sequence, and determining a plurality of pose groups according to the plurality of target waypoints, the first rough pose, and the second rough pose, each pose group corresponding to a target waypoint, each pose group including a third pose of the end of the manipulator in the base coordinate system obtained based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system; An iterative calibration operation is performed according to the multiple posture groups to obtain a calibration value of the second coarse posture when the cross-validation error is less than a first error threshold.

2. The camera pose calibration method of a robot system according to claim 1, characterized in that: The selecting a plurality of target waypoints from the moving waypoint sequence, and determining a plurality of posture groups according to the plurality of target waypoints, the first rough posture, and the second rough posture, comprises: Randomly selecting a target waypoint from the motion waypoint sequence; Acquire a third posture of the corresponding end of the robotic arm according to the target waypoint, the first rough posture and the second rough posture; Determining whether a fourth posture of the calibration object in the camera coordinate system can be read from an image captured by the camera when the end of the robotic arm is in a third posture; If not, return to the step of randomly selecting a target waypoint from the motion waypoint sequence; If so, the third posture and the corresponding fourth posture are stored as a posture group until a plurality of posture groups are determined.

3. The camera pose calibration method of the robot system according to claim 2, characterized in that: The step of obtaining a corresponding third posture of the end of the manipulator according to the target waypoint, the first rough posture, and the second rough posture comprises: Calculate a third preliminary pose of the corresponding robotic arm end according to the target waypoint, the first rough pose and the second rough pose; Determine whether the end of the robotic arm can reach the position corresponding to the third preliminary posture; If not, return to the step of randomly selecting a target waypoint from the motion waypoint sequence; If so, the third preliminary pose is set as the third pose.

4. The camera pose calibration method of a robot system according to claim 1, characterized in that: The iterative calibration operation is performed according to the multiple posture groups to obtain a calibration value of the second coarse posture when the cross-validation error is less than the first error threshold, including: Calculate a first calibration value of the second rough pose according to the multiple pose groups; Calculating a cross validation error corresponding to the first calibration value; If the cross-validation error is less than a first error threshold, using the first calibration value as a calibration value for the second coarse pose; If the cross-validation error is greater than the first error threshold, the second coarse pose is continuously calibrated until the cross-validation error is less than the first error threshold.

5. The camera pose calibration method of a robot system according to claim 4, characterized in that: If the cross-validation error is greater than the first error threshold, continuing to calibrate the second coarse pose until the cross-validation error is less than the first error threshold, comprising: If the cross-validation error is greater than the first error threshold and less than the second error threshold, the first coarse pose and the second coarse pose are updated according to the first calibration value, and a target waypoint is randomly selected from the motion waypoint sequence again to determine a new cross-validation error.

6. The camera pose calibration method of a robot system according to claim 5, characterized in that: The updating of the first coarse pose and the second coarse pose according to the first calibration value includes: The first calibration value is used as the new second rough pose, and a new first rough pose is determined based on the new second rough pose.

7. A camera pose calibration device for a robot system, characterized in that: include: A first acquisition module is used to acquire a first rough pose of the calibration object in a first coordinate system and a second rough pose of the camera in a second coordinate system, wherein the first coordinate system and the second coordinate system are respectively a robot end coordinate system and a base coordinate system; wherein, if the camera is set on the robot, the first coordinate system is the base coordinate system, and the second coordinate system is the robot end coordinate system; if the camera is set outside the robot, the first coordinate system is the robot end coordinate system, and the second coordinate system is the base coordinate system; A setting module, used to set a distance parameter from the camera to the calibration object, and to set a motion waypoint sequence of the robot arm based on the distance parameter; A second acquisition module is used to select multiple target waypoints from the motion waypoint sequence, and determine multiple pose groups according to the multiple target waypoints, the first rough pose and the second rough pose, each pose group corresponds to a target waypoint, and each pose group includes a third pose of the end of the manipulator in the base coordinate system acquired based on the corresponding target waypoint and a fourth pose of the calibration object in the camera coordinate system; The calibration module is used to perform an iterative calibration operation according to the multiple posture groups to obtain a calibration value of the second coarse posture when the cross-validation error is less than a first error threshold.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 6 are executed.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.

Images