A teaching pose determination method, device and electronic equipment

Abstract

This application provides a teaching pose determination method, apparatus, and electronic device. The method includes: based on the current hand-eye calibration result and a first current teaching pose, transforming the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain first transformed coordinates; adjusting the first current teaching pose based on a first error between the first transformed coordinates and the target coordinates; and if a first convergence condition is met, determining the first current teaching pose as the target teaching pose of the robotic arm. This improves the accuracy of the determined teaching pose.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a teaching pose determination method, apparatus and electronic device. Background Technology

[0002] To enable the robotic arm to accurately grasp an object, it needs to be adjusted to a position corresponding to the object's location, such as 1 cm directly above the object. The object's position is typically represented by coordinates in a real-world coordinate system (hereinafter referred to as the object coordinate system). The robotic arm's position is controlled by adjusting the pose of its flange in the robotic arm's base coordinate system. Therefore, to adjust the robotic arm to the position corresponding to the object, the transformation relationship between the flange coordinate system and the object coordinate system needs to be understood; this transformation relationship is referred to as the teaching pose.

[0003] The ability to accurately determine the teaching pose directly affects the ability to accurately control the robotic arm to grasp objects. Therefore, how to accurately determine the teaching pose has become an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a teaching pose determination method, apparatus, and electronic device to improve the accuracy of the determined teaching pose. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a method for determining a teaching pose is provided, the method comprising:

[0006] Based on the current hand-eye calibration result and the first current teaching pose, the target coordinates in the object coordinate system are transformed back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates. The current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose.

[0007] Adjust the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates;

[0008] If the first convergence condition is met, then the first current teaching pose is determined as the target teaching pose of the robotic arm.

[0009] In one possible embodiment, the method further includes:

[0010] Acquire at least one grasping pose and an object image corresponding to each grasping pose, wherein the object image is an image of the object captured by the visual positioning component of the robotic arm when the robotic arm grasps the object in the grasping pose;

[0011] The teaching pose is determined based on the initial hand-eye calibration results, each grasping pose, and the object image corresponding to each grasping pose, and is used as the teaching pose corresponding to the grasping pose.

[0012] The initial teaching pose is determined based on the teaching pose corresponding to each of the described grasping poses.

[0013] In one possible embodiment, determining the initial teaching pose based on the teaching pose corresponding to each of the grasping poses includes:

[0014] The teaching poses with accuracy lower than a preset accuracy threshold are filtered out from the teaching poses corresponding to each of the grasping poses to obtain the initial screening teaching poses.

[0015] The initial teaching pose is determined based on the initial screening teaching pose.

[0016] In one possible embodiment, the at least one grasping pose includes a plurality of different grasping poses.

[0017] In one possible embodiment, the step of transforming the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates includes:

[0018] Transform the target coordinates in the object coordinate system to the camera coordinate system to obtain the camera projection point;

[0019] The camera projection point is transformed to the base coordinate system of the robotic arm to obtain the base projection point;

[0020] The projection point of the base is transformed to the flange coordinate system of the robotic arm to obtain the projection point of the workpiece;

[0021] The projection points of the workpiece are transformed to the physical coordinate system to obtain the first transformed coordinates.

[0022] In one possible embodiment, adjusting the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates includes:

[0023] Based on the first error between the first transformed coordinates and the target coordinates, adjust the first current teaching pose and the current hand-eye calibration result;

[0024] The method further includes:

[0025] Update the hand-eye calibration results used by the visual positioning component.

[0026] In one possible embodiment, the method further includes:

[0027] Based on the initial hand-eye calibration result and the second current teaching pose, the target coordinates in the object coordinate system are transformed back to the object coordinate system through multiple coordinate transformations to obtain the second transformed coordinates. The second current teaching pose is initially the initial teaching pose.

[0028] Adjust the second current teaching pose based on the second error between the second transformed coordinates and the target coordinates;

[0029] The updating of the hand-eye calibration results used by the visual positioning component includes:

[0030] If the first error is less than the second error and the difference between the first error and the second error is greater than a preset difference threshold, then the hand-eye calibration result used by the visual positioning component is updated.

[0031] In one possible embodiment, the method further includes:

[0032] If the first error is not less than the second error or the difference is not greater than the preset threshold, then the hand-eye calibration result used by the visual positioning component is not updated.

[0033] In one possible embodiment, adjusting the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates includes:

[0034] Determine the gradient descent direction of the first error;

[0035] Adjust the first current teaching pose according to the gradient descent direction.

[0036] In a second aspect of the invention, a teaching pose determination device is also provided, the device comprising:

[0037] The first transformation module is used to transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations based on the current hand-eye calibration result and the first current teaching pose, to obtain the first transformed coordinates. The current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose.

[0038] The full optimization module is used to adjust the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates;

[0039] The pose determination module is used to determine the first current teaching pose as the target teaching pose of the robotic arm if the first convergence condition is met.

[0040] In one possible embodiment, the device further includes:

[0041] An initial teaching pose determination module is used to acquire at least one grasping pose and an object image corresponding to each grasping pose, wherein the object image is an image of the object captured by the visual positioning component of the robotic arm when the robotic arm grasps the object in the grasping pose.

[0042] The teaching pose is determined based on the initial hand-eye calibration results, each grasping pose, and the object image corresponding to each grasping pose, and is used as the teaching pose corresponding to the grasping pose.

[0043] The initial teaching pose is determined based on the teaching pose corresponding to each of the described grasping poses.

[0044] In one possible embodiment, the initial teaching pose determination module determines the initial teaching pose based on the teaching pose corresponding to each of the grasping poses, including:

[0045] The teaching poses with accuracy lower than a preset accuracy threshold are filtered out from the teaching poses corresponding to each of the grasping poses to obtain the initial screening teaching poses.

[0046] The initial teaching pose is determined based on the initial screening teaching pose.

[0047] In one possible embodiment, the at least one grasping pose includes a plurality of different grasping poses.

[0048] In one possible embodiment, the first transformation module re-transforms the target coordinates in the object coordinate system to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates, including:

[0049] Transform the target coordinates in the object coordinate system to the camera coordinate system to obtain the camera projection point;

[0050] The camera projection point is transformed to the base coordinate system of the robotic arm to obtain the base projection point;

[0051] The projection point of the base is transformed to the flange coordinate system of the robotic arm to obtain the projection point of the workpiece;

[0052] The projection points of the workpiece are transformed to the physical coordinate system to obtain the first transformed coordinates.

[0053] In one possible embodiment, the full optimization module adjusts the first current teaching pose based on a first error between the first transformed coordinates and the target coordinates, including:

[0054] Based on the first error between the first transformed coordinates and the target coordinates, adjust the first current teaching pose and the current hand-eye calibration result;

[0055] The device further includes:

[0056] An update module is used to update the hand-eye calibration results used by the visual positioning component.

[0057] In one possible embodiment, the device further includes:

[0058] The local optimization module is used to transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations based on the initial hand-eye calibration result and the second current teaching pose, to obtain the second transformed coordinates. The second current teaching pose is initially the initial teaching pose.

[0059] Adjust the second current teaching pose based on the second error between the second transformed coordinates and the target coordinates;

[0060] The update module updates the hand-eye calibration results used by the visual positioning component, including:

[0061] If the first error is less than the second error and the difference between the first error and the second error is greater than a preset difference threshold, then the hand-eye calibration result used by the visual positioning component is updated.

[0062] In one possible embodiment, the device further includes:

[0063] The update module is further configured to refuse to update the hand-eye calibration result used by the visual positioning component if the first error is not less than the second error or the difference is not greater than the preset threshold.

[0064] In one possible embodiment, the full optimization module adjusts the first current teaching pose based on a first error between the first transformed coordinates and the target coordinates, including:

[0065] Determine the gradient descent direction of the first error;

[0066] Adjust the first current teaching pose according to the gradient descent direction.

[0067] In a third aspect of the invention, an electronic device is also provided, comprising:

[0068] Memory, used to store computer programs;

[0069] The processor, when executing a program stored in memory, implements any of the above-described teaching pose determination methods.

[0070] In a fourth aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the above-described teaching pose determination methods.

[0071] Beneficial effects of the embodiments in this application:

[0072] The teaching pose determination method, apparatus, and electronic device provided in this application embodiment can use the teaching pose as a variable to be optimized. By adjusting the first current teaching pose based on the first error, iterative process is formed. It can be understood that if the first current teaching pose is more accurate, the first transformed coordinates obtained by re-transforming the target coordinates to the object coordinate system should be closer to the target coordinates, that is, the first error is smaller. Therefore, the first error can reflect the difference between the first current teaching pose and the real teaching pose. Thus, in the aforementioned iteration, the first current teaching pose will converge to the real teaching pose. When the first convergence condition is met, it can be considered that the first current teaching pose is close enough to the real teaching pose. Therefore, the first current teaching pose at this time can be determined as the target teaching pose of the robotic arm.

[0073] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0075] Figure 1 A schematic diagram of a scenario for the teaching pose determination method provided in an embodiment of the present invention;

[0076] Figure 2 A flowchart of a teaching pose determination method provided in an embodiment of the present invention;

[0077] Figure 3 A flowchart of another teaching pose determination method provided in an embodiment of the present invention;

[0078] Figure 4a This is a schematic diagram of the current first teaching pose being the initial teaching pose, provided in an embodiment of the present invention.

[0079] Figure 4b This is a schematic diagram of the current first teaching pose being the target teaching pose, provided in an embodiment of the present invention.

[0080] Figure 5A flowchart illustrating another teaching pose determination method provided in an embodiment of the present invention;

[0081] Figure 6 This is a schematic diagram of a teaching pose determination device provided in an embodiment of the present invention;

[0082] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0083] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0084] To more clearly explain the teaching pose determination method provided in this application, some terms used in the following text will be explained first:

[0085] robotic arms: such as Figure 1 As shown, the robotic arm includes a base, a workpiece, and a flange for connecting the workpiece and the base.

[0086] Visual positioning components, such as Figure 1 As shown, images of objects are captured to determine the position of the objects through visual positioning, thereby assisting the robotic arm in grasping the objects. The position of the visual positioning component can be integrated on the robotic arm or independent of the robotic arm.

[0087] Teaching position: The coordinate transformation relationship between the flange coordinate system of the robotic arm and the object coordinate system.

[0088] Grasping pose: The coordinate transformation relationship between the flange coordinate system of the robotic arm and the base coordinate system of the robotic arm.

[0089] Visual positioning pose: The coordinate transformation relationship between the object's coordinate system and the camera's coordinate system of the visual positioning component.

[0090] Hand-eye calibration results: Coordinate transformation relationship between the base coordinate system of the robotic arm and the camera coordinate system of the vision positioning component.

[0091] To accurately determine the teaching posture, this application provides a method for determining the teaching posture, such as... Figure 2 As shown, it includes:

[0092] S201, based on the current hand-eye calibration result and the first current teaching pose, the target coordinates in the object coordinate system are transformed back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates.

[0093] Among them, the current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose.

[0094] S202, adjust the first current teaching pose according to the first error between the first transformed coordinates and the target coordinates.

[0095] S203, if the first convergence condition is met, then the first current teaching pose is determined as the target teaching pose of the robotic arm.

[0096] In this embodiment, the teaching pose is used as the variable to be optimized. The adjustment of the first current teaching pose based on the first error forms an iteration. It can be understood that if the first current teaching pose is more accurate, the first transformed coordinates obtained by re-transforming the target coordinates to the object coordinate system should be closer to the target coordinates, that is, the first error is smaller. Therefore, the first error can reflect the difference between the first current teaching pose and the real teaching pose. Thus, in the aforementioned iteration, the first current teaching pose will converge to the real teaching pose. When the first convergence condition is met, it can be considered that the first current teaching pose is close enough to the real teaching pose. Therefore, the first current teaching pose at this time can be determined as the target teaching pose of the robotic arm.

[0097] Furthermore, the teaching pose determination method provided in this application is obtained by iteratively determining the initial teaching pose. Therefore, even if the initial teaching pose has some inaccuracies, the target teaching pose can still be determined relatively accurately. Thus, even if the robotic arm's occlusion or grasping pose is not accurate enough, resulting in an inaccurate determined teaching pose, it can be optimized using the teaching pose determination method provided in this application to obtain an accurate target teaching pose.

[0098] The following will explain S201-S203 respectively:

[0099] In S201, the initial teaching pose can be set according to user experience or actual needs, or the teaching pose can be determined according to any teaching pose determination method.

[0100] In one possible embodiment, the initial teaching pose is determined as follows: Figure 3 As shown, it includes:

[0101] S301, acquire at least one grasping pose and the object image corresponding to each grasping pose.

[0102] The object image is an image of the object captured by the robotic arm's vision positioning component when the robotic arm grasps the object in a grasping pose. At least one grasping pose can be used, or multiple grasping poses can be used. Furthermore, to ensure accurate determination of the object's position in the object image, occlusion of the object by the robotic arm should be minimized.

[0103] Furthermore, in the case of multiple grasping poses, there may be identical grasping poses among the various grasping poses, but the grasping poses should be as different as possible. For example, in one possible embodiment, any two grasping poses are not the same. It is understood that different grasping poses can enrich the information included in the object image as much as possible, so that the teaching pose can be optimized more accurately in the future.

[0104] S302, determine the teaching pose based on the initial hand-eye calibration results, each grasping pose, and the object image corresponding to each grasping pose, and use it as the teaching pose corresponding to the grasping pose.

[0105] The image of the object can reflect the position of the object in the camera coordinate system. Combined with the visual positioning pose, the position of the object in the object coordinate system can be determined. Combined with the hand-eye calibration results, the position of the object in the robot arm base coordinate system can be determined. Combined with the grasping pose, the position of the object in the flange coordinate system can be determined. Based on the position of the object in the object coordinate system and the position of the object in the flange coordinate system, the teaching pose can be determined.

[0106] Understandably, as explained in the preceding steps, the accuracy of the teaching pose determined in S302 is relatively low.

[0107] S303, determine the initial teaching pose based on the teaching pose corresponding to each grasping pose.

[0108] This can involve statistically analyzing the teaching poses corresponding to each grasping pose and using the statistical results as the initial teaching pose. For example, the average / median value of the teaching poses corresponding to the grasping pose can be calculated, and the calculated average / median value can be used as the initial teaching pose.

[0109] Alternatively, the teaching poses corresponding to each grasping pose can be filtered first, and the initial teaching pose can be determined based on the preliminary filtered teaching poses. The filtering method can vary depending on the application scenario. For example, it is understood that although the accuracy of the teaching poses corresponding to each grasping pose is relatively low, theoretically, the determined teaching poses should be distributed within a certain range of the actual teaching poses. Therefore, theoretically, the teaching poses corresponding to each grasping pose should be similar. Conversely, if the teaching pose corresponding to one grasping pose is not similar to the teaching poses corresponding to other grasping poses, it can be considered that the teaching pose corresponding to that grasping pose is abnormal. To avoid the teaching pose corresponding to that grasping pose affecting the accuracy of the initial teaching pose, the teaching pose corresponding to that grasping pose should be filtered out during the filtering process. Therefore, in this embodiment, teaching poses that are not similar to other teaching poses can be filtered out.

[0110] In S201, the method of multiple coordinate transformations may vary depending on the application scenario. During the coordinate transformation process, the current hand-eye calibration result and the first current teaching pose should be used. In addition to the current hand-eye calibration result and the first current teaching pose, the coordinate transformation relationship in each coordinate transformation should be accurate enough.

[0111] Understandably, the hand-eye calibration result is used to represent the coordinate transformation relationship between the robot arm's base coordinate system and the camera coordinate system of the vision positioning component, while the taught pose is used to represent the coordinate transformation relationship between the robot arm's flange coordinate system and the object coordinate system. Therefore, in order to use the current hand-eye calibration result and the first current taught pose during the coordinate transformation process, multiple coordinate transformations should include: coordinate transformation between the base coordinate system and the camera coordinate system, and coordinate transformation between the flange coordinate system and the object coordinate system.

[0112] Since multiple coordinate transformations ultimately require a re-transformation to the object's coordinate system, in one possible implementation, the multiple coordinate transformation process is as follows:

[0113] 1. Transform from object coordinate system to camera coordinate system

[0114] 2. Transform from camera coordinate system to base coordinate system

[0115] 3. Transform from the base coordinate system to the flange coordinate system

[0116] 4. Transform from flange coordinate system to object coordinate system

[0117] The target coordinates can be any coordinate system. For ease of description, the following explanation will use the fixed-point coordinates of the object's bounding box as an example. Let's assume the fixed-point coordinates are labeled P. cor The first transformed coordinates are as follows:

[0118]

[0119] in, This represents the coordinate transformation relationship between the flange coordinate system and the object coordinate system. This represents the coordinate transformation relationship between the base coordinate system and the flange coordinate system. This represents the coordinate transformation relationship between the camera coordinate system and the base coordinate system. This represents the coordinate transformation relationship between the object's coordinate system and the camera's coordinate system.

[0120] In some cases, the current hand-eye calibration results do not need to be adjusted. For example, if the hand-eye calibration results are considered accurate enough, they do not need to be adjusted. In other cases, the hand-eye calibration results may be considered inaccurate, requiring adjustment.

[0121] Examples include application scenario one and application scenario two below.

[0122] Application Scenario 1:

[0123] The robotic arm is controlled to grasp an object. Once the robotic arm successfully grips the object, an image of the object is captured using the robotic arm's vision positioning component. This image reflects the object's position in the camera coordinate system. By combining the vision positioning pose, the object's position in the object space coordinate system can be determined. Furthermore, based on the object's position in the camera coordinate system and the hand-eye calibration results, the object's position in the robotic arm base coordinate system can be determined. By combining the grasping pose, the object's position in the flange coordinate system can be determined. Based on the object's position in both the object space coordinate system and the flange coordinate system, the teaching pose can be determined.

[0124] Application Scenario 2:

[0125] First, the robotic arm's vision positioning component captures an image of the object. Then, the robotic arm is controlled to grasp the object, so that it is in a state where it can grasp the object but is not holding it. Based on the vision positioning pose and the object's position in the captured image, the object's position in the object space coordinate system is determined. Based on the object's position in the camera coordinate system, combined with the hand-eye calibration results, the object's position in the robotic arm base coordinate system can be determined. Combined with the grasping pose, the object's position in the flange coordinate system can be determined. Based on the object's position in the object space coordinate system and its position in the flange coordinate system, the teaching pose can be determined.

[0126] In application scenario one, there is a problem: because the hand-eye calibration results are used in the process of determining the teaching posture, the error of the hand-eye calibration results will lead to the inaccuracy of the determined teaching posture.

[0127] In application scenario two, the following two problems exist:

[0128] Question 1 mentioned above;

[0129] Question 2: Since the robotic arm is not holding an object, there is a certain error between the grasping pose at this time and the actual grasping pose when the robotic arm grasps the object, which leads to the inaccuracy of the determined teaching pose.

[0130] Based on the above application scenarios and existing problems, it can be seen that when the hand-eye calibration results are not accurate enough, the teaching posture will also be affected.

[0131] Based on this, in one possible embodiment, adjusting the first current teaching pose according to the first error between the first transformed coordinates and the target coordinates includes adjusting the first current teaching pose and the current hand-eye calibration result according to the first error between the first transformed coordinates and the target coordinates.

[0132] In S202, the first error can be calculated using the following formula:

[0133]

[0134] Where f represents the first error. The first convergence condition can vary depending on the application scenario. For example, it could be that the number of iterations reaches a preset threshold, or the first error is less than a preset error threshold. Alternatively, it could be that the ratio between the first error obtained in the last iteration and the first error in the previous iteration is less than a preset ratio threshold.

[0135] The adjustment of the first current teaching pose should be in the direction of reducing the first error. The specific adjustment method can be any backpropagation method. For example, the gradient of the first error can be determined, and the first current teaching pose can be adjusted according to the gradient direction to improve iteration efficiency, thereby improving the efficiency of determining the target teaching pose. Similarly, when adjusting the first current teaching pose and the current hand-eye calibration result, the adjustment should also be in the direction of reducing the first error, and the specific adjustment method can be any backpropagation method.

[0136] Specifically, if the adjustment is for the first current teaching pose, it can be calculated by multiplying the gradient with the second learning rate set for the first current teaching pose, and the sum of the first current teaching pose and the product is determined as the adjusted first current teaching pose.

[0137] Alternatively, the Jacobian matrix of the objective function with respect to the first current teaching pose can be calculated, a Hessian matrix can be constructed, and then the optimal step size for the current variable can be calculated by combining the Jacobian matrix and the error of the current objective function. The optimal step size is then added to the first current teaching pose to obtain a new first current teaching pose, which is used as the adjusted first current teaching pose.

[0138] If the adjustment is made for the first current teaching pose and the current hand-eye calibration pose, it can be done by calculating the product of the gradient and the first learning rate set for the current hand-eye calibration result, and then determining the adjusted current hand-eye calibration result by summing the current hand-eye calibration result and the product of the gradient and the second learning rate set for the first current teaching pose.

[0139] Alternatively, the Jacobian matrix of the objective function with respect to the first current teaching pose and the current hand-eye coordinate positioning can be calculated to construct a Hession matrix. Then, the error between the Jacobian matrix and the current objective function can be used to calculate the optimal step size for the current variables. Finally, the optimal step size is added to the first current teaching pose and the current hand-eye coordinate positioning to obtain a new first current teaching pose and the current hand-eye coordinate positioning, which are then used as the adjusted first current teaching pose and the current hand-eye coordinate positioning.

[0140] To more intuitively illustrate the effectiveness of the teaching pose determination method provided in this application, please refer to... Figure 4a and Figure 4b In this system, white dots represent target coordinates, and gray dots represent transformed coordinates. Therefore, the more white dots and gray dots overlap, the smaller the first error; conversely, the smaller the overlap, the larger the first error.

[0141] Figure 4a This is a schematic diagram showing the current first teaching position as the initial teaching position. Figure 4b This is a diagram illustrating the target teaching position when the current first teaching position is used. Figure 4a and Figure 4b It is evident that the first error corresponding to the target teaching pose after iteration is significantly smaller than the first error corresponding to the initial teaching pose. Therefore, compared to the initial teaching pose, the target teaching pose is closer to the true teaching pose. In other words, the teaching pose determination method provided in this application can optimize the initial teaching pose into a more accurate target teaching pose.

[0142] As explained above, the hand-eye calibration results have a certain degree of inaccuracy. However, the aforementioned iterative process can determine not only a more accurate teaching pose but also a more accurate hand-eye calibration result. Therefore, in one possible embodiment, when the first convergence condition is met, the hand-eye calibration result used by the visual positioning component can be updated to the current hand-eye calibration result. In another possible embodiment, it can also be done as follows... Figure 5 As shown:

[0143] S501, based on the current hand-eye calibration result and the first current teaching pose, transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates.

[0144] This step is the same as S201 above, and you can refer to the relevant description of S201 above, so it will not be repeated here.

[0145] S502, based on the first error between the first transformed coordinates and the target coordinates, adjust the first current teaching pose and the current hand-eye calibration result.

[0146] The adjustment method can be found in the relevant explanation of S202 above, and will not be repeated here.

[0147] S503, based on the initial hand-eye calibration result and the second current teaching pose, transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain the second transformed coordinates.

[0148] S504, adjust the second current teaching pose based on the second error between the second transformed coordinates and the target coordinates.

[0149] The adjustment method can be found in the relevant description of adjusting the first current teaching posture in the aforementioned S202. The only difference is that the teaching posture being adjusted changes from the first current teaching posture to the second current teaching posture.

[0150] S505, if the first error is less than the second error and the difference between the first error and the second error is greater than a preset difference threshold, the first current teaching pose is determined as the target teaching pose of the robotic arm, and the hand-eye calibration result used by the visual positioning component is updated. The preset difference threshold can be set by those skilled in the art based on actual needs and / or experience. Therefore, if the first error is less than the second error and the difference between the first error and the second error is greater than the preset difference threshold, the first error can be considered much smaller than the second error. If the first error is not less than the second error and the difference between the first error and the second error is not greater than the preset difference threshold, the first error and the second error are considered equal or close. If the first error is not less than the second error and the difference between the first error and the second error is greater than the preset difference threshold, the first error is considered much larger than the second error.

[0151] Updating the hand-eye calibration results used by the vision positioning component means updating the hand-eye calibration results used by the vision positioning component to the current hand-eye calibration results. For example, initially, the hand-eye calibration results are the first hand-eye calibration results. After executing S505, the initial first hand-eye calibration results are adjusted to the second hand-eye calibration results, and the first error is much smaller than the second error. Therefore, the hand-eye calibration results used by the robotic arm are updated to the second hand-eye calibration results.

[0152] S506, if the first error is not less than the second error or the difference between the first error and the second error is not greater than a preset difference threshold, the first current teaching pose is determined as the target teaching pose of the robotic arm, and the hand-eye calibration result used by the visual positioning component is not updated.

[0153] Specifically, the second convergence condition can vary depending on the application scenario. For example, it could be that the number of iterations reaches a preset threshold, or the second error is less than a preset error threshold. Alternatively, it could be that the ratio between the second error obtained in the last iteration and the second error in the previous iteration is less than a preset ratio threshold.

[0154] In this embodiment, the hand-eye calibration results and the teaching pose are used together as variables to be optimized. This allows for simultaneous optimization of both the hand-eye calibration results and the teaching pose (hereinafter referred to as full optimization) while maintaining the initial hand-eye calibration results (hereinafter referred to as local optimization). It is understood that if the initial hand-eye calibration results, i.e., the hand-eye calibration results currently used by the visual positioning component, are sufficiently accurate, the effects of full optimization and local optimization should be similar; therefore, the first error should be equal to or slightly less than the second error. Conversely, if the hand-eye calibration results currently used by the visual positioning component are inaccurate, the effects of full optimization and local optimization will differ significantly; therefore, the first error should be much smaller than the second error. Therefore, if the first error is much smaller than the second error, it can be considered that the hand-eye calibration results currently used by the visual positioning component are inaccurate and need to be updated. Conversely, if the first error is slightly less than or equal to the second error, it can be considered that the hand-eye calibration results currently used by the visual positioning component are sufficiently accurate and do not need to be changed. Furthermore, if the first error is significantly greater than the second error, it can be assumed that the hand-eye calibration result used by the visual positioning component is more accurate than the adjusted hand-eye calibration result, thus eliminating the need to update the hand-eye calibration result used by the visual positioning component. Therefore, this embodiment minimizes the number of times the hand-eye calibration result needs to be changed, while ensuring that the hand-eye calibration result used by the visual positioning component is sufficiently accurate.

[0155] Furthermore, in the embodiments of this application, the current hand-eye calibration result can also converge to the real hand-eye calibration result. Therefore, when the first convergence condition is met, it can be considered that the current hand-eye calibration result is close enough to the real hand-eye calibration result. Thus, it can be considered that the determined target teaching pose is determined based on the accurate hand-eye calibration result. That is, the teaching pose determination method provided by this application can solve the problems of the aforementioned application scenario one and application scenario two.

[0156] Understandable Figure 5 This is merely a possible flowchart of the teaching pose determination method provided in this application embodiment. In other possible embodiments, S503 and S504 may be executed before S501 and S502, or may be executed in parallel or alternately with S501 and S502.

[0157] Based on the same inventive concept, this application correspondingly provides a teaching pose determination device, such as... Figure 6 As shown, Figure 6 A schematic diagram of a teaching pose determination device provided in an embodiment of the present invention includes:

[0158] The first transformation module 601 is used to transform the target coordinates in the object coordinate system to the object coordinate system through multiple coordinate transformations based on the current hand-eye calibration result and the first current teaching pose, so as to obtain the first transformed coordinates. The current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose.

[0159] The full optimization module 602 is used to adjust the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates;

[0160] The pose determination module 603 is used to determine the first current teaching pose as the target teaching pose of the robotic arm if the first convergence condition is met.

[0161] In this embodiment, the teaching pose is used as the variable to be optimized. The adjustment of the first current teaching pose based on the first error forms an iteration. It can be understood that if the first current teaching pose is more accurate, the first transformed coordinates obtained by transforming the target coordinate system back to the object coordinate system should be closer to the target coordinates, that is, the smaller the first error. Therefore, the first error can reflect the difference between the first current teaching pose and the real teaching pose. Thus, in the aforementioned iteration, the first current teaching pose will converge to the real teaching pose. When the first convergence condition is met, it can be considered that the first current teaching pose is close enough to the real teaching pose. Therefore, the first current teaching pose at this time can be determined as the target teaching pose of the robotic arm.

[0162] In one possible embodiment, the device further includes:

[0163] The initial teaching pose determination module is used to acquire at least one grasping pose and the object image corresponding to each grasping pose. The object image is an image of the object captured by the visual positioning component of the robotic arm when the robotic arm grasps the object in the grasping pose. The teaching pose is determined based on the initial hand-eye calibration result, each grasping pose, and the object image corresponding to each grasping pose, and is used as the teaching pose corresponding to the grasping pose. The initial teaching pose is determined based on the teaching pose corresponding to each grasping pose.

[0164] In one possible embodiment, the initial teaching pose determination module determines the initial teaching pose based on the teaching pose corresponding to each grasping pose, including:

[0165] The teaching poses with accuracy lower than a preset accuracy threshold are filtered out from the teaching poses corresponding to each grasping pose to obtain the initial teaching poses; the initial teaching poses are determined based on the initial teaching poses.

[0166] In one possible embodiment, at least one grasping pose includes multiple different grasping poses.

[0167] In one possible embodiment, the first transformation module 601 transforms the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates, including:

[0168] Transform the target coordinates in the object coordinate system to the camera coordinate system to obtain the camera projection point;

[0169] Transform the camera projection point to the base coordinate system of the robotic arm to obtain the base projection point;

[0170] Transform the base projection point to the flange coordinate system of the robotic arm to obtain the workpiece projection point;

[0171] Transform the workpiece projection points to the physical coordinate system to obtain the first transformed coordinates.

[0172] In one possible embodiment, the full optimization module 602 adjusts the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates, including:

[0173] Based on the first error between the first transformed coordinates and the target coordinates, adjust the first current teaching pose and the current hand-eye calibration result;

[0174] The device also includes:

[0175] The update module is used to update the hand-eye calibration results used by the visual positioning component.

[0176] In one possible embodiment, the device further includes:

[0177] The local optimization module is used to transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations based on the initial hand-eye calibration results and the second current teaching pose, to obtain the second transformed coordinates. The second current teaching pose is initially the initial teaching pose.

[0178] Based on the second error between the second transformed coordinates and the target coordinates, adjust the second current teaching pose.

[0179] The update module updates the hand-eye calibration results used by the visual positioning component, including:

[0180] If the first error is less than the second error and the difference between the first error and the second error is greater than a preset difference threshold, then update the hand-eye calibration result used by the visual positioning component.

[0181] In one possible embodiment, the device further includes:

[0182] The update module is also used to refuse to update the hand-eye calibration results used by the visual positioning component if the first error is not less than the second error or the difference is not greater than a preset threshold.

[0183] In one possible embodiment, the full optimization module 602 adjusts the first current teaching pose based on a first error between the first transformed coordinates and the target coordinates, including:

[0184] Determine the gradient descent direction of the first error;

[0185] Adjust the first current teaching pose according to the gradient descent direction.

[0186] This application also provides an electronic device, such as... Figure 7 As shown, it includes:

[0187] Memory 701 is used to store computer programs;

[0188] When processor 702 executes a program stored in memory 701, it performs the following steps:

[0189] Based on the current hand-eye calibration result and the first current teaching pose, the target coordinates in the object coordinate system are transformed back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates. The current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose.

[0190] Adjust the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates;

[0191] If the first convergence condition is met, the first current teaching pose is determined as the target teaching pose of the robotic arm.

[0192] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 702, the communication interface, and the memory 701 communicating with each other via the communication bus.

[0193] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0194] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0195] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0196] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0197] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described teaching pose determination methods.

[0198] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the teaching pose determination methods described above.

[0199] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), etc.

[0200] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0202] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for determining a teaching pose, characterized in that, The method includes: Based on the current hand-eye calibration result and the first current teaching pose, the target coordinates in the object coordinate system are transformed back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates. The current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose. Adjust the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates; If the first convergence condition is met, then the first current teaching pose is determined as the target teaching pose of the robotic arm. The step of transforming the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates includes: Transform the target coordinates in the object coordinate system to the camera coordinate system to obtain the camera projection point; The camera projection point is transformed to the base coordinate system of the robotic arm to obtain the base projection point; The projection point of the base is transformed to the flange coordinate system of the robotic arm to obtain the projection point of the workpiece; The projection points of the workpiece are transformed to the object coordinate system to obtain the first transformed coordinates.

2. The method according to claim 1, characterized in that, The method further includes: Acquire at least one grasping pose and an object image corresponding to each grasping pose, wherein the object image is an image of the object captured by the visual positioning component of the robotic arm when the robotic arm grasps the object in the grasping pose; The teaching pose is determined based on the initial hand-eye calibration results, each grasping pose, and the object image corresponding to each grasping pose, and is used as the teaching pose corresponding to the grasping pose. The initial teaching pose is determined based on the teaching pose corresponding to each of the described grasping poses.

3. The method according to claim 2, characterized in that, The step of determining the initial teaching pose based on the teaching pose corresponding to each of the grasping poses includes: The teaching poses with accuracy lower than a preset accuracy threshold are filtered out from the teaching poses corresponding to each of the grasping poses to obtain the initial screening teaching poses. The initial teaching pose is determined based on the initial screening teaching pose.

4. The method according to claim 2, characterized in that, The at least one grasping pose includes multiple different grasping poses.

5. The method according to claim 1, characterized in that, The step of adjusting the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates includes: Based on the first error between the first transformed coordinates and the target coordinates, adjust the first current teaching pose and the current hand-eye calibration result; The method further includes: Update the hand-eye calibration results used by the visual positioning component.

6. The method according to claim 5, characterized in that, The method further includes: Based on the initial hand-eye calibration result and the second current teaching pose, the target coordinates in the object coordinate system are transformed back to the object coordinate system through multiple coordinate transformations to obtain the second transformed coordinates. The second current teaching pose is initially the initial teaching pose. Adjust the second current teaching pose based on the second error between the second transformed coordinates and the target coordinates; The updating of the hand-eye calibration results used by the visual positioning component includes: If the first error is less than the second error and the difference between the first error and the second error is greater than a preset difference threshold, then the hand-eye calibration result used by the visual positioning component is updated.

7. The method according to claim 6, characterized in that, The method further includes: If the first error is not less than the second error or the difference is not greater than a preset threshold, then the hand-eye calibration result used by the visual positioning component is not updated.

8. The method according to claim 1, characterized in that, The step of adjusting the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates includes: Determine the gradient descent direction of the first error; Adjust the first current teaching pose according to the gradient descent direction.

9. A teaching posture determination device, characterized in that, The device includes: The first transformation module is used to transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations based on the current hand-eye calibration result and the first current teaching pose, to obtain the first transformed coordinates. The current hand-eye calibration result is initially the initial hand-eye calibration result used by the vision positioning component of the robotic arm, and the first current teaching pose is initially the initial teaching pose. The full optimization module is used to adjust the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates; The pose determination module is used to determine the first current teaching pose as the target teaching pose of the robotic arm if the first convergence condition is met. The first transformation module transforms the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations to obtain the first transformed coordinates, including: Transform the target coordinates in the object coordinate system to the camera coordinate system to obtain the camera projection point; The camera projection point is transformed to the base coordinate system of the robotic arm to obtain the base projection point; The projection point of the base is transformed to the flange coordinate system of the robotic arm to obtain the projection point of the workpiece; The projection points of the workpiece are transformed to the object coordinate system to obtain the first transformed coordinates.

10. The apparatus according to claim 9, characterized in that, The device further includes: An initial teaching pose determination module is used to acquire at least one grasping pose and an object image corresponding to each grasping pose, wherein the object image is an image of the object captured by the visual positioning component of the robotic arm when the robotic arm grasps the object in the grasping pose; a teaching pose is determined based on the initial hand-eye calibration result, each grasping pose, and the object image corresponding to each grasping pose, and is used as the teaching pose corresponding to the grasping pose; an initial teaching pose is determined based on the teaching pose corresponding to each grasping pose. The initial teaching pose determination module determines the initial teaching pose based on the teaching pose corresponding to each of the grasping poses, including: The teaching poses with accuracy lower than a preset accuracy threshold are filtered out from the teaching poses corresponding to each of the grasping poses to obtain the initial screening teaching poses. The initial teaching pose is determined based on the initial screening teaching pose; The at least one grasping pose includes multiple different grasping poses; The full optimization module adjusts the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates, including: Based on the first error between the first transformed coordinates and the target coordinates, adjust the first current teaching pose and the current hand-eye calibration result; The device further includes: The update module is used to update the hand-eye calibration results used by the visual positioning component; The device further includes: The local optimization module is used to transform the target coordinates in the object coordinate system back to the object coordinate system through multiple coordinate transformations based on the initial hand-eye calibration result and the second current teaching pose, to obtain the second transformed coordinates. The second current teaching pose is initially the initial teaching pose. The module adjusts the second current teaching pose based on the second error between the second transformed coordinates and the target coordinates. The update module updates the hand-eye calibration results used by the visual positioning component, including: If the first error is less than the second error and the difference between the first error and the second error is greater than a preset difference threshold, then the hand-eye calibration result used by the visual positioning component is updated. The update module is further configured to refuse to update the hand-eye calibration result used by the visual positioning component if the first error is not less than the second error or the difference is not greater than a preset threshold. The full optimization module adjusts the first current teaching pose based on the first error between the first transformed coordinates and the target coordinates, including: Used to determine the gradient descent direction of the first error; Adjust the first current teaching pose according to the gradient descent direction.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images