A labeling positioning method, device, electronic device and storage medium

Abstract

The present disclosure provides a labeling positioning method, apparatus, electronic device, and storage medium. The method includes: determining the center point coordinates of a label suction head in the mechanical arm tool coordinate system; determining the pose conversion relationship between the mechanical arm tool coordinate system and the pixel coordinate system based on the center point coordinates of the label suction head; determining the center point coordinates of the adsorbed label and the workpiece to be labeled in the mechanical arm tool coordinate system based on the pose conversion relationship; and adjusting the pose of the adsorbed label based on the center point coordinates of the adsorbed label and the workpiece to be labeled.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image data processing, and in particular, to a labeling positioning method, apparatus, electronic device, and storage medium. Background Art

[0002] In the production and manufacturing process, manually labeling products not only has a slow labeling speed, but also the labeling accuracy will decrease with the accumulation of working hours, affecting the overall production efficiency. Currently, a vision sensor is often used to guide a robotic arm for labeling.

[0003] Therefore, improving the efficiency and accuracy of robotic arm labeling has always been the pursued goal. Summary of the Invention

[0004] The present disclosure provides a labeling positioning method, apparatus, electronic device, and storage medium to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a labeling positioning method is provided. The method includes: determining the center point coordinates of a label suction head in a robotic arm tool coordinate system; determining a pose transformation relationship between the robotic arm tool coordinate system and a pixel coordinate system based on the center point coordinates of the label suction head; determining the center point coordinates of an adsorbed label and a workpiece to be labeled in the robotic arm tool coordinate system based on the pose transformation relationship; and adjusting the pose of the adsorbed label based on the center point coordinates of the adsorbed label and the workpiece to be labeled.

[0006] In an implementable embodiment, determining the center point coordinates of a label suction head in a robotic arm tool coordinate system includes: controlling the robotic arm to move the label suction head to touch any point in the operating space of the robotic arm from different orientations to obtain two sets of calibration points; using the two-point calibration method to determine the position transformation relationship between the end point of the label suction head and the end point of the robotic arm; and calibrating the center point coordinates of the label suction head in the robotic arm tool coordinate system based on the position transformation relationship.

[0007] In an implementable embodiment, determining the pose transformation relationship between the robotic arm tool coordinate system and a pixel coordinate system based on the center point coordinates of the label suction head includes: performing hand-eye calibration on a first industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and a first pixel coordinate system; performing hand-eye calibration on a second industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and a second pixel coordinate system; wherein the first industrial camera is an upper camera, and the hand-eye calibration method is eye-to-hand; the second industrial camera is a lower camera, and the hand-eye calibration method is eye-in-hand.

[0008] In an implementable embodiment, performing hand-eye calibration on the first industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system includes: acquiring an image of the label suction head using the first industrial camera; selecting nine pixel points from the acquired image and recording the corresponding pixel coordinates; aligning the label suction head with the centers of the nine pixel points respectively, and using the label suction head to read the coordinates of the nine pixel points in the robotic arm tool coordinate system; and obtaining the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system based on the pixel coordinates and the coordinates of the pixel points in the robotic arm tool coordinate system.

[0009] In an implementable embodiment, performing hand-eye calibration on the second industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system includes: acquiring an image of the calibration plate using the second industrial camera, where the height of the calibration plate is the same as the height of the workpiece to be pasted; selecting nine points on the calibration plate and recording the corresponding pixel coordinates; aligning the label suction head with the centers of the nine points on the calibration plate respectively, and using the label suction head to read the coordinates of the nine points on the calibration plate in the robotic arm tool coordinate system; and obtaining the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system based on the pixel coordinates and the coordinates of the points on the calibration plate in the robotic arm tool coordinate system.

[0010] In an implementable embodiment, determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system based on the pose transformation relationship includes: determining the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted; and determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system based on the pose transformation relationship, the center point pixel coordinates of the adsorbed label, and the center point pixel coordinates of the workpiece to be pasted.

[0011] In an implementable embodiment, determining the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted includes: acquiring images of the adsorbed label and the workpiece to be pasted; and determining the contours of the adsorbed label and the workpiece to be pasted based on the images of the adsorbed label and the workpiece to be pasted, so as to obtain the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted.

[0012] In an implementable embodiment, adjusting the pose of the adsorption label based on the center point coordinates of the adsorption label and the workpiece to be labeled includes: determining the pose offset between the suction head and the center point of the adsorption label in the robot arm tool coordinate system; using the suction head as the center point of the robot arm tool coordinate system, and adjusting the pose of the adsorption label based on the pose offset between the suction head and the center point of the adsorption label; determining the pose offset between the adsorption label and the workpiece to be labeled in the robot arm tool coordinate system; switching the robot arm tool coordinate system, and using the center point of the adsorption label as the center point of the switched robot arm tool coordinate system; and adjusting the pose of the adsorption label based on the pose offset between the adsorption label and the workpiece to be labeled so that the adsorption label is aligned with the workpiece to be labeled.

[0013] In the above implementable embodiment, the method further includes: collecting multiple groups of sample pose data corresponding to the adsorption label, and performing data fitting on the multiple groups of sample pose data by using the least squares method to correct the pose of the adsorption label after adjustment.

[0014] According to a second aspect of the present disclosure, there is provided a labeling positioning device, including: a determination module, configured to determine the center point coordinates of the suction head in the robot arm tool coordinate system; determining a pose conversion relationship between the robot arm tool coordinate system and the pixel coordinate system based on the center point coordinates of the suction head; determining the center point coordinates of the adsorption label and the workpiece to be labeled in the robot arm tool coordinate system based on the pose conversion relationship; and a positioning module, configured to adjust the pose of the adsorption label based on the center point coordinates of the adsorption label and the workpiece to be labeled.

[0015] In an implementable embodiment, the determination module is specifically configured to control the robot arm to move the suction head so that the suction head touches any point in the operation space of the robot arm from different orientations to obtain two sets of calibration points; using the two-point calibration method to determine the position transformation relationship between the end point of the suction head and the end point of the robot arm; and calibrating the center point coordinates of the suction head in the robot arm tool coordinate system based on the position transformation relationship.

[0016] In an implementable embodiment, the determination module is specifically configured to perform hand-eye calibration on the first industrial camera by using the nine-point calibration method to obtain the pose conversion relationship between the robot arm tool coordinate system and the first pixel coordinate system; performing hand-eye calibration on the second industrial camera by using the nine-point calibration method to obtain the pose conversion relationship between the robot arm tool coordinate system and the second pixel coordinate system; wherein, the first industrial camera is the upper camera, and the hand-eye calibration method is eye-to-hand; the second industrial camera is the lower camera, and the hand-eye calibration method is eye-in-hand.

[0017] In an implementable embodiment, the determining module is specifically configured to collect an image of the label suction head by using a first industrial camera; select nine pixel points from the collected image and record the corresponding pixel coordinates; align the label suction head with the centers of the nine pixel points respectively, and use the label suction head to read the coordinates of the nine pixel points in the robotic arm tool coordinate system; obtain the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system based on the pixel coordinates and the coordinates of the pixel points in the robotic arm tool coordinate system.

[0018] In an implementable embodiment, the determining module is specifically configured to collect an image of a calibration board by using a second industrial camera, where the height of the calibration board is the same as the height of the workpiece to be pasted; select nine points on the calibration board and record the corresponding pixel coordinates; align the label suction head with the centers of the nine points on the calibration board respectively, and use the label suction head to read the coordinates of the nine points on the calibration board in the robotic arm tool coordinate system; obtain the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system based on the pixel coordinates and the coordinates of the points on the calibration board in the robotic arm tool coordinate system.

[0019] In an implementable embodiment, the determining module is specifically configured to determine the central point pixel coordinates of the adsorbed label and the central point pixel coordinates of the workpiece to be pasted; determine the central point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system based on the pose transformation relationship, the central point pixel coordinates of the adsorbed label, and the central point pixel coordinates of the workpiece to be pasted.

[0020] In an implementable embodiment, the determining module is specifically configured to collect images of the adsorbed label and the workpiece to be pasted; determine the contours of the adsorbed label and the workpiece to be pasted based on the images of the adsorbed label and the workpiece to be pasted, and obtain the central point pixel coordinates of the adsorbed label and the central point pixel coordinates of the workpiece to be pasted.

[0021] In an implementable embodiment, the positioning module is specifically configured to determine the pose offset between the label suction head and the central point of the adsorbed label in the robotic arm tool coordinate system; use the label suction head as the center point of the robotic arm tool coordinate system, and adjust the pose of the adsorbed label based on the pose offset between the label suction head and the central point of the adsorbed label; determine the pose offset between the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system; switch the robotic arm tool coordinate system, and use the central point of the adsorbed label as the center point of the switched robotic arm tool coordinate system; adjust the pose of the adsorbed label based on the pose offset between the adsorbed label and the workpiece to be pasted, so that the adsorbed label is aligned with the workpiece to be pasted.

[0022] In the above-described feasible embodiments, the device further includes: a calibration module, configured to collect multiple groups of sample pose data corresponding to the adsorption label, and perform data fitting on the multiple groups of sample pose data by using the least squares method to correct the pose of the adsorption label after adjustment.

[0023] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0024] at least one processor; and a memory communicatively connected to the at least one processor;

[0025] wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the present disclosure.

[0026] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the method described in the present disclosure.

[0027] A labeling positioning method, device, electronic device and storage medium of the present disclosure convert the coordinates of the adsorption label and the workpiece to be labeled into the mechanical arm tool coordinate system, and utilize the automatic switching characteristic of the mechanical arm tool coordinate system to adjust the pose of the adsorption label, so as to efficiently and accurately achieve automatic labeling positioning.

[0028] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] By referring to the accompanying drawings and reading the following detailed description, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become easily understood. In the drawings, several embodiments of the present disclosure are shown in an exemplary rather than restrictive manner, wherein:

[0030] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0031] Figure 1 A schematic diagram of a processing flow of the labeling positioning method according to an embodiment of the present disclosure is shown;

[0032] Figure 2 A schematic diagram of a pose relationship between label adsorption and the workpiece to be labeled in the labeling positioning method according to an embodiment of the present disclosure is shown;

[0033] Figure 3 A schematic diagram of a coordinate system relationship in the labeling positioning method according to an embodiment of the present disclosure is shown;

[0034] Figure 4 The figure shows a schematic structural diagram of a labeling positioning device according to an embodiment of the present disclosure;

[0035] Figure 5 The figure shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0036] To make the objectives, features, and advantages of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0037] During the process of a robotic arm labeling guided by a vision sensor, high-precision fixing components are usually used to fix the label and the workpiece to be labeled. However, high-precision components are not only expensive but also do not meet the requirements of flexible production. Errors are inevitably generated during the process of fixing the poses of the label and the workpiece to be labeled. Therefore, it is necessary to use vision calibration and related positioning algorithms to calculate the pose relationship between the end of the robotic arm where the label is adsorbed and the workpiece to be labeled.

[0038] An embodiment of the present disclosure proposes a labeling positioning method, which converts the coordinates of the adsorbed label and the workpiece to be labeled into the robotic arm tool coordinate system, and uses the automatic switching characteristic of the robotic arm tool coordinate system to adjust the pose of the adsorbed label, so as to efficiently and accurately achieve automatic labeling positioning.

[0039] Figure 1 The figure shows a schematic processing flow diagram of the labeling positioning method according to an embodiment of the present disclosure.

[0040] Reference Figure 1 , a processing flow of the labeling positioning method according to an embodiment of the present disclosure may at least include the following steps:

[0041] Step S101: Determine the center point coordinates of the label suction head in the robotic arm tool coordinate system.

[0042] In some embodiments, the calibration of the robot tool coordinate has two-point method, three-point method, twenty-point method, twenty-three-point method, fifty-point method, and multi-point method; the robotic arm tool coordinate system is calibrated by the two-point calibration method, and the coordinate values of the tool (label suction head) end in the robotic arm end coordinate system are used to obtain the coordinate values of the tool (label suction head) end in the robotic arm tool coordinate system; in the labeling scenario, the relative heights of all hardware are fixed, and the Z-axis coordinates can be ignored.

[0043] Therefore, the specific implementation process of determining the center point coordinates of the suction head in the robotic arm tool coordinate system may at least include the following steps:

[0044] Step S201: Control the robotic arm to move the suction head so that the suction head touches any point in the operating space of the robotic arm from different orientations, obtaining two sets of calibration points.

[0045] Step S202: Use the two-point calibration method to determine the position transformation relationship between the end point of the suction head and the end point of the robotic arm.

[0046] Step S203: Calibrate the center point coordinates of the suction head in the robotic arm tool coordinate system based on the position transformation relationship.

[0047] Step S102: Determine the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system based on the center point coordinates of the suction head.

[0048] In some embodiments, the nine-point calibration method is adopted to perform hand-eye calibration on the first industrial camera and the second industrial camera respectively. Taking the center point coordinates of the suction head as the starting point, the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system is obtained. Among them, the first industrial camera is the upper camera, and the hand-eye calibration method is eye-outside-hand; the second industrial camera is the lower camera, and the hand-eye calibration method is eye-on-hand.

[0049] Specifically, the specific implementation process of determining the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system based on the center point coordinates of the suction head may at least include the following steps:

[0050] Step S301: Use the nine-point calibration method to perform hand-eye calibration on the first industrial camera to obtain the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system.

[0051] In some embodiments, it is inconvenient to fix the visual calibration board at the end point of the suction head. The first industrial camera can be used to collect images of the suction head, select nine pixel points from the collected images and record the corresponding pixel coordinates. For example, nine grid intersection points with the center of the suction head as the center point are used to replace the points on the calibration board; the suction head is aligned with the centers of the nine pixel points respectively, and the coordinates of the nine pixel points in the robotic arm tool coordinate system are read by the suction head. Based on the pixel coordinates and the coordinates of the pixel points in the robotic arm tool coordinate system, the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system is obtained.

[0052] Step S302: Use the nine-point calibration method to perform hand-eye calibration on the second industrial camera to obtain the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system.

[0053] In some embodiments, the calibration board is placed at the same height as the workpiece to be pasted, and the second industrial camera is used to collect images of the calibration board. Nine points are selected on the calibration board and the corresponding pixel coordinates are recorded. The suction nozzle is aligned with the centers of the nine points on the calibration board respectively, and the coordinates of the nine points on the calibration board in the robotic arm tool coordinate system are read by the suction nozzle. Based on the pixel coordinates and the coordinates of the points on the calibration board in the robotic arm tool coordinate system, the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system is obtained.

[0054] Step S103: Based on the pose transformation relationship, determine the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system.

[0055] In some embodiments, calculating the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system requires two types of parameters. One is the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system, and the other is the center point pixel coordinates of the adsorbed label and the workpiece to be pasted.

[0056] Therefore, the specific implementation process of determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system based on the pose transformation relationship may at least include the following steps:

[0057] Step S401: Determine the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted.

[0058] Step S402: Based on the pose transformation relationship, the center point pixel coordinates of the adsorbed label, and the center point pixel coordinates of the workpiece to be pasted, determine the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system.

[0059] In some embodiments, images of the adsorbed label and the workpiece to be pasted are collected, the collected images are subjected to grayscale processing and binary processing, and the contours of the adsorbed label and the workpiece to be pasted are located; further, the images including the contour of the adsorbed label and the contour of the workpiece to be pasted are filtered and area screened to remove noise signals, and accurate contours of the adsorbed label and the workpiece to be pasted are obtained, thereby obtaining the center point pixel coordinates of the adsorbed label and the workpiece to be pasted.

[0060] Step S104: Based on the center point coordinates of the adsorbed label and the workpiece to be pasted, adjust the pose of the adsorbed label.

[0061] In some embodiments, the specific implementation process of adjusting the pose of the adsorbed label based on the center point coordinates of the adsorbed label and the workpiece to be pasted may at least include the following steps:

[0062] Step S501: Determine the pose offset between the suction nozzle and the center point of the adsorbed label in the robotic arm tool coordinate system.

[0063] Step S502: Using the suction head as the center point of the robotic arm tool coordinate system, adjust the pose of the adsorbed label based on the pose offset between the suction head and the center point of the adsorbed label.

[0064] Step S503: Determine the pose offset between the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system.

[0065] Step S504: Switch the robotic arm tool coordinate system, and use the center point of the adsorbed label as the center point of the switched robotic arm tool coordinate system.

[0066] Step S505: Adjust the pose of the adsorbed label based on the pose offset between the adsorbed label and the workpiece to be pasted, so that the adsorbed label is aligned with the workpiece to be pasted.

[0067] As Figure 2 shown, an exemplary description of steps S501 - S505 is as follows:

[0068] Use the first industrial camera to collect the initial pose of the adsorbed label. The initial pose of the adsorbed label includes the initial position and angle of the robotic arm adsorbing the label. Among them, in the robotic arm tool coordinate system, the center point coordinates of the adsorbed label are obtained from the above embodiments and denoted as (X c , Y c ); the rotation angle of the adsorbed label relative to the horizontal direction is denoted as θ1; in the robotic arm tool coordinate system, the center point coordinates of the suction head 1 can be read from the calibrated tool coordinate system and denoted as (X1, Y1); the offset of the suction head 1 relative to the center point of the adsorbed label is ΔX = X1 - X C , ΔY = Y1 - Y C . To align the adsorbed label with the workpiece to be pasted, move the adsorbed label above the position to be pasted, and use the suction head 1 as the center point of the robotic arm tool coordinate system, so that the robotic arm moves ΔY and ΔX in the X and Y directions respectively.

[0069] When the adsorbed label moves above the position to be pasted, use the second industrial camera to collect the pose of the label to be pasted and the pose of the workpiece to be pasted. The rotation angle of the workpiece to be pasted relative to the vertical direction is denoted as θ2; the center point coordinates of the workpiece to be pasted are also obtained from the above embodiments and denoted as (X w , Y w ); in the robotic arm tool coordinate system, the center point coordinates of the rotated adsorbed label are denoted as (X c ′, Y c ′); in the robotic arm tool coordinate system, the center point coordinates of the rotated suction head 1 are also read from the calibrated tool coordinate system and denoted as (X1′, Y1′); the angle offset between the adsorbed label and the workpiece to be pasted is Δθ = θ1 - θ2, and the position offset is ΔX1 = X w - X C ′, ΔY1 = Y C ′ - Yw On the basis of the above-mentioned pose adjustment, switch the tool coordinate system of the robotic arm, with the center point of the adsorbed label as the center point of the tool coordinate system of the robotic arm, move the robotic arm to offset by ΔX1 in the X direction and ΔY1 in the Y direction, so that the center point of the adsorbed label is aligned with the center point of the workpiece to be pasted. Finally, rotate the robotic arm around the center point of the label by an angle of Δθ, and the adsorbed label can be aligned with the workpiece to be pasted.

[0070] It should be understood that the pose relationship between the adsorbed label and the workpiece to be pasted in the labeling scenario is not limited to the pose relationship described in the exemplary description, and the adjustment methods corresponding to different pose relationships are not listed one by one here.

[0071] The method provided in the above embodiment can achieve automatic labeling; at the same time, by using the dual-camera system, the errors accumulated by the robotic arm due to long-term high-speed operation and the calibration error corresponding to the first industrial camera are reduced. On this basis, the calibration error corresponding to the second industrial camera can be further reduced by using multiple groups of sample pose data corresponding to the adsorbed label.

[0072] In some embodiments, design multiple groups of experimental data, calculate the theoretical distances of the robotic arm offset in the X direction and Y direction, and the rotation angle, denoted as (X 计i , Y 计i , θ 计i ); collect the actual distances of the robotic arm offset in the X direction and Y direction, and the rotation angle, denoted as (X 实i , Y 实i , θ 实i ); where i = 1, 2, 3... n, and then use the least squares method for data fitting to correct the distances of the robotic arm offset in the X direction and Y direction, and the rotation angle. The calculation formulas are as follows:

[0073]

[0074] Among them, a, b, c, d, j, and k are unknown coefficients. At least two groups of experiments are required to solve the unknown coefficients, and more groups of data can obtain a more accurate calibration result. Sending the calibrated data to the robotic arm can efficiently and accurately achieve automatic labeling.

[0075] In the related art, some methods use a single camera to convert the coordinates of the adsorbed label and the workpiece to be pasted into the world coordinate system of the robotic arm, and then perform calculations such as alignment rotation and offset. The coordinate conversion calculation of this method is cumbersome, and it cannot reduce the cumulative error within a certain range, visual calibration error, and mechanical error caused by repeated pressing at the end of the robotic arm during high-speed repeated movement. These errors will lead to a decrease in labeling accuracy and affect the product yield.

[0076] The automatic labeling positioning method based on vision calibration proposed in the embodiments of the present disclosure uses dual cameras to collect images of the adsorbed label and the label to be pasted, converts the coordinates of the adsorbed label and the workpiece to be pasted into the mechanical arm tool coordinate system, calculates the relative position and angular deviation between the adsorbed label and the workpiece to be pasted, does not require excessive complicated coordinate conversion processes and alignment calculations, and can quickly perform labeling alignment by utilizing the automatic switching characteristic of the mechanical arm tool coordinate system. Finally, positioning error compensation is performed to eliminate the calibration error and the cumulative error generated during the long-term operation of the mechanical arm, realizing accurate and efficient labeling. At the same time, it can improve the flexible production capacity, meet the requirements for production line changes and different labeling scenarios, has strong flexible production capacity, high universality, simple operation and low cost.

[0077] Figure 3 Fig. shows a coordinate system relationship diagram of the labeling positioning method in the embodiments of the present disclosure.

[0078] Reference Figure 3 , the base coordinate system is a coordinate system with the base of the mechanical arm as the origin; the end joint of the mechanical arm is the end coordinate system, and usually a layer of flange is covered, so the end coordinate system of the mechanical arm is also called the flange end coordinate system; during the labeling process of the mechanical arm, the end of the mechanical arm will carry a label-absorbing workpiece, and two symmetrically positioned label-absorbing heads can be set on the label-absorbing workpiece. A coordinate system is established for the label-absorbing heads to facilitate the description of the operation of the label-absorbing heads, that is, the tool coordinate system; similarly, a coordinate system is established for the workpiece to facilitate the description of the position of the workpiece, that is, the workpiece coordinate system; the camera coordinate system is a coordinate system with the optical center of the camera as the origin. The mechanical arm usually carries a camera as a vision sensor. When the camera is carried at the end of the mechanical arm, it is called eye-in-hand, and when it is carried at a fixed position outside the mechanical arm, it is called eye-to-hand.

[0079] Figure 4 Fig. shows a schematic structural diagram of the composition of a labeling positioning device in an embodiment.

[0080] Reference Figure 4 , in an embodiment of a labeling positioning device, the device 40 includes: a determination module 401, configured to determine the center point coordinates of the label-absorbing head in the mechanical arm tool coordinate system; based on the center point coordinates of the label-absorbing head, determine the pose conversion relationship between the mechanical arm tool coordinate system and the pixel coordinate system; based on the pose conversion relationship, determine the center point coordinates of the adsorbed label and the workpiece to be pasted in the mechanical arm tool coordinate system; a positioning module 402, configured to adjust the pose of the adsorbed label based on the center point coordinates of the adsorbed label and the workpiece to be pasted.

[0081] In some embodiments, the determination module 401 is specifically configured to control the robotic arm to move the suction nozzle to touch any point in the operation space of the robotic arm from different orientations, so as to obtain two sets of calibration points; use the two-point calibration method to determine the position transformation relationship between the end point of the suction nozzle and the end point of the robotic arm; calibrate the center point coordinates of the suction nozzle in the tool coordinate system of the robotic arm based on the position transformation relationship.

[0082] In some embodiments, the determination module 401 is specifically configured to perform hand-eye calibration on the first industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the tool coordinate system of the robotic arm and the first pixel coordinate system; perform hand-eye calibration on the second industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the tool coordinate system of the robotic arm and the second pixel coordinate system; wherein, the first industrial camera is the upper camera, and the hand-eye calibration method is eye-to-hand; the second industrial camera is the lower camera, and the hand-eye calibration method is eye-in-hand.

[0083] In some embodiments, the determination module 401 is specifically configured to use the first industrial camera to collect images of the suction nozzle; select nine pixel points from the collected images and record the corresponding pixel coordinates; align the suction nozzle with the centers of the nine pixel points respectively, and use the suction nozzle to read the coordinates of the nine pixel points in the tool coordinate system of the robotic arm; based on the pixel coordinates and the coordinates of the pixel points in the tool coordinate system of the robotic arm, obtain the pose transformation relationship between the tool coordinate system of the robotic arm and the first pixel coordinate system.

[0084] In some embodiments, the determination module 401 is specifically configured to use the second industrial camera to collect images of the calibration plate, and the height of the calibration plate is the same as the height of the workpiece to be pasted; select nine points on the calibration plate and record the corresponding pixel coordinates; align the suction nozzle with the centers of the nine points on the calibration plate respectively, and use the suction nozzle to read the coordinates of the nine points on the calibration plate in the tool coordinate system of the robotic arm; based on the pixel coordinates and the coordinates of the points on the calibration plate in the tool coordinate system of the robotic arm, obtain the pose transformation relationship between the tool coordinate system of the robotic arm and the second pixel coordinate system.

[0085] In some embodiments, the determination module 401 is specifically configured to determine the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted; based on the pose transformation relationship, the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted, determine the center point coordinates of the adsorbed label and the workpiece to be pasted in the tool coordinate system of the robotic arm.

[0086] In some embodiments, the determination module 401 is specifically configured to collect images of the adsorbed label and the workpiece to be pasted; based on the images of the adsorbed label and the workpiece to be pasted, determine the contours of the adsorbed label and the workpiece to be pasted, and obtain the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted.

[0087] In some embodiments, the positioning module 401 is specifically configured to determine the pose offset between the suction head and the center point of the adsorbed label in the robotic arm tool coordinate system; with the suction head as the center point of the robotic arm tool coordinate system, adjust the pose of the adsorbed label based on the pose offset between the suction head and the center point of the adsorbed label; determine the pose offset between the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system; switch the robotic arm tool coordinate system, and use the center point of the adsorbed label as the center point of the switched robotic arm tool coordinate system; adjust the pose of the adsorbed label based on the pose offset between the adsorbed label and the workpiece to be pasted, so that the adsorbed label is aligned with the workpiece to be pasted.

[0088] In some embodiments, the device 40 further includes a calibration module 403, configured to collect multiple groups of sample pose data corresponding to the adsorbed label, and perform data fitting on the multiple groups of sample pose data by using the least squares method to correct the pose of the adsorbed label after adjustment.

[0089] Figure 5 FIG. shows a schematic block diagram of an exemplary electronic device 700 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable electronic device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0090] As Figure 5 shown, the electronic device 700 includes a computing unit 701, which can execute various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0091] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other electronic devices via a computer network such as the Internet and / or various telecommunication networks.

[0092] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as a labeling positioning method. For example, in some embodiments, a labeling positioning method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the labeling positioning method described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute a labeling positioning method in any other suitable manner (e.g., by means of firmware).

[0093] The various embodiments of the systems and technologies described above in this article can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, and the programmable processor can be a special or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or server.

[0095] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0098] A computer system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating blockchain.

[0099] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this disclosure can be achieved. No limitation is imposed herein.

[0100] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" can explicitly or implicitly include at least one of such features. In the description of this disclosure, "a plurality" means two or more, unless otherwise specifically defined.

[0101] As described above, the above are only specific embodiments of this disclosure, but the protection scope of this disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed in this disclosure can easily think of changes or substitutions, which should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A labeling positioning method, characterized in that, The method includes: Determining the center point coordinates of the label suction head in the robotic arm tool coordinate system; Based on the center point coordinates of the label suction head, determining the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system; Based on the pose transformation relationship, determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system; Based on the center point coordinates of the adsorbed label and the workpiece to be pasted, adjusting the pose of the adsorbed label; The determining the center point coordinates of the label suction head in the robotic arm tool coordinate system includes: Controlling the robotic arm to move the label suction head so that the label suction head touches any point in the operation space of the robotic arm from different orientations, obtaining two sets of calibration points; Using the two-point calibration method to determine the position transformation relationship between the end point of the label suction head and the end point of the robotic arm; Based on the position transformation relationship, calibrating the center point coordinates of the label suction head in the robotic arm tool coordinate system; The determining the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system based on the center point coordinates of the label suction head includes: Performing hand-eye calibration on the first industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system; Performing hand-eye calibration on the second industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system; Wherein, the first industrial camera is the upper camera, and the hand-eye calibration method is eye-outside-hand; the second industrial camera is the lower camera, and the hand-eye calibration method is eye-on-hand; The adjusting the pose of the adsorbed label based on the center point coordinates of the adsorbed label and the workpiece to be pasted includes: Determining the pose offset between the center points of the label suction head and the adsorbed label in the robotic arm tool coordinate system; Taking the label suction head as the center point of the robotic arm tool coordinate system, and adjusting the pose of the adsorbed label based on the pose offset between the center points of the label suction head and the adsorbed label; Determining the pose offset between the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system; Switching the robotic arm tool coordinate system, and taking the center point of the adsorbed label as the center point of the switched robotic arm tool coordinate system; Adjusting the pose of the adsorbed label based on the pose offset between the adsorbed label and the workpiece to be pasted so that the adsorbed label is aligned with the workpiece to be pasted.

2. The method according to claim 1, wherein The performing hand-eye calibration on the first industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system includes: Using the first industrial camera to collect images of the label suction head; Selecting nine pixel points from the collected images and recording the corresponding pixel coordinates; Aligning the label suction head with the centers of the nine pixel points respectively, and using the label suction head to read the coordinates of the nine pixel points in the robotic arm tool coordinate system; Based on the pixel coordinates and the coordinates of the pixel points in the robotic arm tool coordinate system, obtaining the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system.

3. The method according to claim 1, characterized in that, Performing hand-eye calibration on the second industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system includes: Using the second industrial camera to collect images of the calibration board, where the height of the calibration board is the same as the height of the workpiece to be pasted; Selecting nine points on the calibration board and recording the corresponding pixel coordinates; Aligning the suction nozzle head with the centers of the nine points on the calibration board respectively, and using the suction nozzle head to read the coordinates of the nine points on the calibration board in the robotic arm tool coordinate system; Based on the pixel coordinates and the coordinates of the points on the calibration board in the robotic arm tool coordinate system, obtaining the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system.

4. The method according to claim 1, characterized in that, Based on the pose transformation relationship, determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system includes: Determining the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted; Based on the pose transformation relationship, the center point pixel coordinates of the adsorbed label, and the center point pixel coordinates of the workpiece to be pasted, determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system.

5. The method according to claim 1, characterized in that, Determining the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted includes: Collecting images of the adsorbed label and the workpiece to be pasted; Based on the images of the adsorbed label and the workpiece to be pasted, determining the contours of the adsorbed label and the workpiece to be pasted, and obtaining the center point pixel coordinates of the adsorbed label and the center point pixel coordinates of the workpiece to be pasted.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Collecting multiple groups of sample pose data corresponding to the adsorbed label, and using the least squares method to perform data fitting on the multiple groups of sample pose data to correct the adjusted pose of the adsorbed label.

7. A labeling positioning device, characterized in that, The device includes: A determination module, configured to determine the center point coordinates of the suction nozzle head in the robotic arm tool coordinate system; based on the center point coordinates of the suction nozzle head, determining the pose transformation relationship between the robotic arm tool coordinate system and the pixel coordinate system; based on the pose transformation relationship, determining the center point coordinates of the adsorbed label and the workpiece to be pasted in the robotic arm tool coordinate system; A positioning module, configured to adjust the pose of the adsorbed label based on the center point coordinates of the adsorbed label and the workpiece to be pasted; The determination module is further configured to control the robotic arm to move the suction nozzle head so that the suction nozzle head touches any point in the operation space of the robotic arm from different orientations, obtaining two sets of calibration points; using the two-point calibration method to determine the position transformation relationship of the end point of the suction nozzle head relative to the end point of the robotic arm; based on the position transformation relationship, calibrating the center point coordinates of the suction nozzle head in the robotic arm tool coordinate system; The determination module is further configured to perform hand-eye calibration on the first industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the first pixel coordinate system; perform hand-eye calibration on the second industrial camera using the nine-point calibration method to obtain the pose transformation relationship between the robotic arm tool coordinate system and the second pixel coordinate system; wherein, the first industrial camera is the upper camera, and the hand-eye calibration method is eye-to-hand; the second industrial camera is the lower camera, and the hand-eye calibration method is eye-in-hand. The positioning module is further configured to determine the pose offset between the label suction head and the center point of the adsorbed label in the robotic arm tool coordinate system; with the label suction head as the center point of the robotic arm tool coordinate system, adjust the pose of the adsorbed label based on the pose offset between the label suction head and the center point of the adsorbed label; determine the pose offset between the adsorbed label and the workpiece to be labeled in the robotic arm tool coordinate system; switch the robotic arm tool coordinate system, and use the center point of the adsorbed label as the center point of the switched robotic arm tool coordinate system; adjust the pose of the adsorbed label based on the pose offset between the adsorbed label and the workpiece to be labeled, so that the adsorbed label is aligned with the workpiece to be labeled.

8. An electronic device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the label positioning method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the label positioning method according to any one of claims 1-6.

Images