Workpiece grasping method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a workpiece grasping method, device, electronic device, and storage medium. The workpiece grasping method includes: acquiring a first workpiece image captured by a camera; determining at least one first feature point from the first workpiece image; determining the first spatial coordinates of the first workpiece in the spatial coordinate system of a robotic arm based on the pixel coordinates and calibration information of the at least one first feature point; generating control information based on the first spatial coordinates, and sending the control information to the robotic arm so that the robotic arm grasps the first workpiece based on the control information. This solution identifies the position of the workpiece through the mapping relationship between the pixel coordinate system and the spatial coordinate system, and controls the robotic arm to automatically grasp the workpiece. This can solve the problem of manually placing the target object at a specific angle and position during the grasping process, which takes a long time and leads to low production efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a workpiece grasping method, device, electronic device, and storage medium. Background Art

[0002] To reduce labor costs and improve safety during the workpiece production process, robotic arms are used to replace manual labor in grasping workpieces for transfer, flipping, welding, and other tasks. These arms simulate the way the human eye sees objects and drive the robotic arms to grasp the workpieces. However, the grasping performance of robotic arms is significantly different from that of manual labor.

[0003] Currently, in order to deal with the above differences, a common method is to place the workpiece directly under the robotic arm and manually adjust the position of the workpiece to a specific angle and position to ensure the grasping effect of the robotic arm.

[0004] However, manually placing the workpiece at a specific angle and position takes a long time, resulting in low production efficiency. Summary of the Invention

[0005] In order to solve the above technical problems, embodiments of the present invention provide a workpiece grasping method, device, electronic device and storage medium to at least solve or alleviate the above problems.

[0006] According to a first aspect of an embodiment of the present application, a workpiece grasping method is provided, comprising: acquiring a first workpiece image captured by a camera, wherein the first workpiece image includes an image of a first workpiece located on a stage; determining at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate a position of the image of the first workpiece in the first workpiece image; determining a first spatial coordinate of the first workpiece in a spatial coordinate system of a robotic arm based on pixel coordinates and calibration information of the at least one first feature point, wherein the calibration information is used to indicate a mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system; generating control information based on the first spatial coordinate, and sending the control information to the robotic arm, so that the robotic arm grasps the first workpiece based on the control information.

[0007] According to a second aspect of an embodiment of the present application, there is provided an object grasping device, comprising: an acquisition module for acquiring a first workpiece image captured by a camera, wherein the first workpiece image comprises an image of a first workpiece located on a stage; an extraction module for determining at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate the position of the image of the first workpiece in the first workpiece image; a determination module for determining a first spatial coordinate of the first workpiece in a spatial coordinate system of a robotic arm based on the pixel coordinates and calibration information of the at least one first feature point, wherein the calibration information is used to indicate a mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system; a grasping module for generating control information based on the first spatial coordinates, and sending the control information to the robotic arm, so that the robotic arm grasps the first workpiece based on the control information.

[0008] According to the third aspect of the embodiments of the present application, an electronic device is provided, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to any method in the above-mentioned multiple method embodiments.

[0009] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any one of the methods in the above-mentioned multiple method embodiments is implemented.

[0010] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, which instruct a computing device to perform operations corresponding to any one of the above-mentioned method embodiments.

[0011] As can be seen from the above technical solution, at least one first feature point determined from the first workpiece image can indicate the position of the image of the first workpiece in the first workpiece image, and the calibration information can indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm. Through the calibration information and the pixel coordinates of the first feature point, the position of the first workpiece in the spatial coordinate system of the robotic arm can be determined, and then control information can be generated based on the position of the first workpiece in the spatial coordinate system. After the control information is sent to the robotic arm, the robotic arm grasps the first workpiece according to the control information. It can be seen that since the position of the workpiece in the spatial coordinate system of the robotic arm can be determined, the robotic arm can automatically grasp the workpiece placed arbitrarily on the stage, without having to place the workpiece on the stage at a specific angle and position, thereby saving the time required to place the workpiece on the stage and improving the efficiency of grasping the workpiece. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 is a flow chart of a workpiece grasping method according to an embodiment of the present application;

[0014] Figure 2 is a flow chart of a method for obtaining second spatial coordinates according to an embodiment of the present application;

[0015] Figure 3 is a flow chart of a method for determining an offset according to an embodiment of the present application;

[0016] Figure 4 is a flow chart of a method for determining calibration information according to an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of the movement sequence of a robotic arm according to one embodiment of the present application;

[0018] Figure 6 Schematic diagram of the coordinates of the movement points of a robotic arm according to one embodiment of the present application;

[0019] Figure 7 is a schematic diagram of an object grasping device according to an embodiment of the present application;

[0020] Figure 8 It is a schematic diagram of an electronic device according to an embodiment of the present application.

[0021] List of reference numerals:

[0022] 100: Workpiece Grasping Method 200: Control Information Generation Method 300: Offset Determination Method

[0023] 400: Method for determining calibration information 700: Object grasping device 701: Acquisition module

[0024] 702: Extraction module 703: Determination module 704: Capture module

[0025] 800: Electronic device 801: Processor 802: Communication interface

[0026] 803: Storage 804: Communication bus 805: Program

[0027] 101: Get the first workpiece image captured by the camera

[0028] 102: Determine at least one first feature point from the first workpiece image

[0029] 103: Determine the first spatial coordinate of the first workpiece in the spatial coordinate system of the robot arm

[0030] 104: Generate control information according to the first spatial coordinate and send the control information to the robotic arm

[0031] 201: Acquire a second workpiece image previously captured by a camera

[0032] 202: Determine at least one second feature point from the second workpiece image

[0033] 203: Determine the second spatial coordinates of the first workpiece in the spatial coordinate system

[0034] 301: Determine the X-axis displacement offset of the position of the first workpiece in the spatial coordinate system relative to the standard position

[0035] 302: Determine the Y-axis displacement of the first workpiece relative to the standard position in the spatial coordinate system

[0036] 303: Determine the Z-axis rotation offset of the position of the first workpiece in the spatial coordinate system relative to the standard position

[0037] 401: Determine the calibration reference point on the second workpiece

[0038] 402: Control the robot arm to perform N translations along the X-axis and / or Y-axis in the spatial coordinate system

[0039] 403: Get the translation calibration image captured by the camera after each translation of the robotic arm

[0040] 404: Control the robotic arm to rotate M times around the Z axis in the spatial coordinate system

[0041] 405: Get the rotation calibration image captured by the camera after each rotation of the robotic arm

[0042] 406: Confirm calibration information DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0044] Workpiece grasping method

[0045] Figure 1 This is a workpiece grasping method 100 according to an embodiment of the present application, such as Figure 1 As shown, the workpiece grasping method 100 includes the following steps:

[0046] Step 101: Acquire a first workpiece image captured by a camera.

[0047] The first workpiece image includes an image of the first workpiece, which may be a workpiece such as a circuit board. When capturing the first workpiece image, the first workpiece image is placed on a stage. After the robotic arm moves above the stage, a camera mounted on the robotic arm and moving synchronously with the robotic arm captures an image of the first workpiece to obtain the first workpiece image.

[0048] Step 102: Determine at least one first feature point from the first workpiece image.

[0049] After acquiring a camera image of a first workpiece to be captured, the position of the image of the first workpiece within the first workpiece image needs to be determined. At least one first feature point is determined from the first workpiece image. A first feature point is a point in the first workpiece image where the grayscale value changes dramatically or a point on the edge of the first workpiece image where the curvature is significant. This first feature point can indicate the position of the image of the first workpiece to be captured within the first workpiece image.

[0050] Step 103: Determine the first spatial coordinates of the first workpiece in the spatial coordinate system of the robot arm.

[0051] The robotic arm has a corresponding spatial coordinate system, based on which the robotic arm can move from one position to another. In order for the robotic arm to grasp a first workpiece, it is necessary to determine the first spatial coordinates of the first workpiece in the robotic arm's spatial coordinate system. The robotic arm can then determine the position of the first workpiece in the spatial coordinate system based on the first spatial coordinates of the first workpiece and then grasp the first workpiece.

[0052] The calibration information can indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm. Since the robotic arm and the camera move synchronously, the calibration information is fixed. Since the pixel coordinates of each first feature point can indicate the position of the image of the first workpiece within the first workpiece image, the pixel coordinates of each first feature point can be transformed into spatial coordinates in the spatial coordinate system based on the pixel coordinates of each first feature point and the calibration information, thereby obtaining the first spatial coordinates that can indicate the position of the first workpiece in the spatial coordinate system.

[0053] Step 104: Generate control information according to the first spatial coordinates, and send the control information to the robotic arm.

[0054] The first spatial coordinates indicate the position of the first workpiece in the spatial coordinate system. Corresponding control information can be generated based on the first spatial coordinates. After the control information is sent to the robotic arm, the robotic arm can move to a suitable position based on the control information to grasp the first workpiece. The control information includes, but is not limited to, commands for grasping, translation, rotation, and extension.

[0055] In an embodiment of the present application, at least one first feature point determined from the first workpiece image can indicate the position of the image of the first workpiece in the first workpiece image, and the calibration information can indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm. Through the calibration information and the pixel coordinates of the first feature point, the position of the first workpiece in the spatial coordinate system of the robotic arm can be determined, and then control information can be generated based on the position of the first workpiece in the spatial coordinate system. After the control information is sent to the robotic arm, the robotic arm grasps the first workpiece according to the control information. It can be seen that since the position of the workpiece in the spatial coordinate system of the robotic arm can be determined, the robotic arm can automatically grasp the workpiece placed arbitrarily on the stage without having to place the workpiece on the stage at a specific angle and position, thereby saving the time required to place the workpiece on the stage and improving the efficiency of grasping the workpiece.

[0056] In one possible implementation, when generating control information based on the first spatial coordinates, a predetermined second spatial coordinate can be obtained. The second spatial coordinates are used to indicate the coordinates of the first workpiece in the spatial coordinate system when the first workpiece is located at the standard position on the stage. Then, based on the first spatial coordinates and the second spatial coordinates, the offset of the position of the first workpiece in the spatial coordinate system relative to the standard position is determined, and the offset is determined as the control information.

[0057] It should be understood that the second spatial coordinates are pre-determined and indicate the coordinates of the first workpiece in the spatial coordinate system if the first workpiece is located at the standard position on the stage. A workpiece of the same type as the first workpiece to be grasped can be placed at the standard position on the stage in advance to determine the second spatial coordinates. When controlling the robotic arm to grasp the first workpiece, it is not necessary to place the first workpiece at the standard position on the stage every time a grasp is performed.

[0058] Since the second spatial coordinate indicates the position of the first workpiece located at the standard position in the spatial coordinate system, and the first spatial coordinate indicates the actual position of the first workpiece in the spatial coordinate system, the offset of the actual position of the first workpiece in the spatial coordinate system relative to the standard position can be determined based on the first spatial coordinate and the second spatial coordinate.

[0059] A preset motion flow for the robotic arm is set for the standard position. When the robotic arm moves above the stage according to this motion flow, it can grasp the first workpiece placed at the standard position. During the actual grasping process, the offset of the first workpiece's actual position in the spatial coordinate system relative to the standard position is transmitted as control information to the robotic arm. Based on this control information, the robotic arm can translate or rotate accordingly within the preset motion flow to grasp the first workpiece placed at any position on the stage.

[0060] In an embodiment of the present application, the first spatial coordinate indicates the actual position of the first workpiece in the spatial coordinate system, and the second spatial coordinate system indicates the position of the first workpiece in the spatial coordinate system when the first workpiece is in the standard position. Based on the first spatial coordinate and the second spatial coordinate, the offset of the actual position of the first workpiece relative to the standard position can be determined, and the offset is sent to the robotic arm as control information, so that the robotic arm can perform corresponding translation or rotation on a preset motion process according to the offset, wherein the preset operation process corresponds to the standard position, and then the workpiece located in a non-standard position on the stage is grasped, which facilitates the motion control of the robotic arm.

[0061] Figure 2 This is a flow chart of a method for obtaining the second spatial coordinates according to an embodiment of the present application. Figure 2 As shown, the second space coordinate acquisition method 200 includes the following steps:

[0062] Step 201: Acquire a second workpiece image captured in advance by a camera.

[0063] When the camera is capturing the image of the second workpiece, the first workpiece is placed in a standard position on the stage, the robotic arm moves above the first workpiece, and the camera, which moves synchronously with the robotic arm, captures the image of the first workpiece to obtain the second workpiece image.

[0064] Step 202: Determine at least one second feature point from the second workpiece image.

[0065] After acquiring a second image of the workpiece to be captured by the camera, the position of the first workpiece image within the second workpiece image needs to be determined. At least one second feature point is determined from the second workpiece image. This second feature point is a point in the second workpiece image where the grayscale value changes dramatically or where the edge of the second workpiece image has a significant curvature. This second feature point can indicate the position of the first workpiece image within the second workpiece image.

[0066] Step 203: Determine the second spatial coordinates of the first workpiece in the spatial coordinate system.

[0067] Each second feature point indicates the position of the image of the first workpiece located at the standard position in the second workpiece image. The calibration information is used to indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system. Therefore, based on each second feature point and the calibration information, the second spatial coordinate used to indicate the first workpiece located at the standard position in the spatial coordinate system of the robotic arm can be determined.

[0068] In an embodiment of the present application, the first workpiece is placed in a standard position on the worktable in advance, and the image of the second workpiece is captured by a camera. One or more second feature points are determined from the second workpiece image, and the second spatial coordinates are determined based on the second feature points and the calibration information. It is ensured that the determined second spatial coordinates can accurately indicate the position of the first workpiece in the spatial coordinate system of the robotic arm when the first workpiece is in the standard position, thereby ensuring the accuracy of the determined control information, so that the robotic arm can accurately grasp the first workpiece according to the received control information.

[0069] In a possible implementation, the first feature point is a point on the contour line of the first workpiece in the first workpiece image, and the second feature point is a point on the contour line of the first workpiece in the second workpiece image.

[0070] In general, there is a significant color difference between the workpiece and the stage. The workpiece's contour can be accurately identified in the workpiece image captured by the camera, and points along the workpiece's contour can be identified as feature points. In this case, all possible first feature points lie along the contour of the first circuit board, and all possible second feature points lie along the contour of the second circuit board.

[0071] In the embodiment of the present application, since the first feature point is located on the contour line of the first workpiece and the second feature point is located on the contour line of the first workpiece, the first workpiece can be accurately positioned according to each first feature point, and the accuracy of the determined second spatial coordinates can be guaranteed according to each second feature point, thereby ensuring that the robot arm can smoothly grasp the workpiece.

[0072] In one possible implementation, the spatial coordinate system of the robotic arm can be a three-dimensional Cartesian coordinate system, in which the planes of the X-axis and the Y-axis are parallel to the stage, and the first spatial coordinate and the second spatial coordinate both include a horizontal coordinate along the X-axis direction of the three-dimensional Cartesian coordinate system, a vertical coordinate along the Y-axis direction of the three-dimensional Cartesian coordinate system, and a rotation angle around the Z-axis direction of the three-dimensional Cartesian coordinate system. On this basis, Figure 3 A method for determining the offset is provided, such as Figure 3 As shown, the offset determination method 300 includes the following steps:

[0073] Step 301: Determine the X-axis displacement offset of the position of the first workpiece in the spatial coordinate system relative to the standard position.

[0074] Since the second spatial coordinate indicates the position of the standard position in the spatial coordinate system, the difference between the horizontal coordinate included in the first spatial coordinate and the horizontal coordinate included in the second spatial coordinate can be calculated, and the difference can be determined as the X-axis displacement offset of the position of the first workpiece relative to the standard position.

[0075] Step 302: Determine the Y-axis displacement offset of the position of the first workpiece in the spatial coordinate system relative to the standard position.

[0076] Since the standard position indicated by the second spatial coordinate is in the spatial coordinate system, the difference between the vertical coordinate included in the first spatial coordinate and the vertical coordinate included in the second spatial coordinate can be calculated, and the difference can be determined as the Y-axis offset of the position of the first circuit board relative to the standard position.

[0077] Step 303: Determine the Z-axis rotation offset of the position of the first workpiece in the spatial coordinate system relative to the standard position.

[0078] Because the second spatial coordinate indicates the position of the standard position in the spatial coordinate system, the difference between the rotation angle included in the first spatial coordinate and the rotation angle included in the second spatial coordinate can be calculated, and the difference can be determined as the Z-axis rotation offset of the position of the first circuit board relative to the standard position. Since the second spatial coordinate indicates the position of the standard position in the spatial coordinate system, with the standard position as a reference, the rotation angle included in the second spatial coordinate can be equal to zero.

[0079] In an embodiment of the present application, the X-axis displacement offset, Y-axis displacement offset and Z-axis rotation offset of the first workpiece relative to the standard position in the spatial coordinate system are calculated as control information of the robotic arm. The robotic arm can translate or rotate relative to the standard position according to the control information, so that the robotic arm can grasp the workpiece at any position on the worktable.

[0080] Figure 4 This is a flow chart of a method for determining calibration information according to an embodiment of the present application. Figure 4 As shown, the calibration information determination method 400 includes the following steps:

[0081] Step 401: Determine a calibration reference point located on a second workpiece.

[0082] The robotic arm's spatial coordinate system is a three-dimensional Cartesian coordinate system, in which the X, Y, and Z axes are perpendicular to each other, and the plane containing the X and Y axes is parallel to the stage. When a first workpiece and a second workpiece are placed on the stage, they are at the same height along the Z axis. The second workpiece can be the same type as the first, or different, such as the same type of circuit board.

[0083] When calibrating the camera using the second workpiece, since the camera captures a planar image, the different distances between the workpiece and the camera will affect the position of the workpiece's image in the image captured by the camera, thereby affecting the calibration result. Therefore, it is necessary to ensure that the second workpiece and the first workpiece have the same height in the Z-axis direction.

[0084] In order to calibrate the camera using the image of the second workpiece, calibration reference points are determined on the upper surface of the second workpiece, and the determined calibration reference points have corresponding images in the image captured by the camera.

[0085] Step 402: Control the robotic arm to perform N translations along the X-axis and / or Y-axis in the spatial coordinate system.

[0086] During the camera calibration process, in order to determine the conversion relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm, it is necessary to control the robotic arm to drive the camera fixed to the robotic arm to perform N translations along the X-axis and / or Y-axis in the spatial coordinate system according to the preset X-axis translation and Y-axis translation, so that the camera can capture images of the second workpiece placed on the stage at different positions. In order to solve the conversion relationship between the pixel coordinate system and the spatial coordinate system, at least three sets of pixel coordinates and spatial coordinates are required. Therefore, the number of times the robotic arm translates in the spatial coordinate system, N, is a positive integer greater than or equal to 3. The camera captures an image of the second workpiece after each translation of the robotic arm as a translation calibration image.

[0087] It should be noted that when the robotic arm translates along the X-axis and / or Y-axis in the spatial coordinate system, the position of the robotic arm in the Z-axis direction remains unchanged, ensuring that the distance between the camera and the stage in the Z-axis direction remains unchanged.

[0088] Step 403: Obtain a translation calibration image captured by the camera after each translation of the robotic arm.

[0089] Obtain the translation calibration images captured by the camera. Since the camera captures at least one translation calibration image after each translation of the robotic arm, at least N translation calibration images can be obtained.

[0090] Step 404: Control the robotic arm to rotate M times around the Z axis in the spatial coordinate system.

[0091] To improve camera calibration accuracy and ensure that the camera is not limited to X- and Y-axis motion, the robotic arm can be controlled to rotate the camera around the Z axis M times in the spatial coordinate system according to a preset Z-axis rotation amount. To determine the conversion relationship between pixel coordinates and spatial coordinates during robotic arm rotation, at least three sets of pixel coordinates and spatial coordinates of the calibration reference point after the robotic arm rotates are required. Therefore, M is a positive integer greater than or equal to 3.

[0092] Step 405: Obtain a rotation calibration image captured by the camera after each rotation of the robotic arm.

[0093] Obtain the rotation calibration image captured by the camera. Since the camera captures at least one rotation calibration image after each rotation of the robotic arm, at least M rotation calibration images can be obtained.

[0094] Step 406: Determine calibration information.

[0095] After obtaining the translation calibration image and the rotation calibration image, the calibration information is determined according to the position offset of the image of the calibration reference point in different translation calibration images, the position offset of the image of the calibration reference point in different rotation calibration images, the X-axis translation and Y-axis translation of the robotic arm when each translation calibration image is captured, and the Z-axis rotation of the robotic arm when each rotation calibration image is captured.

[0096] Since the second workpiece is located on the stage and remains stationary, after the robotic arm drives the camera to translate or rotate, the image of the calibration reference point on the second workpiece will be offset in the image captured by the camera, and the offset is positively correlated with the translation of the robotic arm along the X-axis and / or Y-axis and the rotation around the Z-axis. Therefore, based on each translation calibration image and rotation calibration image, calibration information indicating the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system can be determined.

[0097] In an embodiment of the present application, the robotic arm drives the camera to translate and rotate. After each translation and / or rotation, the camera captures an image of the second workpiece. According to the translation and / or rotation amount of the camera and the position offset of the image of the calibration reference point in the image captured by the camera, the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm is determined, thereby obtaining calibration information reflecting the mapping relationship. After the camera captures the image of the workpiece placed on the stage, the pixel coordinates of the workpiece in the image captured by the camera can be converted into spatial coordinates in the spatial coordinate system through the calibration information, and then the robotic arm can grab the workpiece according to its spatial coordinates, thereby realizing that the robotic arm can grab workpieces placed arbitrarily on the stage.

[0098] In addition, the camera is calibrated through multiple translation calibration images and multiple rotation calibration images to ensure that the generated calibration information can accurately reflect the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm. Then, the calibration information can be used to accurately determine the spatial coordinates of the workpiece in the spatial coordinate system, thereby ensuring the success rate of the robotic arm in grasping the workpiece.

[0099] In one possible implementation, when determining calibration information, the i-th pixel translation offset is calculated for the i-th translation among the N translations performed by the manipulator based on the position of the image of the calibration reference point in the i-1th translation calibration image captured by the camera before the manipulator performed the i-th translation, and the position of the image of the calibration reference point in the i-1th translation calibration image captured by the camera after the manipulator performed the i-th translation. The j-th pixel rotation offset is calculated for the j-th rotation among the M rotations performed by the manipulator based on the position of the image of the calibration reference point in the j-1th rotation calibration image captured by the camera before the manipulator performed the j-th rotation, and the position of the j-th rotation calibration image captured by the camera after the manipulator performed the j-th rotation. The calibration information is determined based on the i-th pixel translation offset and the X-axis and Y-axis translations corresponding to the i-th translation of the manipulator, and the j-th pixel rotation offset and the Z-axis rotation corresponding to the j-th rotation of the manipulator.

[0100] For example, the number of translations N is 8 and the number of rotations M is 4.

[0101] During the eight translations of the robotic arm, the pixel translation offset of the calibration reference point image is calculated before and after each translation. After the first translation, the pixel translation offset of the calibration reference point image is calculated before and after the first translation. The calculation method for subsequent pixel translation offsets is the same as above and is not repeated here.

[0102] During the four rotations of the robotic arm, the pixel rotation offset of the calibration reference point image is calculated before and after each rotation. After the first rotation, the pixel rotation offset of the calibration reference point image before and after the first rotation is calculated. The calculation method for the subsequent j-th pixel rotation offset is the same as the above method and is not repeated here.

[0103] The calibration information is determined based on the translation offsets of the 1st to 8th pixels, the X-axis translation and Y-axis translation corresponding to each translation, the rotation offsets of the 1st to 4th pixels, and the Z-axis rotation corresponding to each rotation.

[0104] In an embodiment of the present application, by comparing the pixel translation offsets in the translation calibration image before and after each translation of the image of the calibration reference point, and the pixel rotation offsets in the rotation calibration image before and after each rotation of the image of the calibration reference point, the offsets of the calibration reference points that match the number of movements are obtained. The accuracy of camera calibration is further improved by the movement offsets in the spatial coordinate system each time and the corresponding offsets of the calibration reference points.

[0105] Figure 5 This is a schematic diagram of the movement sequence of the robot arm in one embodiment of the present application. Figure 5 As shown in the figure, the number of translations N of the robot arm is 9. When the robot arm performs 9 translations, the translation combinations along the X-axis and the Y-axis are (x, 0), (x, 0), (0, y), (-x, 0), (-x, 0), (0, y), (x, 0), (x, 0) and (-x, -y), respectively. Both x and y are not equal to 0.

[0106] In an embodiment of the present application, by planning the translation path of the robotic arm, it is possible to avoid the robotic arm exceeding the range of the stage during the translation process, causing the camera to capture invalid images that do not contain the workpiece, thereby improving the efficiency of camera calibration.

[0107] In a possible implementation, the preset X-axis translation amount is equal to the Y-axis translation amount, that is, x equals y.

[0108] Figure 6 Schematic diagram of the coordinates of the robot arm motion points according to an embodiment of the present application. Figure 6 As shown in the figure, the X-axis translation and the Y-axis translation are equal to 5. When the robot arm performs 9 translations, the translation combinations along the X-axis and along the Y-axis are (5,0), (5,0), (0,5), (-5,0), (-5,0), (0,5), (5,0), (5,0), and (-5,-5).

[0109] In the embodiment of the present application, by setting the X-axis translation and the Y-axis translation to equal offsets, the computational complexity is reduced and the efficiency of camera calibration is improved.

[0110] In one possible implementation, during the movement of the robotic arm, at least one movement process simultaneously performs translation and rotation operations.

[0111] In the embodiment of the present application, by completing translation and rotation synchronously, the amount of images captured by the camera can be reduced, the amount of calculation is reduced while ensuring the accuracy of the calculation, and the efficiency of camera calibration is improved.

[0112] Object gripping device

[0113] Figure 7 Schematic diagram of an object grasping device according to an embodiment of the present application. Figure 7 As shown, the object grasping device 700 includes an acquisition module 701 , an extraction module 702 , a determination module 703 and a grasping module 704 .

[0114] After the acquisition module 701 acquires the first workpiece image acquired by the camera, the extraction module 702 determines at least one first feature point according to the first workpiece image acquired by the camera.

[0115] The determination module 703 determines the first spatial coordinates of the first workpiece in the spatial coordinate system of the robot arm according to the first feature point determined by the extraction module 702 .

[0116] The grabbing module 704 generates control information according to the first spatial coordinates determined by the determining module 703 .

[0117] In this embodiment of the present application, the at least one first feature point determined by the extraction module 702 can indicate the position of the first workpiece in the first workpiece image. The determination module 703 can determine the position of the first workpiece in the robotic arm's spatial coordinate system using the calibration information and the pixel coordinates of the first feature point. The grasping module 704 controls the robotic arm to grasp the workpiece based on the workpiece's position information. Without having to position the workpiece at a specific angle and position, the robotic arm can automatically grasp workpieces anywhere on the stage, saving labor costs.

[0118] electronic devices

[0119] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. Figure 8 As shown, the electronic device 800 may include: a processor 801, a communications interface 802, a memory 803, and a communication bus 804.

[0120] The processor 801 , the communication interface 802 , and the memory 803 communicate with each other via the communication bus 804 .

[0121] The communication interface 802 is used to communicate with other electronic devices or servers.

[0122] The processor 801 is used to execute the program 805, and specifically can execute the relevant steps in any of the aforementioned multiple method embodiments.

[0123] Specifically, the program 805 may include program codes, which include computer operation instructions.

[0124] The processor 801 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0125] The memory 803 is used to store the program 805. The memory 803 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0126] The program 805 can be specifically used to enable the processor 801 to execute any one of the multiple method embodiments in the aforementioned embodiments.

[0127] The specific implementation of each step in program 805 can be found in the corresponding descriptions of the corresponding steps and units in the aforementioned workpiece grasping method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiment, and will not be repeated here.

[0128] Through the electronic device of the embodiment of the present application, at least one first feature point determined from the first workpiece image can indicate the position of the image of the first workpiece in the first workpiece image, and the calibration information can indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robotic arm. Through the calibration information and the pixel coordinates of the first feature point, the position of the first workpiece in the spatial coordinate system of the robotic arm can be determined, and then control information can be generated based on the position of the first workpiece in the spatial coordinate system. After the control information is sent to the robotic arm, the robotic arm grasps the first workpiece according to the control information. It can be seen that since the position of the workpiece in the spatial coordinate system of the robotic arm can be determined, the robotic arm can automatically grasp the workpiece placed arbitrarily on the stage without having to place the workpiece on the stage at a specific angle and position, thereby saving the time required to place the workpiece on the stage and improving the efficiency of grasping the workpiece.

[0129] Computer storage media

[0130] This application also provides a computer-readable storage medium storing instructions for causing a machine to perform any of the method embodiments described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can be caused to read and execute the program code stored in the storage medium.

[0131] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.

[0132] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0133] Computer program product

[0134] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0135] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0136] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or be implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0137] It should be noted that not all steps and modules in the above processes and system structure diagrams are required, and certain steps or modules can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0138] In the above embodiments, the hardware module can be implemented mechanically or electrically. For example, a hardware module can include a permanent dedicated circuit or logic (such as a dedicated processor, FPGA or ASIC) to complete the corresponding operation. The hardware module can also include programmable logic or circuits (such as a general-purpose processor or other programmable processors), which can be temporarily set by software to complete the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0139] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A workpiece grasping method (100), comprising: Determine calibration information; Acquire a first workpiece image captured by a camera, wherein the first workpiece image includes an image of the first workpiece located on the stage; determining at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate a position of the image of the first workpiece in the first workpiece image; determining a first spatial coordinate of the first workpiece in a spatial coordinate system of the robotic arm based on the pixel coordinates of the at least one first feature point and calibration information, wherein the calibration information is used to indicate a mapping relationship between a pixel coordinate system of an image captured by the camera and the spatial coordinate system; generating control information according to the first spatial coordinates, and sending the control information to the robotic arm, so that the robotic arm grasps the first workpiece according to the control information; Determining the calibration information includes: Determining a calibration reference point on a second workpiece, wherein the second workpiece and the first workpiece have the same height in the Z-axis direction of the spatial coordinate system, and the plane containing the X-axis and the Y-axis in the spatial coordinate system is parallel to the stage; According to preset X-axis translation amounts and Y-axis translation amounts, control the robotic arm to perform N translations along the X-axis and / or Y-axis directions in the spatial coordinate system, and obtain a translation calibration image captured by the camera after each translation of the robotic arm, wherein the translation calibration image includes an image of the second workpiece located on the stage, and N is a positive integer greater than or equal to 3; According to a preset Z-axis rotation amount, control the robotic arm to rotate M times around the Z-axis in the spatial coordinate system, and obtain a rotation calibration image captured by the camera after each rotation of the robotic arm, wherein the rotation calibration image includes an image of the second workpiece located on the stage, and M is a positive integer greater than or equal to 3; The calibration information is determined according to the position offset of the image of the calibration reference point in different translation calibration images, the position offset of the image of the calibration reference point in different rotation calibration images, the X-axis translation and Y-axis translation corresponding to each translation calibration image, and the Z-axis rotation corresponding to each rotation calibration image.

2. The method according to claim 1, wherein The generating control information according to the first spatial coordinates includes: Acquire a second spatial coordinate, wherein the second spatial coordinate is used to indicate a coordinate of the first workpiece in the spatial coordinate system when the first workpiece is located at a standard position on the stage; determining, based on the first spatial coordinate and the second spatial coordinate, an offset of a position of the first workpiece in the spatial coordinate system relative to the standard position; The offset is determined as control information.

3. The method according to claim 2, further comprising: Acquire a second workpiece image previously captured by the camera, wherein the second workpiece image includes an image of the first workpiece located at a standard position on a stage; determining at least one second feature point from the second workpiece image, wherein the second feature point is used to indicate a position of the image of the first workpiece in the second workpiece image; Determine the second spatial coordinates of the first workpiece in the spatial coordinate system according to the pixel coordinates of the at least one second feature point and the calibration information.

4. The method according to claim 3, wherein: The first feature point is a point on the contour line of the first workpiece in the first workpiece image, and the second feature point is a point on the contour line of the first workpiece in the second workpiece image.

5. The method according to claim 2, wherein: The spatial coordinate system is a three-dimensional Cartesian coordinate system, in which the planes where the X-axis and the Y-axis are located are parallel to the stage, and the first spatial coordinate and the second spatial coordinate each include a horizontal coordinate along the X-axis direction of the three-dimensional Cartesian coordinate system, a vertical coordinate along the Y-axis direction of the three-dimensional Cartesian coordinate system, and a rotation angle around the Z-axis direction of the three-dimensional Cartesian coordinate system; Determining the offset of the position of the first workpiece in the spatial coordinate system relative to the standard position based on the first spatial coordinate and the second spatial coordinate includes: determining a difference between the abscissa included in the first spatial coordinate system and the abscissa included in the second spatial coordinate system as an X-axis displacement offset of the position of the first workpiece in the spatial coordinate system relative to the standard position; determining a difference between a longitudinal coordinate included in the first spatial coordinate system and a longitudinal coordinate included in the second spatial coordinate system as a Y-axis displacement offset of a position of the first workpiece in the spatial coordinate system relative to the standard position; The difference between the rotation angle included in the first spatial coordinate and the rotation angle included in the second spatial coordinate is determined as the Z-axis rotation offset of the position of the first workpiece in the spatial coordinate system relative to the standard position.

6. The method according to claim 1, wherein The determining the calibration information according to the position offset of the image of the calibration reference point in different translation calibration images, the position offset of the image of the calibration reference point in different rotation calibration images, the X-axis translation and Y-axis translation corresponding to each translation calibration image, and the Z-axis rotation corresponding to each rotation calibration image includes: For the i-th translation among the N translations performed by the robotic arm, calculate the i-th pixel translation offset of the image of the calibration reference point in the i-th translation calibration image and the i-th translation calibration image, where the i-th translation calibration image is the translation calibration image captured by the camera before the robotic arm performs the i-th translation, and the i-th translation calibration image is the translation calibration image captured by the camera after the robotic arm performs the i-th translation, and i is a positive integer less than N; For the j-th rotation among the M rotations performed by the robotic arm, calculate the j-th pixel rotation offset of the image of the calibration reference point in the j-1-th rotation calibration image and the j-th rotation calibration image, where the j-1-th rotation calibration image is the rotation calibration image captured by the camera before the robotic arm performs the j-th rotation, and the j-th rotation calibration image is the rotation calibration image captured by the camera after the robotic arm performs the j-th rotation, and j is a positive integer less than M; The calibration information is determined according to the i-th pixel translation offset and the X-axis translation and Y-axis translation corresponding to the i-th translation of the robotic arm, and the j-th pixel rotation offset and the Z-axis rotation corresponding to the j-th rotation of the robotic arm.

7. The method according to claim 6, wherein: N is equal to 9, M is equal to 3; When the robotic arm performs N translations, the combinations of translation amounts along the X-axis and along the Y-axis are (x, 0), (x, 0), (0, y), (-x, 0), (-x, 0), (0, y), (x, 0), (x, 0), and (-x, -y), respectively, where x is used to represent the translation amount of the robotic arm along the X-axis in the spatial coordinate system, and y is used to represent the translation amount of the robotic arm along the Y-axis in the spatial coordinate system, and both x and y are not equal to 0.

8. The method according to claim 7, wherein: x equals y.

9. The method according to any one of claims 6 to 8, wherein: The robot arm translates along the X-axis and / or Y-axis and rotates around the Z-axis at least once.

10. An object grasping device (700), comprising: An acquisition module (701) is configured to acquire a first workpiece image captured by a camera, wherein the first workpiece image includes an image of the first workpiece located on a stage; an extraction module (702) for determining at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate a position of the image of the first workpiece in the first workpiece image; a determination module (703) for determining a first spatial coordinate of the first workpiece in the spatial coordinate system of the robotic arm based on the pixel coordinates of the at least one first feature point and calibration information, wherein the calibration information is used to indicate a mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system; a grasping module (704), configured to generate control information according to the first spatial coordinates, and send the control information to the robotic arm, so that the robotic arm grasps the first workpiece according to the control information; The calibration information is determined by: Determining a calibration reference point on a second workpiece, wherein the second workpiece and the first workpiece have the same height in the Z-axis direction of the spatial coordinate system, and the plane containing the X-axis and the Y-axis in the spatial coordinate system is parallel to the stage; According to preset X-axis translation amounts and Y-axis translation amounts, control the robotic arm to perform N translations along the X-axis and / or Y-axis directions in the spatial coordinate system, and obtain a translation calibration image captured by the camera after each translation of the robotic arm, wherein the translation calibration image includes an image of the second workpiece located on the stage, and N is a positive integer greater than or equal to 3; According to a preset Z-axis rotation amount, control the robotic arm to rotate M times around the Z-axis in the spatial coordinate system, and obtain a rotation calibration image captured by the camera after each rotation of the robotic arm, wherein the rotation calibration image includes an image of the second workpiece located on the stage, and M is a positive integer greater than or equal to 3; The calibration information is determined according to the position offset of the image of the calibration reference point in different translation calibration images, the position offset of the image of the calibration reference point in different rotation calibration images, the X-axis translation and Y-axis translation corresponding to each translation calibration image, and the Z-axis rotation corresponding to each rotation calibration image.

11. An electronic device (800), comprising: A processor (801), a communication interface (802), a memory (803) and a communication bus (804), wherein the processor (801), the memory (803) and the communication interface (802) communicate with each other via the communication bus (804); The memory (803) is used to store at least one executable instruction, and the executable instruction enables the processor (801) to execute an operation corresponding to the workpiece grasping method according to any one of claims 1 to 9.

12. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the workpiece grasping method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the workpiece grasping method according to any one of claims 1 to 9.

Images