Method of positioning an item, gripping system, storage medium and program product

Abstract

This invention discloses a method for locating an item, a grasping system, a storage medium, and a program product. The method includes: acquiring a first image of an item being transported by a conveying device using a camera; determining a target robotic arm among various robotic arms for grasping the item; acquiring a target transformation matrix between the machine coordinate system of the target robotic arm and the image coordinate system of the camera; obtaining the first coordinates of the item in the image coordinate system based on the first image; and transforming the first coordinates according to the target transformation matrix to obtain the second coordinates of the item in the machine coordinate system. In this invention, the position of the item can be accurately located even when the robotic arm is not within the camera's field of view, enabling the robotic arm to accurately grasp the item.

Description

Technical Field

[0001] This invention relates to the field of object grasping technology, and more particularly to an object positioning method, grasping system, storage medium, and program product. Background Technology

[0002] In recent years, with the continuous improvement of industrial standards, the requirements for speed and precision have also become increasingly stringent. In order to improve cycle time and reduce production costs, the conveying device needs to be constantly running, and the robotic arm on the conveying device needs to follow the movement of the target object to grasp it during the conveying process.

[0003] However, because the robotic arm on the conveyor is far from the camera, its grasping range is outside the camera's field of view, making it impossible for the robotic arm to accurately locate the object and thus impossible for it to accurately grasp the object. Summary of the Invention

[0004] The main objective of this invention is to provide a method for locating an object, a gripping system, a storage medium, and a program product, which aims to solve the problem that robotic arms cannot accurately locate the position of an object.

[0005] To achieve the above objectives, the present invention provides a method for locating an item, applied to a gripping system. The gripping system includes a conveying device, on which a camera and several robotic arms are mounted. The method for locating the item includes the following steps:

[0006] Acquire a first image of the conveying device transporting the item captured by the camera, and determine the target robotic arm among the robotic arms to grasp the item;

[0007] Obtain the target transformation matrix between the machine coordinate system where the target robotic arm is located and the image coordinate system where the camera is located. The target transformation matrix is ​​obtained based on a first transformation matrix and a rotation matrix. The first transformation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system where the conveying device is located. The rotation matrix is ​​obtained based on the transformation relationship between the machine coordinate system and the first coordinate system.

[0008] The first coordinates of the item in the image coordinate system are obtained based on the first image, and the first coordinates are transformed according to the target transformation matrix to obtain the second coordinates of the item in the machine coordinate system.

[0009] In one embodiment, before the step of obtaining the target transformation matrix between the machine coordinate system where the target robotic arm is located and the image coordinate system of the camera, the method further includes:

[0010] The camera periodically captures a second image of the calibration plate, which is positioned above the conveying surface of the conveying device.

[0011] A first transformation matrix between the image coordinate system and the first coordinate system is determined based on the calibration data in each of the second images, wherein the calibration data includes the actual coordinates of the corner point of the calibration plate in the first coordinate system and the coordinates of the corner point in the image coordinate system where the camera is located;

[0012] Based on the first transformation matrix and the second image, the movement of each robotic arm is controlled to obtain the readings of the teach pendant of each robotic arm;

[0013] The rotation matrix corresponding to each robotic arm is determined based on the readings of each robotic arm and the coordinates of the first origin of the first coordinate system.

[0014] Based on the rotation matrix corresponding to each robotic arm, the first origin coordinates, and the second origin coordinates of the machine coordinate system, a target transformation matrix between the machine coordinate system of each robotic arm and the image coordinate system of the camera is determined.

[0015] In one embodiment, the step of determining the first transformation matrix between the image coordinate system and the first coordinate system based on the calibration data in each of the second images includes:

[0016] Multiple calibration data are obtained from each of the second images, and the target calibration data is determined from each of the calibration data.

[0017] Based on the target calibration data, determine the second transformation matrix to be determined between the first coordinate system and the image coordinate system;

[0018] The second transformation matrix is ​​used to transform each remaining calibration data to obtain the transformed coordinates of the corner point corresponding to each remaining calibration data in the first coordinate system, wherein the remaining calibration data is the calibration data other than the target calibration data;

[0019] Determine the error between the transformed coordinates and the actual coordinates of each corner point, and determine a target error among the errors, wherein the target error is the error that is less than a preset error;

[0020] When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the second transformation matrix is ​​used to determine the first transformation matrix.

[0021] In one embodiment, the step of determining the target error among the various errors includes:

[0022] When the ratio between the number of target errors and the number of errors is less than a set ratio, the target calibration data is redefined in each of the calibration data.

[0023] Return to the step of determining the second transformation matrix to be determined between the first coordinate system and the image coordinate system based on the respective target calibration data.

[0024] In one embodiment, the step of determining the first transformation matrix from the second transformation matrix when the ratio between the number of target errors and the number of errors is greater than or equal to a predetermined ratio includes:

[0025] When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the set ratio and the preset error are reduced.

[0026] Return to the step of determining the second transformation matrix to be determined between the first coordinate system and the image coordinate system based on each of the target calibration data;

[0027] When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the set ratio is less than a first threshold, and the preset error is less than a second threshold, or when the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio and the number of iterations of the set ratio reaches a set number, the second transformation matrix is ​​used to determine the first transformation matrix.

[0028] In one embodiment, the step of controlling the movement of each robotic arm to obtain the respective readings of the teach pendant of each robotic arm based on the first transformation matrix and the second image includes:

[0029] According to the first transformation matrix, the third coordinates of the target corner point on the calibration board in the adjacent second image are transformed into the fourth coordinates in the first coordinate system;

[0030] Control each robotic arm to move to the target corner point corresponding to each robotic arm, and obtain the first reading of the teach pendant of each robotic arm;

[0031] After the conveying device moves a first distance, the robotic arm is controlled to move to the target corner point corresponding to each robotic arm, so as to obtain the second reading of the teach pendant of each robotic arm and the second encoder value of the encoder.

[0032] The target distance is determined based on the fourth coordinates corresponding to each of the robotic arms and the first distance, and each robotic arm is controlled to move the target distance corresponding to the robotic arm in the vertical direction to obtain the third reading of the teach pendant of each robotic arm.

[0033] In one embodiment, the step of determining the target distance based on the respective third coordinates corresponding to each of the robotic arms and the first distance includes:

[0034] Obtain the first encoder value and the second encoder value of each robotic arm's encoder, wherein, when a first reading is obtained, the value of the robotic arm's encoder is read to obtain the first encoder, and when a second reading is obtained, the value of the robotic arm's encoder is read to obtain the second encoder;

[0035] The encoder scaling factor for each robotic arm is determined based on each of the third coordinates of each robotic arm, the first encoder value, and the second encoder value.

[0036] The target distance for each robotic arm is determined based on the encoder scaling factor and the first distance.

[0037] To achieve the above objectives, the present invention also provides a grasping system, comprising: a conveying device, a processor, and a memory, wherein the conveying device is equipped with a camera and several robotic arms;

[0038] The conveying device is used to convey items;

[0039] The camera is used to capture images of the items being transported by the conveying device;

[0040] The robotic arm is used to grasp items being transported on the conveying device;

[0041] The memory stores computer-executed instructions;

[0042] The processor executes computer execution instructions stored in the memory, causing the processor to perform the item positioning method as described above.

[0043] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the article positioning method described above.

[0044] To achieve the above objectives, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the item positioning method described above.

[0045] The present invention provides a method for locating items, a grasping system, a storage medium, and a program product. The method involves acquiring an image of an item being transported by a conveyor system captured by a camera, identifying a target robotic arm among various robotic arms for grasping the item, obtaining a target transformation matrix between the machine coordinate system of the target robotic arm and the image coordinate system of the camera, obtaining the first coordinate of the item in the image coordinate system based on the image, and then transforming the first coordinate using the target transformation matrix to obtain the second coordinate of the item in the machine coordinate system. In this invention, the target transformation matrix is ​​obtained based on a first transformation matrix and a rotation matrix. The first transformation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system of the conveyor system, and the rotation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system. That is, even when the robotic arm is not within the camera's field of view, the coordinate system of the conveyor system allows the robotic arm to accurately locate the item's position, thereby enabling the robotic arm to accurately grasp the item. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the structure of the article positioning method of the present invention;

[0047] Figure 2 This is a flowchart illustrating the first embodiment of the article positioning method of the present invention;

[0048] Figure 3 This is a flowchart illustrating the second embodiment of the article positioning method of the present invention;

[0049] Figure 4 This is a detailed flowchart of step S50 in the third embodiment of the article positioning method of the present invention;

[0050] Figure 5 This is a schematic diagram of the hardware structure of the grasping system of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention provides a method for locating an item, implemented through a grasping system.

[0053] Reference Figure 1The grasping system includes a camera 1, a conveyor device 3, and several robotic arms 4. The conveyor device 3 has a conveyor belt. The camera 1 is positioned at the top of the enclosed darkroom of the grasping device, allowing it to capture images including the front of the calibration plate 2. Each robotic arm 4 is mounted on the conveyor device 3. During conveying, the camera acquires images at 2-second intervals (intervals include, but are not limited to, 2 seconds). Corner points are extracted from adjacent images and processed to obtain a first transformation relationship between the camera coordinate system and the conveyor line coordinate system. The camera 1 then moves to the working area of ​​the conveyor device 3 and the robotic arms 4, calibrating the machine coordinate system of the robotic arms 4 against the coordinate system of the conveyor device 3 (the first coordinate system) to obtain a second transformation relationship. Based on the first and second transformation relationships, a third transformation relationship between the camera coordinate system and the machine coordinate system is finally obtained. This third transformation relationship can be represented by a target transformation matrix.

[0054] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0055] Reference Figure 2 , Figure 2 This is a first embodiment of the article positioning method of the present invention, which includes the following steps:

[0056] Step S10: Obtain the first image of the conveyor transporting the item captured by the camera, and determine the target robotic arm for grabbing the item among the various robotic arms.

[0057] In this embodiment, the executing entity is the grasping system; for ease of description, the grasping system will be referred to as "system" below. The system includes a conveying device, on which a camera and several robotic arms are mounted. When the conveying device transports items, the camera begins operation to capture images of the items being transported; these images are defined as the first image.

[0058] The conveying device is equipped with multiple robotic arms, each with its own machine coordinate system. The system needs to identify the robotic arm that will grasp the item, which is defined as the target robotic arm. The system then needs to locate the item to be grasped by the target robotic arm based on its machine coordinate system.

[0059] Step S20: Obtain the target transformation matrix between the machine coordinate system where the target robotic arm is located and the image coordinate system where the camera is located. The target transformation matrix is ​​obtained based on the first transformation matrix and the rotation matrix. The first transformation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system where the conveying device is located. The rotation matrix is ​​obtained based on the transformation relationship between the machine coordinate system and the first coordinate system.

[0060] In this embodiment, the system has calibrated the coordinate system for each robotic arm. Specifically, the system determines the transformation relationship between the machine coordinate system of each robotic arm and the image coordinate system of the camera. This transformation relationship can be represented by a transformation matrix, which is defined as the target transformation matrix. The system stores the target transformation matrix corresponding to each robotic arm. After determining the target robotic arm, the system obtains the target transformation matrix corresponding to that target robotic arm.

[0061] The target transformation matrix is ​​obtained based on the first transformation matrix and the rotation matrix. The first transformation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system in which the conveying device is located. The rotation matrix is ​​obtained based on the transformation relationship between the machine coordinate system and the first coordinate system. The first transformation matrix represents the transformation relationship between the image coordinate system and the first coordinate system in which the conveying device is located, while the rotation matrix represents the transformation relationship between the machine coordinate system and the first coordinate system. The first coordinate system is the coordinate system corresponding to the conveying device.

[0062] Step S30: Obtain the first coordinates of the item in the image coordinate system based on the first image, and transform the first coordinates according to the target transformation matrix to obtain the second coordinates of the item in the machine coordinate system.

[0063] After obtaining the target transformation matrix, the system can acquire the coordinates of the object in the image coordinate system based on the first image. These coordinates are defined as the first coordinates. The system can then transform the first coordinates using the target transformation matrix to obtain the second coordinates of the object in the machine coordinate system. The second coordinates represent the position of the object in the machine coordinate system.

[0064] In the technical solution provided in this embodiment, an image of an item being transported by a conveyor device is captured by a camera. A target robotic arm is identified among the various robotic arms that will grasp the item. A target transformation matrix is ​​obtained between the machine coordinate system of the target robotic arm and the image coordinate system of the camera. The first coordinate of the item in the image coordinate system is obtained based on the image. Then, the first coordinate is transformed using the target transformation matrix to obtain the second coordinate of the item in the machine coordinate system. In this invention, the target transformation matrix is ​​obtained based on the first transformation matrix and a rotation matrix. The first transformation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system of the conveyor device. The rotation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system. That is, even when the robotic arm is not within the camera's field of view, the coordinate system of the conveyor device allows the robotic arm to accurately locate the item, thereby enabling the robotic arm to accurately grasp the item.

[0065] Reference Figure 3 , Figure 3 This is a second embodiment of the article positioning method of the present invention. Based on the first embodiment, before step S20, it further includes:

[0066] Step S40: At regular intervals, the camera captures a second image of the calibration plate, which is positioned above the conveying surface of the conveying device.

[0067] In this embodiment, the system periodically acquires a second image of the calibration board captured by the camera. The time interval can be any composite number, for example, 2 seconds, meaning the camera acquires an image of the calibration board every 2 seconds, and this image is defined as the second image.

[0068] Step S50: Determine the first transformation matrix between the image coordinate system and the first coordinate system based on the calibration data in each second image. The calibration data includes the actual coordinates of the corner points of the calibration plate in the first coordinate system and the coordinates of the corner points in the image coordinate system where the camera is located.

[0069] The device can determine the transformation matrix between the image coordinate system and the first coordinate system based on the calibration data in the second image. This transformation matrix is ​​defined as the first transformation matrix. The calibration data includes the actual coordinates of the corner points of the calibration plate in the first coordinate system, and the coordinates of the corner points in the image coordinate system. Specifically, the first coordinate system in which the conveying device is located is established on the calibration plate, and the three-dimensional coordinates of each corner point on the calibration plate are defined as P = [X, Y, Z]. T Identify the corner points of the calibration board in the second image acquisition, and denote the image coordinates of the corner points of the calibration board as m = [u, v, 1]. T Then the first transformation matrix between the calibration plate plane and the image plane can be expressed as:

[0070]

[0071] The system can calculate the first transformation matrix H based on P and m. m In this embodiment, the calibration plate is placed above the conveying surface of the conveying device, so the plane containing the calibration plate is parallel to the plane containing the conveying surface of the conveying device. Since the distance between the calibration plate and the conveying device is small, the coordinate system containing the calibration plate can be considered as the first coordinate system containing the conveying device.

[0072] Step S60: Based on the first transformation matrix and the second image, control the movement of each robotic arm to obtain the readings of each robotic arm's teach pendant.

[0073] Based on a first transformation matrix and a second image, the system controls the movement of each robotic arm to obtain the readings of the teach pendant for each arm. Specifically, the system uses the first transformation matrix to convert the third coordinates of the target corner point on the calibration board in adjacent second images into fourth coordinates in the first coordinate system. For example, the system extracts the target corner point from two adjacent second images, obtaining image coordinates m1 and m2 of the target corner point, and then uses the first transformation matrix H... m Convert the second image into a bird's-eye view to obtain the fourth coordinates of the target corner point in the first coordinate system: P1 = [x1, y1, 1]. T And P2 = [x2, y2, 1] T .

[0074] The system controls each robotic arm to move to its corresponding target corner point to obtain the first reading of each robotic arm's teach pendant. For example, the target corner point for each robotic arm is defined as P. t Then control the end effector of each robotic arm to move to P. t At this point, the first reading P1 of each robotic arm is obtained by reading the value from the teach pendant of each robotic arm. R While reading the first reading, it is also necessary to read the encoder value of each robotic arm to obtain the first encoder value V1 of each robotic arm.

[0075] After the conveyor has moved the first distance, control the end effector of each robotic arm to move to the corresponding target corner point P. t At this point, the second reading P2 of each robotic arm is obtained by reading the value from the teach pendant of each robotic arm. R While reading the second reading, it is also necessary to read the encoder value of each robotic arm to obtain the second encoder value V2 for each robotic arm. The first distance can be any composite number.

[0076] The system determines the target distance based on the fourth coordinates and the first distance of each robotic arm, and controls the robotic arm to move vertically to the target distance. At this time, the system reads the value of the teach pendant of each robotic arm to obtain the third reading P3 of each robotic arm. R .

[0077] It should be noted that the target distance requires obtaining both the first encoder value and the second encoder value. Specifically, upon obtaining the first reading, the value of the robotic arm's encoder is read to obtain the first encoder value; upon obtaining the second reading, the value of the robotic arm's encoder is read to obtain the second encoder value. The system determines the encoder scaling factor for each robotic arm based on its respective third coordinates, the first encoder value, and the second encoder value. The encoder scaling factor F is calculated using the following formula:

[0078]

[0079] The system can determine the target distance for each robotic arm based on F and the first distance.

[0080] Step S70: Determine the rotation matrix corresponding to each robotic arm based on the readings of each robotic arm and the coordinates of the first origin of the first coordinate system.

[0081] After determining the various readings of the robotic arm, the system can determine the rotation matrix of the robotic arm based on these readings and the coordinates of the origin of the first coordinate system. Specifically, the origin O of the conveying device coordinate system (the first coordinate system) is set. s O, the origin of the robot coordinate system r ,but:

[0082]

[0083] The rotation matrix from the machine coordinate system to the first coordinate system is expressed as:

[0084]

[0085] Therefore, the rotation matrix corresponding to each robotic arm can be obtained.

[0086] Step S80: Based on the rotation matrix corresponding to each robotic arm, the first origin coordinates, and the second origin coordinates of the machine coordinate system, determine the target transformation matrix between the machine coordinate system where each robotic arm is located and the image coordinate system of the camera.

[0087] The system determines the target transformation matrix between the robot arm's machine coordinate system and the camera's image coordinate system based on the robot arm's rotation matrix, the first origin coordinates, and the second origin coordinates of the machine coordinate system. The target transformation matrix is:

[0088]

[0089] In the technical solution provided in this embodiment, the system accurately calibrates the transformation relationship between the coordinate system of each robotic arm and the camera coordinate system by using images acquired periodically by the camera from the calibration board. This enables one camera to calibrate the transformation relationship of multiple robotic arms simultaneously, reducing calibration costs.

[0090] In one embodiment, the distance between the calibration plate and the conveying device is relatively large, requiring a transformation relationship between the coordinate system of the calibration plate plane and the first coordinate system of the conveying device. This is achieved through matrix H. m The second image can be converted into a bird's-eye view with the calibration board plane as the reference plane, thereby obtaining more accurate point location information. The system then extracts corner points from two adjacent second images, obtaining the image coordinates m1 and m2 of the corner points, and uses H in the above formula. m Obtain the coordinates of the corresponding corner point in the bird's-eye view: P1 = [x1, y1, 1] T P2 = [x2, y2, 1] T Since P2 moves only 2 seconds relative to P1 along the direction of the conveyor, the relationship between points P1 and P2 is as follows:

[0091] P2 = R·P1 + T

[0092] Where R is the rotation matrix and T is the translation matrix from P1 to P2, a first coordinate system for the conveying device is established with the movement direction of the conveying device as the x-direction, and the angle between the calibration plate coordinate system and the first coordinate system is a vector. The angle between the coordinate system and the y-axis of the image is used to obtain the transformation relationship from the calibration plate coordinate system to the first coordinate system where the conveying device is located.

[0093] Reference Figure 4 , Figure 4 This is a third embodiment of the positioning method for the article of the present invention. Based on the second embodiment, step S50 includes:

[0094] Step S51: Obtain multiple calibration data based on each second image, and determine the target calibration data from each calibration data.

[0095] The system obtains multiple calibration data points from each second image and uses an adaptive RANSAC (Random Sample Consensus) algorithm to randomly select calibration data from these data points. The randomly selected calibration data is defined as the target calibration data. The number of target calibration data points, n, is at least 5. During the acquisition of target calibration data, the probability z = 0.02 that the calibration data is inaccurate.

[0096] Step S52: Based on the calibration data of each target, determine the second transformation matrix to be determined between the first coordinate system and the image coordinate system.

[0097] The system determines the transformation matrix between the first coordinate system and the image coordinate system based on the calibration data of each target. This transformation matrix is ​​the one that needs to be verified and is defined as the second transformation matrix to be determined. The calculation method of the second transformation matrix is ​​the same as that of the first transformation matrix, and will not be repeated here.

[0098] Step S53: The second transformation matrix is ​​used to transform each remaining calibration data to obtain the transformed coordinates of the corner point corresponding to each remaining calibration data in the first coordinate system. The remaining calibration data are calibration data other than the target calibration data.

[0099] The system uses a second transformation matrix to transform each remaining calibration data point, obtaining the transformed coordinates of the corner point in the first coordinate system for each remaining calibration data point. Remaining calibration data refers to calibration data other than the target calibration data. The calibration data includes the actual coordinates of the corner point in the first coordinate system and its coordinates in the image coordinate system. The system converts the image coordinates in the calibration data into transformed coordinates using the second transformation matrix.

[0100] Step S54: Determine the error between the transformed coordinates and the actual coordinates of each corner point, and determine the target error among each error. The target error is the error that is less than the preset error.

[0101] The system stores preset errors, which can be any composite number; for example, the preset error e = 0.01. The system determines the error between the transformed coordinates and the actual coordinates of a corner point, and identifies a target error from among these errors. The target error is the error that is less than the preset error.

[0102] Step S55: When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the second transformation matrix is ​​used to determine the first transformation matrix.

[0103] The system stores a set ratio, which is the in-point ratio w. The set ratio can be any composite number. For example, the set ratio w = 0.5. The system calculates the ratio between the number of target errors and the number of errors. If the ratio is greater than or equal to the set ratio, the second transformation matrix is ​​determined to be accurate. Therefore, the second transformation matrix can be used as the first transformation matrix.

[0104] If the ratio between the number of target errors and the number of errors is less than a set ratio, the second transformation matrix needs to be recalculated. At this time, the device uses the RANSAC algorithm to randomly reacquire the target calibration data from each calibration data. The system then returns to the step of determining the second transformation matrix to be determined between the first coordinate system and the image coordinate system based on each target calibration data. That is, steps S52-S54 are executed again until the accuracy of the second transformation matrix is ​​high or the number of iterations reaches the set number, at which point the iteration stops to obtain the accurate first transformation matrix.

[0105] In the technical solution provided in this embodiment, the system acquires multiple calibration data through each second image, then uses RANSAC sampling to extract n data point pairs to calculate the second transformation matrix, then uses the second transformation matrix to obtain the transformation matrix of the corner points to determine the transformation error, and finally accurately determines the first transformation matrix based on the transformation error.

[0106] In one embodiment, step S55 includes:

[0107] When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, reduce the set ratio and the preset error.

[0108] Return to the step of determining the second transformation matrix to be determined between the first coordinate system and the image coordinate system based on the calibration data of each target;

[0109] When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the set ratio is less than a first threshold, and the preset error is less than a second threshold, or when the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio and the number of iterations of the set ratio reaches a set number, the second transformation matrix is ​​determined to be the first transformation matrix.

[0110] In this embodiment, when the ratio between the number of target errors and the number of errors is greater than or equal to a set proportion, the second transformation matrix may be able to accurately perform coordinate transformation, but this possibility is small. Therefore, when the ratio between the number of target errors and the number of errors is greater than or equal to the set proportion, the set proportion and the preset error are reduced, thereby performing iterative calculation again, i.e., returning to steps S52-S54. Each time the ratio between the number of target errors and the number of errors is greater than or equal to the set proportion, e and w are reduced.

[0111] When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the set ratio is less than a first threshold, and the preset error is less than a second threshold, or when the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio and the number of iterations of the set ratio reaches a set number, the current second transformation matrix is ​​determined as the first transformation matrix.

[0112] Figure 5 This is a hardware structure diagram of a grasping system according to an exemplary embodiment.

[0113] The grasping system 500 may include: a processor 501, such as a CPU; a memory 502; a transceiver 503; a conveying device 3; a camera 1; and a robotic arm 4 (the grasping device is equipped with several robotic arms). Figure 5 (Only one robotic arm is identified in the text). Those skilled in the art will understand that... Figure 5 The structure shown does not constitute a limitation on the grasping system and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Conveying device 3 is used to transport items; camera 1 is used to capture images of the items transported by the conveying device; robotic arm 4 is used to grasp the items transported on the conveying device.

[0114] The processor 501 can call the computer program stored in the memory 502 to complete all or part of the steps of the above-described method for locating the item.

[0115] Transceiver 503 is used to receive information sent by external devices and to send information to external devices.

[0116] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a grasping system, enable the grasping system to perform the aforementioned method for locating an item.

[0117] A computer program product includes a computer program that, when executed by a processor of a grasping system, enables the grasping system to perform the method for locating the aforementioned item.

[0118] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0119] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0121] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for locating an item, characterized in that, An application is made in a grasping system, the grasping system including a conveying device, a camera and several robotic arms mounted on the conveying device, the method for positioning the item includes the following steps: Acquire a first image of the conveying device transporting the item captured by the camera, and determine the target robotic arm among the robotic arms to grasp the item; Obtain the target transformation matrix between the machine coordinate system where the target robotic arm is located and the image coordinate system where the camera is located. The target transformation matrix is ​​obtained based on a first transformation matrix and a rotation matrix. The first transformation matrix is ​​obtained based on the transformation relationship between the image coordinate system and the first coordinate system where the conveying device is located. The rotation matrix is ​​obtained based on the transformation relationship between the machine coordinate system and the first coordinate system. The first coordinates of the item in the image coordinate system are obtained based on the first image, and the first coordinates are transformed according to the target transformation matrix to obtain the second coordinates of the item in the machine coordinate system; Before the step of obtaining the target transformation matrix between the machine coordinate system where the target robotic arm is located and the image coordinate system where the camera is located, the method further includes: The camera periodically captures second images of the calibration board, and determines a first transformation matrix between the image coordinate system and the first coordinate system based on the calibration data in each of the second images. The step of determining the first transformation matrix between the image coordinate system and the first coordinate system based on the calibration data in each of the second images includes: Multiple calibration data are obtained from each of the second images, and target calibration data is determined from each of the calibration data. Based on the target calibration data, determine the second transformation matrix to be determined between the first coordinate system and the image coordinate system; The second transformation matrix is ​​used to transform each remaining calibration data to obtain the transformed coordinates of the corner point corresponding to each remaining calibration data in the first coordinate system, wherein the remaining calibration data is the calibration data other than the target calibration data; Determine the error between the transformed coordinates and the actual coordinates of each corner point, and determine a target error among the errors, wherein the target error is the error that is less than a preset error; When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the second transformation matrix is ​​used to determine the first transformation matrix.

2. The method for locating an item according to claim 1, characterized in that, Before the step of obtaining the target transformation matrix between the machine coordinate system where the target robotic arm is located and the image coordinate system where the camera is located, the method further includes: Based on the first transformation matrix and the second image, the movement of each robotic arm is controlled to obtain the readings of the teach pendant of each robotic arm; The rotation matrix corresponding to each robotic arm is determined based on the readings of each robotic arm and the coordinates of the first origin of the first coordinate system. Based on the rotation matrix corresponding to each robotic arm, the first origin coordinates, and the second origin coordinates of the machine coordinate system, a target transformation matrix between the machine coordinate system of each robotic arm and the image coordinate system of the camera is determined; wherein, the calibration plate is disposed above the conveying surface of the conveying device; the calibration data includes the actual coordinates of the corner point of the calibration plate in the first coordinate system and the coordinates of the corner point in the image coordinate system of the camera.

3. The method for locating an item according to claim 1, characterized in that, The step of determining the target error among each of the errors includes: When the ratio between the number of target errors and the number of errors is less than a set ratio, the target calibration data is redefined in each of the calibration data. Return to the step of determining the second transformation matrix to be determined between the first coordinate system and the image coordinate system based on the respective target calibration data.

4. The method for locating an article according to claim 1, characterized in that, The step of determining the first transformation matrix from the second transformation matrix when the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio includes: When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the set ratio and the preset error are reduced. Return to the step of determining the second transformation matrix to be determined between the first coordinate system and the image coordinate system based on each of the target calibration data; When the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio, the set ratio is less than a first threshold, and the preset error is less than a second threshold, or when the ratio between the number of target errors and the number of errors is greater than or equal to a set ratio and the number of iterations of the set ratio reaches a set number, the second transformation matrix is ​​used to determine the first transformation matrix.

5. The method for locating an article according to claim 2, characterized in that, The step of controlling the movement of each robotic arm to obtain the respective readings of the teach pendant of each robotic arm based on the first transformation matrix and the second image includes: According to the first transformation matrix, the third coordinates of the target corner point on the calibration board in the adjacent second image are transformed into the fourth coordinates in the first coordinate system; Control each robotic arm to move to the target corner point corresponding to each robotic arm, and obtain the first reading of the teach pendant of each robotic arm; After the conveying device moves a first distance, the robotic arm is controlled to move to the target corner point corresponding to each robotic arm, so as to obtain the second reading of the teach pendant and the second encoder value of the encoder for each robotic arm. The target distance is determined based on the fourth coordinates corresponding to each of the robotic arms and the first distance, and each robotic arm is controlled to move the target distance corresponding to the robotic arm in the vertical direction to obtain the third reading of the teach pendant of each robotic arm.

6. The method for locating an article according to claim 5, characterized in that, The step of determining the target distance based on the third coordinates corresponding to each of the robotic arms and the first distance includes: Obtain the first encoder value and the second encoder value of each robotic arm's encoder, wherein, when a first reading is obtained, the value of the robotic arm's encoder is read to obtain the first encoder, and when a second reading is obtained, the value of the robotic arm's encoder is read to obtain the second encoder; The encoder scaling factor for each robotic arm is determined based on each of the third coordinates of each robotic arm, the first encoder value, and the second encoder value. The target distance for each robotic arm is determined based on the encoder scaling factor and the first distance.

7. A grasping system, characterized in that, include: The system includes a conveying device, a processor, and a memory, wherein the conveying device is equipped with a camera and several robotic arms. The conveying device is used to convey items; The camera is used to capture images of the items being transported by the conveying device; The robotic arm is used to grasp items being transported on the conveying device; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the article positioning method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the article positioning method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for locating the article as described in any one of claims 1 to 6.

Images