Manipulator-based material taking method, manipulator material taking system, device and medium

Abstract

This invention discloses a material handling method, system, equipment, and medium based on a robotic arm. The method includes: acquiring an image of the product to be handled by an image acquisition device; determining the product center point based on the image; determining the product offset based on the product center point, the product template center point, and the pixel equivalent; and sending the product offset to the robotic arm so that the robotic arm can handle the product according to the product offset and a predetermined handling position. The product template center point and pixel equivalent are determined after the robotic arm performs tool coordinate calibration. Tool coordinate calibration includes aligning the robotic arm's suction cup rotation center, the image acquisition field of view center, and the product template center. This solves the problems of complex offset calculation and cumbersome calibration processes, eliminating the need for center rotation to calculate the offset. The calibration process is simple and easy to operate, improving the efficiency of the robotic arm's material handling.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, and in particular to a material handling method, a material handling system, equipment and medium based on a robotic arm. Background Technology

[0002] Visual positioning typically involves pre-setting a template with coordinates that are based on calibrated mechanical coordinates. The pixel coordinates of the product are acquired using a visual imaging device (e.g., a CCD industrial camera), converted to mechanical coordinates, and then the mechanical coordinates of the template are subtracted from the product's mechanical coordinates to obtain the mechanical coordinate offset. This offset is sent to the robot arm, which adjusts its trajectory based on the offset for precise material handling. Traditional visual algorithms suffer from complex pixel-to-mechanical coordinate calibration, which in turn affects the offset calculation, leading to further complexity. Furthermore, changing the visual imaging device or motion control module necessitates repeating this complex calibration process, resulting in low efficiency for the robot arm's material handling process. Summary of the Invention

[0003] This invention provides a material handling method, a material handling system, equipment, and medium based on a robotic arm to solve the problems of complex offset calculation and cumbersome calibration process.

[0004] According to one aspect of the present invention, a material handling method based on a robotic arm is provided, applied to a robotic arm material handling system. The robotic arm system includes a robotic arm suction cup, a robotic arm, an image acquisition device, and an image processing device. The robotic arm suction cup is rotatably disposed at the free end of the robotic arm and is used to adsorb and acquire the product to be picked up. The image acquisition device is used to acquire an image of the product to be picked up. The method is executed by the image processing device and includes:

[0005] The image of the product to be retrieved is acquired by the image acquisition device, and the product to be retrieved in the image is placed in the suction cup of the robotic arm;

[0006] Determine the center point of the product based on the image of the product to be retrieved;

[0007] The product offset is determined based on the product center point, the product template center point, and the pixel equivalent. The product offset is then sent to the robot arm so that the robot arm can pick up the product to be picked up based on the product offset and the predetermined picking position.

[0008] The product template center point and pixel equivalent are determined after the robot arm performs tool coordinate calibration. The tool coordinate calibration includes aligning the robot arm's suction cup rotation center, the image acquisition field of view center, and the product template center.

[0009] According to another aspect of the present invention, a robotic arm material handling system is provided, comprising a robotic arm suction cup, a robotic arm, an image acquisition device, and an image processing device. The robotic arm suction cup is rotatably disposed at the free end of the robotic arm and is used to adsorb and acquire a product to be picked up. The image acquisition device is used to acquire an image of the product to be picked up. The image processing device includes:

[0010] The image acquisition module is used to acquire the image of the product to be acquired by the image acquisition device, wherein the product to be acquired in the image is placed in the suction cup of the robotic arm;

[0011] The product center point determination module is used to determine the product center point based on the image of the product to be retrieved.

[0012] The offset determination module is used to determine the product offset based on the product center point, the product template center point, and the pixel equivalent, and send the product offset to the robot arm so that the robot arm can pick up the product to be picked up according to the product offset and the predetermined picking position.

[0013] The product template center point and pixel equivalent are determined after the robot arm performs tool coordinate calibration. The tool coordinate calibration includes aligning the robot arm's suction cup rotation center, the image acquisition field of view center, and the product template center.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the robotic arm-based material handling method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the robotic arm-based material handling method according to any embodiment of the present invention.

[0019] The technical solution of this invention involves acquiring an image of the product to be retrieved by an image acquisition device, placing the product in the suction cup of a robotic arm; determining the product's center point based on the image; determining the product offset based on the product's center point, the product template's center point, and the pixel equivalent; and sending the product offset to the robotic arm so that the robotic arm can retrieve the product according to the product offset and a predetermined picking position. The product template's center point and pixel equivalent are determined after the robotic arm undergoes tool coordinate calibration. This tool coordinate calibration involves aligning the robotic arm's suction cup rotation center, the image acquisition field of view center, and the product template center. This solves the problems of complex offset calculation and cumbersome calibration processes. Pre-calibrating the robotic arm's tool coordinates, including aligning the robotic arm's suction cup rotation center, the image acquisition field of view center, and the product template center, ensures that the rotation centers coincide. Therefore, the rotation center does not need to be involved in the offset calculation, making the calibration process simple and easy to operate. After calibrating the robotic arm's tool coordinates, the product template's center point and pixel equivalent are determined. When the robotic arm picks up materials, an image acquisition device captures an image of the product to be picked up, which is placed in the robotic arm's suction cup. The center point of the product is determined by processing the image. Then, the product offset is determined based on the product center point, the product template center point, and the pixel equivalent, and sent to the robotic arm. The robotic arm picks up the product to be picked up according to the product offset and the predetermined picking position. This application only requires the product center point, the product template center point, and the pixel equivalent to calculate the product offset. The calculation process is simple and improves the working efficiency of the robotic arm in picking up materials.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a material handling method based on a robotic arm according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of a material handling method based on a robotic arm according to Embodiment 2 of the present invention;

[0024] Figure 3This is a schematic diagram of a robotic arm material handling system according to Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the material handling method based on a robotic arm according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1

[0029] Figure 1 This document provides a flowchart of a material handling method based on a robotic arm, as described in Embodiment 1 of the present invention. This embodiment is applicable to situations where materials are automatically handled by a robotic arm. The method can be executed by an image processing device within the robotic arm material handling system. This image processing device can be implemented in hardware and / or software and can be configured in an electronic device. The robotic arm system also includes a robotic arm suction cup, a robotic arm, and an image acquisition device. The robotic arm suction cup is rotatably mounted at the free end of the robotic arm and is used to adsorb and acquire the product to be picked up. The image acquisition device is used to acquire an image of the product to be picked up. Figure 1 As shown, the method includes:

[0030] S101. Acquire the image of the product to be acquired by the image acquisition device.

[0031] In this embodiment, the image acquisition device can be a camera, video recorder, etc.; the image of the product to be picked up can be specifically understood as an image containing the product to be picked up; the product to be picked up can be specifically understood as a product that needs to be picked up by a robotic arm. For example, in the photovoltaic industry, equipment such as electroplating equipment, exposure equipment, and etching equipment all need to achieve precise loading and unloading of silicon wafers, and all need to use a robotic arm to pick up the silicon wafers and transfer them between two set points.

[0032] Specifically, the image acquisition device is positioned above the robotic arm's picking position. Products to be picked can be automatically transferred to the robotic arm's picking position via a production line or other automated methods, or placed there manually. The image acquisition device can activate at a certain frequency, or upon detecting a new product being placed at the robotic arm's picking position, to capture an image of the product. An image processing device is communicatively connected to the image acquisition device and receives the images of the products to be picked transmitted from the image acquisition device.

[0033] S102. Determine the center point of the product based on the image of the product to be picked up.

[0034] In this embodiment, the product center point can be specifically understood as the location or coordinates of the center point of the product to be retrieved. The image of the product to be retrieved is processed and analyzed to determine its position within the image. Further analysis and calculation of this position determine the product center point. For example, if the product to be retrieved is a quadrilateral, the center points of each side are determined, and the product center point is determined based on the intersection of the lines connecting these center points. Alternatively, the intersection of the lines connecting the vertices of the product to be retrieved is determined, and this intersection is used as the product center point.

[0035] S103. Determine the product offset based on the product center point, the product template center point, and the pixel equivalent. Send the product offset to the robot arm so that the robot arm can pick up the product to be picked up according to the product offset and the predetermined picking position.

[0036] The center point and pixel equivalent of the product template are determined after the robot arm performs tool coordinate calibration. Tool coordinate calibration includes aligning the rotation center of the robot arm's suction cup, the center of the image acquisition field of view, and the center of the product template.

[0037] In this embodiment, the product template center point can be specifically understood as the center point of the product used as the reference template for the product to be taken during the calibration process; the pixel equivalent can be specifically understood as the proportional relationship between the pixel value and the actual size, that is, the actual size corresponding to one pixel value; the product offset can be specifically understood as the value of the offset of the product to be taken relative to the product template.

[0038] In this embodiment, the rotation center of the robotic arm suction cup is the center of the robotic arm suction cup, the image acquisition field of view center is the center of the image acquisition device, and the product template center is the center of the product template. The rotation center of the robotic arm suction cup, the image acquisition field of view center, and the product template center can be determined accordingly after the hardware is determined.

[0039] Specifically, the tool coordinates of the robotic arm are pre-calibrated. In this embodiment, the calibration includes aligning the rotation center of the robotic arm's suction cup, the center of the image acquisition field of view, and the center of the product template. Marks, such as needles, can be set on the rotation center of the robotic arm's suction cup and the center of the product template. The center of the image acquisition field of view can be marked by projection when the image acquisition device takes a picture. By aligning the rotation center of the robotic arm's suction cup and the center of the product template, and aligning the center of the image acquisition field of view with the above two centers, the three points coincide, completing the tool coordinate calibration of the robotic arm and fixing the position of the image acquisition device. The position of the image acquisition device will not change during subsequent material handling.

[0040] For example, this application provides an implementation method for calibrating the working coordinates of a robotic arm, including the following steps:

[0041] 1. Install the product template onto the robotic arm suction cup, aligning the center of the product template with the center of the robotic arm suction cup until they coincide. Then, move the center of the product template and the center of the robotic arm suction cup to the center of the image acquisition field of view. Mark the center of the product template and the calibration point on the product template.

[0042] 2. Control the robotic arm suction cup and the product template on the robotic arm suction cup to rotate at 3 angles, collect images of the product template at the 3 angles, and obtain the coordinates of the calibration point in the 3 images respectively.

[0043] 3. Obtain the fitted rotation center based on the coordinates of the calibration points in the three images. The error between the fitted rotation center and the product template center is controlled below the set threshold.

[0044] When the rotation center of the control robot's suction cup, the center of the image acquisition field of view, and the center of the product template coincide, the tool coordinate calibration is completed, and the rotation center is determined. In the subsequent work of the robot, the rotation center does not participate in the offset calculation.

[0045] As described above, the product offset can be calculated based on the product center point, the product template center point, and the pixel equivalent. The calculation formula or algorithm for the product offset can be predetermined. After determining the product center point, the product template center point and pixel equivalent determined during calibration are used in the calculation formula or algorithm to obtain the product offset. The product offset is then sent to the robotic arm. Upon receiving the product offset, the robotic arm compensates for the predetermined material picking position based on the offset. The compensated product is then picked up using the robotic arm's suction cup and moved to the corresponding position.

[0046] This invention provides a robotic arm-based material handling method that solves the problems of complex offset calculation and cumbersome calibration processes. The robotic arm is pre-calibrated using tool coordinates, which involves aligning the rotation center of the robotic arm's suction cup, the center of the image acquisition field of view, and the center of the product template to ensure coincidence of the rotation centers. Therefore, the rotation center is not required in the offset calculation, making the calibration process simple and easy to operate. After tool coordinate calibration, the center point and pixel equivalent of the product template are determined. During material handling, an image of the product to be picked up is acquired using an image acquisition device. The image is processed to determine the product's center point. Then, based on the product's center point, the product template's center point, and the pixel equivalent, the product offset is determined and sent to the robotic arm. The robotic arm picks up the product according to the product offset and the pre-determined picking position. This application only requires the product's center point, the product template's center point, and the pixel equivalent to calculate the product offset, simplifying the calculation process and improving the efficiency of the robotic arm's material handling.

[0047] Example 2

[0048] Figure 2 This is a flowchart of a material handling method based on a robotic arm, provided in Embodiment 2 of the present invention. This embodiment is a refinement of the above embodiments. Figure 2 As shown, the method includes:

[0049] S201. After the robot arm performs tool coordinate calibration, it acquires the product template image captured by the image acquisition device. The product template image includes the product template.

[0050] In this embodiment, the product template is used as a reference template for the product to be taken during the calibration process; the product template image can be specifically understood as an image containing the product template.

[0051] Specifically, after calibrating the robot's tool coordinates, the center point of the product template is determined. The image acquisition device is already positioned accordingly. The product template is placed in the robot's suction cup, and the image acquisition device is controlled to capture an image of the product template. The image processing device communicates with the image acquisition device and receives the product template image transmitted by the image acquisition device.

[0052] S202. Determine the midpoint of each side of the product template based on the product template image.

[0053] The product template image is identified and analyzed to determine the location of the product template in the image. Each edge of the product template is identified, and the center point of each edge is determined. The center point of each edge can be calculated based on the coordinates of the pixels that make up each edge. For example, for each edge, the average coordinates of each pixel that makes up the edge are calculated to obtain the center point of the edge, or the coordinates of the pixels at the two endpoints of the edge are calculated to obtain the center point of the edge.

[0054] S203. Determine the center point of the product template based on the intersection of the lines connecting the midpoints.

[0055] Connect all center points with lines to determine the intersection points of these lines, and use these intersection points as the center points of the product template. Since the lines connecting the center points may or may not intersect, this application can choose to connect two diagonally opposite center points when connecting the center points. In this case, for a quadrilateral, only two lines are needed to determine the center point of the product template. Alternatively, connect all center points and then determine the number of intersection points. If there is only one intersection point, it is directly determined as the center point of the product template. If there are multiple intersection points, they are filtered to obtain the center point of the product template.

[0056] It should be noted that in this application embodiment, determining the center point of the product template and determining the center point of the product can be done in the same way.

[0057] S204. Control the robotic arm to perform nine-point calibration to obtain pixel equivalents.

[0058] After calibrating the tool coordinates of the robotic arm, the pixel equivalent is also determined. This embodiment uses a nine-point calibration method to determine the pixel equivalent. The robotic arm carries the product to the center of the image acquisition device's field of view, automatically obtaining nine positions in all directions. These nine positions are then used to perform an affine transformation to fit a circle, yielding pixel coordinates. These pixel coordinates are then combined with the machine coordinates for nine-point calibration.

[0059] Optionally, the robotic arm system also includes a photomask and a laser emitting device. The photomask includes light-transmitting holes and is positioned between the laser emitting device and the product template. Correspondingly, the robotic arm is controlled to perform nine-point calibration, including:

[0060] A. Control the laser emitting device to emit laser light so that the laser light passes through the light-transmitting hole on the mask and is projected onto the product template.

[0061] In this embodiment, the laser emitting device can be specifically understood as a device capable of emitting laser light. The shape of the light-transmitting hole can be set according to actual needs, and can be any shape such as circle, square, rectangle, or rhombus. The position of the light-transmitting hole can be set in the middle of the mask plate or any other position. After the mask plate is placed between the laser emitting device and the product template, the laser emitting device is controlled to emit laser light. At this time, the laser light can be projected onto the product template through the light-transmitting hole in the mask plate, forming a laser dot on the product template.

[0062] B. Control the product template to move to the movable area, and control the product template to move in the forward and backward and left and right directions within the movable area to perform nine-point laser printing, and then move the laser-printed product template to the field of view of the image acquisition device for nine-point calibration.

[0063] In this embodiment, the movable area can be specifically understood as the area within which the product template can move forward, backward, left, and right. The movable area can be predetermined based on the size and position of the hardware. The product template is first moved within this movable area. This can be done automatically by directly inputting control parameters or commands, or by responding to manual user control. Within this movable area, the product template moves forward and backward and left and right, allowing the laser to be projected onto the template through the light-transmitting holes in the mask, thus performing laser printing. Since this application uses nine-point calibration, nine points need to be printed, requiring three coordinates each for the forward / backward and left / right directions, resulting in nine laser points.

[0064] This application embodiment can also perform rotational calibration using a mask with light-transmitting holes. This application requires multiple light-transmitting holes for rotational calibration, such as four. The positions of the light-transmitting holes can be set according to requirements; for example, the mask can be square or rectangular, and the holes can be positioned near the four vertices of the mask. The product template is controlled to move to a movable area and rotated within that area for laser printing. The laser-printed product template is then moved into the field of view of the image acquisition device for rotational calibration.

[0065] In existing technologies, when the image acquisition device is stationary for calibration, the calibration object is moved left, right, forward, and backward on a mobile platform to complete the calibration of nine points, and the mobile platform is rotated to complete the rotational calibration. However, in existing technologies, the platform can only move left and right within the field of view, and cannot achieve the traditional nine-point calibration through forward and backward movement, nor can it achieve the traditional rotational calibration through rotation. This invention allows the mobile platform to be moved outside the camera (i.e., within the movable area) and moved left, right, forward, and backward. A laser is then used to print nine corresponding points on the product template through the light-transmitting holes (e.g., the central circular hole) of a custom-made mask. The product template is then moved into the camera's field of view for nine-point calibration. Alternatively, the platform can be moved outside the camera and rotated, and a laser is used to print twelve corresponding points on the product template through the light-transmitting holes (e.g., the four corner circular holes) of the mask. The product template is then moved into the field of view of the image acquisition device for rotational calibration. This application achieves nine-point calibration and visual guidance positioning through a mask with light-transmitting holes, and can also perform rotational calibration simultaneously. This application embodiment achieves nine-point calibration and rotation calibration by laser printing a product template outside the image acquisition device, solving the problem in the prior art that the platform can only move left and right under the field of view, and cannot achieve traditional nine-point calibration by moving forward and backward, nor can it achieve traditional rotation calibration by rotating the angle.

[0066] S205. Acquire an image of the product to be retrieved by the image acquisition device, and place the product to be retrieved in the robotic arm suction cup in the image of the product to be retrieved.

[0067] S206. Determine the center point of the product based on the image of the product to be picked up.

[0068] S207. Determine the pixel difference based on the pixel coordinates of the product center point and the pixel coordinates of the product template center point.

[0069] During the automatic material handling process, after determining the center point of the product to be picked up, the pixel coordinates of the product center point are determined accordingly. A pixel difference is calculated based on the pixel coordinates of the product center point and the pixel coordinates of the product template center point. In this embodiment, the pixel difference can be the difference between the pixel coordinates of the product center point and the pixel coordinates of the product template center point, or vice versa. Accordingly, the method of calculating the pixel coordinates determines how the robotic arm performs material handling compensation. In this embodiment, the pixel difference can be a difference in horizontal coordinates, a difference in vertical coordinates, or the distance and angle between two points.

[0070] S208. Determine the product offset based on the product of the pixel difference and the pixel equivalent.

[0071] Taking the pixel difference as the difference between the horizontal and vertical coordinates as an example, the difference between the horizontal and vertical coordinates is multiplied by the pixel equivalent to obtain the offset in the horizontal and vertical directions, and the offset in the horizontal and vertical directions is used as the product offset.

[0072] S209. Send the product offset to the robot arm so that the robot arm can pick up the product to be picked up according to the product offset and the predetermined picking position.

[0073] Optionally, the material pick-up location is taught and determined relative to the center point of the product template.

[0074] This application pre-teachs the material picking position during the calibration process. After determining the center of the product template, the material picking position is taught to the center of the product template. For example, the material picking trajectory is determined based on the coordinates of the material picking position and the coordinates of the product template center. When picking up the material, the robot first moves to the material picking position, then moves to the center of the product template according to the material picking trajectory, and then moves to the position of the product to be picked up according to the offset; or, the material picking trajectory is re-planned based on the product template center, the offset, and the material picking position, and the product to be picked up is then picked up.

[0075] It should be noted that if the product template is changed in this application embodiment, it can be directly compared with the original product template without the need to teach the material picking position again, making the operation simple. Furthermore, the teaching of the material picking position in this application is simple, requiring only the material picking position.

[0076] Optionally, before acquiring the image of the product to be acquired by the image acquisition device or the product template image acquired by the image acquisition device, the method further includes: performing distortion correction on the image acquisition device.

[0077] Image acquisition devices can exhibit distortion; therefore, this application requires distortion correction for the image acquisition device. Distortion correction can be performed during calibration, specifically before the image acquisition device acquires the product template image. In this case, the distortion correction is already completed when the image acquisition device acquires the image of the product to be acquired. Alternatively, if image acquisition is not required during calibration, then it is only necessary to ensure that correction is performed before the image acquisition device acquires the image of the product to be acquired.

[0078] The image acquisition device acquires multiple pose images. A software distortion correction assistant calibrates the intrinsic and extrinsic parameters of the image acquisition device. Intrinsic parameters are those related to the camera's own characteristics, such as focal length and pixel size. Extrinsic parameters are those in the world coordinate system, such as the camera's position and rotation direction. After obtaining the calibrated intrinsic and extrinsic parameters, they can be evaluated. The image acquisition device, after distortion correction, automatically performs distortion correction during image acquisition, resulting in distortion-free images.

[0079] For example, this application provides an implementation process for distortion correction, taking distortion correction using Halcon software as an example:

[0080] 1. Prepare a Halcon-specific calibration plate (the size of the calibration plate pattern can be (1 / 4-3 / 4) times the field of view);

[0081] 2. Adjust the position, aperture, focal length, and other parameters of the image acquisition device to the correct position (if the above parameters change, the calibration will fail and distortion correction must be performed again).

[0082] 3. Place the calibration board in the center of the field of view of the image acquisition device, ensuring the calibration board is upright. Take one image as the pose image, which will be used as the main pose. Then, ensure the calibration board remains within the field of view and does not move it outside the field of view. Take 15-25 images from different positions for distortion correction.

[0083] 4. Use the Halcon software distortion correction assistant to calibrate the camera's intrinsic and extrinsic parameters. This involves processing the pose image corresponding to the main pose and images from different positions to obtain a map image for distortion correction.

[0084] 5. During application, after the image acquisition device captures a picture, it performs distortion correction on the captured picture through map image.

[0085] This invention provides a robotic arm-based material handling method that solves the problems of complex offset calculation and cumbersome calibration. The method pre-calibrates the robotic arm to ensure the rotation center of the suction cup, the center of the image acquisition field of view, and the center of the product template coincide, thus avoiding the rotation center's involvement in offset calculation. The calibration process is simple and easy to operate. After calibrating the robotic arm's tool coordinates, the center point and pixel equivalent of the product template are determined, and the offset is calculated based on these. When the robotic arm handles material handling, the product center point is determined from the image of the product to be handled. Then, the product offset is determined based on the product center point, the product template center point, and the pixel equivalent and sent to the robotic arm. The robotic arm then handles the product according to the product offset and the pre-determined handling position. This application only requires the product center point, the product template center point, and the pixel equivalent to calculate the product offset, simplifying the calculation process and allowing for rapid acquisition of the product offset. This improves the efficiency of robotic arm material handling, simplifies personnel setup and maintenance, and reduces costs.

[0086] It should be noted that Embodiments 1 and 2 of this application describe placing an image acquisition device at the robotic arm's material-picking position. This can resolve positional errors of the product to be picked up at the picking position, and the robotic arm corrects these errors. Before the robotic arm reaches the picking position, the image acquisition device pre-completes the image acquisition process, and the image processing device pre-completes the processing process. During the robotic arm's movement, this pre-compensation action allows the robotic arm to immediately pick up the product upon reaching the picking position, improving its efficiency. However, during the robotic arm's movement from the picking position to the unloading position along a predetermined path, positional errors may still occur in the product. To correct these errors during movement, at least one image acquisition device can be placed along the predetermined path of the robotic arm to acquire the positional errors of the product along the predetermined path and correct them using the methods described in Embodiments 1 and / or 2. To avoid affecting the robotic arm's efficiency, the image acquisition device along the predetermined path can acquire images of the products at intervals, for example, once every 100 products.

[0087] Example 3

[0088] Figure 3 This is a schematic diagram of a robotic arm material handling system provided in Embodiment 3 of the present invention. Figure 3 As shown, the system includes: a robotic suction cup 31, a robotic arm 32, an image acquisition device 33, and an image processing device 34. The robotic suction cup 31 is rotatably mounted on the free end of the robotic arm 32. The robotic suction cup 31 is used to adsorb and acquire the product to be picked up, and the image acquisition device 33 is used to acquire the image of the product to be picked up.

[0089] The image processing device 34 includes:

[0090] The image acquisition module is used to acquire the image of the product to be acquired by the image acquisition device 33;

[0091] The product center point determination module is used to determine the product center point based on the image of the product to be retrieved.

[0092] The offset determination module is used to determine the product offset based on the product center point, the product template center point, and the pixel equivalent, and send the product offset to the robot arm 32 so that the robot arm 32 can pick up the product to be picked up according to the product offset and the predetermined picking position.

[0093] The product template center point and pixel equivalent are determined after the robot arm performs tool coordinate calibration. The tool coordinate calibration includes aligning the robot arm's suction cup rotation center, the image acquisition field of view center, and the product template center.

[0094] In this embodiment, the image acquisition device 33 can be positioned above the material handling position of the robotic arm. The robotic arm suction cup 31 is used to adsorb the product to be picked up. After acquiring the image of the product to be picked up, the image acquisition device 33 sends it to the image processing device 34. After determining the product offset, the image processing device 34 sends the product offset to the robotic arm 32. The robotic arm 32 picks up the product to be picked up according to the product offset and the predetermined material handling position.

[0095] This invention provides a robotic arm material handling system that solves the problems of complex offset calculation and cumbersome calibration processes. The robotic arm is pre-calibrated using tool coordinates, which involves aligning the rotation center of the robotic arm's suction cup, the center of the image acquisition field of view, and the center of the product template to ensure these centers coincide. Therefore, the rotation center is not required in the offset calculation, making the calibration process simple and easy to operate. After tool coordinate calibration, the center point and pixel equivalent of the product template are determined. During material handling, an image of the product to be picked up is acquired using an image acquisition device. The image is processed to determine the product's center point. Then, based on the product center point, the product template center point, and the pixel equivalent, the product offset is determined and sent to the robotic arm. The robotic arm picks up the product according to the product offset and the pre-determined picking position. This application only requires the product center point, the product template center point, and the pixel equivalent to calculate the product offset, simplifying the calculation process and improving the efficiency of the robotic arm's material handling.

[0096] Optionally, the image processing device 34 further includes:

[0097] The template image acquisition module is used to acquire the product template image acquired by the image acquisition device after the robot arm performs tool coordinate calibration. The product template image includes a product template.

[0098] The center point determination module is used to determine the midpoint of each side of the product template based on the product template image;

[0099] The template center point determination module is used to determine the center point of the product template based on the intersection of the lines connecting the midpoints.

[0100] Optionally, the material picking position is taught and determined relative to the center point of the product template.

[0101] Optionally, the image processing device 34 further includes:

[0102] The pixel equivalent determination module is used to control the robotic arm to perform nine-point calibration to obtain the pixel equivalent.

[0103] The robotic arm system also includes a mask and a laser emitting device. The mask includes a light-transmitting hole and is disposed between the laser emitting device and the product template.

[0104] Correspondingly, the pixel equivalent determination module includes:

[0105] A laser emitting unit is used to control the laser emitting device to emit laser light so that the laser light is projected onto the product template through the light-transmitting hole on the mask plate;

[0106] The calibration and determination unit is used to control the product template to move to the movable area, and to control the product template to move in the forward and backward and left and right directions within the movable area to perform nine-point laser printing, and to move the laser-printed product template to the field of view of the image acquisition device for nine-point calibration.

[0107] Optional, the offset determination module includes:

[0108] A pixel difference determination unit is used to determine the pixel difference value based on the pixel coordinates of the product center point and the pixel coordinates of the product template center point;

[0109] An offset determination unit is used to determine the product offset based on the product of the pixel difference and the pixel equivalent.

[0110] Optionally, the image processing device 34 further includes:

[0111] The correction module is used to perform distortion correction on the image acquisition device before acquiring the image of the product to be acquired by the image acquisition device or acquiring the product template image acquired by the image acquisition device.

[0112] The image processing device provided in the embodiments of the present invention can execute the material handling method based on the robotic arm provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0113] Example 4

[0114] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0115] like Figure 4As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0116] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as a robotic arm-based material handling method.

[0118] In some embodiments, the robotic arm-based material handling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded into and / or mounted on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the robotic arm-based material handling method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the robotic arm-based material handling method by any other suitable means (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0123] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0124] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0125] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A material taking method based on a robot, characterized by, This invention is applied to a robotic arm material handling system, which includes a robotic arm suction cup, a robotic arm, an image acquisition device, and an image processing device. The robotic arm suction cup is rotatably mounted at the free end of the robotic arm and is used to adsorb and acquire the product to be picked up. The image acquisition device is used to acquire an image of the product to be picked up. The method is performed by an image processing device and includes: Acquire images of the product to be retrieved by the image acquisition device; Determine the center point of the product based on the image of the product to be retrieved; The product offset is determined based on the product center point, the product template center point, and the pixel equivalent. The product offset is then sent to the robot arm so that the robot arm can pick up the product to be picked up based on the product offset and the predetermined picking position. The product template center point and pixel equivalent are determined after the robot arm performs tool coordinate calibration. The tool coordinate calibration includes aligning the robot arm's suction cup rotation center, the image acquisition field of view center, and the product template center.

2. The method of claim 1, wherein, Determining the center point of the product template includes: After the robotic arm performs tool coordinate calibration, it acquires a product template image captured by the image acquisition device, the product template image including the product template; The midpoint of each side of the product template is determined based on the product template image; The center point of the product template is determined by the intersection of the lines connecting the midpoints.

3. The method according to claim 1, characterized in that, The material picking position is taught and determined relative to the center point of the product template.

4. The method according to claim 1, characterized in that, Determining the pixel equivalent includes: The robotic arm is controlled to perform nine-point calibration to obtain pixel equivalents.

5. The method according to claim 4, characterized in that, The robotic arm material handling system also includes a mask and a laser emitting device. The mask includes light-transmitting holes and is disposed between the laser emitting device and the product template. Accordingly, controlling the robotic arm to perform nine-point calibration includes: The laser emitting device is controlled to emit a laser so that the laser light passes through the light-transmitting hole on the mask and is projected onto the product template; The product template is controlled to move to a movable area, and the product template is controlled to move in the forward and backward and left and right directions within the movable area to perform nine-point laser printing. The laser-printed product template is then moved to the field of view of the image acquisition device for nine-point calibration.

6. The method according to claim 1, characterized in that, The step of determining the product offset based on the product center point, the product template center point, and the pixel equivalent includes: The pixel difference is determined based on the pixel coordinates of the product center point and the pixel coordinates of the product template center point; The product offset is determined by multiplying the pixel difference and the pixel equivalent.

7. The method according to any one of claims 1-6, characterized in that, Before acquiring the image of the product to be acquired by the image acquisition device or the product template image acquired by the image acquisition device, the method further includes: Distortion correction is performed on the image acquisition device.

8. A robotic arm material handling system, characterized in that, The device includes a robotic suction cup, a robotic arm, an image acquisition device, and an image processing device. The robotic suction cup is rotatably mounted at the free end of the robotic arm and is used to pick up a product. The image acquisition device is used to acquire an image of the product to be picked up. The image processing device includes: The image acquisition module is used to acquire images of the product to be acquired by the image acquisition device. The product center point determination module is used to determine the product center point based on the image of the product to be retrieved. The offset determination module is used to determine the product offset based on the product center point, the product template center point, and the pixel equivalent, and send the product offset to the robot arm so that the robot arm can pick up the product to be picked up according to the product offset and the predetermined picking position. The product template center point and pixel equivalent are determined after the robot arm performs tool coordinate calibration. The tool coordinate calibration includes aligning the robot arm's suction cup rotation center, the image acquisition field of view center, and the product template center.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the robotic arm-based material handling method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the robotic arm-based material handling method according to any one of claims 1-7.

Images