A visual guidance method for a mechanical hand for bag unstacking

Abstract

The application discloses a mechanical hand visual guidance method for bag unstacking, comprising the following steps: a first step of calculating the average effective distance of a sensor from the top layer of a material stack; a second step of calculating a photographing point and moving a camera to the position; a third step of collecting a top layer image of the material stack and performing pretreatment; a fourth step of calculating a bag surface center point pixel coordinate; a fifth step of coordinate conversion; and a sixth step of moving a mechanical hand carrying a vacuum chuck to the bag surface to suck up the material bag, so as to realize unstacking. The method realizes bag unstacking based on 2D visual guidance, has the advantages of simple structure, low failure rate, low hardware cost, fast recognition speed and the like, can effectively improve the efficiency and precision of automatic bag unstacking, and is favorable for enterprises to reduce labor costs and improve the intelligent level of a production line.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of mechanical hand visual control, and particularly relates to a mechanical hand visual guiding method for bag material unstacking. BACKGROUND

[0002] Bag material is usually granular or powdery material packaged by a plastic woven bag or a film bag, and is a common raw material supply form in industrial production and food processing industries. The bag material usually appears in the form of a material stack at the feeding end of a production line, and the material stack is usually stacked in a two-horizontal-three-vertical form. Most unstacking links still rely on manual unstacking and transportation. Since the bag material is usually heavy, the labor intensity of workers is large and the efficiency is low. Although some 3D or 2D+3D recognition positioning and unstacking methods have appeared, the price of the 3D sensor is usually slightly high, and the processing of 3D point cloud data will occupy more computing power and time, affecting the further improvement of unstacking efficiency. At the same time, the requirement of computing power and speed is reflected in the higher hardware configuration requirement of the computer.

[0003] In patent [CN202010159834.2], a 3D visual guiding based unstacking method and system are proposed. In patent [CN202010189907.2], an industrial robot application soft bag unstacking and unloading stacking device and a method for unstacking and unloading stacking are proposed. In patent [CN201510114278.6], an industrial robot automatic unstacking system based on 3D machine vision guidance is proposed. In patent [CN201410665285.0], a robot unstacking method based on binocular stereo vision is proposed. In patent [CN201811297323.6], a flexible unstacking and stacking robot system and method based on machine vision are proposed. In patent [CN202111615890.3], a robot unstacking and stacking system based on visual guidance and a unstacking and stacking method thereof are proposed. In patent [CN201910613517.0], a 2D and 3D vision combined unstacking method is proposed. However, the above unstacking patents based on vision all use 3D vision technology, and most of them are for recognizing and unstacking stacked boxes.

[0004] In patent [CN201911107619.1], an intelligent feeding system is proposed, which uses a mechanical hand, a camera and a range finder, but only briefly describes the structure distribution and function, and does not describe the method of visual system guiding the mechanical hand. SUMMARY

[0005] The application aims to provide a mechanical hand visual guidance method for bag unstacking, which can realize unmanned operation in the bag unstacking process, solve the problem of excessive labor cost in the existing unstacking link, and has the advantages of simple structure, low failure rate, low hardware cost, fast recognition speed and the like compared with the existing 3D visual method.

[0006] The mechanical hand visual guidance method for bag unstacking of the application comprises a six-axis mechanical hand, a vacuum suction cup, an equipment support, an industrial camera and a laser displacement sensor.

[0007] In some embodiments, the vacuum suction cup is installed on the sixth joint at the end of the mechanical hand.

[0008] Further, the vacuum suction cup is parallel to the ground, and the suction port direction is downward.

[0009] In some embodiments, the equipment support is installed on the fifth joint of the mechanical hand,

[0010] Further, the installation center of the equipment support and the center of the sixth joint are located on the same vertical line, and the two are distributed at 180° around the axis of the fifth joint.

[0011] In some embodiments, the industrial camera and the laser displacement sensor are both installed on the equipment support.

[0012] Further, the detection directions of the industrial camera and the laser displacement sensor are both perpendicular to the ground, and the bag pile composed of bags is placed on a tray and below the industrial camera and within the imaging range of the industrial camera.

[0013] The mechanical hand visual guidance method for bag unstacking of the application calculates the ideal shooting position of the camera through the distance recognition of the top layer of the pile by the laser displacement sensor, completes image acquisition at the position, realizes pixel coordinate extraction of the center point of the bag area through image processing, and calculates the corresponding mechanical hand coordinates through coordinate conversion, and the mechanical hand with the vacuum suction cup completes the suction and unstacking operation.

[0014] In some embodiments, a mechanical hand visual guidance method for bag unstacking comprises the following steps:

[0015] a. Average effective distance calculation of the top layer of the pile;

[0016] b. Camera shooting position coordinate calculation;

[0017] c. Image acquisition and image preprocessing;

[0018] d. Bag upper surface center pixel coordinate calculation;

[0019] e. Pixel coordinate to mechanical hand coordinate conversion;

[0020] f, the mechanical hand completes bag body suction and unstacking according to the coordinates.

[0021] In some embodiments, the image preprocessing in step c includes image graying and image filtering.

[0022] In some embodiments, the center pixel coordinate calculation of the upper surface of the bag body in step d includes image feature edge extraction, closed and non-closed region pixel point extraction, effective closed region identification, and effective closed region centroid pixel coordinate calculation.

[0023] The beneficial technical effects of the present application include:

[0024] (1) The detection method combined with a 2D industrial camera and a laser displacement sensor, combined with a six-axis mechanical hand and a vacuum suction cup, can realize the positioning and unstacking of the bag body in the multi-attitude unloading stack, and the recognition speed of the method is fast.

[0025] (2) The method of the present application requires simple structure, low hardware cost, convenient deployment and relatively low requirements for the corresponding field application environment, and has strong adaptability and portability. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the overall structure schematic diagram of the hardware required for the bag material unstacking mechanical hand visual guidance method of the present application;

[0027] Figure 2 is the method flowchart of the present application;

[0028] Figure 3 is the laser displacement sensor measurement path schematic diagram of the present application;

[0029] Figure 4 is the seed point diffusion point search algorithm schematic diagram of the present application. DETAILED DESCRIPTION

[0030] The present application will be described below with polyethylene bag material as an example, combined with the attached Figures 1 to 4 The present application is specifically described.

[0031] As Figure 1 shown is the overall structure schematic diagram of the present application, including a six-axis mechanical hand 1, a vacuum suction cup 2, a device support 3, an industrial camera 4, a laser displacement sensor 5, a tray 6, and a bag material 7.

[0032] In the embodiment, the unstacking mechanism is composed of a six-axis robot 1, a vacuum chuck 2, a device support 3, an industrial camera 4 and a laser displacement sensor 5, the vacuum chuck 2 is located on the sixth joint at the end of the robot 1, the suction port is downward, the device support 3 is fixed on the fifth joint of the robot 1, the industrial camera 4 and the laser displacement sensor 5 are on the device support 3, and the detection directions are perpendicular to the ground. The material stack composed of bag materials 7 is placed on a tray 6 below the industrial camera 4 and within the imaging range of the industrial camera 4.

[0033] In the embodiment, the selection of the industrial camera 4 is determined according to the conditions such as bag size, measurement accuracy and imaging speed, the weight of the bag material 7 applied in the application is 50 Kg, the maximum length is 900 mm, the maximum width is 600 mm, considering the factors such as possible gap or disorder of the bag, the horizontal cross-sectional size of the material stack is at least 1800 mm*1800 mm, and because the tray 6 is usually carried by a forklift, the position is not accurate, therefore, a 200 mm margin is left for both length and width, and the actual shooting size of the industrial camera 4 is 2000 mm*2000 mm. The measurement accuracy is set to 1 mm, first, the pixel is estimated, the target surface of the camera is usually a 4:3 rectangle, taking the short side of the target surface as a reference, the short side pixel should be greater than 2000 / 1=2000, according to the estimated pixel, a Hikvision CMOS camera MV-CA050-12UC can be selected, the target surface size is 2 / 3"(8.8 mm*6.6 mm), the resolution is 2448*2048, the pixel size is 3.45 um, and the maximum frame frequency is 47 fps. The distance between the industrial camera 4 and the top layer of the material stack is not greater than 1500 mm, the focal length f is calculated through f=Wh / w, wherein the target surface width W=6.6 mm, the distance h=1500 mm, and the collection width w=2000 mm, and f=4.95 mm is calculated, and a Hikvision MVL-C0420-5MP lens with a focal length of 4 mm can be selected.

[0034] The laser displacement sensor 5 is comprehensively determined according to the conditions such as measurement distance and repeatability, the distance between the top layer of the material stack without unstacking and the laser displacement sensor 5 is greater than 1500 mm, the height of the conventional material stack is about 1500 mm, the distance between the bottom layer of the material stack and the laser displacement sensor 5 is less than 3000 mm, and a BANNER laser displacement sensor LTF12IC2LDQ can be selected, the effective measurement distance of the BANNER laser displacement sensor LTF12IC2LDQ is 50 mm-7 m, and the repeatability is less than 5 mm.

[0035] As shown in Figure 2 The method flowchart of the application is shown in the figure, and the key links are the average effective distance calculation of the top layer of the material stack, the camera shooting position calculation, the image acquisition and preprocessing, the bag surface center pixel coordinate calculation and the coordinate conversion. Specifically, the steps include the following steps:

[0036] (1) After the unstacking instruction is issued, the robot 1 moves to the position of the top layer of the material stack, and the industrial camera 4 is used to shoot the top layer of the material stack. Figure 3The fixed initial position P1 is the starting point, and the laser displacement sensor 5 is driven to move along the fixed Z-shaped path in the horizontal plane to P 2N , and regularly collect 2N (N≥6) distance values.

[0037] (2) Through the analysis of 2N distance values, the average effective distance L of the laser displacement sensor 5 and the top layer of the material pile is calculated.

[0038] (3) The ideal shooting distance of the industrial camera 4, the arm range of the manipulator 1 and the height range of the material pile are comprehensively considered, and the threshold values L H and L L are set according to the actual test analysis. The effective distance L is compared with the threshold values L H and L L . If L L ≤L≤L H , the next shooting position calculation is performed, and if it is not within the threshold range, it is determined that the height of the material pile is too high or there is no bag material 7 on the tray 6, and no destacking operation is performed, and the process is ended.

[0039] (4) The shooting position of the industrial camera 4 is calculated in combination with the effective distance L and the actual shooting distance h of the industrial camera 4, and the manipulator 1 drives the industrial camera 4 to move to the position.

[0040] (5) Image acquisition and preprocessing are performed, and the image preprocessing process includes image graying and image filtering.

[0041] (6) The calculation of the center pixel coordinates of the upper surface of the bag body is performed, which specifically includes image feature edge extraction, closed and non-closed area pixel point extraction, effective closed area identification, and effective closed area centroid pixel coordinate calculation.

[0042] (7) The conversion of the center pixel coordinates of the upper surface of the bag body to the manipulator coordinates is performed.

[0043] (8) After the manipulator 1 drives the vacuum chuck 2 to the position 100 mm above the center point of the upper surface of the bag body, it is slowed down until the vacuum chuck 2 is tightly attached to the surface of the bag body, and the manipulator 1 is lifted to realize the single bag destacking operation.

[0044] (9) After each single bag destacking operation, it is judged whether to continue destacking. If not, the subsequent operation is stopped, and if yes, it is judged whether the current layer has been destacked and transported. If not, the manipulator 1 drives the vacuum chuck 2 to continue the destacking operation. When the current layer has been destacked and transported, the manipulator 1 returns to the initial position P1, and the Z-shaped path 2N distance value collection and subsequent operation are performed.

[0045] 1, calculate the effective distance L

[0046] In order to find a shooting position suitable for the lens focal length of the industrial camera 4, the effective distance information between the industrial camera 4 and the top layer of the stack needs to be obtained. In the embodiment, the manipulator 1 drives the laser displacement sensor 5 to move along a specific Z-shaped path from a fixed initial point P1 to P 2N The laser displacement sensor 5 regularly and high-frequency collects 2N distance values, Figure 3 In the middle, layer A represents the top layer of the stack at a certain moment, and there are two horizontal and three vertical 5 bags. Layer B represents the second top layer of the stack at that moment, and the 5 bags are distributed at 180° with layer A. Since the placement position of the tray 6 is not accurate, Figure 3 In the middle, the dashed line on the periphery of the Z-shaped path represents the actual measurement range, and the dots on the Z-shaped path represent the 2N theoretical distance points. From Figure 3 It can be seen that, without the requirement of placing the tray 6 particularly accurately, regardless of the A or B layer layout, the Z-shaped path measurement in the embodiment can better cover the 5 bags and can representatively reflect the distance information of the measured layer. However, since the surface of the bag is not flat and there is a gap between the bags, the influence of such abnormal data needs to be excluded. Let the 2N distance values be L1, L2, …, L 2N , then the standard deviation In the formula L1, L2, …, L 2N are judged one by one, when the distance value is excluded, otherwise it is retained. The average value of all retained distance values is the effective distance L.

[0047] 2, Camera shooting position calculation

[0048] According to Figure 3 Let the coordinates of the theoretical center point O of the stack at the distance measuring height be (X0, Y0, Z0), according to Figure 1 the coordinate system, the Z-direction effective distance at this height is L. Let O point be the initial position before shooting of the industrial camera 4, then the actual shooting position of the industrial camera 4 is (X0, Y0, Z0-L+h), in the formula, h is the ideal shooting distance of the industrial camera 4, because the lens focal length of the industrial camera 4 is 4mm, so according to h=fw / W, the ideal shooting distance h of the industrial camera 4 is about 1212mm, then the coordinates of the actual shooting position of the industrial camera 4 are (X0, Y0, Z0-L+1212).

[0049] 3, Image graying

[0050] In order to highlight the image features, simplify the image matrix and improve the calculation efficiency, the image needs to be graying processed. In the embodiment, the weighted average value algorithm is used for graying processing of the image. The weight coefficients P r , P g , Pb The pixel value distribution is focused on the ratio of 0-255, and the value is f(x, y) = P r R+P g G+P b B.

[0051] 4. Image filtering

[0052] The image acquisition and transmission process will be disturbed by noise. In order to suppress noise, reduce noise points and improve image quality, a Gaussian filter is used for image smoothing in the embodiment. The final pixel value of each pixel point in the image is calculated by weighted average of its neighborhood and itself, and the weighted value decreases monotonously with the increase of the distance from the center point. The filter coefficient at (x, y) in the two-dimensional Gaussian filter template is Where σ is the Gaussian distribution parameter, which determines the smoothing degree of the Gaussian function. The filtered pixel value is represented as g(x, y).

[0053] 5. Image feature edge extraction

[0054] In order to better reflect the image feature edge, the improved gradient calculation method is used to calculate the image gradient amplitude and direction in the embodiment. The calculation method uses four angle direction calculation templates of 0°, 45°, 90° and 135°, and the calculation formula is

[0055] The gradient calculation formula in the horizontal direction is The gradient calculation formula in the vertical direction is The gradient amplitude is The gradient direction is After traversal operation, each pixel in the image corresponds to a gradient amplitude and a gradient direction except the edges around the image. Then, a 5x5 pixel array is taken as a judgment unit, the gradient amplitude of the center pixel point of the unit is G i , the gray value is g i (x, y), and the gradient amplitudes of the other four pixels in the gradient direction of the pixel are (G i1 , G i2 , G i3 , G i4 ). By comparing, g i (x, y) is revalued, and the revaluation method is The edges of the image after traversal processing are extracted by binaryzation, the upper threshold is T up , the lower threshold is T down , a 3x3 pixel array is taken as a judgment unit, and the gray value of the center pixel point of the unit is still g i (x, y), which is abbreviated as g i, let the binary value of the pixel point be t i (x, y), let the gray values of the eight pixel points around the pixel point be (g i1 , g i2 , g i8 , g L0 , g R0 , g L0 , g LU0 , g LD0 ), then the binary value of the center pixel point can be calculated according to After traversal operation, the binary processing of the feature edge pixel points in the image is completed, except for the edges of the image.

[0056] 6. Extraction of pixel points in closed and non-closed regions

[0057] According to the seed point diffusion search algorithm in the embodiment, the extraction of all pixel points with a pixel value of 255 in the closed or non-closed region is realized according to the binary contour data, as shown in the method example Figure 4 In the closed region, let the pixel point P0 be the initial seed point, in the first step, find P L0 and P R0 points in the row where P0 is located along the -x and +x directions respectively, the pixel values of the two points are both 255, and the nearest pixel value of the point next to the row is 0 or the point itself is the image boundary point, and the pixel points between the two points are recorded in the set P. In the second step, take P L0 as the reference, find the nearest P LU0 and P LD0 points in the -y and +y directions respectively, and then take P R0 as the reference, find the nearest P RU0 and P RD0 points in the -y and +y directions respectively. In the third step, in the row where P LU0 and P RU0 are located, find the points with a pixel value of 255 in the +x direction from P LU0 and P RU0 respectively, and the points are next to the points with a pixel value of 0 or the points themselves are the image boundary points, find P1 and P2 points in the figure (if there is no contour point between P LU0 and P RU0 , only one common point can be found), and then find the points with a pixel value of 255 in the +x direction from P LD0 and P RD0 respectively, and the points are next to the points with a pixel value of 0 or the points themselves are the image boundary points, the target point coincides with P RD0 in the figure. Take P1, P2 and P RD0 as new seed points, repeat the search process of P0, and record the corresponding P Li and P RiPixels within the central region are recorded in set P (where i∈[0,N]). This process continues by generating new seed points and expanding the set until all pixels within the region are counted. Since none of the points in the set are image boundary points, the region to which set P belongs is a closed region. In the non-closed region, let K0 be the initial seed point. Following the above point-finding method, all pixels with a value of 255 within the region are found and recorded in set K. After point finding, since some pixels in set K are image boundary points, the region to which set K belongs is a non-closed region. Because non-closed regions like set K often belong to the image background, only the set data of closed regions are retained in the actual analysis process.

[0058] 7. Effective closed area identification

[0059] Since the number of closed regions in the image usually exceeds the number of bags, it is necessary to filter the M closed regions to extract the effective closed regions corresponding to each bag. This implementation uses an area method combined with a roundness method to achieve the filtering of closed regions. Through actual experiments, the area of ​​a typical bag region is analyzed, and a threshold S is set for the area. up and lower threshold S down Let the number of pixels within the enclosed region be the area of ​​that region, and so on. Figure 4 Taking the closed region containing point P0 as an example, the number of pixels can be obtained from the set P, and it can be represented as the area S of the region. P If S down ≤S P ≤S up Then the area of ​​region P in set P is determined to be valid. Next, the roundness of this region is calculated, and the roundness value is... Where S is the area of ​​the region, and C is the perimeter of the region's edges. The perimeter C is calculated as follows: Taking an edge pixel within set P as the initial point, create a 3×3 pixel array unit centered on that point (empty pixels are allowed). Starting from the top left corner of the unit, search for the first non-empty pixel in a clockwise direction. Then, using that point as the new starting point, search for the next edge point, and so on, until the initial point is found again to complete the loop. Count the number of pixels in the loop, and record this as the perimeter C of the region. The roundness value of the region in set P. Let the roundness threshold R be... up and lower threshold R down If R down ≤R P ≤R up If the area and roundness of the region of set P are both valid, then the region to which set P belongs is determined to be a valid closed region.

[0060] 8. Calculation of centroid pixel coordinates of the effective closed region

[0061] First, the pixel values ​​within the effective closed region are normalized, that is, a pixel value of 255 is converted to 1, and a pixel value of 0 remains 0. This is represented by v(x,y), where x and y represent the coordinates of the pixel. In the formula, I and J represent the horizontal and vertical ranges of pixel coordinates within the effective closed region. Let the coordinates of the centroid of this region be (x, J). c ,y c If the area of ​​the region is S, then

[0062] 9. Robot coordinate transformation of the centroid of the effectively enclosed region

[0063] The robotic arm needs to obtain the center coordinates of the surface of each bag (effective closed area) for depalletizing. These coordinates should be the robotic arm coordinates, and the already calculated centroid coordinates (x, y) of the area should also be obtained. c ,y c Since the coordinates are pixel coordinates, a coordinate transformation is required. The transformation process involves sequential calculations between pixel coordinates, image coordinates, camera coordinates, and robot coordinates. Let the pixel coordinates of the image center be (x0, y0), and let E and T represent the rotation matrix and translation vector in the transformation between camera coordinates and robot coordinates, respectively. Then we have... In the formula These are the factory default parameters for the industrial camera 4; the parameters are fixed. The external parameters of the industrial camera 4 are calibrated using relatively mature methods, and will not be elaborated upon in this implementation. c Given the shooting distance, pixel coordinates (x) can be obtained. c ,y c ) to the robot arm coordinates (X) c ,Y c Z c The conversion of ).

[0064] The above-described implementation is merely an example of the present invention having been practically applied and achieving good results, and is not intended to limit the present invention in any specific form. Any person experienced in the field of the present invention may make equivalent modifications using the methods or technologies described in this patent without departing from the scope of the main solution of the present invention. Therefore, any equivalent modifications that do not depart from the substantive content of the above technical solution still fall within the protection scope of the technical solution of the present invention.

Claims

1. A vision-guided method for a robotic arm used for depalletizing bagged materials, characterized in that: The method comprises the following steps: S1: after the unstacking instruction is issued, the manipulator takes the initial position P1 as the starting point, drives the laser displacement sensor to move along a fixed Z-shaped path in a horizontal plane to P 2N , and regularly collects 2N distance values; S2: through analysis of the 2N distance values, the average effective distance L of the laser displacement sensor and the top layer of the material pile is calculated; S3: the ideal shooting distance of the industrial camera, the arm span range of the manipulator and the height range of the material pile are comprehensively considered, and the threshold values L H and L L are set through actual test analysis, the effective distance L is compared with the threshold values L H and L L , if L L ≤L≤L H , the next process is carried out, if it is not within the threshold range, it is determined that the height of the material pile is too high or there is no bag material on the tray, and no unstacking operation is carried out, and the process is ended; S4: the shooting position of the industrial camera is calculated combined with the effective distance L and the actual shooting distance h of the industrial camera, and the manipulator drives the industrial camera to move to the position, the industrial camera is a 2D industrial camera; S5: image acquisition and pretreatment are carried out, the image pretreatment process comprises image graying and image filtering; S6: the calculation of the center pixel coordinates of the upper surface of the bag body is carried out, which specifically comprises image feature edge extraction, closed and non-closed area pixel point extraction, effective closed area identification and effective closed area centroid pixel coordinate calculation; S7: the conversion of the center pixel coordinates of the upper surface of the bag body to the coordinates of the manipulator is carried out; S8: after the manipulator drives the vacuum chuck to the position 100mm above the center point of the upper surface of the bag body, the speed is reduced and the vacuum chuck is lowered until the vacuum chuck is tightly attached to the surface of the bag body, the manipulator is lifted to realize the unstacking operation of the bag body; S9: after each single-bag unstacking operation is completed, it is judged whether to continue unstacking, if not, the subsequent operation is stopped, if yes, it is judged whether the current layer has been unstacked and transported, if not, the manipulator drives the vacuum chuck to continue the unstacking operation, and when the current layer has been unstacked and transported, the manipulator returns to the initial position P1, and the Z-shaped path 2N distance value collection and subsequent operation of the next round are carried out.

2. The vision-guided method for debagging a mechanical hand according to claim 1, wherein: The image feature edge extraction is completed by improved gradient calculation method to calculate image gradient amplitude and direction; the closed and non-closed region pixel point extraction is realized by seed point diffusion seeking point algorithm; and the effective closed region identification is realized by comprehensive analysis of target closed region area and roundness to realize judgment.

Images