A vision-based method for industrial robot sheet grasping

Abstract

This invention discloses a vision-based method for industrial robot sheet grasping. The grasping method includes: performing hand-eye calibration on the robot to determine the fixed geometric relationship between the robot's base coordinate system and an industrial camera; acquiring images of the sheet using the industrial camera; performing noise reduction and edge detection preprocessing on the acquired raw sheet images; locating the sheet position in the raw image using a template matching algorithm; establishing a grasping coordinate system on the sheet contour and determining the grasping point; obtaining the pose of the grasping point in the robot's base coordinate system; and controlling the robot to perform sheet grasping and placement. This invention can minimize the problem of low placement accuracy caused by inconsistent incoming material stability, effectively improving the accuracy and success rate of grasping and placement planning.

Description

Technical Field

[0001] This invention relates to a method for industrial robot sheet grasping based on vision positioning, belonging to the field of intelligent assembly technology for industrial robots. Background Technology

[0002] With the rapid development of technology, especially the continuous advancements in industrial robot technology, machine vision, sensor technology, and automation control technology, industrial robots are increasingly widely used in various complex production scenarios. In industries such as composite material manufacturing, automobile manufacturing, and electronics manufacturing, the precise gripping and placement of materials is a crucial step in the production process. Traditional manual placement methods are not only inefficient but also struggle to guarantee product consistency and quality stability. Therefore, using industrial robots for material gripping and placement has become an effective way to solve this problem.

[0003] In robotic applications involving the gripping and laying of sheet materials, many irregularly shaped workpieces or sheets with symmetrical forms are encountered. These irregularly shaped sheets can be gripped from multiple angles, making them ideal for use with robotic grippers or suction cups. Robotic gripping of such items typically employs machine vision technology for assisted positioning, determining the centroid of the workpiece to be gripped, and thus determining the gripping posture to guide the robot in its gripping operation.

[0004] The centroid-based grasping method requires high workpiece consistency. However, in actual production, when the consistency of incoming materials is unstable, the actual cutting specifications of the sheet material deviate from the theoretical dimensions to varying degrees. If the laying work is carried out solely based on the centroid-based grasping method, there may be misalignment between the upper edge of the sheet material and the laying scale lines, affecting the progress of subsequent production work. In addition, the centroid-based grasping method has high requirements for the shape of the sheet material. For irregularly shaped sheets with large curvatures, the centroid may be located outside the sheet material. In this case, guiding the robot to grasp the sheet material based on the centroid may result in unsuccessful grasping. Therefore, there is an urgent need for a visual inspection method that can flexibly and accurately identify the sheet material, locate it, and guide the robot to grasp it, thereby improving the accuracy of robot grasping and the precision of laying. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a vision-based method for industrial robot sheet grasping. By setting a grasping coordinate system at a specific position on the sheet outline and using the origin of the grasping coordinate system as the grasping point, the accuracy of robot grasping and laying sheet is improved.

[0006] The technical solution of this invention is: a method for industrial robot sheet grasping based on vision positioning, comprising:

[0007] Perform hand-eye calibration on industrial cameras and industrial robots to determine the hand-eye transformation matrix between them.

[0008] The raw image of the material sheet is obtained by acquiring images of the material sheet using an industrial camera; at the same time, the template image of the material sheet generated by CAD software is also acquired.

[0009] The original image of the material sheet is subjected to noise reduction and edge detection to obtain a denoised image of the material sheet.

[0010] Image template matching is performed between the denoised sheet image and the sheet template image to locate the sheet outline position in the denoised sheet image;

[0011] Establish a gripping coordinate system on the outline of the material sheet;

[0012] The origin of the grasping coordinate system is regarded as the grasping point, and any point on the x-axis of the grasping coordinate system that is close to the origin is regarded as the attitude point. The pose of the grasping point in the robot base coordinate system is obtained, and then the robot is controlled to grasp the material based on the above pose.

[0013] The hand-eye calibration of industrial cameras and industrial robots includes:

[0014] Multiple images of a cellular calibration board are acquired from different directions using an industrial camera for hand-eye calibration. The calibration results, including camera intrinsic parameters, extrinsic parameters, and distortion coefficients, are obtained through nonlinear optimization. The hand-eye transformation matrix between the industrial camera and the industrial robot is then determined.

[0015] The noise reduction process employs median filtering, mean filtering, or Gaussian filtering.

[0016] The edge detection employs the Canny operator, Roberts operator, Sobel operator, or Laplace operator.

[0017] Image template matching is performed using an improved gray wolf optimization algorithm, an image pyramid-based search algorithm, or a particle swarm optimization algorithm, and the optimal matching position is output.

[0018] The step of establishing a gripping coordinate system on the outline of the material sheet includes:

[0019] Calculate the minimum bounding rectangle of the material sheet outline and obtain the position information of the rectangle;

[0020] The line connecting the center points of the two long sides of the rectangle is taken as the Y-axis of the grasping coordinate system. The intersection of the Y-axis and the top of the material outline is taken as the origin of the grasping coordinate system. Based on the Y-axis and the origin, an X-axis perpendicular to the Y-axis is generated on the material outline plane. This is set as the grasping coordinate system. The origin of the grasping coordinate system is the grasping point.

[0021] Generate a coordinate system graphic on the original image of the material.

[0022] The minimum bounding rectangle of the material sheet is calculated using the rotating caliper method.

[0023] The process of obtaining the pose of the gripping point in the robot's base coordinate system and then controlling the robot to grip the material piece based on the pose includes:

[0024] By calculating the pixel positions of the grasping point and the attitude point in the image, and combining the depth information of the image, i.e. the actual distance from the image to the industrial camera, and the intrinsic parameters of the industrial camera, the three-dimensional coordinates of the grasping point and the attitude point in the camera coordinate system are calculated.

[0025] The calculated 3D coordinates of the grasping point and the attitude point are converted into homogeneous coordinate matrices and multiplied with the hand-eye transformation matrix to obtain the 3D coordinates of these two points in the robot's base coordinate system.

[0026] Extract the position of the grasping point in the base coordinate system, and calculate the angle between the target vector formed by the grasping point and the attitude point and the X-axis in the robot's base coordinate system, thereby obtaining the rotation angle of the robot's grasping action, i.e. the robot's attitude angle.

[0027] The position and rotation angle of the gripping point are combined to obtain the complete pose information of the gripping point in the robot's base coordinate system; the pose information will be used to guide the industrial robot to perform gripping operations.

[0028] The sheet material is a symmetrical irregularly shaped sheet material.

[0029] The advantages of this invention compared to the prior art are:

[0030] (1) This invention sets up a gripping coordinate system at the intersection of the top of the material sheet outline and one of the axes of symmetry of its smallest circumscribed rectangle, and uses the origin of the gripping coordinate system as the gripping point. This can minimize the problem of low laying accuracy caused by inconsistent material stability, and effectively improve the accuracy of gripping and laying the material sheet. Compared with gripping methods that use the centroid as the gripping point, this invention has lower requirements for workpiece size consistency and shape, and is more widely applicable.

[0031] (2) The present invention uses the improved gray wolf optimization algorithm for image templates. Compared with the traditional traversal search algorithm, it effectively saves computational costs, simplifies the complexity of the image matching search process, and improves the effectiveness in optimizing image matching problems. Attached Figure Description

[0032] Figure 1 This is a system flowchart for implementing this method;

[0033] Figure 2 This is a flowchart illustrating the improved and optimized Grey Wolf algorithm used in template matching in this invention.

[0034] Figure 3 This is a schematic diagram of the original image and the generated image in the capture coordinate system. Detailed Implementation

[0035] like Figure 1 As shown, the method flow of the present invention is as follows:

[0036] Step 1: Hand-eye calibration. Use an industrial camera to acquire multiple images of the cellular calibration board in different directions for hand-eye calibration. Obtain the calibration results of nonlinear optimization, including camera intrinsic parameters, extrinsic parameters, and distortion coefficients, and determine the hand-eye transformation matrix between the industrial camera and the industrial robot.

[0037] Step 2: Image acquisition. The material sheet is captured by an industrial camera, and the original image of the material sheet is transmitted into the visual recognition and positioning system. The material sheet template file DXF model data is also transmitted into the visual recognition and positioning system.

[0038] Step 3: Image preprocessing. The image preprocessing module of the visual recognition and positioning system is used to perform noise reduction and edge detection on the original image of the material sheet. The medium-value filter is applied to the material sheet image to remove noise and make the material sheet features more obvious. The Canny operator is applied to the material sheet for edge detection to obtain the material sheet outline.

[0039] Step 4: Template matching. Using the template matching module in the visual recognition and positioning system, the outline of the material piece is matched with the DXF model data using the improved Grey Wolf optimization algorithm to output the best matching position.

[0040] Step 5: Set up the grasping coordinate system. Calculate the minimum bounding rectangle of the material sheet outline using the rotation method to obtain the position information of the rectangle. Take the line connecting the center points of the two long sides of the rectangle as the Y-axis of the grasping coordinate system. Take the intersection of the Y-axis and the top of the material sheet outline as the origin of the grasping coordinate system. Generate an X-axis perpendicular to the Y-axis on the material sheet outline plane based on the Y-axis and the origin. Set this as the grasping coordinate system and generate the grasping coordinate system graphic on the original image of the material sheet.

[0041] Step 6: Coordinate transformation. The origin of the grasping coordinate system is regarded as the grasping point, and any point on the x-axis of the grasping coordinate system that is close to the origin is regarded as the attitude point. The pose of the grasping point in the robot's base coordinate system is obtained to guide the robot to perform the grasping work.

[0042] The median filtering process in step 3 specifically involves:

[0043] Median filtering processes an image by first sorting all pixels within the filtering window and calculating their median values, then replacing the value of the pixel at the center of the window with the median value. The formula is as follows:

[0044]

[0045] Where g(x, y) is the pixel value after median filtering, and f(i, j) is the pixel value before median filtering.

[0046] Median filtering, essentially a technique developed based on in-depth research into sorting theory, is used to process nonlinear signals present in noise. Its application involves rationally integrating and utilizing the various parameters of a digital image and sequence, and, considering the characteristics of each parameter, selecting the average parameter from surrounding points for replacement. This ensures that surrounding pixels are close to the true parameters, thus eliminating isolated noise points. For all pixels in the image, the median parameter within the filtering window is used to replace the pixel's parameter. Median filtering effectively removes noise from images while maintaining clear image edges.

[0047] The specific steps for edge detection of the material sheet using the Canny operator in step 3 are as follows:

[0048] The Canny operator is one of the most commonly used operators for image edge detection, exhibiting good performance and robustness. The Canny operator first applies a Gaussian filter to the image to reduce the impact of noise; the specific calculation formula is as follows:

[0049]

[0050] Then, the first-order partial derivatives in the horizontal x and vertical y directions of the image are calculated, and the gradient magnitude and direction angle of each pixel are calculated. The specific calculation formula is as follows:

[0051]

[0052] θ(x, y) = arctan(I) y / I x In the above formula, M(x, y) represents the gradient magnitude of a pixel, and θ(x, y) represents the direction of the pixel. The gradient magnitude indicates the degree of color change of a pixel, and the gradient direction indicates the direction of color change. By applying non-maximum suppression and double thresholding to detect image edges, and retaining the pixels with the largest local gradients after scanning the image, a refined edge image can be obtained.

[0053] See Figure 2 The diagram shown is a basic flowchart of the improved and optimized Grey Wolf algorithm.

[0054] In step 4 above, the fitness function based on the improved gray wolf optimization algorithm uses the maximum cross-correlation algorithm, and its formula is shown below:

[0055]

[0056] Among them, X ij and Y i+u，j+v The expressions are shown below, where x and y are the original pixel values ​​of the template sub-image and the reference image, respectively, and (u, v) is the position coordinate of the upper left corner of the template sub-image in the reference image.

[0057]

[0058] The maximum cross-correlation method has advantages such as high accuracy, strong robustness, and wide application range. Therefore, this invention uses it as the fitness function to determine the performance of each gray wolf search individual (subgraph). When R(u, v) approaches 1, the optimal solution can be obtained. Thus, the image matching problem can be understood as follows: in a search space (baseline graph), there are n gray wolf search individuals (candidate subgraphs). The optimized gray wolf algorithm is used as a new search strategy to search for the subgraph position in the baseline graph where R(u, v) is closest to 1, thereby obtaining the optimal matching position. The closer the fitness function value is to 1, the better the performance of the search individual, that is, the closer the search subgraph is to the template subgraph.

[0059] The specific steps for image template matching based on the improved gray wolf optimization algorithm in step 4 above are as follows:

[0060] (1) Input all the information of the original image, i.e. the reference image and the template image.

[0061] (2) Set the algorithm parameters. Set the maximum number of iterations and the population range interval to fixed values.

[0062] (3) Initialization. In the entire baseline image (search area), randomly generate n individual gray wolves, resulting in n sub-images. Generate random initial positions for the gray wolves, with positions ranging from [1, M-m+1] (the dimensions of the baseline image and the template sub-image are M×N and m×n, respectively). Calculate the fitness value of each gray wolf individual, and use the optimal position as... The second-best position is The third best position as Other search individual locations are represented as

[0063] (4) Calculate the maximum cross-correlation value of each individual using the fitness function and obtain the optimal, second-best and third-best solutions at this time.

[0064] (5) Update the individual gray wolves and proceed to the next iteration. If the algorithm's termination condition is not met (i.e., the number of iterations is less than the maximum number of iterations), and the best match has not yet been found, return to step (3); otherwise, output the global optimum and end the search.

[0065] Through comparative experiments, the average values ​​of three experimental results—running time, number of matching iterations, and matching accuracy—were calculated and compared between the improved gray wolf optimization algorithm and the traditional traversal search algorithm, as shown in the table below:

[0066] Traditional traversal search algorithm Improved Gray Wolf Optimization Algorithm Matching time (s) 28.4631 3.1792 Matching accuracy 100％ 96.85％ Maximum number of iterations - 80 Total number of loops - 139.2

[0067] The comparative experiments above show that the search strategy of the optimized Grey Wolf algorithm for image template matching effectively saves computational costs, simplifies the complexity of the image matching search process, and improves its effectiveness in optimizing image matching problems.

[0068] The core idea of ​​using the rotation method to find the minimum bounding rectangle in step 5 above is to select a point on the graphic, calculate the minimum bounding rectangle within a 90° rotation around that point, record the rotation angle, and after the image returns to its original position, the bounding rectangle at that angle is the minimum bounding rectangle. The specific process is as follows:

[0069] (1) Calculate the smallest bounding rectangle before the figure is rotated, and record its area as S. min Record the current rotation angle α. Initially, α = 0 by default, and the rotation angle is recorded as β.

[0070] (2) Select a point on the image, rotate the image by an angle θ, record the area S of the smallest positive circumscribed rectangle at this time, and record the current rotation angle α. At this time, α = α + θ. Compare S with S. min Assign the smaller value to S min Meanwhile, β records the rotation angle when the S value is small.

[0071] (3) Continue rotating by angle θ, repeat step (2), until α = 90°, and record S. mtn The minimum value and the obtained β value.

[0072] (4) After the graphic returns to its original position, the minimum bounding rectangle of the graphic can be obtained by rotating the recorded minimum bounding rectangle by β degrees around the selected point.

[0073] like Figure 3 As shown, this is the process of establishing a gripping coordinate system, where the material sheet is a symmetrical irregularly shaped material sheet with curvature.

[0074] The specific steps for generating the capture coordinate system on the original image of the material in step 5 above are as follows:

[0075] (1) Find the minimum bounding rectangle of the material sheet contour calculated by the rotation method, and obtain the position information of the rectangle;

[0076] (2) Take the line connecting the center points of the two long sides of the rectangle as the Y-axis of the grasping coordinate system. Take the intersection of the Y-axis and the top of the material outline as the origin of the grasping coordinate system. Generate an X-axis perpendicular to the Y-axis on the material outline plane based on the Y-axis and the origin. Set this as the grasping coordinate system.

[0077] (3) Generate a capture coordinate system graphic on the original image of the material sheet.

[0078] The specific steps for obtaining the pose of the gripping point in the robot's base coordinate system in step 6 above are as follows:

[0079] (1) Read the hand-eye calibration data and obtain the transformation matrix from the camera coordinate system to the robot base coordinate system;

[0080] (2) Read the camera's intrinsic parameter data to obtain the camera's focal length, distortion coefficient, and other intrinsic parameter data;

[0081] (3) The origin of the grab coordinate system is regarded as the grab point, and any point close to the origin in the x-axis direction of the grab coordinate system is regarded as the attitude point;

[0082] (4) Based on the pixel positions of the grab point and the attitude point, calculate the three-dimensional coordinates of the grab point and the attitude point in the camera coordinate system by calculating the image depth information, i.e., the actual distance from the image to the camera and the camera intrinsic parameters.

[0083] (5) Convert the coordinates of the grasping point and the attitude point in the camera coordinate system into homogeneous coordinate matrices, and multiply them by the hand-eye transformation matrix to obtain the coordinate information in the robot base coordinate system;

[0084] (6) Extract the position of the grasping point in the robot's base coordinate system;

[0085] (7) Calculate the angle between the target vector of the grasping point and the attitude point in the robot base coordinate system and the X-axis in the robot base coordinate system, determine the rotation angle, and obtain the robot's attitude angle;

[0086] (8) Combine the position and orientation of the gripping point to obtain the complete pose of the gripping point in the robot base coordinate system, and guide the robot to perform gripping work.

[0087] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention based on the above-disclosed technical content without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for industrial robot sheet grasping based on vision positioning, characterized in that, include: Perform hand-eye calibration on industrial cameras and industrial robots to determine the hand-eye transformation matrix between them. The raw image of the material sheet is obtained by acquiring images of the material sheet using an industrial camera; at the same time, the template image of the material sheet generated by CAD software is also acquired. The original image of the material sheet is subjected to noise reduction and edge detection to obtain a denoised image of the material sheet. Image template matching is performed between the denoised sheet image and the sheet template image to locate the sheet outline position in the denoised sheet image; Establishing a gripping coordinate system on the outline of the material sheet includes: Calculate the minimum bounding rectangle of the material sheet outline and obtain the position information of the rectangle; The line connecting the center points of the two long sides of the smallest bounding rectangle is taken as the Y-axis of the grasping coordinate system. The intersection of the Y-axis and the top of the material outline is taken as the origin of the grasping coordinate system. An X-axis perpendicular to the Y-axis is generated on the material outline plane based on the Y-axis and the origin. This is set as the grasping coordinate system. The origin of the grasping coordinate system is the grasping point. Generate a capture coordinate system graphic on the original image of the material sheet; The sheet material is a symmetrical irregularly shaped sheet material; The origin of the grasping coordinate system is regarded as the grasping point, and any point on the x-axis of the grasping coordinate system that is close to the origin is regarded as the attitude point. The pose of the grasping point in the robot base coordinate system is obtained, and then the robot is controlled to grasp the material based on the above pose.

2. The method according to claim 1, characterized in that, The hand-eye calibration of industrial cameras and industrial robots includes: Multiple images of a cellular calibration board are acquired from different directions using an industrial camera for hand-eye calibration. The calibration results, including camera intrinsic parameters, extrinsic parameters, and distortion coefficients, are obtained through nonlinear optimization. The hand-eye transformation matrix between the industrial camera and the industrial robot is then determined.

3. The method according to claim 1, characterized in that, The noise reduction process employs median filtering, mean filtering, or Gaussian filtering.

4. The method according to claim 1, characterized in that, The edge detection employs the Canny operator, Roberts operator, Sobel operator, or Laplace operator.

5. The method according to claim 1, characterized in that, Image template matching is performed using an improved gray wolf optimization algorithm, an image pyramid-based search algorithm, or a particle swarm optimization algorithm, and the optimal matching position is output.

6. The method according to claim 1, characterized in that, The minimum bounding rectangle of the material sheet is calculated using the rotating caliper method.

7. The method according to claim 1, characterized in that, The process of obtaining the pose of the gripping point in the robot's base coordinate system and then controlling the robot to grip the material piece based on the pose includes: By calculating the pixel positions of the grasping point and the attitude point in the image, and combining the depth information of the image, i.e. the actual distance from the image to the industrial camera, and the intrinsic parameters of the industrial camera, the three-dimensional coordinates of the grasping point and the attitude point in the camera coordinate system are calculated. The calculated 3D coordinates of the grasping point and the attitude point are converted into homogeneous coordinate matrices and multiplied with the hand-eye transformation matrix to obtain the 3D coordinates of these two points in the robot's base coordinate system. Extract the position of the grasping point in the base coordinate system, and calculate the angle between the target vector formed by the grasping point and the attitude point and the X-axis in the robot's base coordinate system, thereby obtaining the rotation angle of the robot's grasping action, i.e. the robot's attitude angle. The position and rotation angle of the gripping point are combined to obtain the complete pose information of the gripping point in the robot's base coordinate system; the pose information will be used to guide the industrial robot to perform gripping operations.

Images