Industrial robot material sheet grabbing implementation method based on visual positioning

Abstract

The invention discloses an industrial robot material sheet grabbing implementation method based on visual localization, and the grabbing implementation method comprises the steps: carrying out the hand-eye calibration of a robot; determining a fixed geometrical relationship between a robot-based coordinate system and an industrial camera, performing image acquisition on the material sheet through the industrial camera, performing noise reduction and edge detection preprocessing on the acquired original image of the material sheet, and obtaining an edge detection result through a template matching algorithm; the position of the material sheet is positioned in the original image, a grabbing coordinate system is established on the outline of the material sheet, a grabbing point is determined, the pose of the grabbing point under a robot base coordinate system is obtained, and the robot is controlled to conduct material sheet grabbing and paving work. The problem of low paving precision caused by inconsistent stability of supplied materials can be avoided to the greatest extent, and the precision and success rate of grabbing and paving planning are effectively improved.

Description

Technical Field

[0001] The invention relates to a method for realizing sheet grabbing of an industrial robot based on visual positioning, and belongs to the technical field of intelligent assembly of industrial robots. Background Art

[0002] With the rapid development of science and technology, especially the continuous progress of industrial robot technology, machine vision, sensor technology, and automatic control technology, industrial robots are increasingly used in various complex production scenarios. In industries such as composite product manufacturing, automobile manufacturing, and electronic manufacturing, accurate grabbing and laying of sheets is a crucial part of the production process. The traditional manual laying method is not only inefficient, but also difficult to ensure product consistency and quality stability. Therefore, the use of industrial robots for sheet grabbing and laying has become an effective way to solve this problem.

[0003] In the robot grabbing sheet for paving operations, there are many symmetrical shaped workpieces or sheets. Such shaped sheets can be grabbed from multiple angles, which are very suitable for grabbing with robot grippers or suction cups. When using robots to grab such items, machine vision technology is usually used for auxiliary positioning to determine the center of mass of the workpiece to be grabbed, and then determine the grabbing posture to guide the robot to perform the grabbing work.

[0004] The centroid-based grasping method has high requirements for workpiece consistency. However, in the actual production process, when the consistency of incoming materials is unstable, there is more or less deviation between the actual material specifications and the theoretical size of the sheet. If the paving work is only based on the centroid-based grasping method, there may be a problem that the upper edge of the sheet and the paving scale line cannot be aligned, affecting the progress of subsequent production work. In addition, the centroid-based grasping method has high requirements for the shape of the sheet. For sheets with large curvature, the centroid may exist outside the sheet. At this time, guiding the robot to grasp based on the centroid may result in unsuccessful grasping and other problems. Therefore, there is an urgent need for a visual inspection method that can flexibly and accurately identify sheets for positioning and guide the robot to grasp in order to improve the accuracy of robot grasping and the accuracy of paving. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and propose an industrial robot sheet grabbing implementation method based on visual positioning, by setting a grabbing coordinate system at a specific position of the sheet contour and using the origin of the grabbing coordinate system as the grabbing point, thereby improving the accuracy of the robot grabbing and laying the sheet.

[0006] The technical solution of the present invention is: a method for realizing sheet grabbing of an industrial robot based on visual positioning, comprising:

[0007] Perform hand-eye calibration on industrial cameras and industrial robots to determine the hand-eye transformation matrix between industrial cameras and industrial robots;

[0008] The original image of the sheet is obtained by collecting the image of the sheet through an industrial camera; at the same time, the sheet template image generated by the CAD software is obtained;

[0009] Performing noise reduction and edge detection on the original image of the sheet to obtain a denoised sheet image;

[0010] Performing image template matching on the denoised sheet image and the sheet template image to locate the sheet contour in the denoised sheet image;

[0011] Establishing a grabbing coordinate system on the outline of the web;

[0012] The origin of the grabbing coordinate system is regarded as the grabbing point, and any point close to the origin in the x-axis direction of the grabbing coordinate system is regarded as the posture point, the posture of the grabbing point in the robot base coordinate system is obtained, and then the robot is controlled to grab the sheet based on the above posture.

[0013] The hand-eye calibration of the industrial camera and the industrial robot includes:

[0014] An industrial camera is used to collect multiple images of a honeycomb calibration plate in different directions for hand-eye calibration, and a nonlinear optimized calibration result is obtained, including camera intrinsic parameters, extrinsic parameters and distortion coefficients, to determine the hand-eye transformation matrix between the industrial camera and the industrial robot.

[0015] The noise reduction process adopts median filtering, mean filtering or Gaussian filtering.

[0016] The edge detection adopts and applies Canny operator, Roberts operator, Sobel operator or Laplace operator.

[0017] The improved grey wolf optimization algorithm or the image pyramid-based search algorithm or the particle swarm algorithm is used to perform image template matching and output the best matching position.

[0018] The step of establishing a grabbing coordinate system on the outline of the sheet includes:

[0019] Calculate the minimum circumscribed rectangle of the sheet outline and obtain the position information of the rectangle;

[0020] Take the line connecting the center points of the two long sides of the rectangle as the Y axis of the grabbing coordinate system, and take the intersection of the Y axis and the top of the material outline as the origin of the grabbing coordinate system. Generate an X axis perpendicular to the Y axis on the material outline plane based on the Y axis and the origin, and set it as the grabbing coordinate system. The origin of the grabbing coordinate system is the grabbing point.

[0021] Generate a grabbing coordinate system graphic on the original image of the blank.

[0022] The rotating caliper method is used to calculate the minimum circumscribed rectangle of the sheet outline.

[0023] The obtaining of the position and posture of the grasping point in the robot base coordinate system, and then controlling the robot to grasp the sheet based on the position and posture, comprises:

[0024] By calculating the pixel positions of the grasping points and posture points in the image, combined with the image depth information, i.e. the actual distance from the image to the industrial camera and the internal parameters of the industrial camera, the three-dimensional coordinates of the grasping points and posture points in the camera coordinate system are calculated;

[0025] The calculated three-dimensional coordinates of the grasping point and the posture point are converted into a homogeneous coordinate matrix and multiplied with the hand-eye transformation matrix to obtain the three-dimensional coordinates of the two points in the robot base coordinate system;

[0026] Extract the position of the grasping point in the base coordinate system, and calculate the angle between the target vector formed by the grasping point and the posture point and the X-axis in the robot base coordinate system, and then obtain the rotation angle of the robot's grasping action, that is, the robot's posture angle;

[0027] The position and rotation angle, i.e., posture, of the grasping point are combined to obtain complete posture information of the grasping point in the robot base coordinate system; the posture information will be used to guide the industrial robot to perform grasping operations.

[0028] The material piece is a symmetrical special-shaped material piece.

[0029] The advantages of the present invention compared with the prior art are:

[0030] (1) The present invention sets a grabbing coordinate system at the intersection of the top of the sheet outline and one of the symmetry axes of its minimum circumscribed rectangle, and uses the origin of the grabbing coordinate system as the grabbing point, which can avoid the problem of low paving accuracy caused by inconsistent incoming material stability to the greatest extent, and effectively improve the accuracy of grabbing and paving sheets. Compared with the grabbing method using the center of mass as the grabbing point, the present invention has lower requirements on the consistency of workpiece size and shape, and has wider applicability.

[0031] (2) The present invention uses an improved gray wolf optimization algorithm for image templates. Compared with the traditional traversal search algorithm, it effectively saves computing costs, simplifies the complexity of the image matching search process, and improves the effectiveness in optimizing image matching problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a system flow chart for implementing the method;

[0033] Figure 2 It is a flow chart of the present invention based on the improved and optimized Grey Wolf Algorithm when performing template matching;

[0034] Figure 3 It is a schematic diagram of the original image and the generated grasping coordinate system image. DETAILED DESCRIPTION

[0035] like Figure 1 As shown, the process of the method of the present invention is as follows:

[0036] Step 1: Hand-eye calibration, using an industrial camera to collect multiple images of the honeycomb calibration plate in different directions for hand-eye calibration, obtaining nonlinear optimization calibration results, including camera intrinsic parameters, extrinsic parameters and distortion coefficients, and determining the hand-eye transformation matrix between the industrial camera and the industrial robot;

[0037] Step 2: Image acquisition: Use an industrial camera to acquire images of the sheet, and transfer the original image of the sheet into the visual recognition and positioning system, and transfer the DXF model data of the sheet template file into the visual recognition and positioning system;

[0038] Step 3: Image preprocessing: Use the image preprocessing module in the visual recognition and positioning system to perform noise reduction and edge detection on the original image of the sheet. Apply the median filter to perform noise reduction on the sheet image to remove noise and make the sheet features more obvious. Apply the Canny operator to perform edge detection on the sheet to obtain the sheet contour.

[0039] Step 4: Template matching: using the template matching module in the visual recognition and positioning system, the sheet outline and the DXF model data are matched with the image template using the improved gray wolf optimization algorithm, and the best matching position is output;

[0040] Step 5: Set the grabbing coordinate system, use the rotation method to calculate the minimum circumscribed rectangle of the sheet outline, obtain the position information of the rectangle, take the line connecting the center points of the two long sides of the rectangle as the Y axis of the grabbing coordinate system, and regard the intersection of the Y axis and the top of the sheet outline as the origin of the grabbing coordinate system. Generate an X axis perpendicular to the Y axis on the sheet outline plane based on the Y axis and the origin, and set it as the grabbing coordinate system, and generate the grabbing coordinate system graphics on the original sheet image;

[0041] Step 6: Coordinate transformation, regard the origin of the grasping coordinate system as the grasping point, and regard any point close to the origin in the x-axis direction of the grasping coordinate system as the posture point, obtain the position and posture of the grasping point in the robot base coordinate system, and guide the robot to perform grasping work.

[0042] The median filtering process in step 3 is specifically as follows:

[0043] The median filter processes the image by first sorting all the pixels in the filter window and finding their median values, and then using the median value to replace the pixel value at the center of the window. The formula is as follows:

[0044]

[0045] Among them, g(x, y) is the pixel value after median filtering, and f(i, j) is the pixel value before median filtering.

[0046] The concept of median filtering is actually a technology that can process nonlinear signals in noise based on in-depth research on sorting theory. The process of using it is to reasonably integrate and utilize the various parameters between digital images and digital sequences, and combine the characteristics of each parameter to select the average parameters of the surrounding points for replacement. Ensure that the surrounding pixels are close to the real parameters, so that the individual noise points can be removed. For all the pixels in the image, take the intermediate parameters of each pixel in the filter window to replace the pixel parameters of this point. Median filtering can effectively remove noise in the image while keeping the image edges clear.

[0047] In step 3, the specific operation steps of applying the canny operator to perform edge detection on the sheet are as follows:

[0048] The Canny operator is one of the most commonly used operators for image edge detection, with good performance and robustness. The Canny operator first performs a Gaussian filter on the image to reduce the impact of noise. The specific calculation formula is as follows:

[0049]

[0050] Then, the first-order partial derivatives of the image in the horizontal x and vertical y directions are taken, and then the gradient amplitude and direction angle of each pixel are calculated. The specific calculation formula is as follows:

[0051]

[0052] θ(x, y) = arctan(I y / I x ) In the above formula, M(x, y) is the gradient amplitude of the pixel point, and θ(x, y) is the direction of the pixel point. The gradient amplitude indicates the degree of color change of the pixel point, and the gradient direction indicates the direction of color change. Non-maximum suppression and double thresholds are used to detect the edge of the image. By scanning the image and retaining the pixels with the largest local gradient, a refined edge image can be obtained.

[0053] See also Figure 2 As shown, it is a basic flow chart based on the improved optimized grey wolf algorithm.

[0054] In the above step 4, the fitness function based on the improved grey wolf optimization algorithm uses the maximum cross-correlation algorithm, and its formula is as follows:

[0055]

[0056] Among them, X ij and Y i+u，j+v The expressions are as follows, where x and y are the original pixel values ​​of the template sub-image and the reference image, respectively, and (u, v) is the position coordinate of the upper left corner of the template sub-image in the reference image.

[0057]

[0058] The maximum cross-correlation method has the advantages of high accuracy, strong robustness, and a wide range of applications. Therefore, the present invention uses it as a fitness function to determine the performance of each gray wolf search individual (sub-graph). When R (u, v) approaches 1, the optimal solution can be obtained. In this way, the image matching problem can be understood as n gray wolf search individuals (candidate sub-graphs) distributed in a search space (reference graph). The optimized gray wolf algorithm is used as a new search strategy to search for the sub-graph position with R (u, v) closest to 1 in the reference graph to obtain the optimal matching position. At this time, the closer the fitness function value is to 1, the better the performance of the search individual, that is, the closer the position of the search sub-graph is to the template sub-graph.

[0059] The specific steps of image template matching based on the improved gray wolf optimization algorithm in step 4 above are as follows:

[0060] (1) Input all the information of the original image, i.e., the reference image and the template image.

[0061] (2) Set the algorithm parameters. Set the maximum number of iterations and the population range to fixed values.

[0062] (3) Initialization. Randomly generate n gray wolf individuals in the entire reference image (search area), that is, there are n sub-images, and generate random gray wolf position initial values, whose position range is a random integer in [1, M-m+1] (the sizes of the reference image and template sub-image are M×N and m×n respectively). Calculate the fitness value of each gray wolf individual and use the optimal position as The second best position is The third best position Other search individual positions are represented by

[0063] (4) Through the fitness function, the maximum cross-correlation value of each individual is calculated and the optimal solution, the second optimal solution and the third optimal solution are obtained.

[0064] (5) Update the gray wolf individual and proceed to the next iteration. If the termination condition of the algorithm is not met, that is, the number of cycles is less than the maximum number of iterations, and if the best match has not been reached at this time, return to step (3), otherwise the global optimal value will be output and the search ends.

[0065] After comparative experiments, the average values ​​of the running time, matching iteration times, and matching accuracy obtained based on the improved gray wolf optimization algorithm and the traditional traversal search algorithm in the experiment were calculated and compared, as shown in the following table:

[0066] Traditional traversal search algorithm Improved Gray Wolf Optimization Algorithm Matching time(s) 28.4631 3.1792 Matching accuracy 100％ 96.85％ Maximum number of iterations - 80 Total number of cycles - 139.2

[0067] From the above comparative experiments, it can be concluded that the search strategy using the optimized gray wolf algorithm for image template matching effectively saves its computational cost, simplifies the complexity of the image matching search process, and improves the effectiveness in optimizing image matching problems.

[0068] The core idea of ​​using the rotation method to find the minimum enclosing rectangle in step 5 above is to select a point on the figure, calculate the minimum enclosing rectangle of the figure within a range of 90° rotation around the point, record the rotation angle, and after the image returns to its original position, the enclosing rectangle at this angle is the minimum enclosing rectangle. The specific process is as follows:

[0069] (1) Calculate the minimum circumscribed rectangle of the figure before rotation and record its area as S min , record the current rotation angle α. In the initial state, the default α=0, and the recorded rotation angle is β.

[0070] (2) Select a point in the image, rotate the image by angle θ, record the minimum positive circumscribed rectangle area S at this time, record the current rotation angle α, at which α = α + θ, and compare S with S min , assign the smaller value to S min , and β records the rotation angle when the S value is smaller.

[0071] (3) Continue to rotate by angle θ and repeat step (2) until α = 90°. Record S mtn The minimum value of and the obtained β value.

[0072] (4) After the figure returns to its original position, the recorded minimum enclosing rectangle is rotated in the opposite direction around the selected point by β degrees to obtain the minimum enclosing rectangle of the figure.

[0073] like Figure 3 FIG. 1 shows the process of establishing a grabbing coordinate system, wherein the shape of the blank is a symmetrical, shaped blank with curvature.

[0074] The specific steps for generating the grabbing coordinate system on the original image of the sheet in step 5 above are as follows:

[0075] (1) Calculate the minimum circumscribed rectangle of the sheet outline using the rotation method and obtain the position information of the rectangle;

[0076] (2) Take the line connecting the center points of the two long sides of the rectangle as the Y axis of the grabbing coordinate system, and take the intersection of the Y axis and the top of the sheet outline as the origin of the grabbing coordinate system. Generate an X axis perpendicular to the Y axis on the sheet outline plane based on the Y axis and the origin, and set it as the grabbing coordinate system;

[0077] (3) Generate a grabbing coordinate system graphic on the original image of the sheet.

[0078] The specific steps to obtain the position and posture of the grasping point in the robot base coordinate system in step 6 above are as follows:

[0079] (1) Read the hand-eye calibration data and obtain the transformation matrix from the camera coordinate system to the robot base coordinate system;

[0080] (2) Read the camera's internal parameter data to obtain the camera's focal length, distortion coefficient and other internal parameter data;

[0081] (3) The origin of the grasping coordinate system is regarded as the grasping point, and any point close to the origin in the x-axis direction of the grasping coordinate system is regarded as the posture point;

[0082] (4) According to the pixel positions of the grasping points and the posture points, the three-dimensional coordinates of the grasping points and the posture points in the camera coordinate system are calculated by calculating the image depth information, i.e., the actual distance from the image to the camera and the camera intrinsic parameters;

[0083] (5) Convert the coordinates of the grasping point and the posture point in the camera coordinate system into a homogeneous coordinate matrix and multiply them by the hand-eye transformation matrix to obtain the coordinate information in the robot base coordinate system;

[0084] (6) Extract the position of the grasping point in the robot base coordinate system;

[0085] (7) Calculate the angle between the target vector of the grasping point and the posture point in the robot base coordinate system and the X-axis in the robot base coordinate system, determine the rotation angle, and obtain the posture angle of the robot;

[0086] (8) Combine the position and posture of the grasping point to obtain the complete posture of the grasping point in the robot base coordinate system and guide the robot to perform grasping work.

[0087] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention by using the technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for realizing sheet grabbing by an industrial robot based on visual positioning, characterized in that: include: Perform hand-eye calibration on industrial cameras and industrial robots to determine the hand-eye transformation matrix between industrial cameras and industrial robots; The original image of the sheet is obtained by collecting the image of the sheet through an industrial camera; at the same time, the sheet template image generated by the CAD software is obtained; Performing noise reduction and edge detection on the original image of the sheet to obtain a denoised sheet image; Performing image template matching on the denoised sheet image and the sheet template image to locate the sheet contour in the denoised sheet image; Establishing a grabbing coordinate system on the outline of the web; The origin of the grabbing coordinate system is regarded as the grabbing point, and any point close to the origin in the x-axis direction of the grabbing coordinate system is regarded as the posture point, the posture of the grabbing point in the robot base coordinate system is obtained, and then the robot is controlled to grab the sheet based on the above posture.

2. The method according to claim 1, characterized in that: The hand-eye calibration of the industrial camera and the industrial robot includes: An industrial camera is used to collect multiple images of a honeycomb calibration plate in different directions for hand-eye calibration, and a nonlinear optimized calibration result is obtained, including camera intrinsic parameters, extrinsic parameters and distortion coefficients, to determine the hand-eye transformation matrix between the industrial camera and the industrial robot.

3. The method according to claim 1, characterized in that The noise reduction process adopts median filtering, mean filtering or Gaussian filtering.

4. The method according to claim 1, characterized in that The edge detection adopts and applies Canny operator, Roberts operator, Sobel operator or Laplace operator.

5. The method according to claim 1, characterized in that The improved grey wolf optimization algorithm or the image pyramid-based search algorithm or the particle swarm algorithm is used to perform image template matching and output the best matching position.

6. The method according to claim 1, characterized in that The step of establishing a grabbing coordinate system on the outline of the sheet includes: Calculate the minimum circumscribed rectangle of the sheet outline and obtain the position information of the rectangle; Take the line connecting the center points of the two long sides of the rectangle as the Y axis of the grabbing coordinate system, and take the intersection of the Y axis and the top of the material outline as the origin of the grabbing coordinate system. Generate an X axis perpendicular to the Y axis on the material outline plane based on the Y axis and the origin, and set it as the grabbing coordinate system. The origin of the grabbing coordinate system is the grabbing point. Generate a grabbing coordinate system graphic on the original image of the blank.

7. The method according to claim 6, characterized in that The rotating caliper method is used to calculate the minimum circumscribed rectangle of the sheet outline.

8. The method according to claim 6, characterized in that The obtaining of the position and posture of the grasping point in the robot base coordinate system, and then controlling the robot to grasp the sheet based on the position and posture, comprises: By calculating the pixel positions of the grasping points and posture points in the image, combined with the image depth information, i.e. the actual distance from the image to the industrial camera and the internal parameters of the industrial camera, the three-dimensional coordinates of the grasping points and posture points in the camera coordinate system are calculated; The calculated three-dimensional coordinates of the grasping point and the posture point are converted into a homogeneous coordinate matrix and multiplied with the hand-eye transformation matrix to obtain the three-dimensional coordinates of the two points in the robot base coordinate system; Extract the position of the grasping point in the base coordinate system, and calculate the angle between the target vector formed by the grasping point and the posture point and the X-axis in the robot base coordinate system, and then obtain the rotation angle of the robot's grasping action, that is, the robot's posture angle; The position and rotation angle, i.e., posture, of the grasping point are combined to obtain complete posture information of the grasping point in the robot base coordinate system; the posture information will be used to guide the industrial robot to perform grasping operations.

9. The method according to any one of claims 1 to 8, wherein the tablet is a symmetrical special-shaped tablet.

Images