Robot grasping control method based on image recognition

Abstract

The application discloses a robot grabbing control method based on image recognition. The workpiece images under multiple angles and multiple light conditions are collected through a depth camera, and algorithms such as Canny edge detection and Hough transformation are combined to extract local features such as contour, connected domain and connecting line and overall features such as line density, and the weights of the features are determined; the similarity between the image to be recognized and the pre-stored workpiece image is calculated by using an improved similar distance to determine the workpiece type, and precise grabbing is realized. The application improves the accuracy, reliability and adaptability of robot grabbing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot motion control, and particularly relates to a robot grasping control method based on image recognition. BACKGROUND

[0002] In modern industrial production, robots are increasingly widely used. However, when facing stacked workpieces, the robot often cannot accurately identify the type of the workpiece and determine the position of the exposed part of the workpiece, which seriously affects the production efficiency and the degree of automation. Traditional image recognition methods have many limitations when dealing with complex workpiece scenes, such as relying on a single feature for recognition, resulting in low accuracy and reliability, which is difficult to meet the actual production needs. SUMMARY

[0003] The present application provides a robot grasping control method based on image recognition, characterized in that it comprises the following steps:

[0004] Image acquisition of the grasping object under multiple angles and multiple lighting conditions, and transmitting the acquired images to an image processing unit;

[0005] In the image processing unit, after pre-processing the images, local features and overall features of the workpiece images are extracted, the local features include contour features, connected domain features and connection line features, and the overall features include line density features; the extracted feature parameters are normalized, and the weights of different features are determined to obtain a weighted feature vector;

[0006] The feature vector of the image to be recognized is compared with the feature vectors of various pre-stored workpiece types, the similarity distance is calculated combined with the feature vectors, and the workpiece type is determined;

[0007] According to the matched workpiece type, the best grasping posture and path of the robot are calculated through inverse kinematics algorithm;

[0008] The robot is controlled to move according to the calculated grasping posture and path to realize grasping of the workpiece.

[0009] Specifically, the contour feature extraction adopts Canny edge detection algorithm, and the Hu invariant moment is calculated for the extracted contour, and the calculation formula of the Hu invariant moment is:

[0010] Where (xi, yi) is the coordinate of a point on the contour, i = 1, 2, …, n, n is the number of contour points, p, q are used to calculate the order of the moment, and take values of 0, 1, 2, …, μ pq is the p + q order central moment, and η pqare the normalized central moments, and φ1, φ2, φ3, … are Hu moments.

[0011] Specifically, the line density feature calculation is to divide the image into M*N small regions, each region has a size of w*h, perform edge detection on each region, and count the number of edge pixels n ij , where i = 1, 2, …, M, j = 1, 2, …, N, calculate the line density D ij of each region ij = n / (w*h).

[0012] Specifically, the determination of the feature weight includes constructing a judgment matrix, calculating the maximum eigenvalue and the corresponding eigenvector of the judgment matrix, and normalizing the eigenvector to obtain the weight vector of each feature.

[0013] Specifically, the similarity distance is calculated in combination with the eigenvector, specifically: let the eigenvector of the image to be identified be X = [x1, x2, …, x n ], the eigenvector of the pre-stored workpiece image be Y = [y1, y2, …, y m ], and a similarity distance matrix D be constructed, where D(i,j) represents the distance between x i and y j , and the shortest path from D(1,1) to D(n,m) is found, and the cumulative distance of the path is the similarity distance DT(X,Y).

[0014] Specifically, a weighting coefficient α is introduced to adjust the similarity distance, specifically: , where w i is the weight of feature i, and corresponds to the corresponding feature in the weight vector of each feature, and α is taken in the range of [0,1].

[0015] Specifically, the connected domain feature extraction includes performing adaptive threshold binarization processing on the image, marking each connected domain of the workpiece using a connected domain marking algorithm, and calculating the area and / or perimeter and / or circularity and / or rectangularity and / or face depth ratio of the maximum connected domain.

[0016] Specifically, the connection line feature extraction is to detect straight line segments in the image through Hough transformation, and record the slope, intercept and length parameters of the straight line.

[0017] Specifically, when the exposed part of the workpiece is not convenient for direct grabbing, the workpiece is moved by the manipulator, and then image acquisition and identification are performed again until the appropriate grabbing position is determined.

[0018] Specifically, the image preprocessing includes denoising and grayscale operation.

[0019] Beneficial Technical Effects: This invention comprehensively utilizes multiple local and global features for workpiece recognition. Compared to traditional single-feature recognition methods, it improves the accuracy and reliability of recognition and can better handle complex and varied workpiece shapes and textures. By setting feature weights, it fully considers the differences in importance of different features in workpiece recognition, making the recognition results more in line with actual needs. Furthermore, the weight allocation is flexible and can be adjusted according to different workpiece types and application scenarios. The similarity distance calculation is improved by introducing weighted coefficients and feature vector difference terms. While considering the overall similarity of the feature sequence, it also takes into account the degree of difference of individual features, further improving the accuracy of image comparison and enhancing the ability to distinguish similar workpieces. Attached Figure Description

[0020] Appendix Figure 1 This is a flowchart of the robot grasping control method based on image recognition according to the present invention;

[0021] Appendix Figure 2 This is a schematic diagram of a robot grasping operation based on image recognition. Detailed Implementation

[0022] like Figure 1 As shown, this invention provides a robot grasping control method based on image recognition, the specific steps of which include:

[0023] 1. Initialization: Pre-store workpiece data and calibrate robot parameters.

[0024] The image processing unit pre-stores feature vectors and three-dimensional geometric models of various workpiece types. These data can be obtained by acquiring and analyzing images of a large number of workpiece samples, and then processed and stored using the aforementioned feature extraction and weight allocation methods.

[0025] Perform parameter calibration on the robot, including setting parameters such as joint angle range, movement speed, and grasping force, to ensure that the robot can accurately perform grasping tasks.

[0026] like Figure 2As shown, the end effector of the robot can select a gripper for grabbing the workpiece. The parameters of the gripper such as shape, size, opening and closing mode, and clamping force are designed according to different grabbing tasks and workpiece characteristics. For example, for small precision parts, small parallel grippers with high precision opening and closing control and small clamping force are used to avoid damaging the workpiece; for large heavy objects, hydraulic grippers or pneumatic grippers with large clamping force and opening range are used. The workpiece refers to various objects that need to be grabbed by the robot and subsequent operations in the industrial production or logistics process, for example, in a machining workshop, the workpiece may be various metal parts such as bolts, nuts, gears, etc.; in an electronic manufacturing workshop, the workpiece may be small electronic components such as circuit boards, chips, etc.; in a logistics warehouse, the workpiece may be various packaged goods or cargo. Considering the diversity of workpieces in actual scenarios, the material and shape characteristics of the workpiece can be pre-analyzed before operation.

[0027] 2. Image acquisition and processing.

[0028] A depth camera is used to acquire images of the stacked workpieces under predetermined multiple angles and multiple lighting conditions, and the acquired images are transmitted in real time to an image processing unit. The image acquisition angles include at least three angles, and are spaced apart by more than 60°, and the lighting conditions can include strong light, weak light, and natural light, to ensure that clear and complete workpiece image information is obtained.

[0029] In the image processing unit, the image is first preprocessed, including denoising, grayscale operation, etc., then the local features and overall features of the workpiece image are extracted according to the above feature extraction method, and the feature parameters are normalized and weighted to obtain the weighted feature vector.

[0030] The local features include contour, connected domain, and connection line features, wherein the local features are extracted as follows:

[0031] Contour feature extraction: Canny edge detection algorithm is used to detect the edges of the workpiece image to obtain the gradient amplitude and direction information of the image, and the contour information of the workpiece is obtained through non-maximum suppression and double threshold detection. For the extracted contour, the Hu moment is calculated, and the contour point set is ((x i ,y i ), i = 1,2,……,n, and the Hu moment calculation formula is as follows:

[0032] , wherein (xi,yi): represents the coordinates of the points on the contour, i = 1,2,……,n, n is the number of contour points, p,q: the order for calculating the moment, taking values of 0,1,2,……, used to construct moments of different orders to describe the shape features of the contour, μ pq: p + q order central moments, obtained by specific operation on the contour point coordinates, reflecting the distribution of the contour relative to its barycenter, η pq : normalized central moments; φ1, φ2, φ3, …: Hu invariant moments, obtained by specific combination, having rotation, translation and scale invariance, used to describe the shape features of the contour, different φ values reflecting the shape characteristics of the contour in different aspects. pq : normalized central moments; φ1, φ2, φ3, …: Hu invariant moments, obtained by specific combination, having rotation, translation and scale invariance, used to describe the shape features of the contour, different φ values reflecting the shape characteristics of the contour in different aspects.

[0033] Connected domain feature extraction: the image is binarized, and the adaptive threshold method is used to determine the threshold according to the local gray level characteristics of the image, and the image is converted into a binary image. The connected domain marking algorithm (such as the connected domain marking algorithm based on scan line) is used to mark each connected domain of the workpiece, and the area S (the number of pixel points in the connected domain), the perimeter P (the number of pixel points on the boundary of the connected domain calculated by the contour tracking algorithm), the circularity C = 4πS / P 2 (the degree of similarity between the connected domain and the circle, the closer the closer to the circle), the rectangularity R = S / S rect (the area of the smallest rectangle containing the connected domain, the rectangularity is used to describe the similarity between the connected domain and the rectangle), the maximum connected domain face depth ratio and other features. rect : normalized central moments; φ1, φ2, φ3, …: Hu invariant moments, obtained by specific combination, having rotation, translation and scale invariance, used to describe the shape features of the contour, different φ values reflecting the shape characteristics of the contour in different aspects.

[0034] : normalized central moments; φ1, φ2, φ3, …: Hu invariant moments, obtained by specific combination, having rotation, translation and scale invariance, used to describe the shape features of the contour, different φ values reflecting the shape characteristics of the contour in different aspects.

[0035] Connection line feature extraction: straight line segments in the image are detected by Hough transform, which maps straight lines in image space to parameter space, and the parameters of the straight line are determined by finding peaks in the parameter space. For the detected straight line, record its slope k = tan θ, intercept b = ρ / sin θ, and length L (calculated according to the coordinates of the two endpoints of the straight line).

[0036] Global features include but are not limited to line density features, which are extracted as follows: divide the image into M*N small regions, each with a size of w*h. Perform edge detection (such as Sobel operator) on each region, and count the number of edge pixels n ij (i = 1,2, …, M, j = 1,2, …, N), calculate the line density D ij of each region ij =n norm / (w*h).

[0037] Feature normalization and weight assignment: normalize the extracted feature parameters so that their numerical range is between [0, 1], using the linear normalization method x min =(x – x max ) / (x min -x min ), where x is the original feature value, x max and x ij are the minimum and maximum values of the feature in all samples, respectively. The following will take four features as an example to introduce how to determine the weight of different features. First, construct the judgment matrix A:

[0038] where a max represents the importance of feature i relative to feature j. Then calculate the maximum eigenvalue λ ij and the corresponding eigenvector of the judgment matrix, normalize the eigenvector to obtain the weight vector W = [w1, w2, w3, w4] of each feature. The specific calculation steps are as follows:

[0039]

[0040] where i = 1,2, …, n, n is the number of features, and for this example, n = 4.

[0041] Perform consistency check, calculate consistency index CI , where is the i-th element of the vector . According to the order n of the judgment matrix, determine the average random consistency index RI, calculate the consistency ratio CR = CI / RI, when CR < 0.1, consider the judgment matrix to have satisfactory consistency, otherwise need to adjust the judgment matrix.

[0042] 3. Image comparison and similarity distance calculation.

[0043] The feature vector of the current acquired image to be identified is compared with the feature vectors of various pre-stored workpiece types, and the similarity distance is calculated. When the similarity distance is less than the set distance threshold, it is determined that the image to be identified matches a certain pre-stored workpiece type successfully, thereby identifying the workpiece type. Let the feature vector of the image to be identified be X = [x1, x2, …, xn], and the feature vector of the pre-stored workpiece image be Y = [y1, y2, …, yn], and a similarity distance matrix D is constructed, where D(i, j) represents the distance between xi and yj. The shortest path from D(1, 1) to D(n, m) is found, and the cumulative distance of the path is the similarity distance DT(X, Y). n m i j

[0044] In order to further improve the accuracy and adaptability of distance calculation, a weighting coefficient a is introduced to adjust the similarity distance: where w i is the weight of feature i, corresponding to the relative weight in the weight vector calculated above, used to consider the importance difference of different features in distance calculation, and a is taken in the range of [0, 1] according to actual situation. Set the similarity distance threshold T, when DT α is less than T, it is considered that the image to be identified matches a certain pre-stored workpiece type successfully, and the workpiece type is determined.

[0045] 4. Grasping position determination and adjustment.

[0046] According to the identified workpiece type, its pre-stored three-dimensional geometric model and feature information are retrieved, and the actual state of the workpiece in the current image is combined to analyze the exposed part position of the workpiece in the stacking state.

[0047] The optimal grasping pose and path of the robot are calculated through inverse kinematics algorithm, if it is found that the exposed part position of the workpiece is not convenient for direct grasping, the robot is used to move the workpiece appropriately, such as pushing, pulling, etc., the position of the workpiece is adjusted, and then image acquisition and identification are performed again until the appropriate grasping position is determined.

[0048] 5. Grasping operation execution and feedback.

[0049] The robot is controlled to move according to the calculated grasping pose and path, and precise motion commands are sent to the drive of the robot through the robot control system to realize accurate grasping of the workpiece.

[0050] ​​​​During the grabbing process, the grabbing state is monitored in real time through visual feedback or sensor feedback (such as a force sensor) to ensure the stability and reliability of the grabbing action. After successful grabbing, the workpiece is transported to the designated position, and the workpiece is again confirmed to be placed in place through vision or sensor.

[0051] Through the above embodiments, the present application can effectively realize the accurate grabbing of the stacked workpiece by the robot based on image recognition, and improve the automation level and production efficiency of industrial production.

Claims

1. A robot grasping control method based on image recognition, characterized in that, Includes the following steps: The system acquires images of the target object from multiple angles and under various lighting conditions, and then transmits the acquired images to the image processing unit. In the image processing unit, after preprocessing the image, local and global features of the workpiece image are extracted. The local features include contour features, connected component features, and connecting line features, and the global features include line density features. The extracted feature parameters are normalized, and the weights of different features are determined to obtain a weighted feature vector. The contour feature extraction uses the Canny edge detection algorithm, and the Hu invariant moments are calculated for the extracted contours. The formula for calculating the Hu invariant moments is as follows: Where (xi,yi) are the coordinates of a point on the contour, i = 1,2,...,n, where n is the number of contour points, and p,q are used to calculate the order of the moments, taking values ​​of 0,1,2,...,μ. pq Let η be the central moment of order p + q. pq φ1, φ2, φ3, ... are the normalized central moments, and φ1, φ2, φ3, ... are Hu invariant moments; The line density feature calculation involves dividing the image into M*N small regions, each with a size of w*h, performing edge detection on each region, and counting the number of edge pixels n. ij Let i = 1,2,……,M, j = 1,2,……,N, and calculate the linear density D for each region. ij =n ij / (w*h); The feature vector of the image to be identified is compared with the feature vectors of various pre-stored workpiece types. The similarity distance is calculated by combining the feature vectors to determine the workpiece type. Specifically, the similarity distance is calculated by combining the feature vectors: Let the feature vector of the image to be identified be X=[x1,x2,……,x…]. n The feature vector of the pre-stored workpiece image is Y=[y1,y2,……,y m Construct a similarity distance matrix D, where D(i,j) represents x. i With y j The similarity distance DT(X,Y) is calculated by finding the shortest path from D(1,1) to D(n,m). The cumulative distance of this path is the similarity distance. A weighting coefficient α is introduced to adjust the similarity distance, specifically: , where w i α represents the weight of feature i, corresponding to the weight vector of each feature, and takes values ​​in the range [0,1]. Based on the type of workpiece that is successfully matched, the optimal grasping posture and path of the robot are calculated using inverse kinematics algorithms. The robot is controlled to grasp the workpiece according to the calculated grasping posture and path.

2. The method according to claim 1, characterized in that, Determining feature weights involves constructing a judgment matrix, calculating the maximum eigenvalue and the corresponding eigenvector of the judgment matrix, and normalizing the eigenvectors to obtain the weight vectors for each feature.

3. The method according to claim 1, characterized in that, The connected component feature extraction includes performing adaptive thresholding binarization on the image, marking each connected component of the workpiece using a connected component labeling algorithm, and calculating the area and / or perimeter and / or circularity and / or rectangularity and / or the surface-to-depth ratio of the largest connected component.

4. The method according to claim 1, characterized in that, The feature extraction of the connecting line involves detecting straight line segments in the image using Hough transform and recording the slope, intercept, and length parameters of the straight line.

5. The method according to claim 1, characterized in that, When the exposed part of the workpiece is not convenient to be grasped directly, the workpiece is moved by a robot and then the image is acquired and recognized again until a suitable grasping position is determined.

6. The method according to claim 1, characterized in that, Image preprocessing includes denoising and grayscale conversion.

Images