Robot grabbing control method based on image recognition

Abstract

The invention discloses a robot grabbing control method based on image recognition. Workpiece images under multi-angle and multi-illumination conditions are collected through a depth camera, local features such as contours, connected domains and connecting lines and linear density overall features are extracted in combination with algorithms such as Canny edge detection and Hough transformation, the weight of each feature is determined, and the similarity between an image to be recognized and a pre-stored workpiece image is calculated in combination with an improved similarity distance to determine the type of a workpiece. And accurate grabbing is achieved. The grabbing accuracy, reliability and adaptability of the robot are improved.

Description

Technical Field

[0001] The invention relates to the field of robot motion control, and in particular to a robot grasping control method based on image recognition. Background Art

[0002] In modern industrial production, robots are increasingly used. However, when faced with piled workpieces, robots are often unable to accurately identify the type of workpieces and determine the location of the exposed parts, which seriously affects production efficiency and automation. Traditional image recognition methods have many limitations when dealing with complex workpiece scenes. For example, they rely only on a single feature for recognition, resulting in low accuracy and reliability, making it difficult to meet actual production needs. Summary of the invention

[0003] The present invention provides a robot grasping control method based on image recognition, which is characterized by comprising the following steps: Capture images of the grasped object at multiple angles and under multiple lighting conditions, and transmit the captured images to the image processing unit; In the image processing unit, after preprocessing the image, local features and overall features of the workpiece image are extracted, the local features include contour features, connected domain features and connecting line features, and the overall features include line density features; the extracted feature parameters are normalized, and the weights of different features are determined to obtain a weighted feature vector; Compare the feature vector of the image to be identified with the pre-stored feature vectors of various workpiece types, calculate the similarity distance based on the feature vectors, and determine the workpiece type; According to the type of workpiece that is successfully matched, the robot's optimal grasping posture and path are calculated through the inverse kinematics algorithm; The robot is controlled to move according to the calculated grasping posture and path to grasp the workpiece.

[0004] Specifically, the contour feature extraction uses the Canny edge detection algorithm to calculate the Hu invariant moment of the extracted contour, where the calculation formula of the Hu invariant moment is: , Where (xi,yi) are the coordinates of the points on the contour, i = 1,2,...,n, n is the number of contour points, p,q are used to calculate the order of the moment, and the values ​​are 0,1,2,...,μ pq is the p + q order central moment, η pq is the normalized central moment, φ1, φ2, φ3, … are Hu invariant moments.

[0005] Specifically, the line density feature calculation is to divide the image into M*N small areas, each area size is w*h, perform edge detection on each area, and count the number of edge pixels n ij , where i = 1,2,…,M, j = 1,2,…,N, calculate the line density D of each area ij =n ij / (w*h).

[0006] Specifically, determining the feature weight includes constructing a judgment matrix, calculating the maximum eigenvalue of the judgment matrix and the corresponding eigenvector, and normalizing the eigenvector to obtain the weight vector of each feature.

[0007] Specifically, the similarity distance is calculated in combination with the feature vector, specifically: suppose the feature vector of the image to be identified is X=[x1,x2,……,x n ], the feature vector of the pre-stored workpiece image is Y=[y1,y2,……,y m ], construct a similarity distance matrix D, where D(i,j) represents x i With y j The distance between them is used to find the shortest path from D(1,1) to D(n,m). The cumulative distance of this path is the similarity distance DT(X,Y).

[0008] Specifically, the weighting coefficient α is introduced to adjust the similarity distance, which is: , where w i is the weight of feature i, corresponding to the weight vector of each feature, and α takes values ​​in the range of [0,1].

[0009] Specifically, the connected domain feature extraction includes performing adaptive threshold method binarization processing on the image, marking each connected domain of the workpiece using a connected domain marking algorithm, and calculating the area and / or perimeter and / or circularity and / or rectangularity and / or the surface-to-depth ratio of the largest connected domain.

[0010] Specifically, the connecting line feature extraction is to detect the straight line segment in the image through Hough transform, and record the slope, intercept and length parameters of the straight line.

[0011] Specifically, when a part of the workpiece is exposed and is not convenient for direct grasping, the workpiece is moved by a manipulator and image acquisition and recognition are performed again until a suitable grasping position is determined.

[0012] Specifically, image preprocessing includes denoising and grayscale operations.

[0013] Beneficial technical effects: The present invention uses a variety of local and overall features to identify workpieces. Compared with traditional single feature recognition methods, it improves the accuracy and reliability of recognition and can better cope with complex and changeable situations such as workpiece shapes and textures. By setting feature weights, the importance differences of different features in workpiece recognition are fully considered, so that the recognition results are more in line with actual needs, and the weight distribution is flexible and can be adjusted according to different workpiece types and application scenarios. The similarity distance calculation is improved, and weighting coefficients and feature vector difference terms are introduced on this basis. While considering the overall similarity of the feature sequence, the degree of difference of a single feature is also taken into account, further improving the accuracy of image comparison and enhancing the ability to distinguish similar workpieces. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Attached Figure 1 is a flow chart of a robot grasping control method based on image recognition of the present invention; Attached Figure 2 Schematic diagram of robot grasping operation based on image recognition. DETAILED DESCRIPTION

[0015] like Figure 1 As shown, the present invention provides a robot grasping control method based on image recognition, and its specific steps include: 1. Initialization: Pre-store workpiece data and calibrate robot parameters.

[0016] Feature vectors and three-dimensional geometric models of various workpiece types are pre-stored in the image processing unit. These data can be obtained by collecting and analyzing images of a large number of workpiece samples, and are processed and stored through the above-mentioned feature extraction and weight distribution methods.

[0017] Calibrate the robot parameters, including setting parameters such as joint angle range, movement speed, and grasping force, to ensure that the robot can accurately perform grasping tasks.

[0018] like Figure 2As shown in the figure, the end effector of the robot can select a gripper to grasp the workpiece. The shape, size, opening and closing mode, and clamping force of the gripper are designed according to different grasping tasks and workpiece characteristics. For example, for small precision parts, a small parallel gripper with high-precision opening and closing control and a small clamping force is used to avoid damage to the workpiece; for large heavy objects, a hydraulic gripper or a pneumatic gripper with a large clamping force and an opening range is used. Workpieces refer to various objects that need to be grasped and subsequently operated by robots in industrial production or logistics processes. For example, in a machining workshop, workpieces may be various metal parts such as bolts, nuts, gears, etc.; in an electronic manufacturing workshop, workpieces may be small electronic components such as circuit boards and chips; in a logistics warehouse, workpieces may be various packaged goods or cargoes. Considering the diversity of workpieces in actual scenarios, the material and shape characteristics of the workpiece can be pre-analyzed before operation.

[0019] 2. Image acquisition and processing.

[0020] The depth camera is used to collect images of the stacked workpieces according to the predetermined multi-angle and multi-lighting conditions, and the collected images are transmitted to the image processing unit in real time. The image collection angles include at least three angles, with an interval of more than 60°. The lighting conditions can include strong light, weak light, and natural light to ensure that clear and complete workpiece image information is obtained.

[0021] In the image processing unit, the image is first preprocessed, including denoising, grayscale and other operations, and then the local features and overall features of the workpiece image are extracted respectively according to the above-mentioned feature extraction method, and the feature parameters are normalized and weighted to obtain a weighted feature vector.

[0022] Local features include contours, connected domains, and connecting line features. The extraction of local features is as follows: Contour feature extraction: Use the Canny edge detection algorithm to detect the edge of the workpiece image, obtain the gradient amplitude and direction information of the image, and obtain the contour information of the workpiece through non-maximum suppression and double threshold detection. For the extracted contour, calculate its Hu invariant moment, and set the contour point set as ((x i ,y i ), i = 1,2,……,n, the Hu invariant moment calculation formula is as follows: , Among them, (xi,yi): represents the coordinates of the points on the contour, i = 1,2,...,n, n is the number of contour points, p,q: used to calculate the order of the moment, the value is 0,1,2,..., used to construct moments of different orders to describe the shape characteristics of the contour, μ pq: p + q order central moment, obtained by specific operations on the coordinates of the contour points, reflects the distribution of the contour relative to its center of gravity, η pq is the normalized central moment; φ1, φ2, φ3, ...: Hu invariant moments, which are given by η pq It is calculated through a specific combination and has rotation, translation and scale invariance. It is used to describe the shape characteristics of the contour. Different φ values ​​reflect the shape characteristics of the contour in different aspects.

[0023] Connected domain feature extraction: Binarize the image, use the adaptive threshold method to determine the threshold according to the local grayscale characteristics of the image, and convert the image into a binary image. Use a connected domain marking algorithm (such as a scan line-based connected domain marking algorithm) to mark each connected domain of the workpiece, and calculate the area S (the number of pixels in the connected domain), perimeter P (the number of pixels at the border of the connected domain is calculated by the contour tracing algorithm), and circularity C = 4πs / P of each connected domain. 2 (Used to measure the similarity between the connected domain and the circle, the closer it is, the closer it is to the rectangularity of the circle), rectangularity R = S / S rect (Among them, S rect is the minimum rectangular area that contains the connected domain, and the rectangularity is used to describe the similarity between the connected domain and the rectangle), the surface depth ratio of the maximum connected domain, and other characteristics.

[0024] The surface depth ratio of the maximum connected domain is Hr=Smax*Hmax / Hmin, where Smax is the area of ​​the maximum connected domain of the separable image, the maximum connected domain is the connected domain with the largest area of ​​the image of the independent separable grasped object, Hmax is the depth of the maximum depth point of the maximum connected domain from the reference point, Hmin is the depth of the minimum depth point of the maximum connected domain from the reference point, and the maximum depth point and the minimum depth point of the maximum connected domain are obtained by calculating the depth of each point on the contour of the maximum connected domain from the reference point, and sorting the maximum depth point and the minimum depth point on the contour of the maximum connected domain, that is, the maximum depth point and the minimum depth point of the maximum connected domain. The maximum depth point and the minimum depth point of the maximum connected domain can also be determined by other methods, specifically: calculating the depth of each point in the maximum connected domain from the reference point, connecting the maximum depth point and the minimum depth point in the maximum connected domain, and the line connecting the two points and the contour of the connected domain generate two intersections, where the intersection with a larger depth from the reference point is recorded as the maximum depth point of the maximum connected domain, and the intersection with a smaller depth from the reference point is recorded as the minimum depth point of the maximum connected domain.

[0025] Connection line feature extraction: Detect straight line segments in the image through Hough transform, map the straight line in the image space to the parameter space, and determine the parameters of the straight line by finding the peak in the parameter space. For the detected straight line, record its slope k = tanθ, intercept b =ρ / sinθ) and length L (calculated according to the coordinates of the two end points of the straight line).

[0026] The overall features include but are not limited to line density features, which are extracted as follows: divide the image into M*N small regions, each of which is w*h in size. Perform edge detection (such as Sobel operator) on each region and count the number of edge pixels n ij (i = 1,2,...,M, j = 1,2,...,N), calculate the line density D of each area ij =n ij / (w*h).

[0027] Feature normalization and weight assignment: Normalize the extracted feature parameters so that their value range is between [0,1], using the linear normalization method x norm =(x – x min ) / (x max -x min ), where x is the original eigenvalue, x min and x max are the minimum and maximum values ​​of the feature in all samples respectively. The following will take four features as an example to introduce how to determine the weights of different features. First, construct the judgment matrix A: , where a ij Indicates the importance of feature i relative to feature j. Then, the maximum eigenvalue λ of the judgment matrix is ​​calculated max and the corresponding eigenvector , normalize the feature vector and get the weight vector W=[w1,w2,w3,w4] of each feature. The specific calculation steps are as follows:

[0028] Where i=1,2,…,n, n is the number of features, for this example, n=4.

[0029] Perform consistency test and calculate consistency index CI , in For vector The i-th element of . According to the order n of the judgment matrix, the average random consistency index RI is determined, and the consistency ratio CR=CI / RI is calculated. When CR<0.1, the judgment matrix is ​​considered to have satisfactory consistency, otherwise the judgment matrix needs to be readjusted.

[0030] 3. Image comparison and similarity distance calculation.

[0031] Compare the feature vector of the currently acquired image to be identified with the feature vectors of various pre-stored workpiece types to calculate the similarity distance. When the similarity distance is less than the set distance threshold, it is determined that the image to be identified successfully matches a pre-stored workpiece type, thereby identifying the workpiece type. Suppose the feature vector of the image to be identified is X=[x1,x2,……,x n ], the feature vector of the pre-stored workpiece image is Y=[y1,y2,……,y m ], construct a similarity distance matrix D, where D(i,j) represents x i With y j Find the shortest path from D(1,1) to D(n,m), and the cumulative distance of the path is the similarity distance DT(X,Y).

[0032] In order to further improve the accuracy and adaptability of distance calculation, the weighting coefficient α is introduced to adjust the similarity distance: , where w i is the weight of feature i, which corresponds to the weight vector calculated above. It is used to consider the importance difference of different features in distance calculation. α takes a value in the range of [0,1] according to the actual situation. Set the similarity distance threshold T. When DT α When it is less than T, it is considered that the image to be identified successfully matches a pre-stored workpiece type and the workpiece type is determined.

[0033] 4. Determine and adjust the grabbing position.

[0034] According to the identified workpiece type, its pre-stored 3D geometric model and feature information are retrieved, and combined with the actual state of the workpiece in the current image, the position of the exposed part of the workpiece in the stacked state is analyzed.

[0035] The robot's optimal grasping posture and path are calculated through the inverse kinematics algorithm. If it is found that the exposed part of the workpiece is not convenient for direct grasping, the robot is used to perform appropriate pushing, pulling and other moving operations on the workpiece. After adjusting the position of the workpiece, image acquisition and recognition are performed again until a suitable grasping position is determined.

[0036] 5. Crawl operation execution and feedback.

[0037] The robot is controlled to move according to the calculated grasping posture and path, and precise motion instructions are sent to the robot's driver through the robot control system to achieve accurate grasping of the workpiece.

[0038] During the grasping process, the grasping status is monitored in real time through visual feedback or sensor feedback (such as force sensor) to ensure the stability and reliability of the grasping action. After the grasping is successful, the workpiece is moved to the specified location and the visual or sensor feedback is used to confirm whether the workpiece is in place.

[0039] Through the above implementation modes, the present invention can effectively realize the precise grasping of stacked workpieces by robots based on image recognition, thereby improving the automation level and production efficiency of industrial production.

Claims

1. A robot grasping control method based on image recognition, characterized in that: The following steps are involved: Capture images of the grasped object at multiple angles and under multiple lighting conditions, and transmit the captured images to the image processing unit; In the image processing unit, after preprocessing the image, local features and overall features of the workpiece image are extracted, the local features include contour features, connected domain features and connecting line features, and the overall features include line density features; the extracted feature parameters are normalized, and the weights of different features are determined to obtain a weighted feature vector; Compare the feature vector of the image to be identified with the pre-stored feature vectors of various workpiece types, calculate the similarity distance based on the feature vectors, and determine the workpiece type; According to the type of workpiece that is successfully matched, the robot's optimal grasping posture and path are calculated through the inverse kinematics algorithm; The robot is controlled to move according to the calculated grasping posture and path to grasp the workpiece.

2. The method according to claim 1, characterized in that The contour feature extraction adopts the Canny edge detection algorithm to calculate the Hu invariant moment of the extracted contour, wherein the calculation formula of the Hu invariant moment is: ， Where (xi,yi) are the coordinates of the points on the contour, i = 1,2,...,n, n is the number of contour points, p,q are used to calculate the order of the moment, and the values ​​are 0,1,2,...,μ pq is the p + q order central moment, η pq is the normalized central moment, φ1, φ2, φ3, … are Hu invariant moments.

3. The method according to claim 1, characterized in that The line density feature calculation is to divide the image into M*N small areas, each area size is w*h, perform edge detection on each area, and count the number of edge pixels n ij , where i = 1,2,…,M, j = 1,2,…,N, calculate the line density D of each area ij =n ij / (w*h).

4. The method according to claim 1, characterized in that: The determination of feature weights includes constructing a judgment matrix, calculating the maximum eigenvalue of the judgment matrix and the corresponding eigenvector, and normalizing the eigenvector to obtain the weight vector of each feature.

5. The method according to claim 1, characterized in that Combine the feature vector to calculate the similarity distance, specifically: suppose the feature vector of the image to be identified is X=[x1,x2,……,x n ], the feature vector of the pre-stored workpiece image is Y=[y1,y2,……,y m ], construct a similarity distance matrix D, where D(i,j) represents x i With y j The distance between them is used to find the shortest path from D(1,1) to D(n,m). The cumulative distance of this path is the similarity distance DT(X,Y).

6. The method according to claim 5, characterized in that Introduction The weighting coefficient α adjusts the similarity distance, specifically: , where w i is the weight of feature i, corresponding to the weight vector of each feature, and α takes values ​​in the range of [0,1].

7. The method according to claim 1, characterized in that The connected domain feature extraction includes performing adaptive threshold method binarization processing on the image, marking each connected domain of the workpiece using a connected domain marking algorithm, and calculating the area and / or perimeter and / or circularity and / or rectangularity and / or the surface-to-depth ratio of the largest connected domain.

8. The method according to claim 1, characterized in that The connecting line feature extraction is to detect the straight line segment in the image through Hough transformation, and record the slope, intercept and length parameters of the straight line.

9. The method according to claim 1, characterized in that: When part of the workpiece is exposed and not convenient for direct grasping, the workpiece is moved by a manipulator and image acquisition and recognition are performed again until a suitable grasping position is determined.

10. The method according to claim 1, characterized in that Image preprocessing includes denoising and grayscale operations.

Images