A regular polygonal prism detection method for a robotic arm

Abstract

The present invention discloses a regular polygonal prism detection method for a robotic arm, comprising the following steps: Step 1: Obtaining a top-view image of an operating table; Step 2: Obtaining an object mask; Step 3: Obtaining the object edge; Step 4: For each object, using the mask obtained in the second step to denoise the edge obtained in the third step; Step 5: Determining whether there is a circle; if so, obtaining the upper surface circle and jumping to Step 9, otherwise Step 6; Step 6: Obtaining information about the straight line where the edge is located from the edge detection result; Step 7: Selecting the straight line of the upper surface edge of the object; Step 8: Shortening the straight line to an object edge segment; Step 9: Calculating the shape, centroid, and angle of the object. The present invention can accurately obtain the edge of the upper surface of a regular polygonal prism object, and obtain the suction position, object shape, and object angle required for the placement task, thereby helping the placement task to be completed smoothly.

Description

Technical Field

[0001] The present invention belongs to the technical field of robotic arms, and in particular relates to a regular polygonal prism detection method for a robotic arm, so as to accurately detect the edge, shape, centroid and angle of the upper surface of the regular polygonal prism. Background Art

[0002] A robotic arm is typically a complex system characterized by high precision, multiple inputs and outputs, high nonlinearity, and strong coupling. Due to its unique operational flexibility, it has found widespread application in industrial assembly, safety, and explosion-proofing. When the target object is always in a fixed position, fixed, repetitive position control can be used to enable the robotic arm to perform fixed-position pick-and-place procedures. However, when the position of the target object changes, the robotic arm needs to be able to sense changes, automatically locate the object, and complete the assembly task.

[0003] To increase the perception capability of the robotic arm, sensors are used in hardware to obtain external information, and the required information is calculated based on the data obtained. For example, patent application 202120538755.2 discloses a distribution network live operation robot system, comprising: an operating table device, a support device, and a control cabinet; the operating table device comprises an operating table, a robotic arm assembly, and a sensor assembly; the robotic arm assembly and the sensor assembly are mounted on the operating table; the robotic arm assembly comprises a first number of robotic arms and a robotic arm base; the robotic arm base is fixed to the operating table, and the robotic arm is fixed to the robotic arm base; the support device comprises an insulating bucket and a support frame; the support frame is clamped around the insulating bucket; the operating table is slidably mounted on the support frame; a chamber is formed on the insulating bucket, and the control cabinet is mounted in the chamber; a processor and a communication module are mounted in the control cabinet; the robotic arm and the sensor assembly are connected to the processor, and the processor is connected to the communication module.

[0004] In assembly tasks, the robot arm is provided with sensing capabilities. An RGB camera can be used as a sensor to obtain images of the objects to be assembled and analyze the information required for picking and placing the objects, such as the position, angle, and shape of the target objects, and finally enable the robot arm to perform the placement action. Whether the assembly task can be successfully completed depends on the accuracy of the position and angle.

[0005] The process of analyzing the position, angle, and shape of the target object can be implemented using either target detection or instance segmentation models. However, both models have their own shortcomings. The detection boxes output by the target detection model are mostly horizontal or vertical, without tilt angles, and cannot well reflect the object angle, making it difficult to meet assembly requirements. The segmentation results of the instance segmentation model often cannot accurately fit the actual edges of the object, and the object vertices are not clear enough, resulting in the inability to accurately obtain the center of gravity of the object, affecting the success rate of robotic arm assembly. In addition, when the object is at a certain height, the side surface of the object exists in the input image, and both models will be affected. Current technology cannot accurately filter out the upper surface edge of the object to be assembled, making it difficult to obtain the accurate angle and center of gravity of the object, affecting the success rate of the assembly task. Therefore, improvement is urgently needed. Summary of the Invention

[0006] Based on this, the primary purpose of the present invention is to provide a regular polygonal prism detection method for a robotic arm, which can provide perception capabilities for the robotic arm and overcome the shortcomings of the current target detection model and instance segmentation model, obtain the suction position, object shape and object angle required for the placement task, and help complete the placement task smoothly.

[0007] Another object of the present invention is to provide a regular polygonal prism detection method for a robotic arm that can detect the edge of an object's top surface while eliminating the influence of side surfaces. Adaptive threshold segmentation is performed using the saturation S channel of the HSV color space to obtain an object mask. Grabcut segmentation is then performed in the Lab and YCrCb color spaces to maximize the detection of the object's top surface mask. Simultaneously, edge detection using the L component of the Lab color space and the V component of the HSV color space is integrated to obtain a more complete top surface edge, ensuring detection accuracy.

[0008] To achieve the above object, the technical solution of the present invention is:

[0009] A method for detecting regular polygonal prisms for a robotic arm, characterized by comprising the following steps:

[0010] Step 1: Obtain a top-down image of the workbench. Use an RGB camera to capture the workbench and the objects to be assembled on it from a top-down perspective to obtain a picture of the workbench.

[0011] In order to distinguish the operating table and the objects to be assembled, they are marked with different colors; the operating table is designed to be white, and the objects to be assembled are other colors.

[0012] Step 2: Get the object mask,

[0013] Wherein, obtaining the object mask refers to obtaining at least a mask of the upper surface of the object.

[0014] The specific steps include:

[0015] S21. Convert the image to HSV color space and use the saturation S channel to perform adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask will be obtained.

[0016] S22, denoising, using operations such as dilation, erosion, and removal of connected domains with excessively large areas to remove some noise;

[0017] S23, then input the denoised mask and the table top image converted to Lab color space into grabcut, so that the module detection process is fully automated. Converting the image to Lab color space can obtain a better upper surface segmentation effect than RGB color space;

[0018] S24. Next, check whether the area of the connected domain in the segmentation mask is too large. If it is too large, it is suspected that the masks of multiple objects are adhered. The image is transferred to the YCrCb space and a grabcut is performed again and used as the upper surface mask of the object. Otherwise, the segmentation result in the current Lab color space is directly used as the upper surface mask of the object.

[0019] Step 3: Detect the edges of objects in the image.

[0020] Using Canny edge detection, we obtain a binary edge image. Because the color variation on the sides and edges of some objects is minimal, these edges can be difficult to detect. To address this issue, we exploit the fact that the top and side surfaces primarily vary in brightness. We perform edge detection on the L component in Lab color space and the V component in HSV color space, respectively. We then take the union of the detection results from these two components.

[0021] Step 4: Detect object edges in the image and use the mask obtained in Step 2 to remove noise. This step aims to minimize noise in the edge detection results. The noise removed includes both internal and external noise. Dilate the mask and intersect it with the edge detection result to remove external edge noise. Erode the mask and subtract it from the edge detection result to remove internal noise.

[0022] Furthermore, mask noise is achieved by dilating the mask and finding the intersection with the edge detection result to obtain the edge detection result with the external noise of the object removed; then the mask is eroded and the difference with the edge detection result is found to obtain the edge detection result with the internal noise of the object removed.

[0023] Step 5: Determine whether there is a circle.

[0024] Determine whether the edge image of the current target object contains a circle. If so, go to step eight, otherwise go to step six. Specifically, use Hough circle detection to detect whether there is a circle in the intersection result obtained by applying the object mask to the Lab color space. If so, get the most suitable upper surface circle and go to step nine, otherwise go to step six. The process of obtaining the upper surface circle is to use Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively find the circle with the largest param2 in the color space, and then use the circle with the outer center as the upper surface circle. Param2 is the input parameter of the Hough circle detection method, which indicates the regularity of the circle. Only shapes larger than param2 can be detected as circles. The outer circle is selected as the upper surface circle based on the fact that the upper surface is always relatively close to the periphery of the picture when viewed from a bird's-eye view.

[0025] Step 6: Obtain the information of the straight line where the edge is located from the edge detection result.

[0026] The edge detection result uses the Hough line detection technology to adaptively obtain the analytical expression of the line where the line segment with a length greater than th1 in the current edge detection image is located.

[0027] The specific process is as follows:

[0028] S41, obtaining the line segment th=th1;

[0029] S42, detecting line segments whose length is greater than th;

[0030] S43, determine whether a straight line is detected; if no straight line is detected, proceed to the next step; if a straight line is detected, set th+=4 and return to S42;

[0031] S44, set th-=1;

[0032] S45, further determine th>th1, if yes, proceed to the next step, otherwise end;

[0033] S46, detecting line segments whose length is greater than th;

[0034] S47. Determine whether a straight line is detected; if a straight line is detected, record the straight line information and th, and delete the detected straight line in the original image; if not, return to step S44 and set th-=1.

[0035] After deleting the detected straight lines, repeat the above steps until all detections are completed.

[0036] Through this process, the straight line information of all line segments with a length greater than th1 is recorded in reverse order of the line segment length. The information includes the inclination angle, the distance from the line to the image origin, and the length threshold th at the time of detection.

[0037] Step 7: Screen out the straight lines on the upper surface edge of the object. This step eliminates interference from the side surface edge, bottom surface edge and other noise of the object, and selects the straight line that best fits the actual edge of the upper surface of the object.

[0038] The three operations of line grouping, line denoising and line selection are performed in sequence.

[0039] Straight line grouping: All detected straight lines are grouped according to their angles. All straight lines with an angle difference within th2 are grouped into one group, and each group contains at least one straight line.

[0040] Furthermore, nearly parallel lines with a distance difference greater than th3 are grouped separately. th3 needs to be adjusted so that parallel edges on the upper surface of an object are divided into two groups, while parallel edges on the upper and lower surfaces are grouped together. The rule for determining nearly parallel lines is that the two lines have no intersection within the frame.

[0041] Line denoising: This function mainly removes side edge lines and other noise. Lines are removed based on the shorter side edge length and shorter noise lines when viewed from a bird's-eye view. The rule for determining a line is that there is only one line in a group and the length of the identified line segment is less than th4.

[0042] Select lines: From the remaining sets of upper and lower surface edge lines, select the upper surface lines. Compare the difference between the object framed by each line and the mask by comparing the upper surface lines with the lower surface lines. For each group, select the line with the smallest difference and consider it to be the upper surface edge.

[0043] Furthermore, when selecting lines, it is necessary to first construct a closed box for each line, and then compare the difference between the closed range and the mask; when constructing the closed box, each line is matched with the line closest to the center of the mask in the other line group to form a closed box; compare all pixels in the closed box A with all pixels in the grabcut mask B, and record the number n of pixels in A but not in B for each line; then, select a line with the smallest n in each group and consider it as the edge line of the upper surface.

[0044] Step 8: Shorten the straight lines to the object's edge segments. Since the straight line information obtained in step 4 does not include endpoints, this step requires finding the intersection of the edge lines, which are the vertices of the object's upper surface, to obtain line segments that fit the object's edge.

[0045] To do this, we traverse all current pairwise combinations of lines and calculate the minimum distance between the intersection of these two lines and the mask. If the distance is less than th5, the two lines are considered adjacent. The distance from a point to the mask is calculated by drawing circles of increasing radius centered on the point. When the circle begins to intersect the mask, the radius of the circle is the distance from the point to the mask. After determining the adjacency relationship for all lines, we calculate the vertices of the adjacent lines and connect them sequentially to obtain the edge of the object's surface.

[0046] 9: Calculate an object's shape, centroid, and angles. Based on the one-to-one correspondence between the shape and the number of edges of a regular polyhedron, the shape can be determined from the number of vertices. The centroid can be calculated from the closed figure formed by the edges of the top surface. The angle of an object can be calculated from the rotation angle of the minimum circumscribed rectangle of the top surface.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention utilizes the transformation of color space and the fusion of multi-color space information to solve the problem of poor segmentation and detection effects in RGB color space. Specifically, for image segmentation, the S component of HSV is used to segment the initial mask, and then further grabcut segmentation is performed in Lab and YCrCb color spaces according to the situation to obtain the upper surface mask of the object as much as possible. For edge detection, the edge detection of the L component in the Lab color space and the V component in the HSV color space is integrated to obtain a more complete upper surface edge as much as possible, which can accurately obtain the accurate edge, improve the accuracy of detection, and obtain the suction position, object shape and object angle required for the placement task, helping to complete the placement task smoothly. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart implemented by the present invention.

[0050] Figure 2 This is a flow chart of the Hough line detection implemented by the present invention.

[0051] Figure 3 This is an overhead view of the operating console.

[0052] Figure 4 This is a diagram of all object masks obtained by the present invention.

[0053] Figure 5 It is a schematic diagram of detecting the edges of all objects in an image implemented by the present invention.

[0054] Figure 6 Schematic diagram of triangle denoising edge implemented in the present invention.

[0055] Figure 7 Schematic diagram of pentagonal denoising edge implemented in the present invention.

[0056] Figure 8 Schematic diagram of circular denoising edge implemented in the present invention.

[0057] Figure 9 Schematic diagram of square denoising edge implemented in the present invention.

[0058] Figure 10 Schematic diagram of all graphic denoising edges implemented in the present invention.

[0059] Figure 11 Schematic diagram of the intersection of the triangle mask and the image implemented in the present invention.

[0060] Figure 12 Schematic diagram of the intersection of a quadrilateral mask and an image implemented in the present invention.

[0061] Figure 13 Schematic diagram of the intersection of the pentagonal mask and the image implemented in the present invention.

[0062] Figure 14 Schematic diagram of the intersection of a cylindrical mask and an image implemented in the present invention.

[0063] Figure 15 It is a straight line graph for detecting triangle edges implemented by the present invention.

[0064] Figure 16 It is a straight line graph for detecting pentagon edges implemented in the present invention.

[0065] Figure 17 It is a straight line graph for detecting square edges implemented by the present invention.

[0066] Figure 18 It is a straight line schematic diagram of the upper surface of the triangle implemented by the present invention.

[0067] Figure 19 It is a straight line schematic diagram of the upper surface of the pentagon implemented by the present invention.

[0068] Figure 20 It is a straight line schematic diagram of the upper surface of the square implemented by the present invention.

[0069] Figure 21 Schematic diagram of triangle edge segments implemented in the present invention.

[0070] Figure 22 Schematic diagram of pentagonal edge segments implemented in the present invention.

[0071] Figure 23 Schematic diagram of the square edge segments implemented in the present invention.

[0072] Figure 24 It is a schematic diagram of calculating object information implemented by the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0074] Figure 1 As shown in FIG, a flow chart of the detection method of a regular polygonal prism for a robotic arm implemented by the present invention is shown. Since most objects to be assembled are cylinders, the present invention uses a regular polygonal prism as an example for detection.

[0075] As shown in the figure, the present invention takes as input an image of a workbench viewed from above, where the workbench is designed to be white and the objects to be assembled are cylinders of a different color. The output is the top surface edge of each object in the image, as well as its shape, position, and angle information. Existing techniques used include color space conversion, adaptive threshold segmentation, grabcut segmentation, Canny edge detection, Hough line detection, and morphological processing. In addition to combining these techniques, the present invention creates processing rules based on the prior knowledge that the objects to be assembled are cylinders, which are used to filter out the true edges of the object's top surface from the noisy edge detection results.

[0076] The specific steps include:

[0077] Step 1: Get a top-view image of the workbench. Use an RGB camera to shoot the workbench and the objects to be assembled on it from a top-down perspective to get a picture of the workbench, such as Figure 3 As shown;

[0078] In order to distinguish the operating table and the objects to be assembled, they are marked with different colors; the operating table is designed to be white, and the objects to be assembled are other colors.

[0079] Step 2: Obtain an object mask, where the object mask refers to obtaining at least a mask of the upper surface of the object.

[0080] The specific steps include:

[0081] S21. Convert the image to the HSV color space and use the saturation S channel therein for adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask is obtained. First, use the different color saturations of the white table and the object to be assembled to convert the image to the HSV color space and use the saturation S channel therein for adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask will be obtained.

[0082] S22, denoising, using operations such as dilation, erosion, and removal of connected domains with excessively large areas to remove some of the noise; denoising technology is an existing technology and will not be described in detail here.

[0083] S23. Then, the denoised mask and the table top image converted to the Lab color space are input into grabcut together. When the denoised mask and the table top image are input into grabcut together, the mask plays the role of manually inputting the foreground area for the user, making the module detection process fully automated. Converting the image to the Lab color space can obtain a better upper surface segmentation effect than the RGB color space.

[0084] S24. Next, check whether the area of the connected domain in the segmentation mask is too large. If it is too large, it is suspected that the masks of multiple objects are stuck together. The image is transferred to the YCrCb space and a grabcut is performed again and used as the upper surface mask of the object. Otherwise, the segmentation result in the current Lab color space is directly used as the upper surface mask of the object. The masks of all objects are obtained as follows Figure 4 shown.

[0085] Step 3: Detect the edges of objects in the image. Use Canny edge detection to obtain an edge binary image, such as Figure 5 As shown in the figure, all object edges are displayed. Even when viewed from above, the sides and edges of objects are still visible. In these cases, edges connected to the side surfaces have little color variation, making them difficult to detect. To address this issue, we exploit the fact that the top and side surfaces primarily vary in brightness. We perform edge detection on the L component in Lab color space and the V component in HSV color space, respectively, as these components better reflect brightness variations. The detection results from these two components are then combined.

[0086] Step 4: Use the mask obtained in step 2 to denoise the edge obtained in step 3. This step aims to obtain an edge with as little noise as possible. The noise removed includes both internal and external noise at the edge of the object. Dilate the mask and find the intersection with the edge detection result to remove external edge noise. Then erode the mask and find the difference with the edge detection result to remove internal noise. Figure 6-11 As shown, Figure 6 This is a schematic diagram of triangle denoising. Figure 7 This is a schematic diagram of pentagonal denoising. Figure 8 is a schematic diagram of circular denoising, Figure 9 This is a schematic diagram of square denoising. Figure 10 This is a schematic diagram of all graphics denoising.

[0087] Step 5: Detect whether there is a circle. Since the upper surface of a cylinder is circular and does not contain straight lines, subsequent detection and processing of straight lines cannot be performed. Therefore, in this step, it is necessary to determine whether the target object currently being processed is a cylinder.

[0088] The detection process is as follows Figure 11-14 As shown, Figure 11 is a schematic diagram of the intersection of the triangle mask and the image, Figure 12 is a schematic diagram of the intersection of the quadrilateral mask and the image, Figure 13 is a schematic diagram of the intersection of the pentagonal mask and the image, Figure 14 It is a schematic diagram of the intersection of the cylindrical mask and the image.

[0089] When detecting whether there is a circle, the object mask is applied to the intersection result obtained in the Lab color space, and the Hough circle transform is used to detect whether there is a circle. If so, the most suitable upper surface circle is obtained and step nine is entered, otherwise step six is entered. The process of obtaining the upper surface circle is to use Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively find the circle with the largest param2 in the color space, and then use the circle with the outer center as the upper surface circle. Param2 is the input parameter of the Hough circle detection function, which indicates the regularity of the circle. Only shapes larger than param2 can be detected as circles. The outer circle is selected as the upper surface circle based on the fact that the upper surface is always relatively close to the periphery of the image when viewed from a bird's-eye view.

[0090] Step six: obtain the information of the straight line where the edge is located. The binary mask of the edge detection result image is not the mathematical expression of each line segment. For subsequent calculations, the analytical expression of the straight line must be obtained from the binary mask. The present invention uses Hough line detection technology to adaptively obtain the analytical expression of the straight line where the line segment with a length greater than th1 in the current edge detection image is located. Among them, the threshold th1 is used to limit the shortest length of the line segment, and the straight line shorter than th1 on the image will not be detected. In actual use, it is necessary to make adjustments based on the length of the edge of the object to be assembled in the image, so that th1 is slightly smaller than the length of the edge segment of the object, so as to achieve the purpose of detecting all edge segments and filtering out shorter noise segments. Experienced, it can be set to 0.6 to 0.8 times the length of the object edge. For example, when the shortest length of the edge of the image object is 100 pixels, th1 can be set to a value between 60 and 80.

[0091] The specific process is as follows Figure 2 As shown, through this process, all line information containing line segments with lengths greater than th1 is recorded in reverse order of line segment lengths. This information includes the tilt angle theta, the distance rho from the line to the image origin, and the length threshold th used for detection. th is the input parameter of the Hough line detection function. Only line segments with lengths greater than th are detected. The specific process is as follows:

[0092] S41, obtaining the line segment th=th1;

[0093] S42, detecting line segments whose length is greater than th;

[0094] S43, determine whether a straight line is detected; if no straight line is detected, proceed to the next step; if a straight line is detected, set th+=4 and return to S42;

[0095] S44, set th-=1;

[0096] S45, further determine th>th1, if yes, proceed to the next step, otherwise end;

[0097] S46, detecting line segments whose length is greater than th;

[0098] S47. Determine whether a straight line is detected; if a straight line is detected, record th and the straight line inclination angle theta and the distance rho to the image origin output by the Hough line detection, and delete the detected straight line from the original image; if not, return to step S44 and set th-=1.

[0099] After deleting the detected straight lines, repeat the above steps until all detections are completed.

[0100] Detect the triangle edge straight line Figure 15 As shown, the detection of pentagon edge straight line is as follows Figure 16 As shown, the detection of square edge straight lines is as follows Figure 17 As shown in Figure 3. Step 7: Filter out the top edge lines according to the rules. This step aims to eliminate interference from the side and bottom edges of the object, as well as other noise, and select the lines that best match the actual top edge of the object. Because the lines obtained by Hough detection in Step 5 are affected by the edge detection results, when the line segments of the edge binary mask have pixel-level curvature, a line width greater than 1, or noise, a series of lines will be detected near each edge of the object. To select the lines that are closest to the true edge, three operations are performed, line grouping, line denoising, and line selection. Line grouping aims to map all edge lines to actual edges. After classification, each group of lines corresponds to an actual edge on the object's surface. Line denoising removes noise segments and line groups corresponding to side edge segments based on the grouping, ensuring that each remaining line group corresponds to an edge on the object's top surface. At this point, each line group may contain lines that are close to the true top edge, as well as lines that are close to the bottom edge. These lines also have varying angles, so not every line group will perfectly match the actual surface edge. The last step, line selection, is to select a line in each group of lines that best matches the true edge of the object and use it as the upper surface line detection result.

[0101] Straight line grouping: The purpose of grouping is to make each group of straight lines correspond to an edge of the target object, and it is necessary to find the features used for grouping. Since the object is a regular polygon, the upper surface is a regular polygon. It is known that the slopes of the straight lines of the regular polygons with odd sides are different, so the angle features can be used to distinguish the sides; the opposite sides of the regular polygons with even sides are parallel to each other and have a large spacing, and the angles between different opposite sides are different, so the angle and distance features can be used to distinguish the sides at the same time. Further considering the edge of the side surface of the polygon, when the side surface is exposed, there is an edge that is parallel to and close to the corresponding edge of the upper surface, that is, the edge of the lower surface, and two edges that are perpendicular to the operating table in reality, that is, the edges that belong only to the side surface, and their angles are different from those of the edges of the upper surface. Therefore, in any case, considering the angle and distance features at the same time, the straight lines can be corresponded to the sides of the regular polygon.

[0102] First, set the angle threshold th2 and distance threshold th3 respectively. th2 reflects the tolerance for angle deviation. Lines with an angle difference within th2 are grouped together. The larger th2 is, the more lines with a larger angle difference will be grouped together. The specific value should be suitable for distinguishing different sides while tolerating a certain angle error. A good value is th2 = 0.3 * min (interior angle, exterior angle). For a triangle, for example, its interior angle is 60° and its exterior angle is 120°. Taking the minimum of the two, 60°, and multiplying by 0.3, we can set th2 = 0.3 * 60° = 18°. The distance threshold th3 is used to distinguish between two opposite sides. When the distance difference between two straight lines is less than th3, they are considered to belong to the same side; otherwise, they belong to different opposite sides. The value of Th3 needs to be adjusted according to the size of the target object in the image, so that the parallel edges on the upper surface of the object can be divided into two groups, while the parallel edges on the upper and lower surfaces can be combined into the same group. The value th3 can be 0.5*the width of the minimum circumscribed rectangle of the regular polygon on the upper surface.

[0103] Next, we use th2 and th3 to perform grouping. Based on the recorded line information theta, we group all lines whose angle difference is within th2 into a group, with each group containing at least one line. Furthermore, we group nearly parallel lines whose distance difference is greater than th3 into separate groups. The rule for determining nearly parallel lines is that the two lines have no intersection within the image.

[0104] In actual use, it is also necessary to judge the rationality of the values of th2 and th3 based on the grouping results. If necessary, the above two steps can be repeated until the grouping effect reaches the expected effect, that is, each group of straight lines corresponds to an edge of the object.

[0105] Straight line denoising: The purpose of this step is to remove the two edges perpendicular to the operation table in the side edges and other noises. By using the rule that the lengths of these two edges and the noises are shorter in the top view, a length threshold th4 can be set to remove them. If there is only one straight line in a group and the corresponding th < th4, then this group is removed. The role of th4 is to separate the shorter vertical edges and noises from all the straight lines. Its value range is (the length of the vertical edge and the noise straight line, the length of the upper surface edge). The median value of this range can be taken first, and then adjusted repeatedly according to the actual denoising result until the purpose of this step can be achieved.

[0106] Selecting a straight line: After the previous straight line grouping and straight line denoising steps, the current straight line group only includes the straight lines corresponding to the upper surface edge and the lower surface edge parallel to it. The purpose of this step is to select the most suitable straight line from the detected straight lines as the detection result of the corresponding upper surface edge. By using the rule that the straight line on the upper surface encloses less part of the object than the straight line on the lower surface, and the possibility that the enclosed part is not within the mask range output in the second step is smaller, compare the differences between the objects enclosed by each straight line and the mask respectively, and select the straight line with the smallest difference for each group, which is considered as the upper surface edge. Specifically, a closed frame needs to be constructed for each straight line first, and then the difference between the closed range and the mask is compared. The edge of the closed frame is composed of this straight line and the straight line in other straight line groups that is closest to the center of the mask, aiming to ensure that the difference between the enclosed closed figure and the mask is only determined by this straight line. Compare all the pixels in the closed frame A with all the pixels in the grabcut mask B, and record the number n of pixels that are in A but not in B for each straight line. Then, select a straight line with the smallest n in each group, which is considered as the real upper surface edge straight line.

[0107] As Figure 18-20 shown, where Figure 18 is the triangular upper surface straight line, Figure 19 is the pentagonal upper surface straight line, Figure 20 is the square upper surface straight line.

[0108] Step 8: Shorten the lines to object edge segments. Since the fourth step only yielded line information, not line segment information, and true edges are line segments, this step first finds the vertices of the regular polygon and then connects adjacent vertices to obtain edge segments. Vertices are the intersection points of adjacent edge lines, so the adjacency relationship of the lines must be determined first. Specifically, a distance threshold th5 is set and all current pairwise line combinations are iterated over. The distance from the line intersection to the mask is calculated. If the distance is less than th5, the two lines are considered adjacent. This is based on the principle that the distance from a vertex to the mask is closer, while the intersection of non-adjacent lines is farther from the mask. Th5 is set to distinguish these two distances and can generally be set to 30 pixels. The distance from a point to the mask is calculated by drawing circles of increasing radius centered at the point. When the circle begins to intersect the mask, the radius of the circle equals the distance from the point to the mask. After determining the adjacency relationship of all lines, the vertices of the adjacent lines are calculated and connected sequentially to obtain the edge of the object's top surface.

[0109] like Figure 21-23 As shown, Figure 21 is the edge segment of the triangle, Figure 22 is the edge segment of the pentagon, Figure 23 is a line segment on the edge of the square.

[0110] Step 9: Calculate the shape, centroid, and angle of the object. According to the one-to-one correspondence between the shape of a regular polyhedron and the number of vertices, the shape can be obtained from the number of vertices. For example, if there are three vertices, it means it is a regular triangular prism; the centroid can be calculated based on the closed figure formed by the edges of the upper surface; the angle of the object is calculated by the rotation angle of the minimum circumscribed rectangle of the upper surface. Figure 24 shown.

[0111] Therefore, the present invention can accurately segment the upper surface of a regular polygonal object with straight and sharp edges and clear vertices. On this basis, it can more accurately calculate the information required for the robotic arm assembly task, including the obtained position, angle and shape information, so that the automated assembly task can be completed smoothly, reducing the possibility of assembly failure due to unsatisfactory segmentation and detection results, thereby obtaining the accurate suction position, object shape and object angle required for the placement task, helping to complete the placement task smoothly.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting regular polygonal prisms for a robotic arm, characterized in that The steps include: Step 1: Get the current state image. Specifically, use an RGB camera to shoot the operating table from above to obtain the image of the object to be assembled. Step 2: Obtain object mask in the image; Step 3: Get the edge of the object in the image; Step 4: For each target object, use the mask obtained in step 2 to denoise the edges obtained in step 3; Step 5: Determine whether the edge image of the current target object contains a circle. If so, proceed to step 9; otherwise, proceed to step 6. Step 6: Obtain the information of the straight line where the edge is located from the edge detection result; Specifically, the Hough line detection technology is used to adaptively obtain the analytical expression of the line where the line segment with a length greater than th1 in the current edge detection image is located; Step 7: Screen out the straight lines along the upper surface edge of the object. Specifically, exclude interference from the side surface edge, bottom surface edge, and other noise, and select the straight line that best fits the actual edge of the upper surface of the object. Step 8: Obtain edge line segments on the object's upper surface. Specifically, find the intersection points of the edge lines, i.e., the vertices on the object's upper surface, and connect the vertices to obtain line segments that fit the object's edge. Step 9: Calculate the shape, centroid, and angle of the object. The shape can be determined by the number of vertices, because the shape of a regular polyhedron corresponds to the number of edges. The centroid can be calculated from the closed figure enclosed by the edges of the upper surface. The angle of the object is calculated from the rotation angle of the minimum circumscribed rectangle of the upper surface.

2. The method for detecting regular polygonal prisms for a robotic arm according to claim 1, wherein In step 1, in order to distinguish the operating table and the objects to be assembled, they are marked with different colors; the operating table is designed to be white, and the objects to be assembled are other colors.

3. The regular polygonal prism detection method for a robotic arm according to claim 1, characterized in that In step 2, obtaining the object mask means obtaining at least a mask of the upper surface of the object.

4. The method for detecting regular polygonal prisms for a robotic arm according to claim 3, wherein Step 2 specifically includes the following steps: S21. Convert the captured RGB image to the HSV color space and use the saturation S channel to perform adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask will be obtained. S22, denoising, using morphological operations such as dilation, erosion, and removal of connected domains with excessively large areas to remove some noise; S23, then input the denoised mask and the table image converted to the Lab color space into grabcut to obtain the grabcut segmentation result of the foreground; S24. Next, check whether the area of the connected domain in the segmentation mask is too large. If it is too large, transfer the image to the YCrCb space and perform another grabcut and use it as the upper surface mask of the object. Otherwise, directly use the segmentation result in the current Lab color space (obtained in S23) as the upper surface mask of the object.

5. The method for detecting regular polygonal prisms for a robotic arm according to claim 1, wherein In step three, the edges of objects in the image are detected. Specifically, edge detection is performed on the L component in the Lab color space and the V component in the HSV color space, and then the detection results obtained from the two components are combined.

6. The method for detecting regular polygonal prisms for a robotic arm according to claim 5, wherein In step 4, a mask is used to remove noise from the edge detection results, including removing both internal and external noise from the object edge. Specifically, the mask is expanded and its intersection with the edge detection result is calculated to remove external edge noise. The mask is then eroded and its difference with the edge detection result is calculated to remove internal noise.

7. The method for detecting regular polygonal prisms for a robotic arm according to claim 5, wherein In step 5, Hough circle detection is used to detect whether there is a circle in the intersection result obtained by applying the object mask to the Lab color space. If so, the most suitable upper surface circle is obtained and step 9 is entered. Otherwise, step 6 is entered. The process of obtaining the upper surface circle is as follows: using Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively finding the circle with the largest param2 in the color space, and then taking the circle with the outermost center as the upper surface circle. Param2 is the input parameter of the Hough circle detection method, which indicates the regularity of the circle. Only shapes larger than param2 can be detected as circles. The outer circle is selected as the upper surface circle because the upper surface is always relatively close to the periphery of the image when viewed from a bird's-eye view.

8. The method for detecting regular polygonal prisms for a robotic arm according to claim 1, wherein In step 6, the specific process of adaptively obtaining the longest line segment with a length greater than th1 is as follows: S41, obtaining the line segment th=th1; S42, detecting line segments whose length is greater than th; S43, determine whether a straight line is detected; if no straight line is detected, proceed to the next step; if a straight line is detected, set th+=4 and return to S42; S44, set th-=1; S45, further determine th>th1, if yes, proceed to the next step, otherwise end; S46, detecting line segments whose length is greater than th; S47, determine whether a straight line is detected; if a straight line is detected, record the straight line information and th, and delete the detected straight line in the original image; if not, return to step S44 and set th-=1; After deleting the detected straight lines, repeat the above steps until all detections are completed.

9. The method for detecting regular polygonal prisms for a robotic arm according to claim 1, wherein In step 7, to filter out the edge lines on the object's upper surface, you need to perform three operations: line grouping, line denoising, and line selection: Straight line grouping: All detected straight lines are grouped according to their angles. All straight lines with an angle difference within th2 are grouped into one group, and each group contains at least one straight line. straight line Denoising: Remove side edge lines and other noise. Since the side edge length is shorter and the noise line length is shorter when viewed from a bird's-eye view, the shorter line length is removed. Select a line: From the remaining set of upper and lower surface edge lines, select the upper surface line. Using the rule that upper surface lines are closer to the object's upper surface mask than lower surface lines, compare the difference between the object framed by each line and the mask. For each group, select the line with the smallest difference and consider it the upper surface edge. When selecting lines, you first need to construct an enclosing box for each line, and then compare the enclosed range with the difference between the mask and the enclosed range. To construct the enclosing box, pair each line with the line closest to the center of the mask in the other line group. These lines form the enclosing box. Compare all pixels in the enclosing box A with all pixels in the grabcut mask B. For each line, record the number n of pixels in A but not in B. Then, select the line with the smallest n in each group and consider it to be the edge line of the upper surface.

10. The regular polygonal prism detection method for a robotic arm according to claim 1, wherein In step eight, all current pairings of straight lines are traversed, and the minimum distance from the intersection of the two straight lines to the mask is calculated. If the distance is less than th5, the two straight lines are considered adjacent. The distance from a point to the mask is calculated by drawing a circle with a gradually increasing radius with the point as the center. When the circle begins to intersect with the mask, the radius of the circle is the distance from the point to the mask. After obtaining the adjacent relationship of all straight lines, the vertices of the adjacent straight lines are calculated and connected in sequence to obtain the edge of the upper surface of the object.

Images