A method, device, readable storage medium and system for estimating fastener pose based on visual point cloud

Abstract

The present invention relates to the field of image recognition technology, and in particular to a method, device, readable storage medium, and system for estimating the pose of a fastener based on visual point clouds. The method comprises preprocessing input point cloud data and selecting a point cloud cluster; projecting the point cloud cluster onto a reference plane to obtain a two-dimensional point cloud; performing convex hull detection on the two-dimensional point cloud to extract the edge segments of the fastener; calculating the coordinates (x, y) of the center point of the fastener based on the relationship between the edge segments and the geometric shape of the fastener; selecting a fastener edge segment as a reference edge segment, calculating and correcting the orientation angle of the reference edge segment; and performing a conversion calculation on the orientation angle to obtain the orientation angle of the fastener. The point cloud data is not affected by outdoor light or the position of the fastening robot, and can subsequently be efficiently processed and analyzed to achieve high-precision pose recognition.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, readable storage medium and system for estimating the position and posture of a fastener based on visual point clouds. Background Art

[0002] In recent years, my country's wind power industry has grown rapidly, gradually moving towards deep-sea operations. This has posed significant challenges to wind turbine operation and maintenance. Wind turbines typically consist of multiple tower sections, each connected by flange bolts. A certain amount of preload is applied to maintain rigidity and integrity. Under the influence of long-term alternating wind loads, bolts can loosen and lose preload, potentially leading to tower collapse. Therefore, regular visual inspection and tightening of bolts within wind turbine towers are essential regular maintenance tasks.

[0003] In power systems, manual live-line work poses significant safety risks, especially during equipotential work. Frequent manual work injuries and deaths pose a significant challenge to personnel safety. To prevent the impact of manual operation on fastening quality, while also reducing labor intensity for construction workers, improving the level of digital construction, and accelerating automated bolt tightening, a fastening robot has been developed. The fastening robot is a drone equipped with a power assembly and a sleeve. It uses visual positioning to locate the fastener, then drives the drone to insert the sleeve into the fastener. Finally, the power assembly is driven to twist the fastener, thereby verifying the tightness of the bolt and fastener.

[0004] In the existing patent CN118195986A, the visual positioning technology uses template matching technology to realize the identification of bolts and fasteners. However, wind turbines are all installed outdoors. In outdoor environments, the background of the flange photographed by the drone contains more content than that in the stable indoor environment. In addition, the light outdoors changes over time, which may affect the accuracy of template matching and the recognition effect is poor. Fasteners are usually polygonal nuts. When the sleeve is inserted into the nut, it needs to be aligned according to the position of the nut. When the template matching is inaccurate, the position of the nut cannot be effectively identified, and the sleeve cannot be inserted into the nut surface. In this case, the fastening robot needs to change the angle to take the identification picture and re-match the template for identification. This greatly reduces the efficiency of wind turbine inspection. Therefore, there is an urgent need for an algorithm that can effectively and accurately match the position of nuts and bolts in outdoor environments to cooperate with the fastening robot to realize outdoor wind turbine inspection. Summary of the Invention

[0005] In response to the above-mentioned defects, the purpose of the present invention is to propose a fastener pose estimation method, device, readable storage medium and system based on visual point cloud to solve the problem that fastener pose recognition is easily affected by external environmental conditions, resulting in inaccurate fastener pose recognition.

[0006] To achieve this goal, the present invention adopts the following technical solution: a fastener pose estimation method based on visual point cloud, comprising the following steps:

[0007] Step S1: pre-process the input point cloud data and select a point cloud cluster;

[0008] Step S2: Project the point cloud onto the reference plane to obtain a two-dimensional point cloud; perform convex hull detection on the two-dimensional point cloud to extract the edge segments of the fastener;

[0009] Step S3: Calculate the coordinates (x, y) of the center point of the fastener based on the relationship between the edge segments and the geometric shape of the fastener;

[0010] Step S4: Select an edge segment of a fastener as a reference edge segment, calculate and correct the direction angle of the reference edge segment ; Direction angle Perform conversion calculations to obtain the orientation angle of the fastener .

[0011] Preferably, the pre-processing steps in step S1 are as follows:

[0012] Step S11: perform spatial range cropping on the input point cloud data and complete voxel filtering;

[0013] Step S12: performing a proximity search on the point cloud data according to the Euclidean distance to complete the clustering of the point cloud data and obtain a point cloud cluster;

[0014] Step S13: Count the number of points in each point cloud cluster, and select the point cloud cluster with the largest number of points.

[0015] Preferably, the specific steps of step S2 are as follows:

[0016] Step S21: orthogonally projecting the point cloud cluster onto the reference plane to obtain a two-dimensional point cloud;

[0017] Step S22: performing convex hull detection on the two-dimensional point cloud to extract polygon vertices of the fastener;

[0018] Step S23: Connect the polygon vertices in pairs in order to obtain the first line segment;

[0019] Step S24: Calculate the included angle between adjacent first line segments. When the included angle is less than the angle threshold, merge the line segments. Perform a linear fitting on the merged line segments using the least squares method to generate a second line segment.

[0020] Step S25: According to the geometric relationship of the polygon, the length and angle of the second line segment are checked, abnormal second line segments are eliminated, and the remaining second line segments are determined as edge segments of the fastener.

[0021] Preferably, the specific steps of step S3 are as follows:

[0022] Step S31: Calculating the median of the edge segments and the angle bisectors of adjacent edge segments based on the edge segments of the fastener;

[0023] Step S32: performing pairwise intersection calculations on all medians and angle bisectors to obtain a first set of intersection points;

[0024] Step S33: using the RANSAC algorithm to process the first intersection point set to obtain a second intersection point set;

[0025] Step S34: Calculate the average value of the intersection points of the second intersection point set as the center point coordinates (x, y) of the fastener.

[0026] Preferably, the specific steps of step S4 are as follows:

[0027] Step S41: Calculate the length of each edge segment of the fastener and direction angle , select the longest edge segment as the reference edge segment;

[0028] Step S42: Based on the length of each edge segment of the fastener and direction angle , correct the direction angle of the reference edge segment ;

[0029] Step S43: Direction Angle Perform conversion calculations to obtain the orientation angle of the fastener .

[0030] Preferably, the direction angle of the reference edge segment is calculated and corrected in step S42. The formula is:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, is the exterior angle of a regular polygon, N is the number of sides of the regular polygon, i is an integer from 1 to N, is the direction angle of the reference edge segment before correction, % is the remainder operator, is the length of the i-th edge segment, is the total length of the edge segments, is the direction angle of the i-th edge segment.

[0036] Preferably, in step S43, the direction angle Perform conversion calculations to obtain the orientation angle of the fastener The formula is:

[0037] ;

[0038] ;

[0039] in, is the exterior angle of the regular polygon, and N is the number of sides of the regular polygon.

[0040] A fastener pose estimation device based on visual point cloud comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the fastener pose estimation method when executing the computer program.

[0041] A readable storage medium, characterized in that a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, a fastener pose estimation method as described above is implemented.

[0042] A fastener pose estimation system based on visual point cloud, comprising the fastener pose estimation device camera module and coordinate transformation module;

[0043] The camera module is used to obtain point cloud data of the fastener;

[0044] The coordinate transformation module is used to transform the center point coordinates (x, y) and direction angle of the fastener Transform from camera coordinate system to device coordinate system.

[0045] One of the above technical solutions has the following advantages or beneficial effects: point cloud data is unaffected by deformations such as image rotation and scaling, resulting in greater robustness during use. By identifying the largest point cloud cluster, it can be determined that the cluster is a combination of a fastener and a bolt. The pose of the point cloud cluster is acquired, and the point cloud data is unaffected by outdoor lighting or the position of the fastening robot. Subsequently, the point cloud data can be efficiently processed and analyzed, achieving high-precision pose recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an embodiment of the method of the present invention.

[0047] Figure 2 It is a structural diagram of an embodiment of the system of the present invention.

[0048] Figure 3 This is a schematic structural diagram of a fastening robot observation bolt according to one embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the point cloud of the fastener edge and bolt center positioning;

[0050] Among them: fastening robot 1, depth camera 2, bolt 3, fastener 4. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0052] In the description of the embodiments of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically specified.

[0053] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, "plurality" means two or more. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0054] like Figures 1 to 4 As shown, a fastener pose estimation method based on visual point cloud includes the following steps:

[0055] Step S1: pre-process the input point cloud data and select a point cloud cluster;

[0056] Step S2: Project the point cloud onto the reference plane to obtain a two-dimensional point cloud; perform convex hull detection on the two-dimensional point cloud to extract the edge segments of the fastener;

[0057] Step S3: Calculate the coordinates (x, y) of the center point of the fastener based on the relationship between the edge segments and the geometric shape of the fastener;

[0058] Step S4: Select an edge segment of a fastener as a reference edge segment, calculate and correct the direction angle of the reference edge segment ; Direction angle Perform conversion calculations to obtain the orientation angle of the fastener .

[0059] In some embodiments, a fastening robot equipped with a depth camera moves on the connecting flange layer of the wind turbine tower, and observes the bolts on the flange during its movement; when the fastening robot reaches directly above the target bolt, it rotates the sleeve and makes it fall vertically to cover the target nut, and tightens the target bolt by applying a specified pre-tightening force. During this process, the positions of the fastening robot body, the depth camera, and the sleeve are relatively fixed. The depth camera is required to continuously observe the bolts on the flange to determine the next operation target of the sleeve and provide the fastening robot with a clear nut posture, that is, the position and direction angle of the nut. Figure 3 As shown, a fastening robot moves across the flange connection of a wind turbine tower. During this movement, a depth camera captures the bolts and nuts on the flange, generating point cloud data. This point cloud data is unaffected by deformations such as image rotation and scaling, providing enhanced robustness during use. The pose of this point cloud cluster is captured, unaffected by outdoor lighting or the position of the fastening robot. This data can then be efficiently processed and analyzed, enabling high-precision pose recognition.

[0060] The point cloud cluster is then projected onto a reference plane to obtain a two-dimensional point cloud, where the reference plane can be the plane where the flange is located. The edges in the point cloud cluster are obtained using existing convex hull detection technology. Convex hull detection is a basic algorithm in computational geometry that can analyze the contours of the point cloud cluster to clearly obtain the corresponding edge information, and select the corresponding fastener edge segment from the edge information. The fastener is a regular polygonal nut, and the midline line or angle bisector line of the fastener edge segment falls on the center of the bolt (fastener). Therefore, after convex hull detection, based on the relationship between the fastener edge segment and its geometric shape, the coordinates of the fastener's center point can be easily obtained based on the midline data and angle bisector data, and the center position of the bolt is used as the positioning position of the fastener.

[0061] The fastener edge obtained from the point cloud data can be obtained by constructing a simple coordinate system to obtain the coordinates of the two end points of the fastener edge, and then the direction angle of the fastener edge line is calculated to obtain the angle of each fastener edge, and any fastener edge segment is used as the reference edge segment, and the direction angle of the reference edge segment is used as the reference edge segment. Just the posture of the fastener.

[0062] However, due to the influence of the shooting position of the robot's depth camera, the direction angle is deviated and cannot fit the actual angle. Make corrections to get the corrected direction angle By the direction angle Update and correct the image to improve the accuracy of the fastener pose.

[0063] Finally, in one embodiment, a pre-shipment hand-eye calibration of the device can determine a fixed transformation relationship between the depth camera and the sleeve on the flange. Therefore, the bolt center point position and fastener orientation angle in the sleeve coordinate system can be calculated using the coordinate system transformation matrix, achieving sleeve alignment.

[0064] It should be noted that the above description uses the bolt assembly of a wind turbine tower as an example to obtain the position and angle of the nut on the bolt, but it does not constitute a limitation to this application. This application can also be used to obtain the position and angle of bolts or screws or other fasteners with polygonal features.

[0065] Preferably, the pre-processing steps in step S1 are as follows:

[0066] Step S11: perform spatial range cropping on the input point cloud data and complete voxel filtering;

[0067] Spatial range clipping can limit the points in the point cloud data to a specific spatial range, reducing unnecessary point cloud data input and alleviating the pressure of calculation. It is then downsampled through voxel filtering to reduce the number of points in the point cloud while maintaining its overall shape.

[0068] Step S12: performing a proximity search on the point cloud data according to the Euclidean distance to complete the clustering of the point cloud data and obtain a point cloud cluster;

[0069] When clustering, the Euclidean distance between adjacent point clouds can be calculated to determine whether the Euclidean distance is less than a distance threshold, and the point clouds with distances less than the distance threshold are regarded as point cloud clusters.

[0070] Step S13: Count the number of points in each point cloud cluster, and select the point cloud cluster with the largest number of points.

[0071] After step S11, the processing scope has been narrowed, reducing the input of unnecessary features. However, the flange contains bolts, fasteners, and some small debris such as dust. At this point, the bolts and fasteners are assembled together, forming the largest single object on the flange. By identifying the largest point cloud cluster, it can be determined that this point cloud cluster is the combination of the fastener and bolt. Therefore, only the point cloud cluster with the largest number of points needs to be processed later, and the other point cloud clusters can be deleted.

[0072] Preferably, the specific steps of step S2 are as follows:

[0073] Step S21: orthogonally projecting the point cloud cluster onto the reference plane to obtain a two-dimensional point cloud;

[0074] Step S22: performing convex hull detection on the two-dimensional point cloud to extract polygon vertices of the fastener;

[0075] Step S23: Connect the polygon vertices in pairs in order to obtain the first line segment;

[0076] Step S24: Calculate the included angle between adjacent first line segments. When the included angle is less than the angle threshold, merge the line segments. Perform a linear fitting on the merged line segments using the least squares method to generate a second line segment.

[0077] Step S25: According to the geometric relationship of the polygon, the length and angle of the second line segment are checked, abnormal second line segments are eliminated, and the remaining second line segments are determined as edge segments of the fastener.

[0078] The acquired point cloud is three-dimensional point cloud data, which includes height information in three-dimensional space. However, height information is useless when calculating the orientation angle. Furthermore, points on the fastener edge may be scattered due to height variations. Orthogonally projecting the point cloud onto the reference plane (flange) eliminates the interference of height information on edge extraction, allowing the partially scattered point cloud on the fastener edge to fall within the detection. Furthermore, processing two-dimensional point cloud data is simpler than processing three-dimensional point cloud data, significantly reducing the computational effort. This helps improve the algorithm's speed and meet real-time requirements.

[0079] Then, a convex hull detection is performed. Since the three-dimensional point cloud data is converted into two-dimensional point cloud data, the convex hull detection process includes more vertices than the original three-dimensional point cloud data. Convex hull detection only extracts polygon vertices from the two-dimensional point cloud and cannot accurately determine the multiple vertices of the regular polygon of the fastener. Because regular polygons have many vertices, the lines connecting two adjacent polygon vertices form multiple first line segments of varying lengths. If the angle between adjacent first line segments is less than 20°, it indicates that the two first line segments belong to the same edge and are merged into a single line segment. Linear fitting is performed on the merged first line segments to generate a second line segment. Finally, length and angle thresholds are used to filter out the edge segments of the fastener from the second line segments. Therefore, an angle threshold is also set in the present invention. By obtaining the angle between adjacent first line segments and determining whether the angle is less than the angle threshold, if so, the adjacent first line segments are considered to be extensions of the same contour segment. Linear fitting can then be performed using the least squares method to generate the second line segment.

[0080] Finally, the second line segment is determined to be an edge segment based on the fastener's geometric shape. For example, the length of the second line segment, the angle between adjacent line segments, and the length are verified to determine whether they meet the fastener parameter standards. For example, if the fastener edge length is 60, the standard for the fastener edge length can be set to 30-70. Therefore, as long as the value of the second line segment falls within 30-70, it can be determined to be a fastener edge. Similarly, for angle determination, the specific angle between each edge segment can be determined based on the number of sides of the fastener. For example, in a regular hexagonal fastener, the angle between edge segments is 120°. In this case, the angle determination threshold can be set to 110-130°. When the angle between adjacent second line segments falls within this angle range, it can be determined that the second line segment meets the angle requirements. Only second line segments that meet both the angle and length requirements are confirmed as edge segments. Other line segments are considered abnormal and need to be eliminated.

[0081] It is worth mentioning that when setting the fastener edge length standard, a lower lower limit should be set. Because of the influence of the shooting position of the depth camera, part of the fastener edge will be blocked by the bolt, resulting in shorter fastener edges when performing line segment fitting. The more fastener edges are detected, the higher the accuracy of subsequent positioning and angular error correction can be.

[0082] Preferably, the specific steps of step S3 are as follows:

[0083] Step S31: Calculating the median of the edge segments and the angle bisectors of adjacent edge segments based on the edge segments of the fastener;

[0084] Step S32: performing pairwise intersection calculations on all medians and angle bisectors to obtain a first set of intersection points;

[0085] Step S33: using the RANSAC algorithm to process the first intersection point set to obtain a second intersection point set;

[0086] Step S34: Calculate the average value of the intersection points of the second intersection point set as the center point coordinates (x, y) of the fastener.

[0087] Although in theory, the center of the bolt will fall at the intersection of the line connecting the center lines of the fastener edge or the intersection of the line connecting the angle bisectors. Due to the sampling accuracy of the point cloud data, the polygon vertices of the convex hull detection have multiple points, and the edge line segments of the fastener formed by fitting are not necessarily complete and accurate. Therefore, the intersection of the center line and the angle bisector does not coincide. Figure 4 As shown, at this time, there is more than one intersection point when the midline and the angle bisector intersect each other. In order to more accurately determine the center position, the present invention will process the first intersection point set using the existing RANSAC algorithm to obtain a second intersection point set. The specific RANSAC algorithm will estimate a center point range and then filter out the first intersection point set that falls within the center point range as the second intersection point set. Finally, the center position of the bolt is determined by obtaining the average value of the second intersection point set. The average value of multiple second intersection points can reduce the impact of random errors of individual intersection points and improve the accuracy of center position determination.

[0088] Preferably, the specific steps of step S4 are as follows:

[0089] Step S41: Calculate the length of each edge segment of the fastener and direction angle , select the longest edge segment as the reference edge segment;

[0090] Step S42: Based on the length of each edge segment of the fastener and direction angle , correct the direction angle of the reference edge segment ;

[0091] Step S43: Direction Angle Perform conversion calculations to obtain the orientation angle of the fastener .

[0092] Preferably, the direction angle of the reference edge segment is calculated and corrected in step S42. The formula is:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, is the exterior angle of a regular polygon, N is the number of sides of the regular polygon, i is an integer from 1 to N, is the direction angle of the reference edge segment before correction, % is the remainder operator, is the length of the i-th edge segment, is the total length of the edge segments, is the direction angle of the i-th edge segment.

[0098] The formula for obtaining the direction angle is as follows: , RAD2DEG is the radian to angle operation, and are the ordinate difference and abscissa difference of the two endpoints of the fastener edge segment, respectively.

[0099] Due to the shooting angle of the depth camera, the length of the edge of the fastener cannot be kept consistent. In order to facilitate the determination of the direction reference and calculation, the longest edge segment of the fastener is selected as the reference edge segment in the present invention. The direction angle of the reference edge segment is used as its specific value.

[0100] By error value Correct the angle deviation caused by the depth camera shooting angle so that the direction angle is consistent with the actual angle, ensuring that the sleeve of the fastening robot can effectively align with the fastener and achieve the purpose of detecting whether the fastener is tightened.

[0101] Preferably, in step S43, the direction angle Perform conversion calculations to obtain the orientation angle of the fastener The formula is:

[0102] ;

[0103] ;

[0104] in, is the exterior angle of the regular polygon, N is the number of sides of the regular polygon,

[0105] This solution uses the minimum direction angle of the edge segment of the fastener as the direction angle β of the fastener. Since regular polygons have the property of equal internal angles, the direction angle of the edge segment is has an arithmetical relationship with the direction angle β. In other words, the direction angle of any edge segment The modulus of the tolerance is calculated to obtain the fastener's orientation angle β. Continuing with the example of accurately fitting a nut with a sleeve, when the nut's orientation angle is 0°, the nut's bottom and top edges are horizontal, and the sleeve can simply be dropped vertically to fit the nut. When the nut's orientation angle is β, the sleeve's rotation angle is β, and the sleeve can then be rotated and dropped vertically to fit the nut.

[0106] A fastener pose estimation device based on visual point cloud comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the fastener pose estimation method when executing the computer program.

[0107] A readable storage medium, characterized in that a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, a fastener pose estimation method as described above is implemented.

[0108] A fastener pose estimation system based on visual point cloud, comprising the fastener pose estimation device camera module and coordinate transformation module;

[0109] The camera module is used to obtain point cloud data of the fastener;

[0110] The coordinate transformation module is used to transform the center point coordinates (x, y) and direction angle of the fastener Transform from camera coordinate system to device coordinate system.

[0111] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0112] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A fastener pose estimation method based on visual point cloud, characterized in that: The steps include: Step S1: pre-process the input point cloud data and select a point cloud cluster; Step S2: Project the point cloud onto the reference plane to obtain a two-dimensional point cloud; perform convex hull detection on the two-dimensional point cloud to extract the edge segments of the fastener; Step S3: Calculate the coordinates (x, y) of the center point of the fastener based on the relationship between the edge segments and the geometric shape of the fastener; Step S4: Select an edge segment of a fastener as a reference edge segment, calculate and correct the direction angle of the reference edge segment ; Direction angle Perform conversion calculations to obtain the orientation angle of the fastener ; The specific steps of step S4 are as follows: Step S41: Calculate the length of each edge segment of the fastener and direction angle , select the longest edge segment as the reference edge segment; Step S42: Based on the length of each edge segment of the fastener and direction angle , correct the direction angle of the reference edge segment ; Step S43: Direction Angle Perform conversion calculations to obtain the orientation angle of the fastener ; In step S42, the direction angle of the reference edge segment is calculated and corrected. The formula is: ； ； ； ； in, is the exterior angle of a regular polygon, N is the number of sides of the regular polygon, i is an integer from 1 to N, is the direction angle of the reference edge segment before correction, % is the remainder operator, is the length of the i-th edge segment, L is the total length of the edge segments, is the direction angle of the i-th edge segment.

2. The method for fastener pose estimation based on visual point cloud according to claim 1, characterized in that: The pre-processing steps in step S1 are as follows: Step S11: perform spatial range cropping on the input point cloud data and complete voxel filtering; Step S12: performing a proximity search on the point cloud data according to the Euclidean distance to complete the clustering of the point cloud data and obtain a point cloud cluster; Step S13: Count the number of points in each point cloud cluster, and select the point cloud cluster with the largest number of points.

3. The method for fastener pose estimation based on visual point cloud according to claim 1, characterized in that: The specific steps of step S2 are as follows: Step S21: orthogonally projecting the point cloud cluster onto the reference plane to obtain a two-dimensional point cloud; Step S22: performing convex hull detection on the two-dimensional point cloud to extract polygon vertices of the fastener; Step S23: Connect the polygon vertices in pairs in order to obtain the first line segment; Step S24: Calculate the angle between adjacent first line segments, and when the angle is less than the angle threshold, perform a line segment merging operation; The merged line segments are fitted with a least square method to generate a second line segment; Step S25: According to the geometric relationship of the polygon, the length and angle of the second line segment and / or the first line segment are checked, abnormal second line segments and / or first line segments are eliminated, and the remaining second line segments and / or first line segments are determined as edge segments of the fastener.

4. The method for fastener pose estimation based on visual point cloud according to claim 1, characterized in that: The specific steps of step S3 are as follows: Step S31: Calculating the median of the edge segments and the angle bisectors of adjacent edge segments based on the edge segments of the fastener; Step S32: performing pairwise intersection calculations on all medians and angle bisectors to obtain a first set of intersection points; Step S33: using the RANSAC algorithm to process the first intersection point set to obtain a second intersection point set; Step S34: Calculate the average value of the intersection points of the second intersection point set as the center point coordinates (x, y) of the fastener.

5. The method for fastener pose estimation based on visual point cloud according to claim 1, characterized in that: In step S43, the direction angle Perform conversion calculations to obtain the orientation angle of the fastener The formula is: ； ； in, is the exterior angle of the regular polygon, and N is the number of sides of the regular polygon.

6. A fastener pose estimation device based on visual point cloud, characterized in that: The invention comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the fastener pose estimation method according to any one of claims 1 to 5 when executing the computer program.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the fastener pose estimation method according to any one of claims 1 to 6 is implemented.

8. A fastener pose estimation system based on visual point cloud, characterized in that: comprising the fastener pose estimation device as claimed in claim 6, as well as a camera module and a coordinate transformation module; The camera module is used to obtain point cloud data of the fastener; The coordinate transformation module is used to transform the center point coordinates (x, y) and direction angle of the fastener Transform from camera coordinate system to device coordinate system.

Images