Bolt attitude estimation method based on deep learning and template matching

Abstract

The invention discloses a bolt attitude estimation method based on deep learning and template matching. The bolt attitude estimation method comprises the following steps: S1, collecting an image of a cantilever bolt of a contact network and making a data set; s2, marking a bounding box of the bolt image and hexagonal shape area information of the bolt; s3, identifying six angular points of the bolt through a deep learning method; and S4, obtaining 6D pose information of the bolt. According to the method, the angular points and the 6D poses of the bolts can be accurately recognized by combining deep learning and template matching, the precision of bolt pose estimation is greatly improved, meanwhile, the positioning and pose conditions of a plurality of bolts in one area can be detected in real time, efficient detection and segmentation of the bolts are achieved, and the processing speed and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a bolt posture estimation method based on deep learning and template matching. Background Art

[0002] The application of machine vision in manufacturing has brought higher precision, efficiency and reliability to industrial production, and promoted the transformation of manufacturing towards digitalization and intelligence. In recent years, the application of machine vision in bolt posture estimation has gradually increased. Bolt posture estimation is of great significance in the field of industrial manufacturing and automation. It brings higher precision, efficiency and reliability to the manufacturing and assembly process by accurately determining the position and direction of the bolt in space, and promotes the manufacturing industry to move towards intelligence and automation. The continuous advancement of bolt posture estimation technology has not only improved the automation level of industrial assembly, but also played an important role in aerospace, automobile manufacturing, precision instruments and other fields. The application of these technologies provides a solid foundation for intelligent manufacturing, making production lines more intelligent and automated, while also providing precise guidance for robot operations and improving the safety and efficiency of the production process.

[0003] Although the existing bolt posture estimation methods have made some progress in accuracy and efficiency, there are still some shortcomings and challenges. First, the robustness of machine vision technology in complex environments is poor, and it is greatly affected by illumination changes, occlusion and background interference, resulting in a decrease in the accuracy of posture estimation. Secondly, although lidar and structured light technology can provide high-precision three-dimensional point cloud data, the equipment cost is high and the effect is not good on the surface of highly reflective or transparent materials. Robot vision and control systems rely on the collaborative work of multiple sensors, and the complexity of system integration and real-time processing is high, which increases the difficulty of development and maintenance. In addition, augmented reality technology has high hardware requirements, expensive equipment and poor wearing comfort, which affects its wide application in industrial environments. Artificial intelligence and machine learning methods require a large amount of labeled data for training, and the cost of data acquisition and labeling is high. At the same time, the demand for computing resources is large, and the model training and deployment time is long, which limits its rapid application in actual production. In general, the existing methods face problems such as high cost, poor robustness, complex system, and large data demand in practical applications, which affects the popularization and promotion of bolt posture estimation technology. Summary of the invention

[0004] The purpose of the present invention is to provide a bolt posture recognition method based on deep learning and template matching, which solves the technical problems of poor positioning and posture estimation accuracy, unstable detection accuracy and low matching accuracy in existing bolt posture estimation methods.

[0005] To achieve the above object, the present invention provides the following technical solution: a bolt posture estimation method based on deep learning and template matching, comprising the following steps:

[0006] S1, collect images of overhead line arm bolts and make them into a data set, and expand the image data set through image enhancement;

[0007] S2, annotating the bounding box of the bolt image, the category label corresponding to the bounding box, and the hexagonal shape area information of the bolt;

[0008] S3, identify the six corner points of the bolt through deep learning method;

[0009] S4. Obtain the 6D pose information of the bolt.

[0010] In a specific embodiment, S3 includes the following steps:

[0011] S31: Target detection: Use the labeled bolt image data to train the PP-YOLOE target detection model, and use the trained PP-YOLOE target detection model to perform target detection on the bolts;

[0012] S32: Semantic segmentation: Use the PP-LiteSeg semantic segmentation model to perform semantic segmentation on the bolt image after target detection to obtain the mask image of the bolt;

[0013] S33: LSD line detection: Use the improved LSD line detection algorithm to detect the mask image of the bolt and obtain the six edges of the bolt;

[0014] S34: Ordered corner point detection: The intersection points of the six edges of the bolt image are calculated to obtain the six corner points of the bolt. At this point, the corner point detection of the bolt image is completed.

[0015] In a specific embodiment, in S33, the improved LSD line detection algorithm includes the following steps:

[0016] S331, extracting the edge of the bolt by using Canny edge detection in images of different scales to generate a multi-scale edge image;

[0017] S332, optimizing the bolt edge by morphology to remove noise and fine edges; fusing edge information at different scales by multi-scale fusion to generate a comprehensive edge map;

[0018] S333, performing LSD line segment detection on the optimized edge map to generate an initial line segment candidate set;

[0019] S334. On the initial candidate lines, redundant line segments are removed by non-maximum suppression, and the main line segment candidates are retained; the line segments are verified and enhanced by Hough transform, the main line segment features are identified by Hough transform, the line segments detected by LSD are supplemented and corrected, and the accuracy of the line segments is improved.

[0020] In a specific embodiment, in S34, the ordered corner point detection includes the following steps:

[0021] S341, after grayscale processing of the obtained mask image, line segment detection is performed on the grayscale image using an LSD detection function to obtain the start and end point information of all detectable line segments in the image;

[0022] S342, sorting all the line segment information obtained in S33 by Euclidean length, selecting the longest six groups of line segment information, and calculating the mathematical expression of the line segment;

[0023] S343, sorting the closest points of the starting points or end points in the six groups of line segment information, traversing the remaining 11 points, finding the point closest to the starting point or end point, and obtaining an ordered point sequence;

[0024] S344. According to the point sequence, the intersection of the line segment expression is calculated to obtain the ordered position information of the bolt corner points.

[0025] In a specific embodiment, S4 includes the following steps:

[0026] S41, establishing a CAD model of the object and importing the CAD model into a three-dimensional graphics software in a CAD format as an input;

[0027] S42, performing image preprocessing on the data collected by the depth camera installed at the end of the robotic arm, and detecting the characteristic information of the bolt;

[0028] S43, performing feature matching on the detected initial point of the bolt and the initial point of the object CAD model, and storing the matched feature information;

[0029] S44, if the feature matching in S43 is unsuccessful, return to S42 to detect the bolt feature information until the matching is successful;

[0030] S45, tracking the bolt and estimating the 6D pose to obtain the pose of the bolt.

[0031] In a specific embodiment, the feature information detection of the bolt in S42 includes the feature point, feature line and feature surface detection of the bolt.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. By combining deep learning and template matching methods, the present invention can accurately identify the corner points and 6D pose of the bolt, greatly improving the accuracy of bolt pose estimation.

[0034] 2. The present invention utilizes the PP-YOLOE target detection model and the PP-LiteSeg semantic segmentation model, both of which are lightweight deep learning algorithms that can detect the positioning and posture of multiple bolts in an area in real time, achieve efficient detection and segmentation of bolts, and improve processing speed and efficiency.

[0035] 3. By combining the depth camera and the CAD model, the present invention can be applied in actual industrial environments to provide accurate bolt posture information, assist robot assembly and detection, and improve the intelligence and automation level of the production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall flow chart of bolt posture estimation of the present invention;

[0037] Figure 2 The improved LSD line detection flow chart of the present invention is

[0038] Figure 3 This is a flow chart of bolt ordered corner point detection of the present invention;

[0039] Figure 4 A bolt picture for obtaining corner point position information of the present invention;

[0040] Figure 5 This is a flow chart of obtaining 6D posture information of a bolt according to the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] See also Figure 1 , a bolt posture estimation method based on deep learning and template matching, comprising the following steps:

[0043] S1. Collect images of overhead contact arm bolts and create a data set.

[0044] A realsense D456 camera was used to collect image datasets of contact network arm bolts, and an image enhancement algorithm was used to simulate images of contact network arm bolts in weather or environments such as rain, snow, fog, and night to expand the image dataset.

[0045] S2: annotate the bounding box of the bolt image and the hexagonal shape area information of the bolt.

[0046] The bolt images were annotated using Labelme image data annotation software. The bolt bounding boxes were annotated. Each bounding box had a corresponding bolt category label. The six edges of the bolts were manually annotated to crop the bolts from the original image and obtain a bolt mask image.

[0047] S3. Identifying the six corner points of the bolt by deep learning method includes the following steps: Figure 3 As shown:

[0048] Object detection: Use the bolt image data annotated in S2 to train the PP-YOLOE object detection model, and use the trained PP-YOLOE object detection model to detect bolts;

[0049] Semantic segmentation: Use the PP-LiteSeg semantic segmentation model to perform semantic segmentation on the bolt image after target detection to obtain the mask image of the bolt;

[0050] LSD line detection: Use the improved LSD line detection algorithm to detect the mask image of the bolt and obtain the six edges of the bolt;

[0051] like Figure 2 As shown, the improved LSD line detection algorithm includes the following steps:

[0052] In images of different scales, the edges of bolts are extracted by using Canny edge detection to generate multi-scale edge images; by scaling (usually reducing) the images, images of different resolutions can be generated, and these images of different resolutions are called images of different scales. Edge detection at each scale can capture edge features of different sizes and shapes, thereby improving the robustness and accuracy of image detection.

[0053] The edges of the bolts are optimized through morphology such as dilation and erosion, noise and small edges are removed, and the main structure of the image is retained; the dilation operation will expand and grow the objects in the image, fill the holes inside the objects, and connect the nearby objects. Specifically, the dilation operation will replace each pixel in the image with the maximum value of the pixels around it (even if the white area is expanded). The erosion operation will shrink and contract the objects in the image, eliminate the small protrusions of the objects, and separate the connected objects. Specifically, the erosion operation will replace each pixel in the image with the minimum value of the pixels around it (even if the black area is expanded).

[0054] Through multi-scale fusion (maximum fusion), the edge information at different scales is fused to generate a comprehensive edge map, which improves the detection ability of line segments of different bolt sizes and shapes.

[0055] The following steps are involved:

[0056] Features of different scales are fused, usually using methods such as maximum fusion, average fusion or convolution fusion.

[0057] Different fusion methods:

[0058] Maximum fusion: Take the maximum value of features at different scales. It is suitable for tasks such as edge detection that need to retain strong features.

[0059] Average fusion: Take the average of features of different scales, which is suitable for tasks that require smooth features.

[0060] Convolution fusion: weighted fusion of multi-scale features through convolution operation, suitable for feature fusion in deep learning models.

[0061] Perform LSD line segment detection on the optimized edge map to generate an initial line segment candidate set. Non-maximum suppression is used on the initial candidate lines to remove redundant line segments and retain the main line segment candidates; the line segments are verified and enhanced by Hough transform, the main line segment features are identified by Hough transform, and the line segments detected by LSD are supplemented and corrected to improve the accuracy of the line segments. The Hough transform is achieved by converting points in the image space (image coordinate system) to the parameter space (Hough space).

[0062] Ordered corner point detection: Find the intersection points of the six edges of the bolt image to obtain the six corner points of the bolt. At this point, the corner point detection of the bolt image is completed. It includes the following steps:

[0063] The obtained mask image is grayscale processed to facilitate line segment detection and obtain high-precision straight line information. Initialize the LSD detection function, set the detector parameters, perform line segment detection on the grayscale image through the LSD detection function, and obtain the start and end point information of all detectable line segments in the image;

[0064] In order to eliminate environmental interference in the actual operation site, considering that the bolt has a hexagonal structure, all the obtained line segment information is sorted by Euclidean length, the longest six groups of line segment information are selected, and the mathematical expression of the line segment is calculated.

[0065] In order to obtain the orderly arranged corner point position information, the starting point or end point in the six groups of line segment information is sorted by the nearest point. By traversing the remaining 11 points, the distance between the remaining points and the point is calculated, and the point closest to the starting point or end point is found to obtain an orderly arranged point sequence; based on the orderly arranged sequence, the intersection point calculation is performed on the line segment expression, and the coordinates of the intersection point are found by solving the two line segment equations to obtain the orderly position information of the bolt corner point. The bolt picture with the corner point position information is shown in the figure below. Figure 4 shown.

[0066] S4. Obtain the 6D pose information of the bolt, such as Figure 5 shown.

[0067] Create a CAD model of the object and use the CAD model as input in CAO format. Create the model to be matched through SolidWorks, then import it into Blender for processing, and finally export a CAO file for matching.

[0068] The data collected by the depth camera installed at the end of the robotic arm is subjected to image preprocessing; and the feature information of the bolt is detected, including the feature point, feature line and feature surface detection of the bolt.

[0069] S43, feature matching is performed between the detected initial point of the bolt and the initial point of the object CAD model. Common algorithms include SIFT, SURF, ORB, etc., and the feature information after matching is stored; the 6D pose of the template in the actual scene is calculated through the matched feature points. If the feature matching is unsuccessful, the bolt feature information is detected until the matching is successful. After the matching is successful, the bolt is tracked and the 6D pose is estimated to obtain the bolt's pose.

Claims

1. A bolt posture estimation method based on deep learning and template matching, characterized in that: The following steps are involved: S1, collect images of overhead contact arm bolts and make them into a data set; S2, annotating the bounding box of the bolt image and the hexagonal shape area information of the bolt; S3, identify the six corner points of the bolt through deep learning method; S4. Obtain the 6D pose information of the bolt.

2. The bolt posture estimation method based on deep learning and template matching according to claim 1 is characterized in that: S3 includes the following steps: S31: Target detection: Use the labeled bolt image data to train the PP-YOLOE target detection model, and use the trained PP-YOLOE target detection model to perform target detection on the bolts; S32: Semantic segmentation: Use the PP-LiteSeg semantic segmentation model to perform semantic segmentation on the bolt image after target detection to obtain the mask image of the bolt; S33: LSD line detection: Use the improved LSD line detection algorithm to detect the mask image of the bolt and obtain the six edges of the bolt; S34: Ordered corner point detection: The intersection points of the six edges of the bolt image are calculated to obtain the six corner points of the bolt. At this point, the corner point detection of the bolt image is completed.

3. The bolt posture estimation method based on deep learning and template matching according to claim 2 is characterized in that: In S33, the improved LSD line detection algorithm comprises the following steps: S331, generating a multi-scale edge image by extracting the edge of the bolt at different scales by using Canny edge detection; S332, optimizing the bolt edge by morphology to remove noise and fine edges; fusing edge information at different scales by multi-scale fusion to generate a comprehensive edge map; S333, performing LSD line segment detection on the optimized edge map to generate an initial line segment candidate set; S334. On the initial candidate lines, redundant line segments are removed by non-maximum suppression, and the main line segment candidates are retained; the line segments are verified and enhanced by Hough transform, the main line segment features are identified by Hough transform, the line segments detected by LSD are supplemented and corrected, and the accuracy of the line segments is improved.

4. The bolt posture estimation method based on deep learning and template matching according to claim 2 is characterized in that: In S34, the ordered corner point detection includes the following steps: S341, after grayscale processing of the obtained mask image, line segment detection is performed on the grayscale image using an LSD detection function to obtain the start and end point information of all detectable line segments in the image; S342, sorting all the line segment information obtained in S33 by Euclidean length, selecting the longest six groups of line segment information, and calculating the mathematical expression of the line segment; S343, sorting the closest points of the starting points or end points in the six groups of line segment information, traversing the remaining 11 points, finding the point closest to the starting point or end point, and obtaining an ordered point sequence; S344. According to the point sequence, the intersection of the line segment expression is calculated to obtain the ordered position information of the bolt corner points.

5. The bolt posture estimation method based on deep learning and template matching according to claim 1 is characterized in that: S4 includes the following steps: S41, establishing a CAD model of the object and importing the CAD model into a three-dimensional graphics software in a CAD format as an input; S42, performing image preprocessing on the data collected by the depth camera installed at the end of the robotic arm, and detecting the characteristic information of the bolt; S43, performing feature matching on the detected initial point of the bolt and the initial point of the object CAD model, and storing the matched feature information; S44, if the feature matching in S43 is unsuccessful, return to S42 to detect the bolt feature information until the matching is successful; S45, tracking the bolt and estimating the 6D pose to obtain the pose of the bolt.

6. The bolt posture estimation method based on deep learning and template matching according to claim 5 is characterized in that: The feature information detection of the bolt in S42 includes the detection of feature points, feature lines and feature surfaces of the bolt.

Images