A detection method for the loosening of wire-connected bolts

Abstract

The present invention discloses a detection method for the looseness of wire-connected bolts, belonging to the technical field of machine vision detection. Two-dimensional color data and three-dimensional point cloud data of the wire-connected bolt group to be detected are obtained through a 3D camera, and a neural network is used to perform target detection on the wire-connected bolt group; the position of the wire to be detected is obtained by performing line detection on the detection area, and the point cloud data of the corresponding area is extracted; the dimensionality reduction in the depth direction is performed on the point cloud data; the shape of the wire in the depth direction is obtained, and the curvature of the depth change of the wire is calculated to realize the detection of the connection state of the wire. The present invention locates the position of the wire through a two-dimensional image, judges its fracture or missing failure form, and further combines the three-dimensional point cloud data to judge the curvature of the depth change of the wire for looseness detection, which can effectively improve the detection accuracy and assist in intelligent daily maintenance operations.

Description

Technical Field

[0001] The present invention relates to a method for detecting the failure of the wire stringing anti-loosening method, in particular to a method for detecting the anti-loosening failure of wire-strung bolts based on two-dimensional images and point cloud data, belonging to the technical field of machine vision detection. Background Art

[0002] Wire stringing anti-loosening is a mechanical anti-loosening method, which means that low-carbon steel wires are inserted into the holes in the heads of each screw, and the screws are connected in series to restrict each other. This anti-loosening method is mainly applicable to bolt group connections, with reliable anti-loosening, and is usually applied to the joints in key areas, and is also the key detection item focused on during daily inspections. With the development of intelligent maintenance, the method of defect detection using non-contact two-dimensional images and three-dimensional point clouds is increasingly applied in various scenarios due to its simple and efficient characteristics. However, different from the detection of ordinary bolts through marking lines, in addition to fracture and missing, the failure forms of wire stringing anti-loosening also include the intermediate state of the disappearance of the pre-tightening force. Therefore, it is difficult to obtain the spatial morphological changes of the wire through two-dimensional image detection. The present invention proposes a method for detecting the anti-loosening failure of wire-strung bolts based on two-dimensional image data and point cloud data, which can realize the positioning and state detection of the strung wire, ensure the accuracy of detection, and thus better assist in realizing intelligent daily maintenance operations. Summary of the Invention

[0003] Aiming at the above deficiencies of the existing detection technology, the method for detecting the failure of the wire stringing anti-loosening method for bolt groups provided by the present invention solves the problem that the existing detection methods cannot detect the disappearance of the pre-tightening force of the wire. The present invention is realized through the following technical solutions:

[0004] A detection method for wire-strung bolt loosening, based on the fusion of two-dimensional image data and point cloud data, specifically includes the following steps:

[0005] 1) Obtain a three-dimensional depth image with color information through a 3D color camera;

[0006] 2) Use the trained neural network to perform target detection on the two-dimensional color image to identify the position of the wire-strung bolt group;

[0007] 3) Perform straight line detection and condition screening in the bolt group area identified in step 2):

[0008] If there is a corresponding straight line, it indicates the position of the wire, and enter step 4);

[0009] If no straight line is recognized, it indicates that the wire is missing, broken or bent, indicating that the anti-loosening method has failed;

[0010] 4) Obtain the point cloud data at the corresponding positions of the steel wires detected in step 3), and perform dimensionality reduction in the depth direction on it;

[0011] 5) For the steel wire depth curve obtained in step 4), calculate its depth curvature. By setting a certain threshold, determine whether the steel wire is bent in the depth direction, that is, it shows relaxation, so as to judge whether it fails.

[0012] Furthermore, the specific steps of using the trained neural network to perform target detection on the series-connected steel wire bolt group in the two-dimensional color image in step 2) are as follows:

[0013] 2.1) Perform target dataset annotation on the area of the steel wire series-connected bolt group to obtain the network training dataset;

[0014] 2.2) Use the Yolov5 network model to perform target detection training;

[0015] 2.3) Input the image to be detected containing the steel wire series-connected bolt group into the neural network trained in step 2.2), and obtain the coordinate positioning output of the area of the steel wire series-connected bolt group.

[0016] Furthermore, in step 3), perform straight line detection and screening on the recognized bolt group area, specifically including the following steps:

[0017] 3.1) Perform median filtering on the image of the steel wire series-connected bolt group area obtained in step 2);

[0018] 3.2) Calculate the edges of the processing result obtained in step 3.1) using the Canny operator;

[0019] 3.3) Perform Hough straight line detection on the edge image obtained in step 3.2). By setting the minimum straight line distance and interval, obtain the position coordinates of the two endpoints of the steel wire.

[0020] Furthermore, in step 4), after obtaining the point cloud data at the corresponding positions of the screws, perform dimensionality reduction in the depth direction on it, specifically including the following steps:

[0021] 4.1) Obtain the position coordinates of the two ends of the steel wire from the straight line detection result in step 3), and convert the two-dimensional area coordinates to the coordinate information in the point cloud data;

[0022] 4.2) In the point cloud data, by calculating the distances of the two endpoints along the X and Y directions respectively, use the direction with the larger distance as the reference direction for sampling with a unit of one, and take the point cloud data closest to the straight line according to the nearest principle for the coordinate values in the other direction;

[0023] 4.3) Arrange the depth data in sequence according to the unit distance, and remove the singular points in the data through median filtering;

[0024] 4.4) Obtain the depth values along the straight lines where each steel wire is located after dimensionality reduction through the above steps, and calculate the average value change data along the longest reference direction.

[0025] Further, in step 5), calculate the depth curvature of the steel wire, and determine whether the steel wire is loose in the depth direction by setting a certain threshold, which specifically includes the following steps:

[0026] 5.1) Use the least squares method to calculate the center and radius of the fitting curve;

[0027] 5.2) Calculate the curvature and a preset threshold to judge whether the steel wire is bent.

[0028] In view of the problem that the loose state of the steel wire series-connected bolts cannot be accurately detected by non-contact means during the intelligent inspection of daily key components in the present invention, that is, various failure forms of the steel wire series-connected bolts, including missing, broken, loose, etc., by fusing two-dimensional images and point cloud data, the problem that the state of the steel wire series-connected bolts in the key area cannot be recognized or the recognition rate is low is solved. By adopting the method of the present invention, not only can the time and economic costs of manual detection be saved, the automation level be improved, but also a safer and more reliable intelligent scenario maintenance plan can be helped to be realized. Brief Description of the Drawings

[0029] Figure 1 is a schematic flow chart of the method of the present invention;

[0030] Figure 2 is a color depth map of the steel wire series-connected bolt group taken by a 3D color camera;

[0031] Figure 3 is a detection result map of the steel wire position after the target area is detected and segmented;

[0032] Figure 4 is a depth information map after the dimensionality reduction of the steel wire position along the depth direction. Detailed Embodiments

[0033] To make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the drawings and embodiments.

[0034] As Figure 1 shown, based on the fusion of two-dimensional data and power data, the method for detecting the looseness of the steel wire series-connected bolts of the present invention specifically includes the following step processes:

[0035] Step 1, obtain a three-dimensional depth image with color information through a 3D color camera, and the obtained three-dimensional depth image is as Figure 2 shown;

[0036] Step 2: Use the trained neural network to perform object detection on the two-dimensional color image and identify the position of the wire-connected bolt group;

[0037] The specific steps are as follows:

[0038] 2.1) In the two-dimensional color image, label the target data set for the area of the wire-connected bolt group, and randomly divide the network training data set according to the ratio of 7:3 for the training set and the test set;

[0039] 2.2) Use the Yolov5 network model to perform object detection training to obtain the network parameter weights;

[0040] 2.3) Input the image to be detected containing the wire-connected bolt group into the neural network trained in step 2.2), obtain the coordinate output of the area of the wire-connected bolt group, and crop to obtain the corresponding area image.

[0041] Step 3: Perform line detection in the bolt group area obtained in step 2 and conduct conditional screening. As Figure 3 shown, if there is a corresponding line, it indicates the position of the wire, and proceed to step 4. If no line is recognized, it means that the wire is missing, broken, or bent, indicating that the anti-loosening method has failed;

[0042] Specifically, in step 3.1, perform median filtering processing on the image of the wire-connected bolt group area using the operator;

[0043] Step 3.2: Calculate the edges of the processing result obtained in step 3.1 using the Canny operator;

[0044] Step 3.3: Perform Hough line detection processing on the edge image obtained in step 3.2, set the minimum line distance and the interval , and perform screening through the detected angle range to obtain the position coordinates of the two endpoints of the wire .

[0045] Step 4: Obtain the point cloud data at the corresponding positions of the wire detected in step 3 and perform dimensionality reduction in the depth direction.

[0046] Specifically, in step 4.1, obtain the position coordinates of the two ends of the wire from the line detection result in step 3, and convert the two-dimensional area coordinates to the coordinate information in the point cloud data;

[0047] Step 4.2: In the point cloud data, by calculating the distances of the two endpoints along the X and Y directions respectively, sample with a unit of one based on the direction with the larger distance as the reference direction, and take the point cloud data closest to the line according to the nearest principle for the coordinate values in the other direction to obtain the point cloud data of n units at the position of the wire;

[0048] Step 4.3: Arrange the n depth data in sequence according to the unit distance, and remove the singular points in the data through median filtering;

[0049] Step 4.4: Obtain the depth values along the straight lines where each wire is located after dimensionality reduction through the above steps. As Figure 4 shown, obtain the depth change data along the longest reference direction.

[0050] Step 5: Calculate the depth curvature of the wire depth curve obtained in Step 4 , and determine whether the wire bends in the depth direction, that is, shows relaxation, by setting a certain threshold , so as to judge whether it fails.

[0051] Specifically, in Step 5.1, use the least squares method to calculate the center and radius of the fitting curve O ; R

[0052] Step 5.2: Calculate the curvature , set the threshold , if , then judge that the wire bends. If , then judge that the wire does not bend.

[0053] Furthermore, using the least squares method to calculate the center and radius of the fitting curve in Step 5.1 includes the following steps:

[0054] Step 5.1.1: Calculate the centroid position of the point cloud data on the straight line ;

[0055] Step 5.1.2: Calculate the residual function according to the formula ;

[0056] Step 5.1.3: Optimize through the least squares method to determine the center ;

[0057] Step 5.1.4: Calculate the radius, that is, the average distance from the center s to each point .

Claims

1. A detection method for the loosening of wire-connected bolts, characterized in that, This detection method is based on the fusion of two-dimensional image data and point cloud data, and specifically includes the following steps: 1) Obtain a three-dimensional depth image with color information through a 3D color camera; 2) Use a trained neural network to perform object detection on the two-dimensional color image to identify the position of the series-connected wire bolt group; 3) Perform line detection and conditional screening in the bolt group area identified in step 2): If there is a corresponding line, it indicates the position of the wire, and proceed to step 4); If no line is recognized, it indicates that the wire is missing, broken or bent, indicating that the anti-loosening method has failed; 4) Obtain the point cloud data at the corresponding positions of the wires detected in step 3) and perform dimensionality reduction in the depth direction; after obtaining the point cloud data at the corresponding positions of the screws, perform dimensionality reduction in the depth direction, which specifically includes the following steps: 4.1) Obtain the coordinate information of the two ends of the wire from the line detection result in step 3) and convert the two-dimensional area coordinates to the coordinates in the point cloud data; 4.2) In the point cloud data, calculate the distances of the two endpoints along the X and Y directions respectively, and use the direction with the larger distance as the reference direction for sampling with a unit of one, and the coordinate values in the other direction are taken according to the nearest principle to obtain the point cloud data closest to the line; 4.3) Arrange the depth data in sequence according to the unit distance and remove the singular points in the data through median filtering; 4.4) Obtain the depth values along the lines where each wire is located after dimensionality reduction through the above steps, and calculate the average value change data along the longest reference direction; 5) Calculate the depth curvature of the wire depth curve obtained in step 4), and judge whether the wire is bent in the depth direction, that is, it shows relaxation, by setting a certain threshold, so as to judge whether it fails.

2. The detection method for loosening of wire-connected bolts according to claim 1, characterized in that, The specific steps of using the trained neural network to perform object detection on the series-connected wire bolt group in the two-dimensional color image in step 2) are as follows: 2.1) Label the target data set for the area of the wire series-connected bolt group to obtain the network training data set; 2.2) Use the Yolov5 network model to perform object detection training; 2.3) Input the image to be detected containing the wire series-connected bolt group into the neural network trained in step 2.2) to obtain the coordinate positioning output of the wire series-connected bolt group area.

3. A detection method for the loosening of wire-connected bolts according to claim 1, characterized in that, In step 3), line detection and screening are performed in the recognized bolt group area, which specifically includes the following steps: 3.1) Perform median filtering on the image of the wire series-connected bolt group area obtained in step 2); 3.2) Calculate the edges of the processing result obtained in step 3.1) using the Canny operator; 3.3) Perform Hough line detection on the edge image obtained in step 3.2), and obtain the coordinate positions of the two endpoints of the wire by setting the minimum line distance and interval.

4. The detection method for the loosening of the wire-connected bolt according to claim 1, characterized in that, In step 5), calculate the depth curvature of the wire, and judge whether the wire is relaxed in the depth direction by setting a certain threshold, which specifically includes the following steps: 5.1) Use the least squares method to calculate the center and radius of the fitting curve; 5.2) Calculate the curvature and a preset threshold to judge whether the wire is bent.

Images