Small-angle detection method for bolt loosening based on deep learning

Abstract

The present invention discloses a method for detecting small angles of bolt looseness based on deep learning, comprising the following steps: (1) collecting pictures of different bolt rotation angles, and using the pictures of different bolt rotation angles as training set data; (2) using the training set data to train the Single Shot MultiBox Detector (SSD) deep learning network to obtain an SSD training model; (3) using the SSD training model to detect pictures of bolts to be tested, and identifying the loose angles of the bolts. The present invention can effectively solve the problem of identifying and detecting the loose angles of bolt connection structures. The loose angle detection is accurate, and studies have shown that the method of the present invention has relatively good performance in terms of implementation cost and angle recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical fastener failure detection, and in particular to a method for detecting small-angle bolt loosening based on deep learning. Background Art

[0002] Bolt connections have the advantages of reliable connection and easy assembly and disassembly, and have been widely used in wood structures. Wood structure packaging boxes are subjected to continuous vibration loads during transportation, and the resulting loosening of bolt connections may directly lead to structural failure. Therefore, the research on detection methods for loose bolt connections in wood structures is of great significance. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for detecting small angles of loose bolts based on deep learning in response to the above-mentioned deficiencies in the prior art. The method for detecting small angles of loose bolts based on deep learning can effectively solve the problem of identifying and detecting loose angles of bolted connection structures.

[0004] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0005] A method for detecting small angles of bolt loosening based on deep learning, comprising the following steps:

[0006] (1) Collect pictures of different bolt rotation angles and use them as training set data;

[0007] (2) Use the training set data to train the SSD deep learning network to obtain the SSD training model;

[0008] (3) Use the SSD training model to detect the image of the bolt to be tested and identify the loose angle of the bolt.

[0009] As a further improved technical solution of the present invention, the step (1) comprises:

[0010] Take pictures of different bolt rotation angles, where circular marks are provided on the bolts and the nuts matched therewith;

[0011] Use the labelImg tool to classify and name the targets in the image, where the targets are the circular marks on the bottom of the bolt and the nut;

[0012] The classified and named pictures are used as training set data.

[0013] As a further improved technical solution of the present invention, the step (3) is specifically as follows:

[0014] The SSD training model is used to detect the image of the bolt to be tested, and all the targets in the image are identified. At the same time, the center coordinates of the targets are identified. The default coordinates of the image identified by the SSD training model take the upper left corner as the origin. The center coordinates of the circular mark on the nut are (x1, y1), and the center coordinates of the bottom of the bolt are (x2, y2).

[0015] Move the origin of the default coordinates to the center of the bottom surface of the bolt, then the center coordinates of the circular mark on the nut are: (x3, y3) = (x1-x2, y1-y2), the angle α between the line from the origin to the coordinates (x3, y3) and the negative direction of the Y axis is the identification calculation angle, 0°≤α<360°, the calculation process is shown in formula (1):

[0016]

[0017] When the bolt is installed for the first time, collect the bolt picture, identify and calculate the included angle α, and record the included angle α at this time as the initial angle α0;

[0018] When the bolt loosening angle needs to be detected, the bolt image is collected, and the angle α is identified and calculated. The angle α at this time is recorded as angle α i ;

[0019] The loosening angle of the bolt is θ = α i -α0.

[0020] The beneficial effects of the present invention are as follows: the present invention can effectively solve the problem of loose angle identification and detection of bolt connection structures. The loose angle detection is accurate. Studies have shown that the method of the present invention has relatively good performance in terms of implementation cost and identification angle accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of using the labelImg tool to classify and name objects in an image.

[0022] Figure 2 (a) shows the rotation direction of the nut when loosening (Loosening) and tightening (Tightening).

[0023] Figure 2 (b) is a schematic diagram of the center coordinates of the target object identified by the SSD training model.

[0024] Figure 2 (c) is a schematic diagram of the coordinates (x3, y3) of the target object on the nut when the origin of the original default coordinates is moved to the center of the bolt.

[0025] Figure 2 (d) is a schematic diagram of the angle α between the line from the origin to the coordinate (x3, y3) and the negative direction of the Y axis.

[0026] Figure 3 This is a schematic diagram of the SSD network structure.

[0027] Figure 4 (a) An image from the training set data input to the SSD network structure.

[0028] Figure 4 (b) is a large-scale feature map.

[0029] Figure 4 (c) is a small-scale feature map.

[0030] Figure 5 A diagram showing how to set up the prediction priors during model training.

[0031] Figure 6 (a) is a picture of the nut after it is rotated 50 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0032] Figure 6 (b) is a picture after the nut is rotated 150 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0033] Figure 6 (c) is a picture after the nut is rotated 250 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0034] Figure 6 (d) is a picture of the nut after it is rotated 350 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0035] Figure 7 This is a schematic diagram of the loss reduction curve.

[0036] Figure 8 (a) is a picture after the nut is rotated 1 degree, and is a schematic diagram of the angle α result identified and calculated by the detection method of this embodiment.

[0037] Figure 8 (b) is a picture of the nut after it has been rotated 3 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0038] Figure 8 (c) is a picture of the nut after it is rotated 5 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0039] Figure 8 (d) is a picture of the nut after it is rotated 7 degrees, and is a schematic diagram of the angle α calculated by the detection method of this embodiment.

[0040] Fig. 9 This is a schematic diagram of 4 bolts in this embodiment.

[0041] Fig.10 It is a schematic diagram of the angle α results calculated for four bolt images using the detection method of this embodiment.

[0042] Fig.11 This is a schematic diagram of 8 bolts in this embodiment.

[0043] Fig.12 It is a schematic diagram of the angle α results calculated for 8 bolt images using the detection method of this embodiment. DETAILED DESCRIPTION

[0044] The specific embodiments of the present invention are further described below according to the accompanying drawings:

[0045] This embodiment provides a method for detecting small angles of loose bolts based on deep learning, which can effectively solve the problem of identifying and detecting loose angles of bolt connection structures. Specifically, it includes:

[0046] The first step is to collect pictures of different bolt rotation angles as training set data and design a method to identify the loose angle.

[0047] In the second step, the training set image data is trained using the SSD target detection algorithm based on the deep learning pytorch framework to obtain an SSD training model.

[0048] In the third step, the SSD training model is used to detect the newly taken pictures of different bolt rotation angles, and finally the loose angle of the bolt is identified.

[0049] The specific contents are as follows:

[0050] 1. Image collection and loose angle recognition method design:

[0051] The image acquisition device is a smart phone, and the shooting parameters are: 3024*3024 pixels, 72dpi resolution, and 8-10cm shooting distance. To ensure clear features, the nut occupies at least 1 / 4 of the image. A total of 210 images were collected, of which 70% were used as training sets and 30% as test sets. The labelImg tool was then used to classify and name the targets in the images. The targets were the round marks on the bottom of the bolt and the nut, named "Bolt" and "Sign", respectively. Figure 1 shown.

[0052] After SSD deep learning network training, these targets will be automatically identified and the center coordinates of the targets will be given. The calculation method of the loose angle is as follows: Figure 2 As shown. Figure 2In (a), the direction of rotation for tightening the nut is clockwise, and the direction of rotation for loosening the nut is counterclockwise. Figure 2 In (b), the default coordinates of the image recognized by SSD take the upper left corner as the origin, and the coordinates of the target object are (x1, y1) and (x2, y2), as shown in Figure 2 In (c), when calculating, the coordinate center is moved to the center of the bolt, and the coordinates of the target object on the nut are converted to (x3, y3) = (x1-x2, y1-y2), such as Figure 2 In (d), the angle α between the line connecting the new origin and the coordinates (x3, y3) and the negative direction of the Y axis is the detection angle, which ranges from 0°≤α<360°. The calculation process is shown in formula (1). In real life, when the bolt is first installed, a photo is taken and the angle α is identified as the initial angle α0. When the bolt is loosened, a photo is taken and the angle α is identified as the angle α i , the loosening angle of the bolt is θ=α i -α0.

[0053]

[0054] 2. Deep learning network training process:

[0055] The deep learning network training is based on the Single Shot MultiBox Detector (SSD) algorithm under the pytorch framework. The SSD algorithm is a one-stage method. Its main idea is to evenly perform dense sampling at different locations in the image. Different scales and aspect ratios can be used for sampling, and then Convolutional Neural Networks (CNN) is used to extract features and then perform classification and regression directly. The whole process only requires one step, so its advantage is fast speed.

[0056] The network structure of the SSD algorithm is as follows Figure 3 As shown in the figure, SSD uses VGG16 as the basic model, and then adds a convolutional layer to obtain more feature maps for detection. Its core concepts are mainly three points:

[0057] (1) Use multi-scale feature maps for detection. In CNN, the feature maps are relatively large in the early stage, and the size of the feature maps will be reduced through convolution and pooling in the later stage. Figure 4 (b)) is used to detect small objects, while small-scale feature maps (such as Figure 4 (c) is used to detect large objects, such as Figure 4 As shown in Figure 3, a large-scale feature map can be divided into more units, but the scale of the prior box of each unit is relatively small.

[0058] (2) Use convolution for detection. Unlike the YOLO algorithm, which places the fully connected layer at the end, the SSD algorithm directly uses convolution to extract detection results from different feature maps. For feature maps with a shape of m*n*p, only a relatively small convolution kernel such as 3*3*D is needed to obtain the detection value, which improves the detection speed.

[0059] (3) Setting of prior boxes. In YOLO, each unit predicts multiple bounding boxes, but they are all relative to the unit itself. However, the shape of the real target is changeable. YOLO needs to adapt to the shape of the target during training. SSD borrows the concept of anchor in Faster R-CNN. Each unit sets a prior box with different scales or aspect ratios. The predicted bounding box is based on these prior boxes, which reduces the difficulty of training to a certain extent. In general, each unit will set multiple prior boxes with different scales and aspect ratios, such as Figure 5 shown.

[0060] In the training process of the SSD algorithm, we must first determine which prior box the ground truth (real target feature) in the training image matches. The bounding box corresponding to the matching prior box will be responsible for predicting the feature. There are two main matching principles. First, for each ground truth in the image, find the prior box with the largest IOU (IOU is the ratio of the overlapping area and the union area between the predicted bounding box and the ground truth bounding box). This ensures that each ground truth must match a prior box. The prior box that matches the ground truth is usually called a positive sample. If a prior box does not match any ground truth, then the prior box can only match the background, which is a negative sample. In fact, there are very few ground truths in a picture, but there are many prior frames. If only the first principle is followed for matching, many prior frames are negative samples, resulting in an extremely unbalanced positive and negative samples. Therefore, the second matching principle is proposed. For the remaining unmatched prior frames, if the IOU of a ground truth is greater than a certain threshold (usually 0.5), then the prior frame is also matched with the ground truth. This means that a ground truth may match multiple prior frames, but the reverse is not true, because a prior frame can only match one ground truth. If the IOU of multiple ground truths with a prior frame is greater than the threshold, then the prior frame is only matched with the ground truth with the largest IOU. The second principle must be performed after the first principle. Although a ground truth can be matched with multiple prior frames, there are still too few ground truths relative to the prior frames, and there are still many more negative samples relative to the positive samples. In order to ensure that the positive and negative samples are as balanced as possible, the negative samples are sampled to ensure that the ratio of positive and negative samples is close to 1:3.

[0061] 3. Deep learning network training results:

[0062] The training data set contains a total of 210 images, of which 70% are used as training sets and 30% as test sets. The training is based on the SSD algorithm under the pytorch deep learning framework. During the training process, the model loss decreases with the increase in the number of iterations until it stabilizes, and the recognition accuracy of the model can also reach a high level. In the setting of training parameters, in order to increase the training speed, batch_size is set to 4, the learning rate decay factor is set to 0.94, and the optimizer is SGD. Regarding the setting of learning rate, the learning rate is generally set larger during initial training to avoid being trapped in the local optimal solution. When the loss is small, the learning rate is set smaller to improve the accuracy. Therefore, the initial learning rate is set to 0.001, and the number of training steps is 10,000. Then, based on the trained model, another 10,000 steps are trained at a learning rate of 0.0001.

[0063] In order to identify the accuracy of the angle, new pictures were taken for testing. The nut was rotated 50°, 150°, 250° and 350° respectively using the angle measurement device. Three pictures were taken at each angle. The camera was kept facing the nut and the size of the nut was at least one-third of the picture. The rotation angle was identified by the trained model. The identification angle data is shown in Table 1.

[0064] Table 1, recognition result data:

[0065]

[0066] The best recognition result of the model is as follows Figure 6 As shown in (a) to (d), it can be seen that after two trainings, the recognition accuracy of each target in the single bolt image is 100%. After calculation, the average angle recognition error is 0.38°, so this method is used as the basic method for bolt loosening identification.

[0067] 4. Minimum recognition angle accuracy test:

[0068] In the bolt connection of wooden structure, when the nut is loose, the rotation angle of the nut may be very small. When the rotation angle of the nut is large, it means that the looseness is obvious and the possibility of structural failure is very high. Therefore, the earlier the loosening of the bolt is detected, the more structural failure can be avoided. Therefore, the minimum recognition angle of the proposed detection method is tested. In order to improve the recognition accuracy, the secondary training model was trained again for 50,000 steps at a learning rate of 0.00001. In the deep learning algorithm, the evaluation of the model is mainly based on parameters such as loss, average precision (AP) of a single class, and average precision (MAP) of all classes. Loss refers to the error between the predicted result and the actual result in the training set; AP refers to the recognition accuracy of each class; MAP refers to the average recognition accuracy of all classes. Usually, there are many target categories recognized by deep learning algorithms, but the model trained in this embodiment only recognizes two classes: "sign" and "bolt". The recognition accuracy of the class and the average recognition accuracy are both 0.999. The loss decline curve of the third training is as follows: Figure 7 As shown, the loss curve at this time is almost in a stable state, indicating that the model has converged and the recognition effect of the model is close to stability.

[0069] Since the minimum accuracy of the angle measuring instrument is 1°, we measured and took pictures of 1°, 3°, 5° and 7° and conducted recognition tests. We still took three pictures for each angle and used the self-training model for recognition. The best recognition result is as follows: Figure 8 As shown in (a) to (d).

[0070] Table 2, small angle recognition data:

[0071]

[0072] The small angle recognition data results are shown in Table 2. The recognition errors of the four angles are 0.37°, 0.1°, 0.07° and 0.2° respectively, and the overall average error is 0.185°. Increasing the number of model training times reduces the angle recognition error to a certain extent. In addition, considering the error caused by manual angle measurement, it can be considered that this method has been able to achieve good recognition accuracy, and the minimum recognition angle can reach 1°, which can effectively detect the initial state of bolt loosening.

[0073] 5. Multi-bolt recognition effect:

[0074] In actual working environments, structural connections usually use many bolts. If only one bolt is detected at a time, the work efficiency is relatively low. The simultaneous identification of multiple bolts will greatly improve the work efficiency. Therefore, it is necessary to train and test the loose angle identification of bolts in the case of multiple bolts. This embodiment first uses the case of 4 bolts for testing. Fig. 9 As shown in the figure, the camera equipment is the same, the image pixel is 3024*3024, and the circular feature is selected to identify the target object. A total of 210 images are collected, of which 70% are used as training sets and 30% are used as test sets. The two types of target objects are "Sign" and "Bolt", and they are trained for 10,000 steps with learning rates of 0.001 and 0.0001 respectively, for a total of 20,000 steps.

[0075] The recognition result is as follows Fig.10 As shown in the figure, when the angles of four bolts are identified in one picture at the same time, the recognition accuracy of "bolt" is still close to 100%, but the recognition accuracy of "sign" on the nut is relatively reduced. After calculating the recognition results of the test set, the average recognition accuracy of "sign" is 65.7%. As the number of bolts increases, the "sign" target on the nut becomes smaller than that of a single bolt in a picture, the difficulty of training increases, and the recognition accuracy decreases. However, it can be seen from the recognition results that this method can be used to identify the loose angle of multiple bolts.

[0076] Increase the number of bolts to 8, such as Fig.11 As shown, the model is trained using the same method and parameters. Among all the test results, the best recognition result is as follows Fig.12 As shown in the figure, the recognition accuracy of the “bolt” feature of the 8 bolts is still very high, close to 100%, but only 5 “sign” features on the nuts are recognized, and the accuracy is further reduced to about 35%.

[0077] As the number of bolts increases, the SSD algorithm's recognition accuracy for the small target "sign" gradually decreases. In order to achieve better small target recognition, the visual algorithm can be improved. In order to achieve loose angle recognition of multiple bolts, multiple bolts in the image can be cropped out one by one before training the model, and then angle recognition can be performed. However, the final angle recognition method is the same as that for a single bolt.

[0078] This embodiment of Figure 6 , Figure 8 , Fig.10 and Fig.12 The detected objects are marked with boxes, and the row of data above each box is the name of the object, the recognition accuracy, the center coordinate x, and the center coordinate y; degree represents the angle α. Figure 6 , Figure 8, Fig.10 and Fig.12 The coordinates of the left and bottom of the image are original coordinates, in pixels.

[0079] The protection scope of the present invention includes but is not limited to the above embodiments. The protection scope of the present invention shall be based on the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by technicians in this field shall fall within the protection scope of the present invention.

Claims

1. A method for detecting small angles of loose bolts based on deep learning, characterized in that: The following steps are involved: (1) Collect pictures of different bolt rotation angles and use them as training set data; (2) Use the training set data to train the SSD deep learning network to obtain the SSD training model; (3) Use the SSD training model to detect the image of the bolt to be tested and identify the loose angle of the bolt; The step (1) comprises: Take pictures of different bolt rotation angles, where circular marks are provided on the bolts and the nuts matched therewith; Use the labelImg tool to classify and name the targets in the image, where the targets are the circular marks on the bottom of the bolt and the nut; Use the classified and named pictures as training set data; The step (3) is specifically as follows: The SSD training model is used to detect the image of the bolt to be tested, and all the targets in the image are identified. At the same time, the center coordinates of the targets are identified. The default coordinates of the image identified by the SSD training model take the upper left corner as the origin. The center coordinates of the circular mark on the nut are (x1, y1), and the center coordinates of the bottom of the bolt are (x2, y2). Move the origin of the default coordinates to the center of the bottom surface of the bolt, then the center coordinates of the circular mark on the nut are: (x3, y3) = (x1-x2, y1-y2), the angle α between the line from the origin to the coordinates (x3, y3) and the negative direction of the Y axis is the identification calculation angle, 0°≤α<360°, the calculation process is shown in formula (1): When the bolt is installed for the first time, collect the bolt picture, identify and calculate the included angle α, and record the included angle α at this time as the initial angle α0; When the bolt loosening angle needs to be detected, the bolt image is collected, and the angle α is identified and calculated, and the angle α at this time is recorded as angle α i ; The loosening angle of the bolt is θ = α i -α0.

Images