Bolt looseness detection method based on machine vision

Abstract

The invention is suitable for the technical field of visual inspection, and particularly relates to a bolt looseness detection method based on machine vision, and the method comprises the steps: carrying out the image collection, carrying out the preprocessing of the collected image, and obtaining training data; constructing a target detection model and a key point detection model, and training the target detection model and the key point detection model by using the training data; importing an image to be detected, and outputting a key point marking image; and carrying out key point correction processing on the key point marking image, and calculating a nut rotation angle according to the key point marking image. According to the method, the limitation of a traditional image superposition technology and a key point matching algorithm in view angle deviation processing and perspective transformation correction is overcome by constructing a space reference system of the bolt center point, the deflection angle of each bolt is accurately calculated by comparing the slope difference of a bolt key point connecting line and a fitting straight line, and the correction accuracy is improved. And the loosening state of the nut is effectively monitored.

Description

Technical Field

[0001] The invention belongs to the technical field of visual detection, and in particular relates to a bolt loosening detection method based on machine vision. Background Art

[0002] In modern society, the safe operation of infrastructure in high-risk environments has received great attention. The integrity and stability of the structure are of vital importance, especially the high-strength bolted parts in the structure. These bolts play an irreplaceable role in ensuring the stability and safety of the structure. They are widely used in infrastructure and mechanical devices in various high-risk environments. For example, the parts connected by high-strength bolts in steel bridge structures are affected by factors such as continuous high loads, vibrations and environmental erosion. These bolts may become loose, thereby affecting the strength and stiffness of the structure. Once the bolts are loose, it may not only reduce the function of the structure, but may even cause the structure to become unstable and damaged, thereby threatening public safety and environmental safety.

[0003] In the prior art, three types of detection methods are generally used, namely manual detection, sensor monitoring and computer vision monitoring. However, the detection methods provided by the prior art have the problems of slow bolt identification and positioning and low monitoring accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a method for detecting loose bolts based on machine vision, aiming to solve the problems of slow bolt identification and positioning and low monitoring accuracy in the detection methods provided by the prior art.

[0005] The present invention is implemented as follows: a method for detecting bolt loosening based on machine vision, the method comprising:

[0006] Perform image acquisition, and pre-process the acquired images to obtain training data;

[0007] Build a target detection model and a key point detection model, and use the training data to train the target detection model and the key point detection model;

[0008] Importing the image to be detected into the target detection model and the key point detection model in turn, and outputting the key point marked image through the target detection model and the key point detection model;

[0009] The key point marking image is subjected to key point correction processing, and the nut rotation angle is calculated based on the key point marking image.

[0010] Preferably, the target detection model adopts the Faster R-CNN detection algorithm, the target detection model adopts ResNet-50 as the skeleton network, and the loss function adopts CrossEntropyLoss.

[0011] Preferably, the key point detection model adopts the RTMPose network architecture, the RTMPose network architecture adopts CSPNeXt as the backbone network, and the key point detection model adopts Kullback-Leibler as the loss function.

[0012] Preferably, the step of performing key point correction processing on the key point marked image includes: extracting the coordinates of each key point from each key point marked image, constructing a circle in the coordinate system, performing circular fitting on the key points so that the sum of squared distances from all key points to the constructed circle is minimized, obtaining a fitted circle, and screening the key points based on the sum of squared distances.

[0013] Preferably, the coordinates of all key points are The sum of squared distances is expressed by minimizing the objective function:

[0014]

[0015] Wherein, (a, b) is the coordinate of the center of the fitted circle, and r is the radius of the fitted circle.

[0016] Preferably, when deleting key points, the average distance between all key points and the fitting circle is calculated. If the difference between the distance value between a key point and the fitting circle and the average distance is greater than a preset value, the key point is determined to be an abnormal point and deleted.

[0017] Preferably, in the step of calculating the rotation angle of the nut based on the key point marked image, when detecting the loosening angle of a single nut, the original marked image is imported into the target detection model and the key point detection model, the original key point image is output, the key points in the original key point image and the key points in the key point marked image are superimposed and displayed, the slope of the edge line of each nut is calculated, the slope is converted into an angle, and the average rotation angle of the six corners of the nut is calculated.

[0018] Preferably, in the step of calculating the nut rotation angle according to the key point marked image, when detecting the loose angles of multiple nuts, the original marked image is imported into the target detection model and the key point detection model, the original key point image is output, the line between the center points of the nuts in the original key point image is identified, the first angle between the edge line of the nut and the line in the original key point image is calculated, the line between the center points of the nuts in the key point marked image is identified, the second angle between the edge line of the nut and the line in the key point marked image is calculated, the difference between the second angle and the first angle is calculated, and the nut rotation angle is obtained.

[0019] Preferably, when the image to be detected is imported, orthodontic processing is performed on the image to be detected to correct geometric distortion in the image to be detected.

[0020] Preferably, the calculation formula for the average rotation angle is:

[0021]

[0022] Among them, θ i is the rotation angle of the i-th edge of the nut.

[0023] The machine vision-based bolt loosening detection method provided by the present invention overcomes the limitations of traditional image overlay technology and key point matching algorithm in processing viewing angle deviation and perspective transformation correction by constructing a spatial reference system for the center points of the bolts. By comparing the slope difference between the line connecting the key points of the bolts and the fitting straight line, the deflection angle of each bolt is accurately calculated, thereby effectively monitoring the loosening state of the nut. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a method for detecting loose bolts based on machine vision provided by an embodiment of the present invention;

[0025] Figure 2 An architecture diagram of a target detection model provided by an embodiment of the present invention;

[0026] Figure 3 An architecture diagram of ResNet-50 provided in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the training results of the target detection model provided by an embodiment of the present invention;

[0028] Figure 5 A flowchart of a key point detection model provided by an embodiment of the present invention;

[0029] Figure 6 An architecture diagram of CSPNeXt provided in an embodiment of the present invention;

[0030] Figure 7 A loss function image of a key point detection model provided by an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of evaluation indicators of different models provided by an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of marking an image provided by an embodiment of the present invention;

[0033] Fig.10 A schematic diagram before and after correction provided by an embodiment of the present invention;

[0034] Fig.11 A coordinate schematic diagram of a nut provided in an embodiment of the present invention;

[0035] Fig.12A schematic diagram of a viewing angle range provided by an embodiment of the present invention;

[0036] Fig.13 The nut is viewed from different angles;

[0037] Fig.14 Schematic diagram of nuts under different conditions provided by the embodiments of the present invention;

[0038] Fig.15 A basic angle schematic diagram of a nut provided in an embodiment of the present invention;

[0039] Fig.16 Key point images of the nut at different viewing angles provided by the embodiment of the present invention;

[0040] Fig.17 A schematic diagram of profile comparison provided by an embodiment of the present invention;

[0041] Fig.18 Schematic diagram of the same nut at different shooting angles provided by an embodiment of the present invention;

[0042] Fig.19 An image of a nut set provided for an embodiment of the present invention;

[0043] Fig. 20 A first high-precision image provided by an embodiment of the present invention;

[0044] Fig.21 This is the second high-precision image provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0047] like Figure 1 As shown, it is a flow chart of a method for detecting bolt looseness based on machine vision provided by an embodiment of the present invention, and the method comprises:

[0048] Perform image acquisition, pre-process the acquired images, and obtain training data.

[0049] In this step, if Fig.12 As shown in the figure, the data collection uses a mobile phone to take pictures of the existing nuts from different angles, and obtains 142 pictures with a resolution of 3024×3024, including pictures of nuts taken at different positions. Labelme is used for data annotation, and 142 pictures are annotated, including bounding box information and key point information. The annotation format file is a json file.

[0050] In practice, the number of samples is small. In order to ensure the training capacity of the simulation, data enhancement is performed on the samples to improve the sample quality. The more samples there are, the better the trained model will be and the stronger the generalization ability of the model will be. Increasing the amount of training data improves the generalization ability of the model and adding noise data improves the robustness of the model. Data enhancement uses random changes to the image, such as rotation, mirroring, lighting, noise, etc. to enhance each image five times, as follows Fig.12 As shown, save the output image file and annotation file. Convert the enhanced 710 images and annotated json files into MS COCO annotated json files.

[0051] Build a target detection model and a key point detection model, and use the training data to train the target detection model and the key point detection model.

[0052] In this step, object detection aims to identify all objects of interest in an image or video and determine their locations and categories. It can be divided into two-stage detectors and single-stage detectors. Faster R-CNN is one of the two-stage detection methods, which is specially designed for object detection tasks. It integrates the region proposal network (RPN) with a deep convolutional neural network to automatically generate region proposals and perform efficient and accurate object detection

[30] . Specifically, the Faster R-CNN workflow is as follows: Figure 2 As shown, Figure 2 The framework of the target detection model is shown. The entire image is first input into the network, and features are extracted through the backbone network. The backbone network uses residual connections to avoid the gradient vanishing problem in deep network training, and reduces the amount of calculation through bottleneck structure design. Figure 3 As shown in the figure, in Faster R-CNN, ResNet-50 is used as the backbone network, which is based on the consideration of the ResNet deep convolutional neural network architecture. ResNet, full name residual network, is designed to solve the common gradient disappearance or explosion problems in deep learning models.

[0053] ResNet has many variants, including ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152, depending on the number of layers. The numbers represent the number of layers in the network. As a specific implementation, ResNet-50 uses a bottleneck structure to streamline model parameters while increasing the depth and width of the network to improve feature extraction capabilities. This design enables ResNet-50 to improve the efficiency of the network while retaining as much information as possible. The introduction of ResNet-50 enables Faster R-CNN to learn image features more deeply when processing images, thereby improving the accuracy and efficiency of object detection.

[0054] Loss function: CrossEntropyLoss is used for classification tasks, that is, to determine which category each region belongs to. Here y is the true label of the category (represented by one-hot encoding), p is the probability distribution predicted by the model, and M is the total number of categories. This function calculates the difference between the true label and the predicted probability distribution, which is used to optimize the model to correctly classify the target.

[0055]

[0056] L1Loss is used for bounding box regression tasks, which is to adjust the size and position of the detection box to make it as close to the real target as possible.

[0057]

[0058] Here i are the predicted bounding box properties (e.g. center coordinates, width, height), is the corresponding true bounding box attribute. L1Loss calculates the sum of the absolute differences between the predicted value and the true value, which is used to make the predicted bounding box cover the target object as accurately as possible.

[0059] From the bottom Figure 4The target detection model shown in the figure is trained and evaluated. When the target detection model is trained, the accuracy of the training set increases rapidly and reaches stability. Because the dataset only has a single object "nut", the training of the target detection of the dataset is fast. The accuracy of detecting a single target is good. The figure below shows the total loss (Loss) and its components - including the bounding box loss (loss_bbox), classification loss (loss_cls), region proposal network classification loss (loss_rpn_cls) and region proposal network bounding box loss (loss_rpn_bbox). All loss indicators drop rapidly at the beginning of training as the training steps progress, indicating that the performance of the model is improving rapidly. After the initial decline, the loss values ​​tend to stabilize, which usually means that the model begins to converge at this stage. The loss function of the target detection in the training set is stable, and the model performance is good.

[0060] When training the object detection model, the training is performed for a total of 50 epochs, with a training set processed every 4 batches and a validation set processed every 2 batches. The model weights are evaluated and saved every 10 rounds to monitor the model performance. The Epoch Based Train Loop is used for epoch training, the Val Loop is used for validation, and the Test Loop is used for testing. The SGD optimizer is used with an initial learning rate of 0.02, a momentum of 0.9, and a weight decay of 0.0001. The learning rate adjustment strategy includes linear warm-up and multi-step decay, which first increases the learning rate linearly in the first 500 iterations, and then multiplies the learning rate by 0.1 at the 8th and 11th epochs. As the skeleton network, ResNet-50 is used, combined with the Feature Pyramid Network (FPN) to enhance the feature representation. The RPN head is used to generate region proposals, and the ROI head further performs classification and bounding box regression on each proposal. At the same time, the data loaders for training and testing and the corresponding data augmentation strategies such as random flipping and scaling are defined.

[0061] Keypoint detection models are designed to find keypoints or joints of a specific object from an image, and are usually used to analyze the pose or morphology of an object. Keypoints can be used to estimate human poses, and can identify the positions of human joints such as shoulders, elbows, wrists, hips, knees, etc. in 2D or 3D. Keypoint detection algorithms generally use convolutional neural networks (CNNs) to predict the positions of keypoints in an image. Some popular network architectures include Convolutional Pose Machines, Stacked Hourglass Networks, and OpenPose.

[0062] like Figure 5As shown in Figure 1, RTMPose (Real-time Multi-Person Pose Estimation) is a network architecture for real-time multi-person pose estimation. The network architecture is as follows Figure 5 As shown in the network architecture of the RTMPose model, the input image is first sent into a deep neural network called backbone to extract features, and the extracted features, the so-called neck part, are further refined and enhanced by the Feature Pyramid Network (FPN).

[0063] FPN maintains spatial resolution by integrating feature maps of different scales, allowing the model to better handle objects of different sizes. The feature map processed by FPN is then fed into the RTMCC decoder head (Real-Time Multi-Person Convolutional Pose Machine with Coordinate Convolution), a decoder specifically designed for keypoint detection. It decodes the spatial information on the feature map through convolution operations and predicts a series of keypoint locations for each object. The keypoint location is encoded in an output called a heat map, where each pixel value on the heat map represents the probability of the keypoint existing at the corresponding position. In order to select the final keypoint location from multiple keypoint candidates, RTMPose uses the Non-Maximum Suppression (NMS) algorithm. NMS ensures that each keypoint location is locally optimal by suppressing (i.e. reducing) non-maximum local peaks, thereby reducing overlap and erroneous keypoint predictions. Finally, the heatmap output by the model is converted into predicted keypoint coordinates (x, y), which indicate the exact location of the nut vertex in the image.

[0064] like Figure 6 As shown in the figure, in RTMPose, CSPNeXt is selected as the backbone network to provide efficient and accurate multi-person pose estimation capabilities. CSPNeXt is a further development of the Cross-Stage Partial Network (CSPNet), which aims to reduce computational costs and improve information flow efficiency by optimizing the network structure. Specifically, CSPNet divides the input feature map into two parts, one part directly passes through the next stage of the network, and the other part undergoes a series of convolution operations. The feature maps of these two parts are merged again in the later stages of the network. This design effectively reduces the amount of parameters in the network while maintaining or improving the performance of the model. By cleverly segmenting the network and performing rich transmission and efficient fusion of features on this basis, CSPNeXt ensures full utilization of information in the network.

[0065] In order to achieve a lightweight and efficient network design, CSPNeXt borrows the concept of depthwise separable convolution, which significantly reduces the number of model parameters and computational complexity while ensuring the effectiveness of feature extraction. This strategy makes CSPNeXt an ideal choice for real-time pose estimation tasks.

[0066] Loss function: Kullback-Leibler (KL) divergence, also known as relative entropy, is a method to measure the difference between two probability distributions. It is a basic concept in information theory, used to describe the dissimilarity or information loss of a probability distribution relative to another reference probability distribution, and is an indicator used to measure the similarity of two probability distributions. Given two probability distributions P and Q, defined on the same probability space Ω, KL divergence is expressed by the following formula: This method calculates the KL divergence loss between the predicted output (dec_outs) and the true label (labels).

[0067] Loss=KL(P||Q)=∑P(x)[log P(x)-log Q(x)] (0.2)

[0068] For continuous random variables

[0069]

[0070] Where P is the distribution of true labels and Q is the distribution of predicted outputs. For each pair of predictions and targets, they are first reshaped into a shape suitable for calculation, and then the criterion method is called to calculate the KL divergence loss.

[0071] If target weights are enabled, the loss is multiplied by the weights. The final loss is the weighted average of all joint losses.

[0072] We trained three models with different network depths: RTMPose-S, RTMPose-M, and RTMPose-L. Figure 7 This is the loss function image of the model training process.

[0073] As the number of iterations increases, the loss function of the training set decreases, and the model tends to be stable. At the beginning of training, the loss functions of the models decrease in a similar manner. After a period of time, the training begins to differ. The training of the S model decreases steadily, followed by the M model, and the L model can continue to decrease a little. The final loss function of the S model and the L model is not much different. The final loss function of the L model is better than that of the S and M. The corresponding training and testing time becomes longer. Therefore, it is possible to complete the experiment in the laboratory and adopt a faster model. The evaluation indicators of the model tested on the test machine are as follows Figure 8 shown.

[0074] As shown in the table below, six different evaluation indicators of the three models are shown. From RTMPose-S to RTMPose-L, it can be seen that there is an improvement in all performance indicators (AP, AR, PCK, AUC), which shows that the scale of the model is positively correlated with its performance. RTMPose-L performs best on these indicators, showing the highest average precision, recall, and key point detection accuracy, as well as the largest area under the curve, which means that it performs best in accurately identifying and locating key points. All models show low errors in NME (normalized mean error), but there is still room for improvement. Especially for the RTMPose-L model, although it performs best in other indicators, there is still potential for further optimization in reducing key point prediction errors.

[0075] Table 1

[0076]

[0077] The images to be detected are sequentially imported into the target detection model and the key point detection model, and the key point marked images are outputted through the target detection model and the key point detection model.

[0078] When training the key point detection model, in order to make the key point detection more accurate, 300 training cycles were set, which was determined based on previous experiments and performance evaluation to ensure that the model can fully learn the features in the data. The initial learning rate was set to 4e-10. Combining linear learning rate warm-up and cosine annealing strategy, a smooth transition from a smaller learning rate to the maximum value and then gradually decreased was achieved, which helped the model to stabilize in the early stage and refine the performance in the later stage of training, thereby improving the convergence speed and the final accuracy of the model. The model input size was fixed to 512x512. The SimCCLabel encoder was used to process key point and skeleton information, so that the model can effectively recognize and encode complex patterns of human posture from the input image. For the training set and validation set, this study designed a series of data enhancement strategies, such as random flipping, random scaling and rotation, and HSV random enhancement, which not only increased the diversity of the training data, but also simulated various changes that may be encountered in the real world, thereby improving the generalization ability and robustness of the model.

[0079] like Fig. 9As shown in the figure, although the overall accuracy of key point prediction is high, inaccuracies may still occur due to occlusion or insufficient model training. In order to improve the accuracy of nut key point detection and effectively correct prediction errors, this study proposes a correction method based on the hexagonal features of the nut. For nut images taken from different angles, fitting a circle directly through the six vertices may not be accurate enough due to perspective distortion. Therefore, this method first applies a perspective transformation to make the shape of the nut close to a standard hexagon. Then, an optimal circular hexagon is fitted based on the transformed key points, and six new key points are calculated. These key points are then mapped back to the original image using the inverse of the perspective transformation matrix. This method not only verifies the accuracy of the key point information, but also re-evaluates and corrects the detected key points, significantly improving the accuracy of the results.

[0080] Perspective transformation is a geometric transformation in computer vision and image processing that is used to project points in a two-dimensional image to different viewpoints through perspective projection. The transformation can be described by a 3x3 transformation matrix that maps points in the original image to new coordinates in the target image. Given the coordinates of the point in the original image as (x, y), the transformed coordinates as (X, Y), and the transformation matrix T, the perspective transformation can be expressed by the following formula:

[0081]

[0082] in Let W = 1, and then apply the perspective transformation matrix T to transform the coordinates in the original image into the coordinates in the transformed image. Breaking down these equations yields the following formula:

[0083]

[0084] Therefore, to determine the transformation matrix T, at least eight coordinates need to be known. This includes the coordinates of the four initial points, point 1, point 3, point 4, and point 6, and the coordinates of the corresponding four points after the transformation, point 1', point 3', point 4', and point 6', as shown in Fig.10 As shown. In the case of a rectangular arrangement of bolts, the coordinates of the center points of the four sides of the nut are used as the four points after transformation when photographed from the front view. As a result, after applying the perspective transformation, the generated image will approximate the front position of the nut.

[0085] like Fig.11The following are images of the same nut taken from different angles. (a) shows the initial key points of the two images, while (b) shows the corrected key points after perspective transformation. Table 2 lists the slopes of the key points after perspective transformation. For the same nut in different states, the difference in the corrected nut position obtained from different viewing angles is small, indicating good performance. In general, the accuracy of key point detection can be significantly improved by perspective transformation and hexagonal feature correction methods, and the prediction error caused by different viewing angles and perspective transformation can be effectively corrected.

[0086] Table 2

[0087]

[0088]

[0089] In order to comprehensively evaluate the performance of the proposed method, images of nuts in different states and angles were taken to simulate actual application scenarios, such as Fig.12 shown.

[0090] The experimental structure is divided into four different states to show the diversity and complexity of the nut in terms of orientation and position. This approach allows for a comprehensive evaluation of the performance of the object detection and keypoint localization algorithms under multiple challenges. Fig.13 As shown, images (a) and (b) represent views taken from arbitrary angles above and below, respectively. Images (c) and (d) are taken from left to right at different angles, while images (e) and (f) are taken from right to left at random viewing angles.

[0091] The shooting angle affects the detection of key points, but a certain degree of deviation is allowed. Experiments show that when there are more than three visible key points, the key point recognition is accurate. If the deviation is significant, the key point detection fails. After multiple verifications, the minimum angle that can accurately predict the key points is determined. The feasible shooting angle range is shown in Table 3.

[0092] Table 3

[0093]

[0094] In actual application scenarios, conditions are more complex than in laboratory environments. Therefore, we conducted multiple sets of comparative experiments on key point detection, taking into account factors such as nut position, lighting conditions, and background color contrast. Fig.14 As shown in Figure 2, (a) shows the key point detection of randomly placed nuts, (b) shows the key point detection under low light and overexposure conditions, and (c) experiments with a steel plate as the background. The results show that we have achieved good performance in key point detection.

[0095] The accuracy and robustness of keypoint detection depends on the effectiveness of the algorithm and the diversity of the dataset. In theory, a sufficiently large and diverse dataset can achieve high robustness and accuracy. However, laboratory experiments have inherent limitations. In actual deployment, a large number of images of nuts in real environments need to be collected. This includes images of nuts in actual bolted joints, considering various lighting conditions, different weather conditions, different background color contrasts, and even painted surfaces. Training on these extensive real-world image data will result in keypoint detection algorithms that are more in line with real-world applications and show enhanced robustness.

[0096] Currently, when calculating the rotation angle of a nut in machine vision technology applications, we rely on detecting the angle change by comparing the contour change of the nut from the initial state to the loose state. This is achieved by comparing the change in the slope of the nut in the two images. However, this approach needs to be further optimized in multi-angle detection scenarios involving bolt groups. In order to accurately calculate the rotation angle of the bolt group, the deviation of the image shooting angle must be corrected using a perspective transformation. However, perspective transformation inevitably introduces geometric distortion, which affects the accuracy of angle measurement. In practical applications, especially under conditions of large shooting angles or limited accessibility, relying on key point matching algorithms may have limitations.

[0097] Therefore, we propose a method based on the relative position detection of bolt groups. This method first determines the angle of the nut relative to the fitting line as the initial angle. In subsequent detection, the rotation angle of the nut can be calculated by simply measuring the change of this angle relative to the fitting line. In order to reduce the error caused by the perspective transformation, we adjust the slope to compensate for the error in the rotation angle after the perspective adjustment. Although there may still be small errors, these errors will not significantly affect the detection of bolt loosening, making this method more accurate than the contour comparison method.

[0098] This method uses advanced image processing technology to detect bolt groups in images, followed by target recognition and key point extraction. The extracted key points are used to determine the center coordinates of each nut. Through spatial fitting of these center coordinates, bolt groups installed in a rectangular arrangement can be located and analyzed. Specifically, when the bolts are first tightened, the reference angle of each bolt relative to the fitting line is determined. Fig.15 As shown in (a), the basic angle Ang 1 of nut 1 and nut 2 is obtained. Subsequently, images of the nuts in a loose state can be taken from different angles, such as the left view in (b), the parallel view in (c), and the right view in (d). By analyzing the key points of each nut in the image and comparing the angle between the nut edge line and the fitting line, the relative angle Ang 1' is obtained. The compensation value is then calculated based on the slope of the line. This value is applied according to formula (7) to optimize the process.

[0099] By comparing the key point images of the nut under different viewing angles, such as Fig.16 As shown, using the line of the bolt center point obtained by fitting as a reference, the slope of the connecting line of each bolt key point is compared with the slope of the fitting line.

[0100] According to formula (7), the deviation angle of each bolt relative to the fitted line is calculated. This allows us to determine the relationship between the rotation angles of the nuts at different positions, representing the loosening angle of the nuts.

[0101] We compared the error between contour detection and line fitting methods. Fig.17 In (a), the schematic diagram of contour comparison shows the length of the contour edge, which represents the distance from the camera to the nut. Similarly, in (b), it represents the distance from the center point of each nut in our proposed line fitting method.

[0102] According to the figure, the angles obtained by the two methods are α1 and α2, respectively, as shown in formula (8).

[0103]

[0104] When the key point detection error is the same, both methods obtain the relative error as shown in the following formula.

[0105]

[0106] Obviously, since the fitting line is much larger than the contour line, our proposed line fitting method has more advantages than the contour comparison method.

[0107] In order to verify the effectiveness of our proposed method, we analyzed Fig.18 Two images of nuts in (a) and (b). Although the two images show the nut in the same state, they are taken from different angles. We used two detection methods: one is to compare the rotation angles of the nut contours detected in the images, and the other is to compare the angles using our line fitting method.

[0108] The results show that ideally, the angle difference should be close to 0° since the nuts are in the same state. As shown in Table 4, our method effectively compensates for the error caused by the change in shooting angle.

[0109] Table 4

[0110]

[0111] In the experiment, we used a high-precision method specially designed to measure and analyze the rotation state of nuts. The experiment involved random and equal rotation experiments on six nuts, in which a specific nut experienced a predetermined rotation angle. Initially, the nut was placed in the forward position and then randomly rotated to various angles to the left and right to simulate possible deviations. Fig.19 As shown, UVA acquired images of the nut group.

[0112] A DJI Mini 4Pro drone was used for image capture in the experiment. The drone is equipped with a 1 / 1.3-inch CMOS camera with an effective resolution of 48 megapixels and a gimbal that can be used in pitch angles from -90° to 60° and roll angles from -90° to 0°.

[0113] The experimental steps are divided into two main parts: First, after each rotation, the drone captured high-precision photos of the loose state of the nut, a total of 4 pictures, such as Fig. 20 and Fig.21 , where preset rotation and offset conditions are set. These images are then fed into an advanced analysis module that integrates object detection and key point detection. Using advanced image processing techniques, the module accurately identifies the position and orientation of the nuts in each image and then extracts key point information. Once the center points of the nuts are determined, they are named according to how close they are to the origin; the closest nut is named "nut1", and so on. The key points of the nut are then arranged according to the distance of the six vertices from the origin, with the closest being named "nut1-1", and so on.

[0114] To verify the accuracy of the nut group detection under different angle offsets, the following experiments were conducted: First, a photo of the nut in its initial state was taken, and then the nut was randomly rotated. Next, the rotated nut image was taken from a specific offset direction, and three images were obtained: one from the front, and the other two from the right and left sides. These images were input into the integrated object detection and key point detection module, providing key point detection information for each image, as shown in the two groups of figures. After obtaining the relative rotation angle of the nut, the angular offset of the nut compared to the initial state was calculated, as shown in Table 5 below.

[0115] Table 5

[0116]

[0117] The absolute error represents the difference between the actual measured angle and the standard angle (ideal or expected value). A positive value means that the actual measured value is higher than the standard value, and a negative value means that it is lower than the standard value. In these experiments, the values ​​of the absolute error are generally small, which indicates that the measurement accuracy is relatively high. Comparing the measurement results of the front, right, and left sides, it can be seen that the change in viewing angle has a certain effect on the measurement results, but in most cases, this effect is not significant. This may indicate that the measurement method has a certain robustness to changes in viewing angle. The data shows that the measurement method adopted can provide accurate and consistent angle measurement results at different viewing angles, and in most cases the absolute error and relative error are very small. Although shooting from different sides will have a certain impact on the angle measurement, this impact is generally not significant, indicating that the measurement method has strong robustness. In practical applications, this method may be suitable for scenes where accurate angle measurement from different angles is required.

[0118] In order to solve the existing problems in bolt loosening detection, the present invention proposes a method for key point detection combined with deep learning, so that bolt loosening detection is more applicable and universal. The integrated Fast R-CNN target detection and RTM-Pose key point detection method can more accurately predict the six key points of the nut and detect the looseness of multiple bolts at the same time. By comparing the three key point detection models, considering the training time and accuracy, the RTM-Pose-S model was selected for key point reasoning prediction. It achieved an AP (average precision) of 0.8820, a PCK (cumulative pairing accuracy) of 0.9097, and the NME (normalized mean square error) was reduced to 0.1420.

[0119] A relative position detection method based on spatial fitting of the key points of the nut center is proposed to solve the accuracy and adaptability problems of bolt angle detection under multi-angle shooting conditions. By constructing a spatial reference system of the nut center point, the limitations of traditional image overlay technology and key point matching algorithm in dealing with viewpoint deviation and perspective transformation correction are overcome. By comparing the slope difference between the connection line of the key points of the nut and the fitting line, the deviation angle of each nut is accurately calculated, and the loose state of the nut is effectively monitored.

[0120] The proposed method can detect the rotation angle of the nut group at different angles, eliminates the need for image matching, optimizes the detection method, and cancels the requirement of fixed-point shooting in actual scenes. This nut group detection method is easier to deploy in practical applications.

[0121] Our key point detection technology demonstrates excellent accuracy, with absolute errors between 0.1° and 1.2°, relative errors ranging from -2.43% to 3.97%, and relative errors within ±4% for most measurements. Even though the accuracy fluctuates under different test scenarios, these fluctuations do not affect our ability to determine whether the nut is rotated or loose. It not only accurately identifies the rotation of the nut at various angles, but also provides critical data support to ensure the safety and function of the structure.

[0122] However, the manual annotation of the dataset introduces certain errors, which affects the detection accuracy. Therefore, future developments can include automatic annotation methods for the key points of the nut. In addition, deep learning methods still require larger and more diverse datasets, so obtaining more comprehensive training data to provide better algorithm robustness is one of the future goals.

[0123] Although the integrated Fast R-CNN and RTM-Pose approach provides a powerful framework for nut localization and looseness detection, a detailed analysis of the measurement data shows that optimization is needed when dealing with small angles of rotation (e.g., 20°) or accurately detecting the nut rotation angle.

[0124] Real-time bolt loosening detection is also a consideration, and enabling robots to perform real-time inspections through vision is a meaningful direction for future development. In addition, we will continue to research real-time detection.

[0125] Based on the above method, six key points of the nut are detected by integrating object detection and key point detection, followed by post-processing to determine the rotation angle of the nut. At present, this method has shown good results in the experimental stage. The next step in terms of autonomy is to deploy the algorithm and post-processing on the robot, build an initial database, and use the robot's inspection to detect the actual position and looseness of the bolt to achieve autonomy.

[0126] In reality, bolts or nuts may be damaged, corroded, or broken. The current key point detection algorithm mainly focuses on the position and orientation of the nut. In terms of the nut or bolt status, it can be optimized in the target detection in the future. We plan to gradually collect data on damaged, corroded, and broken bolts or nuts for training. In the target detection stage, various failure modes of bolts will be detected and classified to achieve health monitoring of bolts.

[0127] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting bolt looseness based on machine vision, characterized in that: The method comprises: Perform image acquisition, and pre-process the acquired images to obtain training data; Build a target detection model and a key point detection model, and use the training data to train the target detection model and the key point detection model; Importing the image to be detected into the target detection model and the key point detection model in turn, and outputting the key point marked image through the target detection model and the key point detection model; The key point marking image is subjected to key point correction processing, and the nut rotation angle is calculated based on the key point marking image.

2. The method for detecting bolt looseness based on machine vision according to claim 1, characterized in that: The target detection model adopts the Faster R-CNN detection algorithm, the target detection model adopts ResNet-50 as the skeleton network, and the loss function adopts CrossEntropyLoss.

3. The method for detecting bolt looseness based on machine vision according to claim 1, characterized in that: The key point detection model adopts the RTMPose network architecture, the RTMPose network architecture adopts CSPNeXt as the backbone network, and the key point detection model adopts Kullback-Leibler as the loss function.

4. The method for detecting bolt looseness based on machine vision according to claim 1, characterized in that: The step of performing key point correction processing on the key point marked image includes: extracting the coordinates of each key point from each key point marked image, constructing a circle in the coordinate system, performing circular fitting on the key points so that the sum of the squares of the distances from all key points to the constructed circle is minimized, obtaining a fitting circle, and screening the key points according to the sum of the squares of the distances.

5. The method for detecting bolt looseness based on machine vision according to claim 4 is characterized in that: The coordinates of all key points are The sum of squared distances is expressed by minimizing the objective function: Wherein, (a, b) is the coordinate of the center of the fitted circle, and r is the radius of the fitted circle.

6. The method for detecting bolt looseness based on machine vision according to claim 4, characterized in that: When deleting key points, the average distance between all key points and the fitting circle is calculated. If the difference between the distance between a key point and the fitting circle and the average distance is greater than a preset value, the key point is determined to be an abnormal point and deleted.

7. The method for detecting bolt looseness based on machine vision according to claim 1, characterized in that: In the step of calculating the rotation angle of the nut based on the key point marked image, when detecting the loose angle of a single nut, the original marked image is imported into the target detection model and the key point detection model, the original key point image is output, the key points in the original key point image and the key points in the key point marked image are superimposed and displayed, the slope of the edge line of each nut is calculated, the slope is converted into an angle, and the average rotation angle of the six corners of the nut is calculated.

8. The method for detecting bolt looseness based on machine vision according to claim 1, characterized in that: In the step of calculating the rotation angle of the nut according to the key point marked image, when detecting the loose angles of multiple nuts, the original marked image is imported into the target detection model and the key point detection model, the original key point image is output, the line between the center points of the nuts in the original key point image is identified, the first angle between the edge line of the nut and the line in the original key point image is calculated, the line between the center points of the nuts in the key point marked image is identified, the second angle between the edge line of the nut and the line in the key point marked image is calculated, the difference between the second angle and the first angle is calculated, and the rotation angle of the nut is obtained.

9. The method for detecting bolt looseness based on machine vision according to claim 1, characterized in that: When the image to be detected is imported, orthodontic processing is performed on the image to be detected to correct the geometric distortion in the image to be detected.

10. The method for detecting bolt looseness based on machine vision according to claim 7, characterized in that: The formula for calculating the average rotation angle is: Among them, θ i is the rotation angle of the i-th edge of the nut.

Images