Bolt fracture deformation monitoring method based on machine vision

Abstract

The invention discloses a bolt fracture deformation monitoring method based on machine vision. The method comprises the following steps: acquiring a to-be-detected image by adopting photographing equipment; selecting a region-of-interest image from the to-be-detected image; based on the position information of the bolt, detecting the bolt fracture missing condition; and detecting the inclination deformation condition of the bolt based on the region-of-interest image. According to the bolt fracture deformation monitoring method based on machine vision, the bolt position is identified as the region of interest, the bolt fracture missing position is determined, the bolt center line inclination angle is calculated according to the three key points of the bolt, the bolt deformation result is determined according to the preset threshold value, and the bolt fracture deformation monitoring accuracy is improved. Therefore, automatic and autonomous alarm is realized, the alarm function can be realized without manual participation, and the judgment logic is simple, efficient and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of detection and fastener connection, and particularly to a method for monitoring bolt fracture and deformation based on machine vision. Background Art

[0002] In many industrial application scenarios, such as mechanical equipment manufacturing, large-scale construction facilities, transportation tools, etc., bolts, as key connecting components, their structural integrity plays a crucial role in the safe and stable operation of the overall equipment or structure. However, during long-term use, due to the influence of complex external forces (such as vibration, impact, tension, etc.) and environmental factors (such as corrosion, temperature change, etc.), bolts are prone to problems such as fracture and deformation. If not discovered and processed in time, it is very likely to cause serious safety accidents.

[0003] Traditional bolt detection methods mostly rely on manual inspections, using naked-eye observation and the assistance of simple tools to judge the bolt status. This method not only has low efficiency and high labor intensity, but also the detection results are easily affected by the subjective factors of the inspectors, and the accuracy is difficult to guarantee. Another method is to monitor through contact sensors, such as acoustic-based methods, piezoelectric active sensing methods, and impedance-based methods. This method requires high-precision instruments and built-in algorithms to compensate for the influence of environmental changes, and it is difficult to popularize both from an economic perspective and a deployment perspective. With the rapid development of computer vision technology, laser, radar detection technology, and deep learning algorithms, using them to perform automated and high-precision detection of bolts has become a very promising solution. However, existing related technologies still need to be further improved in terms of feature extraction accuracy, adaptability to bolts under different working conditions, and detection accuracy. Therefore, there is an urgent need for a more advanced and reliable bolt fracture and deformation detection technology. Summary of the Invention

[0004] To solve the deficiencies of the prior art, the present invention provides a method for monitoring bolt fracture and deformation based on machine vision, which identifies the bolt position as the region of interest, and then determines the bolt fracture and missing position. In addition, the inclination angle of the bolt center line is calculated based on three key points of the bolt, and the bolt deformation result is determined through a preset threshold, thereby realizing automated and autonomous alarm, and the alarm function can be realized without manual participation, and the judgment logic is simple, efficient, and accurate.

[0005] The embodiments of the present invention provide the following solutions:

[0006] The embodiments of the present invention provide a method for monitoring bolt fracture and deformation based on machine vision, the method comprising:

[0007] S1. Obtain an image to be detected by using a camera device;

[0008] S2. Select the region of interest image from the image to be detected;

[0009] S3. Detect the breakage and missing situation of bolts based on the position information of the bolts;

[0010] S4. Detect the inclination and deformation situation of bolts based on the region of interest image.

[0011] In an alternative embodiment, the process of selecting the region of interest image from the image to be detected described in step S2 includes the following steps:

[0012] S2.1. Use the object detection algorithm to identify all suspected bolts in the image to be detected and calibrate them with detection frames. The detection frame corresponding to the i-th bolt is a rectangular frame R i ;

[0013] S2.2. After expanding the range of the detection frame by a preset percentage, the intercepted image is used as the region of interest image.

[0014] In an alternative embodiment, the object detection algorithm adopts one of the methods of SSD (Single Shot MultiBoxDetector), YOLO (You Only Look Once), and ViT (Vision Transformer).

[0015] In an alternative embodiment, the process of detecting the breakage and missing situation of bolts based on the position information of the bolts described in step S3 includes the following steps:

[0016] S3.1. Calculate the average width w of the bolts based on the mode of the widths of each rectangular frame b ;

[0017] S3.2. Calculate the distance between adjacent bolts based on the sorting of the abscissas of each rectangular frame. The bolt distance between the i-th bolt and the (i + 1)-th bolt is denoted as d i ;

[0018] S3.3. If d i > 2w b , it is marked as bolt breakage or missing, and the number of missing is

[0019] In an alternative embodiment, if there is no detected bolt rectangular frame R on both sides of the image to be detected i , mark the breakage or missing position of the bolt according to the confidence threshold of the recognition result from the image to be detected by the object detection algorithm and the preset detection area.

[0020] In an alternative embodiment, the inconsistent calculation of the widths of bolts caused by the perspective of the camera shooting angle results in the reference width wb Since there are multiple values for the first preset threshold, the optimal reference bolt w is obtained based on the widths of adjacent bolts b .

[0021] In an alternative embodiment, the process of detecting the inclination and deformation of the bolt based on the image of the region of interest in step S4 includes the following steps:

[0022] S4.1 Identify three key point information, namely the center point of the bolt top, the center point of the upper boundary of the nut, and the center point of the lower boundary of the nut;

[0023] S4.2 Fit a straight line equation y = kx + b based on the three key point information, where the parameter k to be solved represents the slope and b represents the intercept;

[0024] S4.3 Solve the parameters k and b, and then obtain the inclination angle and deformation result of the bolt.

[0025] In an alternative embodiment, step S4.1 uses a trained deep learning model to identify the three key point information.

[0026] In an alternative embodiment, the training of the deep learning model includes the following process: collecting images of a preset area containing bolts; performing data annotation on the bolt nodes in the collected images, and forming a training set and a validation set; ending the training when the verification result of the validation set meets the end condition, and saving the detection model parameters.

[0027] In an alternative embodiment, step S4.3 uses the least squares method to solve for the parameters k and b, where

[0028] The beneficial effects of the present invention based on its technical solution are as follows:

[0029] A method for monitoring bolt fracture and deformation based on machine vision provided by the present invention identifies the bolt position as the region of interest, and then determines the bolt fracture and missing position. In addition, the inclination angle of the bolt center line is calculated based on three key points of the bolt, and the bolt deformation result is determined through a preset threshold, thereby realizing automatic and autonomous alarm, and the alarm function can be realized without manual participation, and the judgment logic is simple, efficient and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1 Schematic flow diagram of a bolt fracture and deformation monitoring method based on machine vision provided by the present invention.

[0032] Figure 2 Schematic diagram of normal bolt information.

[0033] Figure 3 Schematic diagram of bolt fracture information.

[0034] Figure 4 Schematic diagram of bolt deformation information.

[0035] Figure 5 Schematic diagram of point cloud. Specific implementation manner

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art belong to the scope protected by the embodiments of the present invention.

[0037] The embodiments of the present invention provide a bolt fracture and deformation monitoring method based on machine vision. Referring to Figure 1 , the method includes:

[0038] S1. Obtain the image to be detected by using a camera device.

[0039] S2. Select the region of interest image from the image to be detected. Specifically, it includes the following process:

[0040] S2.1. Use the object detection algorithm to identify all suspected bolts in the image to be detected and calibrate them with detection frames, one of the SSD, YOLO, and ViT methods; the detection frame corresponding to the i-th bolt is the rectangular frame R i , as Figure 2 shown.

[0041] S2.2. After expanding the range of the detection frame by a preset percentage, the intercepted image is used as the region of interest image.

[0042] S3. Detect the bolt fracture and missing situation based on the position information of the bolt. Specifically, it includes the following process:

[0043] S3.1. Calculate the average width w of the bolt based on the mode of the widths of each rectangular frame b ;

[0044] S3.2. Calculate the adjacent bolt spacing based on the sorting of the abscissas of each rectangular frame. The bolt spacing between the i-th bolt and the (i + 1)-th bolt is denoted as d i ;

[0045] S3.3. If d i > 2w b , it is marked as bolt fracture or missing, as Figure 3 shown, and the number of missing ones is

[0047] If the corresponding bolt rectangular frames R on both sides of the image to be detected are not detected i , mark the positions of bolt fracture or missing according to the confidence threshold of the recognition result from the image to be detected based on the object detection algorithm and the preset detection area.

[0048] Due to the perspective of the camera shooting angle, the inconsistent width of the bolt at different distances is calculated to obtain the reference width w b Based on the first preset threshold, there are multiple values, and the best reference bolt w is obtained according to the widths of adjacent bolts b .

[0049] S4. Detect the inclination and deformation of the bolt based on the image of the region of interest, which specifically includes the following process:

[0050] S4.1. Use the trained deep learning model to identify 3 key point information, namely the center point of the bolt top, the center point of the upper boundary of the nut, and the center point of the lower boundary of the nut.

[0051] The training of the deep learning model includes the following process: First, collect the images of the preset area containing bolts; perform data annotation on the bolt nodes in the collected images and form a training set and a validation set; Second, during the model training process, use the online data augmentation method. The data augmentation methods adopted include but are not limited to the following: scale change, Gamma transformation, perspective transformation, brightness and contrast adjustment, random rotation and flipping, histogram equalization, adding Gaussian noise, HSV space color transformation, converting to grayscale, etc. At the same time, use the Mosaic method to randomly splice four transformed images into one piece of data, so that the image background information is richer and is conducive to improving the training effect. Among them, use the L2 regularization method to limit the weights of the convolutional neural network model to improve the generalization ability of the model; Finally, when the verification result of the validation set meets the end condition, end the training and save the detection model parameters.

[0052] S4.2. Fit the straight line equation y = kx + b based on the 3 key point information, where the parameter k to be solved represents the slope and b represents the intercept;

[0053] S4.3. Use the least squares method to solve and obtain the parameters k and b, where Furthermore, obtain the inclination angle and deformation result of the bolt, as Figure 4As shown, for example, the second preset threshold is 5°. If the calculated inclination angle of the bolt is 80° and the calibrated bolt angle is 89°, and the absolute difference of 9° is greater than the second preset threshold, it is considered that the bolt is inclined and an alarm is given.

[0054] The method for collecting the image to be detected in this solution can use a camera and be obtained through various methods such as manual collection, drone inspection collection, wall-climbing robot collection, and fixed-position camera collection. It can also use lidar, etc. to photograph the working area of the bolt to obtain point cloud data as shown, and use point cloud rendering models, such as deep learning three-dimensional reconstruction technologies like the NeRF neural radiance field model and the 3D Gaussian splash model, to perform directional reconstruction on the obtained point cloud map to generate a two-dimensional view from the camera's perspective to obtain an equivalent image of the bolt to be detected; the training of the point cloud rendering model includes: first, collecting point cloud data and actual scene data, and designing loss functions including reconstruction error, view consistency, and smoothness loss; second, using standard optimization algorithms such as the Adam optimizer to train the model to adjust the neural network weights. Finally, when the verification result of the validation set meets the end condition, the training ends and the parameters of the detection model are saved. The inference of the point cloud rendering model includes: model quantization and pruning; directional camera perspective rendering avoids reconstructing the entire 3D scene to obtain an image with the same effect as the visible light acquisition image. Figure 5 As shown, for example, the second preset threshold is 5°. If the calculated inclination angle of the bolt is 80° and the calibrated bolt angle is 89°, and the absolute difference of 9° is greater than the second preset threshold, it is considered that the bolt is inclined and an alarm is given.

[0055] A method for monitoring bolt fracture and deformation based on machine vision provided by the present invention identifies the position of the bolt as the region of interest, and then determines the position where the bolt is fractured and missing. In addition, the inclination angle of the bolt center line is calculated based on three key points of the bolt, and the deformation result of the bolt is determined through a preset threshold, so as to achieve automatic and autonomous alarm, and the alarm function can be realized without manual participation, and the judgment logic is simple, efficient, and accurate.

[0056] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (modules, systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded computers, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0058] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0060] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0061] Obviously, those skilled in the art can make various changes and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. A method for monitoring bolt fracture and deformation based on machine vision, characterized in that, The method includes: S1. Obtain the image to be detected by using a camera device; S2. Select the region of interest image from the image to be detected; S3. Detect the breakage and missing condition of the bolt based on the position information of the bolt; S4. Detect the inclination and deformation condition of the bolt based on the region of interest image.

2. The method for monitoring bolt fracture and deformation based on machine vision according to claim 1, characterized in that: The process of selecting the region of interest image from the image to be detected described in step S2 includes the following process: S2.

1. Identify all suspected bolts from the image to be detected using the object detection algorithm and calibrate them with detection frames. The detection frame corresponding to the i-th bolt is a rectangular frame R i ; S2.

2. After expanding the detection frame by a preset percentage, the intercepted image is used as the region of interest image.

3. The method for monitoring bolt fracture and deformation based on machine vision according to claim 2, wherein: The target detection algorithm uses one of the SSD, YOLO, and ViT methods.

4. The method for monitoring bolt fracture deformation based on machine vision according to claim 2, characterized in that: The detection of the breakage and missing condition of the bolt based on the position information of the bolt described in step S3 includes the following process: S3.

1. Calculate the average width w of the bolts based on the mode of the widths of each rectangular box b ; S3.

2. Calculate the adjacent bolt spacing based on the sorting of the abscissas of each rectangular box, where the bolt spacing between the i-th bolt and the (i + 1)-th bolt is denoted as d i ; S3.

3. If d i > 2w b , it is marked as bolt breakage or missing, and the number of missing ones is 5. The method for monitoring bolt fracture and deformation based on machine vision according to claim 4, characterized in that: If the corresponding bolt rectangular frames R are not detected on both sides of the image to be detected i , based on the confidence threshold of the recognition result from the image to be detected by the object detection algorithm, combine with the preset detection area to mark the bolt fracture or missing position.

6. The method for monitoring bolt fracture and deformation based on machine vision according to claim 4, characterized in that: Calculation of the inconsistent width of the bolt in the far and near directions caused by the perspective of the shooting angle of the imaging device results in the reference width w b , which has multiple values based on the first preset threshold, and the optimal reference bolt w is obtained according to the widths of adjacent bolts b .

7. The method for monitoring bolt fracture and deformation based on machine vision according to claim 4, wherein: The detection of the inclination and deformation condition of the bolt based on the region of interest image described in step S4 includes the following process: S4.

1. Identify three key point information, namely the center point of the bolt top, the center point of the upper boundary of the nut, and the center point of the lower boundary of the nut; S4.

2. Fit the straight line equation y = kx + b based on the three key point information, where the parameter k to be solved represents the slope and b represents the intercept; S4.

3. Solve the parameters k and b, and then obtain the inclination angle and deformation result of the bolt.

8. The method for monitoring bolt fracture and deformation based on machine vision according to claim 4, characterized in that: Step S4.1 uses a trained deep learning model to identify the three key point information.

9. The method for monitoring bolt fracture and deformation based on machine vision according to claim 8, wherein: The training of the deep learning model includes the following process: Collect the preset region images containing bolts; perform data annotation on the bolt nodes in the collected images, and form a training set and a validation set; end the training when the verification result of the validation set meets the end condition, and save the detection model parameters.

10. The method for monitoring bolt fracture and deformation based on machine vision according to claim 4, characterized in that: Step S4.3 uses the least squares method to solve for the parameters k and b, where

Images