Bolt looseness identification method and device based on image center point matching and storage medium

Abstract

The invention discloses a bolt looseness identification method and device based on image center point matching and a storage medium. The method comprises the steps of image preprocessing and correction, detection of a bolt head through a YOLOv8 algorithm and positioning of a center point, matching of a point set through an RANSAC algorithm and calculation of a homography matrix H, and correction of a visual angle of a target image through inverse transformation of H. Bolt accurate positioning and image cutting are carried out, a YOLOv8 algorithm is adopted to carry out bolt target detection on the corrected image, each bolt area is positioned, and an independent bolt image is cut according to a detection result; bolt rotation angle calculation and looseness identification are carried out, a SuperUNet algorithm is used to carry out semantic segmentation on the cut bolt head image, and a bolt edge contour is extracted; and calculating the rotation angle of the bolt based on the edge contour, and judging whether the bolt is loosened by comparing the rotation angle difference with a preset threshold value. According to the invention, high-precision and high-efficiency automatic bolt state monitoring is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and computer vision, and particularly relates to a bolt loosening recognition method based on image center point matching, a device and a storage medium. BACKGROUND

[0002] In the field of construction, bridge and other engineering, steel structure is widely used due to its high strength, light weight and convenient construction. As the main connection form of steel structure, bolt connection plays a key role in transmitting and bearing load, and its reliability is directly related to the safety of the overall structure. However, in the long-term use process, bolts are easily affected by temperature changes, dynamic loads, vibrations and corrosion and other factors, and appear loosening phenomenon, especially in high-strength bolts. If not timely discovered and handled, bolt loosening may lead to degradation of connection performance, and even cause structure failure, so it is of great significance to develop an efficient and reliable bolt state detection method.

[0003] Traditional bolt loosening detection methods include contact detection and sensor detection methods based on sound, light and electrical signals. Contact methods such as visual inspection, knocking method and torque test rely on manual operation, have low efficiency and strong subjectivity, are difficult to meet the detection needs of a large number of bolts in large structures, and are prone to cause missed detection. Although the sensor-based detection means such as piezoelectric impedance method and ultrasonic detection have high precision, they have limitations such as complex equipment, high cost and the need for regular maintenance, which limits their wide application in engineering.

[0004] With the development of technology, computer vision-based detection methods have gradually become a research hotspot, and there are currently three types: one is significant loosening feature detection method, which judges loosening by analyzing the gap between the bolt head and the connecting surface, but it is not sensitive to small amplitude loosening and is prone to missed detection; the second is special marker detection method, which uses pre-set marker lines or marks on the bolt surface to identify changes, which improves the detection accuracy to some extent, but requires additional installation of markers, which has problems such as easy wear, obstruction and high maintenance cost; the third is a rotation angle-based measurement method, which determines loosening by calculating the change of bolt rotation angle, which does not rely on external markers and is suitable for 0~60° loosening detection of hexagonal head bolts, and has good applicability and long-term stability. However, this method still faces several technical difficulties, including image distortion caused by irregular arrangement of bolts, influence of low image resolution on angle measurement accuracy, and problems such as interference of bolt targets and insufficient recognition reliability in complex environments.

[0005] Therefore, how to improve image quality, optimize distortion correction and achieve high-precision angle measurement is still a key challenge that needs to be solved in this technical direction. SUMMARY

[0006] The application aims to provide a bolt loosening recognition method based on image center point matching, a device and a storage medium, solve the problems of insufficient bolt loosening detection precision, low efficiency and limited reliability caused by image distortion, low resolution and background interference in the prior art, and realize high-precision, high-efficiency and environment-adaptive automatic bolt state monitoring.

[0007] The technical scheme adopted by the application is a bolt loosening recognition method based on image center point matching, comprising the following steps:

[0008] Step S1: image preprocessing and correction: obtaining target images and reference images, detecting bolt head target frames and calculating center point coordinates to form a source point set and a reference point set; matching the point set and calculating a homography matrix H, and using the inverse transformation of H to perform geometric correction on the target image;

[0009] Step S2: bolt accurate positioning and image cropping: performing bolt target detection on the corrected image, positioning each bolt head region, and cropping an independent bolt head image according to the detection result;

[0010] Step S3: bolt rotation angle calculation and loosening recognition: performing semantic segmentation on the cropped bolt head image, extracting the bolt edge contour; calculating the rotation angle of the bolt based on the edge contour, and determining whether the bolt is loose by comparing the rotation angle difference with a preset threshold.

[0011] Further, in step S1, after calculating the center point coordinates and before forming the source point set and the reference point set, a row-column grouping strategy is adopted to sort and identify the detected target frames, specifically including: vertically grouping according to the y coordinates of the center points of the target frames, and if the y coordinate difference of two points is less than a set threshold, it is determined as the same row; sorting the target frames in each row according to the x coordinates; and assigning a row number and a column number to each target to achieve unique naming.

[0012] Further, in step S1, the specific process of matching the point set includes: constructing a spatial graph structure of the bolt points, identifying the spatial dense area using the DBSCAN clustering algorithm, and connecting the isolated points in combination with the k-nearest neighbor algorithm; generating a multi-modal feature vector combining the geometric features and SIFT appearance features of the bolt; taking the cosine similarity of the multi-modal feature vector as the measure, and using the Hungarian algorithm for point set matching and quality evaluation.

[0013] Further, in step S3, the semantic segmentation network used is a SuperUNet network; the front end of the SuperUNet network performs super-resolution reconstruction on the input bolt head image based on the Real-ESRGAN algorithm, and the back end performs semantic segmentation based on the U-Net architecture to obtain a sub-pixel precision bolt edge contour.

[0014] Further, in the U-Net architecture, a convolution block attention module CBAM is used to replace the original skip connection, and a spatial pyramid pooling ASPP module is integrated in the encoder;

[0015] The SuperUNet network is trained by a boundary-aware loss function BALM, which is composed of a Dice loss, a contour consistency loss, a curvature consistency loss and an adversarial loss.

[0016] Further, in the step S3, the specific process of extracting the bolt edge contour includes: performing an edge extraction operation on the mask graph obtained after the semantic segmentation to obtain the edge contour of the bolt.

[0017] Further, in the step S3, when calculating the rotation angle based on the edge contour, for the edge points on the contour and the contour centroid, the angle of the edge point relative to the centroid is calculated And the distance The formulas are as follows:

[0018]

[0019]

[0020] Wherein, is the contour centroid coordinate, is the contour edge point coordinate.

[0021] Further, in the step S3, the specific process of determining whether the bolt is loose by comparing the rotation angle difference includes: calculating the angle difference of the corresponding corner points in the loose state and the reference state After excluding the difference in the angle difference that exceeds 60°, the average value of the remaining angle difference is calculated to obtain the average angle difference .

[0022] The calculation formulas of the angle difference and the average angle difference are as follows:

[0023]

[0024]

[0025] Wherein, is the angle of the corresponding corner point and the x-axis when tightening, is the angle of the corresponding corner point and the x-axis after loosening, is the angle difference value of the ith pair of effective matching corner points.

[0026] The application also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to enable the processor to perform the bolt loosening recognition method based on image center point matching.

[0027] The application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the bolt loosening recognition method based on image center point matching.

[0028] The application has the following advantages:

[0029] 1. The application adopts the image center point matching method, combines the YOLOv8 target detection and semantic segmentation model, effectively overcomes the recognition inaccuracy problem caused by perspective distortion and complex environment interference in the traditional bolt loosening detection, innovatively introduces the spatial graph structure modeling based on the DBSCAN clustering and k-nearest neighbor algorithm, combines the bolt geometric features and SIFT appearance features to form a multi-modal feature description, solves the optimal matching pair through the Hungarian algorithm, and dynamically adjusts the RANSAC parameters based on the distance consistency analysis, realizes the adaptive perspective transformation, and effectively suppresses the interference of the perspective change and partial occlusion, provides a stable and high-precision coordinate correspondence for subsequent angle calculation, and fundamentally improves the reliability of the detection system.

[0030] 2. The application fuses the YOLOv8 and the improved SuperUNet network to construct an efficient multi-model collaborative processing framework. In the framework, the Real-ESRGAN algorithm is integrated in the front end of the SuperUNet to perform super-resolution reconstruction on the input image to enhance the detail information; the pixel-level bolt head edge segmentation is realized based on the U-Net architecture in the rear end, so as to obtain high-precision contour data; and the channel-spatial attention module (CBAM) and the spatial pyramid pooling module (ASPP) are innovatively integrated, the feature expression ability is enhanced through the multi-scale attention mechanism; and the boundary perception loss module (BALM) is introduced, the softening Dice loss, the contour consistency constraint and the antagonistic edge optimization are combined, and the segmentation accuracy and naturalness of the bolt edge details are significantly improved. Through the cooperation of the multi-scale attention mechanism and the boundary optimization method, the sub-pixel level bolt contour extraction can be realized under the condition of complex background and low-quality image, the segmentation precision and the anti-environment interference ability are significantly improved; at the same time, with the help of algorithm fusion and loss function optimization, the segmentation difficulty under the low-quality image is effectively overcome, and a stable and reliable basis is provided for the high-precision calculation of the bolt angle.

[0031] 3. The present application can accurately identify the small amplitude of bolt rotation changes (such as below 0.5°) by the sub-pixel corner point positioning method based on the bolt head profile distance spectrum, combined with the two-stage interpolation strategy and multi-condition peak merging algorithm, greatly improving the sensitivity of loose identification. Through the synergistic mechanism of dynamic threshold grouping and centroid alignment verification, the identification problem of regular and irregular arranged bolts is effectively solved, ensuring the geometric consistency of the bolt hexagonal structure. This method not only significantly improves the timeliness of bolt loosening identification, but also is insensitive to the arrangement mode of the bolt, has good universality, and can be extended to the health monitoring field of various steel structures such as buildings and bridges. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0033] Figure 1 is a flowchart of the method of the present application.

[0034] Figure 2 is a flowchart of the center point matching method and bolt semantic segmentation of the present application, wherein (a) is the center point matching method and (b) is the bolt semantic segmentation.

[0035] Figure 3 is a schematic diagram of bolt center point positioning.

[0036] Figure 4 is a schematic diagram of bolt sub-image naming, wherein (a) is regular arrangement of bolts and (b) is irregular arrangement of bolts.

[0037] Figure 5 is a schematic diagram of bolt node image distortion correction.

[0038] Figure 6 is a schematic diagram of YOLOv8 network structure.

[0039] Figure 7 is a schematic diagram of SuperUNet network structure.

[0040] Figure 8 is a schematic diagram of bolt loosening angle calculation process, wherein (a) is corner point positioning and (b) is angle difference calculation.

[0041] Figure 9 is a schematic diagram of multi-condition peak calculation result.

[0042] Figure 10is a schematic diagram of a bolt label dataset, where (a) is a target detection label, and (b) is a semantic segmentation label.

[0043] Figure 11 is an example diagram of data augmentation of a bolt dataset.

[0044] Figure 12 is a schematic diagram of an experimental device and a bolt loosening angle.

[0045] Figure 13 is a result diagram of an ablation experiment of a super-resolution algorithm, where (a) is without using a super-resolution algorithm, and (b) is using a super-resolution algorithm.

[0046] Figure 14 is an error diagram of bolt loosening angle estimation under different viewing angles, where (a) is a shooting distance of 0.5 m, (b) is a shooting distance of 1.0 m, and (c) is a shooting distance of 1.5 m. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0048] As shown in Figure 1 , a bolt loosening recognition method, device and storage medium based on image center point matching provided by the embodiment, comprising the following steps:

[0049] Step S1: image preprocessing and correction: this step aims to solve the problem of image perspective distortion caused by differences in shooting angles. By extracting the center point of the bolt head and calculating the homography matrix, geometric correction of the target image is achieved, providing an accurate and consistent visual basis for subsequent bolt positioning and angle analysis. Specifically, the following processes are included:

[0050] S11: image acquisition and enhancement: the bolt connection node image is acquired by a camera device such as a smart phone or an industrial camera, and the image is input to the processing system. The acquired image may have perspective distortion due to different shooting angles and positions.

[0051] In order to improve the generalization ability and robustness of the model under complex working conditions, data enhancement techniques are used to simulate the generation of bolt image samples under various complex weather and environments such as fog, rain, snow and motion blur, in order to construct more comprehensive and diverse training data sets.

[0052] S12: bolt detection and robust matching based on YOLOv8:

[0053] This step aims to accurately identify bolts from the images and establish a highly reliable point correspondence, laying the foundation for geometric correction. It also calculates the Laplacian variance of each image as a sharpness indicator for subsequent correction steps. Figure 2 As shown in (a), the specific process is as follows:

[0054] (1) Target detection and center point localization:

[0055] Based on the YOLOv8 algorithm, the target bounding box of the bolt head is detected, and the coordinates of the center point of the target bounding box in the target view (source point set) and the reference view (reference point set) are extracted as key feature points. For example... Figure 3 As shown, for each calibration box, through its upper left corner and bottom right corner The coordinates are used to calculate the center point using the formula. :

[0056]

[0057] (2) Adaptive bolt row and column grouping and identification:

[0058] To enhance the robustness of the matching process and effectively eliminate outlier interference, before forming the source point set and reference point set, the ordinates of the center points of all bolt target boxes detected by YOLOv8 are first extracted to form a one-dimensional dataset. To automatically estimate the number of rows of bolts in an image, within a preset search range (such as the minimum number of rows). , the maximum number of rows Within the dataset, the K-Means clustering algorithm is applied to cluster the ordinate dataset, and the optimal number of clusters is automatically determined based on metrics such as silhouette coefficient. This is the estimated number of rows; after obtaining the estimated number of rows... Then, a dynamic threshold mechanism is used to achieve intelligent grouping in the vertical direction: threshold = ×Average row height+ ×(image height / expected number of rows) Where N is the total number of bolts; This represents the ordinate of the center point of the bounding box of the Nth bolt detected by the YOLOv8 algorithm; the average row height is obtained by calculating the median difference between the ordinates of all adjacent bolts. and Weighting coefficients (usually set as follows) , If the difference between the vertical coordinates of two points is less than the threshold, they are determined to be in the same row; then the target boxes in each row are horizontally sorted according to their horizontal coordinates x.

[0059] A three-part naming scheme of "row number-column number-feature hash value" is assigned to each target (the feature hash value is generated based on bolt geometric features such as area, aspect ratio), and the accuracy between rows is ensured by combining the centroid alignment verification. At the same time, a topological relationship matrix is constructed to record the adjacency relationship, providing rich features for subsequent matching. The grouping results are shown in Figs. Figure 4 (a), Figure 4 (b).

[0060] (3) Multi-modal graph matching and quality evaluation:

[0061] After completing the grouping and sorting, an innovative center point matching method is applied for intelligent point set matching. DBSCAN clustering algorithm (parameters eps=100, min_samples=2) and k-nearest neighbor algorithm (k=3) are used to construct the spatial graph structure of the bolt points: DBSCAN is used to identify spatial dense regions, and k-nearest neighbor algorithm is used to connect isolated points, thereby forming a robust graph model that combines geometric connection relationship and clustering attributes; combined with the geometric features (such as center coordinates, aspect ratio, area) and SIFT appearance features (texture descriptor) of the bolt, a multi-modal feature vector is generated to significantly enhance the discrimination between different point sets; the Hungarian algorithm is used to solve the optimal matching pair by taking the cosine similarity of the multi-modal features between point sets as the measure.

[0062] To evaluate the reliability of the matching results, distance consistency analysis is performed; a smoothing term and normalization are introduced, and the matching quality score formula is as follows:

[0063]

[0064]

[0065] wherein, represents the matching quality score obtained by distance consistency analysis, the value range is (0, 1], and the value closer to 1 indicates higher matching consistency; , respectively represent the calculation of mean and standard deviation; , are the internal distance vectors of the source point set and the reference point set, respectively; , are the element numbers of vectors , , is the combined standard deviation; is a very small constant (such as 1e-6) to ensure numerical stability; is an exponential function, which plays a role of normalization and smoothing.

[0066] S13: Image distortion correction based on perspective transformation

[0067] Image perspective correction is performed using a center point location matching method. A minimum quality threshold (e.g., score < 0.3) is set, and matching pairs with scores below this threshold are directly eliminated, thus removing obviously erroneous outliers in the initial stage. The RANSAC algorithm parameters are dynamically adjusted based on the aforementioned matching quality score (S12) and image sharpness (calculated via Laplacian variance) to adaptively calculate the homography matrix: high-quality matching (score > 0.8) uses a threshold of 2.0, a confidence level of 0.99, and a maximum number of iterations of 2000; low-quality matching (score ≤ 0.5) uses a threshold of 5.0, a confidence level of 0.90, and a maximum number of iterations of 500. Only the high-quality matching pairs (i.e., inliers) retained after filtering are input into the subsequent RANSAC algorithm for robust estimation of the homography matrix, thereby ensuring the accuracy of geometric correction.

[0068] Perspective transformation is a key technique in computer vision for image registration and geometric alignment. Its core lies in establishing a mapping relationship between two images using a transformation matrix. Specifically, the homography matrix H is calculated using a set of matched center points, and its mathematical expression is:

[0069]

[0070] in, The coordinates of the center point in the reference image; The matrix represents the coordinates of the corresponding center point in the transformed target image. It contains complete geometric transformation information, its elements Together, they determine the effects of all geometric transformations, from translation and rotation to scaling and perspective distortion. , ,and By inversely transforming this matrix, the target image is corrected to match the viewpoint of the reference image, thus achieving image correction and ultimately ensuring consistency between the front and rear shooting viewpoints. Figure 5 As shown in the figure. Where (x', y') are the coordinates of a point in the reference image, and (x, y) are the coordinates of the corresponding point in the transformed target image.

[0071] Step S2: Precise bolt positioning and image cropping based on YOLOv8:

[0072] S21: Precise Bolt Location: Based on the perspective-corrected image obtained in step S1, the YOLOv8 model is used again to detect the bolt in the image, achieving precise location.

[0073] like Figure 6As shown, the YOLOv8 network consists of three main modules: the backbone, the Feature Pyramid Network (FPN), and the YoloHead. In the backbone, an efficient feature extraction network processes the input image, extracting three effective feature layers at different levels as the foundation for subsequent processing. The FPN part continues the PANet structure, fusing multi-scale feature information through upsampling and downsampling operations (e.g., upsampling a 20×20×512 feature layer and combining it with a 40×40×512 feature layer), ultimately generating feature layers P3, P4, and P5 to enhance the model's detection capabilities. The YoloHead then predicts the target's category and location information based on these feature maps.

[0074] To evaluate model performance, accuracy was used. Recall rate and The F1 score serves as the evaluation metric, where TP, FP, and FN represent true positives, false positives, and false negatives, respectively. The closer the F1 score is to 1, the better the overall performance of the model.

[0075] S22: Bolt Image Cropping and Naming: After the detection is completed, the system uses the bounding box coordinates obtained in step S21 to crop out the image of each independent bolt head region from the corrected image and performs standardized naming, providing input for the fine-tuning process in step S3.

[0076] Step S3: Bolt rotation angle calculation and loosening identification based on semantic segmentation:

[0077] S31: Semantic segmentation of bolt heads: such as Figure 2 As shown in (b), this step performs refined semantic segmentation on the bolt head images obtained from S22 cropping. The SuperUNet network structure is adopted, which innovatively integrates multi-scale attention mechanisms and adversarial boundary optimization to overcome the limitations of traditional U-Net. Figure 7 As shown, the network front-end integrates the Real-ESRGAN algorithm for super-resolution reconstruction to enhance image details.

[0078] Real-ESRGAN is a blind super-resolution reconstruction algorithm designed for complex degraded scenes. Its core utilizes a Generative Adversarial Network (GAN) for image reconstruction, comprising a generator network, a discriminator network, and a multi-stage training strategy. The generator uses Residual Dense Blocks (RRDB) as its basic module, mathematically expressed as:

[0079]

[0080] in, and These represent the input and output features, respectively. Convolutional parameters for the mth dense connection; to The output features of all the residual dense connection layers in the current residual dense block are represented as X m-1, X m-2,..., X 1, and X 0. This kind of dense connection structure enables the features to flow and reuse sufficiently within the block, which helps to recover more rich detail information. represents a feature concatenation operation. This operation provides a data basis for subsequent sub-pixel level segmentation.

[0081] The backend of the SuperUNet network implements pixel-level bolt head edge segmentation using a U-Net architecture to obtain high-precision contour information. The U-Net is a fine segmentation network, and its classic structure includes an encoder (down-sampling), a decoder (up-sampling), and a skip connection, which is suitable for accurate segmentation of bolt edge contours in a simple background. However, the original skip connection mechanism of the U-Net has significant limitations: on the one hand, it can only realize the direct association of features at the same level, and cannot deeply mine the complementary value between the details contained in the shallow features and the semantic information carried by the deep features; on the other hand, the original feature fusion only uses a simple concatenation mode, which not only easily causes information redundancy accumulation in the feature channels, but also cannot realize the efficient cooperation of shallow detail features and deep context information, ultimately leading to the difficulty of the model in breaking through the segmentation accuracy bottleneck due to insufficient feature interaction in complex scenes. In view of this core problem, the present application introduces a CBAM (Channel-Spatial Attention Module) to replace the traditional skip connection of the U-Net, which strengthens the representation ability of key features by integrating and optimizing features at different levels step by step.

[0082] Based on the above improvements, the core segmentation architecture is reconstructed in this step: in the encoder part, the network integrates an ASPP (Atrous Spatial Pyramid Pooling) module to capture multi-scale context information through parallel atrous convolutions (with dilation rates of 1, 6, 12, and 18) and enhance the model's adaptability to bolts of different sizes; in the skip connection part, a CBAM (Channel-Spatial Attention Module) is introduced to replace the original connection, which dynamically focuses on the key regions of the bolt edge through the channel attention branch (based on global average pooling and max pooling to generate weights) and the spatial attention branch (based on convolution layers to generate a spatial weight map), suppresses background interference, and improves feature expression ability; the decoder is responsible for up-sampling and feature fusion, and finally outputs a high-precision segmentation mask.

[0083] The convolutional block attention module (CBAM) dynamically optimizes feature fusion through the dual mechanisms of channel attention and spatial attention, significantly improving the bolt edge segmentation accuracy. The channel attention first performs global average pooling and max pooling on the input features to generate two Cx1x1 feature vectors, capturing the global response of each channel; then the pooling results are processed through a shared MLP (containing two 1x1 convolution layers and ReLU activation), and the calculation formula is:

[0084]

[0085] In the channel attention mechanism of the CBAM module, MLP is a shared network that processes the feature vectors generated by global average pooling and global max pooling respectively to calculate the channel attention weights. This represents the feature vector after global average pooling or max pooling. , These are the convolution weights; This represents the activation function; ultimately, it will... The outputs are summed and channel weights are generated using the Sigmoid function. Recalibrate input features:

[0086]

[0087] in, Indicates channel attention weight The calibrated intermediate feature map; The input feature map of the CBAM module has dimensions C×H×W, where C is the number of channels, and H and W are the height and width of the feature map, respectively. This represents channel-by-channel multiplication;

[0088] Spatial attention performs averaging and max pooling on the features along the channel dimension, resulting in two 1×H×W feature maps. These maps are then concatenated and fused through a convolutional layer with a kernel size of k×k (default 7×7) to generate spatial weights. The calculation formula is:

[0089]

[0090] in, Indicates the convolution operation; For Sigmoid function; concat represents tensor concatenation operation, which concatenates the results of different pooling operations along the channel dimension; , These represent the output results of average pooling and max pooling performed along the channel dimension of the feature map, respectively; Output Finally, the weights are applied to optimize the features:

[0091]

[0092] in, This represents the final output feature map of the CBAM module; this design enables the model to adaptively focus on key areas at the bolt edges and suppress background interference.

[0093] The network training employs a boundary-aware loss (BALM) module for collaborative optimization. This module is a core component of the SuperUNet network, specifically designed to address challenges in bolt edge segmentation, such as low sensitivity to small targets, blurred boundaries, loss of details, and interference from complex environments. The BALM module overcomes the limitations of traditional single loss functions by achieving end-to-end joint optimization through four collaboratively working sub-units, significantly improving segmentation accuracy and robustness. The four core components include:

[0094] (1) Dice Loss Unit: A softened Dice coefficient and class weighting strategy is adopted to enhance the segmentation stability of small target bolt regions. Traditional Dice loss is sensitive to class imbalance, which can easily lead to bias, especially in the segmentation of small targets such as bolts. This design effectively alleviates the foreground-background pixel imbalance problem through softening and dynamic weighting; its mathematical expression is:

[0095]

[0096] in, For Dice's loss; To predict probabilities; This is a real label; To avoid extremely small constants with a denominator of 0, a value of 1e-6 is used; class weights The pixel ratio of the bolt foreground to the background is dynamically adjusted to enhance the segmentation stability of small target bolt areas.

[0097] (2) Contour Consistency Loss Unit: By integrating distance transformation and curvature constraints, boundary alignment is enhanced to ensure edge smoothness and the realism of the hexagonal structure. The distance transformation maps of the predicted and actual edges are calculated. , And define pixel-level error:

[0098]

[0099] in, This indicates the loss of contour consistency; This represents the total number of all valid edge pixels; This represents the distance transform map corresponding to the predicted segmentation result in pixels. The value at; This represents the distance transform map corresponding to the real label in pixels. The value at the location is also introduced; simultaneously, a curvature consistency constraint is introduced, and edge curvature features are extracted using the Sobel operator to calculate the curvature difference:

[0100]

[0101] in, This indicates the loss of curvature consistency. This represents the predicted curvature plot. Represents the true curvature diagram.

[0102] (3) Adversarial Edge Optimization Unit: Through adversarial learning, the network learns to generate more natural edge textures, improving robustness in complex environments such as fog, haze, rain, and snow; a lightweight discriminator is used for generative adversarial training based on the Wasserstein GAN-GP framework; adversarial training includes discriminator loss. With generator loss Two parts:

[0103]

[0104] in, This represents the mathematical expectation operator, which calculates the average of the expression within parentheses over the data distribution. This represents the output score of the discriminator for sample x; This represents the predicted edge distribution output by the generator (i.e., the segmentation network); Let represent the marginal distribution predicted by the generator (segmentation network). Discriminator output score for medium sample x Determine the expected value; This represents the distribution of the actual edge labels; Represents the distribution of true edge labels Discriminator output score for medium sample x Determine the expected value; Represents the random interpolation points of the real and generated samples. From real samples With generated samples Linear combination produces: interpolation coefficients It follows a uniform distribution, that is... ; Indicates the gradient penalty coefficient; This indicates that the discriminator D is used for random interpolation points. The gradient; gradient penalty term The discriminant gradient norm is forced to be close to 1; Indicates the random interpolation point In its distribution Calculate the expected value of the corresponding terms.

[0105]

[0106] The generator minimizes This maximizes the discriminator's score on the generated samples, thereby improving the realism of the generated samples.

[0107] (4) Dynamic weight control unit: Adaptively adjusts the loss weights according to the training stage and image content complexity. The weight calculation formula is:

[0108]

[0109] in, These are dynamic weighting coefficients; Image gradient variance (reflecting sharpness); , For hyperparameters; This is the current training round; The total number of training rounds is 1; early training focuses on stable convergence of Dice loss, while later training enhances contour and adversarial loss to refine the boundaries.

[0110] Through the above multi-dimensional optimization, end-to-end joint optimization of reconstruction and segmentation is achieved, outputting a high-precision segmentation mask, providing reliable input for angle calculation.

[0111] To evaluate the performance of the segmentation model, pixel accuracy, average pixel accuracy (mPA), and average intersection-union ratio (mIoU) are used as evaluation metrics:

[0112]

[0113]

[0114]

[0115] Where k is the total number of categories; This represents the number of pixels that were correctly predicted, which is the sum of pixels with both the true class and the predicted class being l. This represents the number of pixels whose true class is l but are predicted as class j. This represents the number of pixels whose true category is j but are predicted as category l.

[0116] S32: Bolt Rotation Angle Calculation and Loosening Determination: The bolt image mask obtained in S31 is converted to the HSV (Hue, Saturation, Brightness) color space, and a specific red area (a pre-defined marker for auxiliary recognition) is extracted. Then, the bolt edge contour is extracted through morphological operations; the bolt rotation angle is calculated and loosening is determined.

[0117] (1) Angle calculation and sub-pixel corner location:

[0118] Calculate the centroid coordinates of the profile For each edge point on the segmented contour, calculate its angle relative to the centroid. and distance This forms the contour distance spectrum, calculated using the following formula:

[0119]

[0120]

[0121] To achieve high-precision positioning, a two-stage interpolation strategy (linear interpolation combined with cubic spline interpolation) is adopted to reconstruct and smooth the range spectrum, and Gaussian filtering is used ( To eliminate noise interference and form a continuous distance-angle curve, sub-pixel accuracy corner positioning is achieved. A multi-condition peak merging algorithm is used to optimize corner detection: local extrema in the distance spectrum are detected as candidate corners; an angle difference threshold mechanism (e.g., 40°) is introduced to filter out spatially adjacent redundant peaks; finally, peak significance evaluation (e.g., amplitude sorting) is combined to retain key corners, ensuring the geometric consistency of the bolt's hexagonal structure. The peaks after curve smoothing correspond to the corner positions of the bolt head, such as... Figure 8 As shown in (a); a schematic diagram of the multi-condition peak value calculation results is shown below. Figure 9 As shown.

[0122] Calculate the angle difference between corresponding corner points before and after loosening. :

[0123]

[0124] in, This refers to the angle between the corresponding corner point and the x-axis when tightened. The angle between the corresponding corner point and the x-axis after loosening.

[0125] like Figure 8 As shown in (b), before calculating the average angle difference, differences exceeding 60° are first excluded. This setting is based on the fact that the bolt has a hexagonal head structure, and ideally, the interval between adjacent corner points is 60°. Therefore, setting the abnormal threshold to 60° can effectively filter out excessive deviations caused by mismatches and prevent missed detections. The maximum and minimum values ​​are removed from the remaining valid differences, and the corrected average angle difference is calculated according to the formula. :

[0126]

[0127] in, Let be the angle difference between the i-th pair of valid matching corner points.

[0128] (2) Loosening identification: The calculated average rotation angle difference Compare with a preset threshold. If If the value exceeds the threshold, the bolt is determined to be loose; otherwise, the bolt is determined to be not loose. The system outputs the final diagnostic result accordingly.

[0129] To verify the effectiveness of the method described in this invention, a series of experiments and analyses are conducted below.

[0130] 1. Dataset Preparation and Augmentation

[0131] Two separate datasets were created for training object detection and semantic segmentation models. A total of 656 images containing actual bridge connection bolts, laboratory equipment bolts, and bolt plate specimens were acquired using a camera and a smartphone, with resolutions of 6000×4000 and 3648×2736, respectively. Bolt targets in the images were manually labeled with green boxes using LabelImg software, such as... Figure 10 As shown in (a), the target detection model was trained. Then, using the model's built-in cropping function, the bounding box was enlarged by 20% and cropped to obtain individual bolt images. These images were then manually labeled with the hexagonal outline of the bolts using LabelMe software, as shown... Figure 10 As shown in (b), a total of 1935 labeled images of individual bolts were obtained.

[0132] To improve the model's generalization ability and reduce the risk of overfitting, the dataset was augmented. For the object detection dataset, complex environments such as severe weather, Gaussian noise, brightness variations, and Gaussian blur were simulated. For the semantic segmentation dataset, processing including mirror flipping, Gaussian noise reduction, brightness variation adjustment, and background replacement was applied. Figure 11 As shown, after enhancement, the object detection dataset contains a total of 5437 images, and the semantic segmentation dataset contains a total of 7373 images. Both datasets are randomly divided into training set (81%), validation set (9%), and test set (10%).

[0133] 2. Model Training and Selection

[0134] The experiment was conducted on a computer equipped with an Ubuntu 20.04.5 LTS operating system, an Intel i9-13900KF CPU (24 cores, 32 threads), and two NVIDIA RTX4090 GPUs (24GB of VRAM). CUDA 12.6 and CUDNN 8.5.0 were used for accelerated computation, and the programming environment was built based on Python 3.11.7 and PyTorch 1.13.1.

[0135] To compare and select the optimal model, comparative experiments were conducted. For object detection models, YOLOv7 and YOLOv8 variants (including YOLOv7-L / X, YOLOv8-N / S / M / L / X) were compared. For semantic segmentation models, U-Net, HRNet, DeepLabv3+, PSPNet, and SuperUNet were compared.

[0136] Table 1 Evaluation indicators of the validation results of each target detection model

[0137] Table 1 shows the parameter size, speed, P, r, and F1 of different YOLOv7 and YOLOv8 models based on the validation set. As the parameter size increases, the model speed decreases. To achieve good results in terms of both accuracy and speed, the YOLOv8 model was selected for bolt head target detection.

[0138] Table 2 Evaluation metrics for the validation results of each semantic segmentation model

[0139] Table 2 shows the parameter size, speed, mPA, accuracy, and mIoU of the U-Net, HRNet, DeepLabv3+, PSPNet, and SuperUNet models based on the validation set. The accuracy differences among the models are small, but considering both accuracy and speed, the SuperUNet model with the ResNet50 backbone is selected for bolt detection in the fine segmentation task of hexagonal bolts.

[0140] To compare and select the optimal model, comparative experiments were conducted. The object detection model was compared with YOLOv7 and YOLOv8 series variants (including YOLOv7-L / X, YOLOv8-N / S / M / L / X). For semantic segmentation, four mainstream algorithms with representative structures—U-Net, HRNet, DeepLabv3+, and PSPNet—were selected as benchmark models, and their performance was compared with the improved U-Net algorithm proposed in this study. Based on performance metrics on the validation set (such as precision P, recall R, F1 score, mPA, mIoU, etc.), and considering both model speed and accuracy, the YOLOv8s model was ultimately selected for bolt head object detection, and the SuperUNet model with ResNet50 as the backbone network was selected for fine-grained bolt segmentation.

[0141] 3. Bolt loosening test results and analysis

[0142] A bolted connection model (steel plate dimensions 320×230×12mm, using 6 sets of M16 bolts) was constructed under laboratory conditions. The model was photographed directly at the center of the steel plate using a smartphone's rear camera (image resolution 4096×3072). A laser rangefinder was used to measure the shooting distance, and a miniature tiltmeter was used to measure the actual rotation angle of the bolts as a reference value. The set loosening angles for some bolts were 6.7°, 32.0°, 38.2°, and 9.4°, respectively. Figure 12 As shown.

[0143] 3.1 Validation of the Super-Resolution Reconstruction Algorithm (Ablation Experiment)

[0144] To verify the effectiveness of the super-resolution reconstruction algorithm (step S31), an ablation experiment was conducted. The experiment analyzed the impact of using the algorithm on the bolt loosening angle detection error at different shooting distances ranging from 0.5 meters to 2 meters. The results showed that, as... Figure 13 As shown in (a), without the super-resolution algorithm, the error increases with distance, reaching a maximum error of approximately 4.8° at 2 meters. Figure 13 As shown in (b), after applying this algorithm, the error is significantly reduced. The error is basically kept within 2° at distances of 0.5 meters and 1 meter, with a maximum error of 3.6°, which effectively improves the detection accuracy.

[0145] 3.2 Method Robustness Testing

[0146] Further testing investigated the effects of different shooting distances (0.5 meters to 1.5 meters) and different viewing angles (20°, 40°, 60°) on the detection results. The measurement results were basically consistent with the actual angles, and the error range remained within an acceptable level. Specific results are as follows... Figure 14 (a) to Figure 14 As shown in (c), under bright lighting conditions at a shooting distance of 1 meter, when the viewing angle is between 20° and 40°, the estimation error of the loosening angle of most bolts is within 3.2°; even when the viewing angle is increased to 60°, the maximum error is 3.0°, and the estimation error tends to stabilize as the shooting distance increases, which proves the robustness and accuracy of the method of the present invention under different conditions.

[0147] In summary, through systematic model comparison, ablation experiments, and robustness tests, the technical solution proposed in this invention has been fully verified. The results show that a model combination based on YOLOv8s and SuperUNet (ResNet50 backbone), supplemented by Real-ESRGAN super-resolution reconstruction preprocessing, can construct an efficient, accurate, and stable automatic bolt loosening detection system under different shooting conditions (such as changes in distance and viewing angle).

[0148] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0149] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A bolt loosening recognition method based on image center point matching, characterized in that, The method comprises the following steps: Step S1: image preprocessing and correction: obtaining target image and reference image, detecting bolt head target frame and calculating center point coordinates, forming source point set and reference point set; matching point set and calculating homography matrix H, using inverse transformation of H to perform geometric correction on target image; Step S2: accurate positioning of bolt and image cropping: performing bolt target detection on the corrected image, positioning each bolt head region, and cropping independent bolt head images according to the detection results; Step S3: bolt rotation angle calculation and looseness identification: performing semantic segmentation on the cropped bolt head image, extracting the bolt edge contour; based on the edge contour, the rotation angle of the bolt is calculated, and whether the bolt is loose is determined by comparing the rotation angle difference with a preset threshold.

2. The bolt loosening recognition method based on image center point matching according to claim 1, characterized in that, In step S1, after calculating the center point coordinates and before forming the source point set and the reference point set, a row-column grouping strategy is used to sort and identify the detected target frame, which specifically includes: vertical direction grouping according to the vertical coordinates y of the center points of each target frame, and if the vertical coordinate difference of two points is less than a set threshold, it is determined as the same row; sorting the target frame in each row according to the horizontal coordinates x; assigning a row number and a column number to each target to achieve unique naming.

3. The bolt loosening recognition method based on image center point matching according to claim 1, characterized in that, In step S1, the specific process of matching the point set includes: constructing a spatial graph structure of bolt points, using DBSCAN clustering algorithm to identify spatial dense regions, and combining k-nearest neighbor algorithm to connect isolated points; generating a multi-modal feature vector combining the geometric features and SIFT appearance features of the bolt; using the cosine similarity of the multi-modal feature vector as the measure, and using the Hungarian algorithm for point set matching and quality evaluation.

4. The bolt loosening recognition method based on image center point matching according to claim 1, characterized in that, In step S3, the semantic segmentation network used is SuperUNet network; the front end of the SuperUNet network is based on Real-ESRGAN algorithm to perform super-resolution reconstruction on the input bolt head image, and the back end is based on U-Net architecture for semantic segmentation to obtain sub-pixel precision bolt edge contour.

5. The bolt loosening recognition method based on image center point matching according to claim 4, characterized in that, In the U-Net architecture, the convolution block attention module CBAM is used to replace the original skip connection, and the spatial pyramid pooling ASPP module is integrated in the encoder; The SuperUNet network is trained by a boundary-aware loss function BALM, and the loss function BALM is composed of Dice loss, contour consistency loss, curvature consistency loss and adversarial loss.

6. The bolt loosening recognition method based on image center point matching according to claim 1, characterized in that, In step S3, the specific process of extracting the bolt edge contour includes: performing edge extraction operation on the mask image obtained after semantic segmentation to obtain the edge contour of the bolt.

7. The bolt loosening recognition method based on image center point matching according to claim 1, characterized in that, In the step S3, when calculating the rotation angle based on the edge contour, the angle of the edge point relative to the centroid is calculated for the edge point on the contour and the centroid of the contour and the distance The formulas are as follows, respectively: ； ； wherein, is the contour centroid coordinate, is the contour edge point coordinate.

8. The bolt loosening recognition method based on image center point matching according to claim 7, characterized in that, The specific process of determining whether the bolt is loose by comparing the rotation angle difference in the step S3 includes: calculating the angle difference of the corresponding angle point in the loose state and the reference state , excluding the difference exceeding 60° in the angle difference, and obtaining the average angle difference by averaging the remaining angle difference . the angle difference the average angle difference the calculation formula is: ； ； wherein, is the angle of the corresponding corner point with the x-axis at the time of tightening, is the angle of the corresponding corner point with the x-axis after loosening, is the angle difference of the ith pair of effective matching corner points.

9. An electronic device comprising a memory and a processor, said memory having stored therein a computer program, characterized in that, The computer program is executed by the processor to make the processor execute the steps of the method of any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

Images