Steel bridge bolt falling detection method based on data enhancement

Abstract

The invention discloses a steel bridge bolt falling-off detection method based on data enhancement, and relates to the technical field of bridge detection, and the method comprises the steps: 1, generating a falling-off bolt pseudo image based on improved FASTGAN network training, and achieving the effective augmentation of a data set under the condition of rare falling-off bolt samples; 2, providing a generated image quality evaluation method based on an HDBSCAN clustering algorithm, and selecting high-quality and diversified generated images; and 3, training is carried out based on a YOLOV8-LSKA network, and steel bridge bolt falling detection is realized. According to the invention, the falling bolt can be rapidly detected with high precision, and the tedious process of a traditional image processing method is avoided.

Description

Technical Field

[0001] This invention belongs to the field of bridge inspection technology, specifically a data-augmented method for detecting loose bolts in steel bridges. Background Technology

[0002] Bolted connections, as one of the main methods of connecting steel bridge nodes and components, are subject to long-term vibration and load conditions during actual service due to the coupling effect of external environment, vehicles, and temperature loads. This leads to bolts being prone to loosening or even breaking and falling off, seriously affecting the safety of personnel and structures below the bridge and significantly reducing the safety of the bridge structure during its service life. Therefore, developing real-time and automated bolt loosening detection technology, while improving the accuracy and efficiency of detection, has become a key issue that urgently needs to be addressed in the field of bridge engineering.

[0003] Traditional detection methods for bolt loosening mainly include manual inspection, bridge dynamic response-based methods, and machine vision-based methods. Manual inspection is currently the most common method for detecting bolt defects in bridges, but it suffers from low efficiency, high false negative rates, and high costs when dealing with long-span steel bridges, making real-time detection difficult. Bridge dynamic response-based methods monitor bridge vibration, local strain, and other dynamic response parameters, using anomaly signals to locate loose bolts. However, the actual stress state of bridges is complex, and different defects can couple and affect the signals of loose bolts, making accurate judgments difficult.

[0004] Currently, vision-based inspection methods have significant advantages in detecting bridge defects due to their non-contact, high efficiency, and low cost. On one hand, traditional image processing methods analyze bolt conditions one by one, focusing on bolt outlines, key points, and shadow areas to detect loose bolts. However, this method requires high-quality bolt images and has low detection efficiency, making real-time and batch detection difficult. On the other hand, methods based on detection models such as YOLO primarily achieve loose bolt detection through model training. However, this method requires a large amount of training data, and is prone to data shortages when dealing with small-sample problems like bolt loosening, leading to model overfitting. Furthermore, the dense distribution of bridge bolts and complex backgrounds can easily cause misjudgments and missed detections, affecting the detection results. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and proposes a data-enhanced method for detecting loose steel bridge bolts. This method aims to effectively improve the detection efficiency and accuracy of loose bolts, thereby solving problems such as high cost, difficulty in real-time and batch detection, and insufficient sample size of loose bolts in traditional detection methods.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention provides a data-augmented method for detecting loose steel bridge bolts, characterized by the following steps: Step 1: Obtain the image set of loose bolts ,in, Indicates the size is The n Image of a loose bolt. N This represents the total number of images of detached bolts. Represents the length of the image of the detached bolt. Represents the width of the image of the detached bolt; Obtain normal bolt image set ,in, Indicates the first m A normal bolt image, M This indicates the total number of normal bolt images; Step 2: Construct an improved FASTGAN network, including a cross-scale feature fusion generator and a discriminator, and handle random noise. Z and X The images are processed to obtain discrimination scores, which are then used to construct a loss function to train an improved FASTGAN network, resulting in a trained image generation model for loose bolts. This model is then used to generate a pseudo-image set of loose bolts. ,in, Indicates the first A fake image of a loose bolt. A This represents the total number of pseudo-images of detached bolts; Step 3: Use the HDBSCAN clustering algorithm to... After filtering, a high-quality image set of detached bolts was obtained. ,in, Indicates the first b High-quality image of the loose bolt. B This represents the total number of high-quality images of the detached bolts; Step 4: For the normal bolt dataset Augmentation is performed to construct a normal bolt training image set. ,in, Indicates the first e A normal bolt training image, E This represents the total number of training images for normal bolts; Will , Merged into a bolt training image set and order T Any bolt training image is I ,and I The size is M I × N I,in, M I Indicates the length of the bolt training image. N I Indicates the width of the bolt training image; Step 5: [Regarding...] I middle s The location and category of each detected target are labeled to obtain... s Location label sequence of targets g and s A sequence of category labels for each target c ; Step 6: Construct the YOLOv8-LSKA network, which includes, in sequence: a feature extraction network, a feature fusion network, and a prediction network, and then... I Processing is performed to obtain I Category prediction label sequence and location prediction label sequences ; Step 7: Based on and as well as and Classification loss and bounding box loss are constructed separately, and the YOLOV8-LSKA network is iteratively trained using the SGD optimizer until the classification loss and bounding box loss converge, thus obtaining a trained bolt detection model for detecting loose bolts in bolt images.

[0007] The data-augmented steel bridge bolt loosening detection method described in this invention is also characterized in that step 2 includes the following steps: Step 2.1: The cross-scale feature fusion generator includes: an initialization layer, an upsampling module, a local skip activation network, a local feature fusion network, and an output convolution module, and handles random noise. Z Processing is performed to obtain a high-resolution pseudo-feature map. and low-resolution pseudo-feature maps ; Step 2.2: The discriminator includes a first downsampling module, a second downsampling module, and a convolutional layer, and performs... as well as , The data is processed to obtain the discrimination score of the high-resolution real image. L Discrimination score of 0 and low-resolution real images L 1 and the discrimination score of high-resolution pseudo-images l Discrimination scores for 0 and low-resolution pseudo-images l 1; Step 2.3: Calculate the discriminator loss using equations (3) and (4). LD and the loss of the cross-scale feature fusion generator L G ; (3) (4) In equations (3) and (4), This indicates a vector concatenation operation.

[0008] Step 2.4: Train the improved FASTGAN network using the Adam optimizer and calculate... L G and L D To update network parameters until L G and L D The process continues until convergence is achieved, resulting in a well-trained model for generating images of detached bolts, and a set of pseudo-images of detached bolts is generated. .

[0009] Furthermore, step 2.1 includes the following steps: Step 2.1.1: The initialization layer handles random noise. Z Process the noise to output an initial noise feature map. P 0; Step 2.1.2: The upsampling module uses bilinear interpolation to sequentially... P 0 proceed H The next upsampling process generates the corresponding... H A set of coarse noise feature maps ,in, The first part represents the no-local-jump excitation and no-local-feature fusion. h A coarse noise feature map; Step 2.1.3: The local skip reinforcement network includes: J One jump stimulus module SLE; No. j The first jump excitation module SLE utilizes equation (1) to perform local jump excitation and local feature fusion on the first jump excitation module. h+j Coarse noise feature map and the j- The first local jump excitation and the first no-local feature fusion H Coarse noise feature map Processing is performed to obtain the first... j The first local jump excitation and the fusion of no local features H Coarse noise feature map Thus, the size is The The first local jump excitation and the fusion of no local features H Coarse noise feature map ; (1) In equation (1), Indicates an upsampling operation; function This includes downsampling operations and Sigmoid gating operations; ; Step 2.1.4: The local feature fusion network includes: K Each feature fusion module CSFPN, and ; No. k The first feature fusion module CSFPN uses equation (2) to perform the first feature fusion. The first local jump excitation and the fusion of no local features H Coarse noise feature map and the The local jump excitation and the first The first local feature fusion H Coarse noise feature map Processing is performed to obtain the first... The local jump excitation and the first The first local feature fusion H Coarse noise feature map Thus, by the first K The output size of the feature fusion module CSFPN is The The local jump excitation and the first The first local feature fusion H coarse noise characteristics ; (2) In equation (2), Down represents the downsampling operation; Align represents the channel alignment operation; + Indicates element-wise addition; function F This includes convolution operations with 3×3 kernels, BatchNorm normalization, and GLU gating operations; Step 2.1.5: The output convolution module... After processing, the size is obtained as follows: High-resolution pseudo-feature map ; The output convolution module for After processing, the size is obtained as follows: Low-resolution pseudo-feature maps .

[0010] Furthermore, step 2.2 includes the following steps: Step 2.2.1: The first downsampling module... conduct After the second downsampling process, the size is obtained as High-resolution pseudo-feature map ; The first downsampling module conduct After the second downsampling process, the size is obtained as True feature map ; The first downsampling module conduct After the second downsampling process, the size is obtained as High-resolution true feature map ; Step 2.2.2: The second downsampling module... conduct After the second downsampling process, the size is obtained as Low-resolution pseudo-feature maps ; The second downsampling module conduct After the second downsampling process, the size is obtained as Low-resolution true feature map ; Step 2.2.3: The convolutional layer will , Converted into high-resolution real image discrimination scores respectively L 0 and low-resolution real image discrimination score L 1; will , Conversion to high-resolution pseudo-image discrimination score l 0 and low-resolution pseudo-image discrimination score l 1.

[0011] Furthermore, step 3 includes the following steps: Step 3.1: Extract using InceptionV3 True feature vector ,in, express The true feature vector, N This represents the total number of true feature vectors; Extracted using InceptionV3 pseudo-feature vectors ,in, express pseudo-feature vectors AThis represents the total number of pseudo-eigenvectors; Step 3.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require and Merge to obtain the merged feature vector. ,in, Indicates the first d The merged feature vectors D This represents the total number of eigenvectors after merging, and D=A+N ; Step 3.3: Define the parameters of the minimum sample and the size of the minimum cluster The current number of clusters is C and initialize C =0; Step 3.4: Calculation and The Euclidean distance between each merged feature vector is calculated, and the first one is selected. s The smallest Euclidean distance is denoted as . core distance Thus, the core distance set is obtained. ; Step 3.5: From core distance , No. The merged feature vectors core distance as well as and The maximum value among the Euclidean distances between them is selected as and mutual reachability distance Thus, the set of mutually reachable distances is obtained. ; Step 3.6: [The sentence is incomplete and requires more context to be translated accurately.] All merged feature vectors are considered as a vertex, and the reachability distance between any two merged feature vectors is... Consider the weights of the edges between two vertices, and generate a standard minimum spanning tree with the goal of minimizing the sum of the edge weights; Add a self-loop edge with the core distance of the corresponding vertex to each vertex in the standard minimum spanning tree to form the extended minimum spanning tree EMST; Step 3.7: Sort the edges in EMST in ascending order according to their weights to obtain the edge list. ,in, Indicates the first g Edge, let express The corresponding weight value, G Represents the total number of edges, and initializes the edge index. g=1, set EMST as the current cluster; Step 3.8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require If the current cluster has not split after being removed from it, proceed to step 3.10; otherwise, proceed to step 3.9. Step 3.9: Determine if the number of vertices in the new cluster split from the current cluster is greater than or equal to the number of vertices in the new cluster. p If so, then C +1 is assigned to C And mark the newly split cluster as the first C If a cluster is split, all vertices in the new cluster will be marked as noise points; otherwise, all vertices in the split cluster will be marked as noise points. Step 3.10: Remove The next cluster will be used as the current cluster. g +1 is assigned to g Then, return to step 3.8 and execute sequentially until... g > G Finally obtained C Each cluster class; Step 3.11: Calculation The quality score is used to obtain the set of quality scores. ; (5) In equation (5), express The mass fraction; express The weight of the edge removed when it is marked as noise; express The weight of the edge removed when the cluster splits; Step 3.12: [The rest of the text appears to be a list of steps or instructions, and doesn't form a co After sorting in descending order and removing true feature vectors, select the highest-scoring ones. B Generate feature vectors, and from Take the highest score from B B pseudo images of detached bolts corresponding to each generated feature vector are used to construct a high-quality image set of detached bolts. .

[0012] Furthermore, step 6 includes the following steps: Step 6.1: The feature extraction network sequentially includes an initial convolutional layer Conv0, n A local feature extraction module and a fast spatial pyramid pooling module SPPF, and for I The process is performed to obtain the characteristics of the reinforced bolt. I SPPF ; Step 6.2: The feature fusion network includes four feature fusion units, and... ISPPF , No. n -1 bolt local features I n-1 and the n -2 bolt local features I n-2 The process was performed to obtain the second bolt fusion feature. Third bolt fusion characteristics Fourth bolt fusion characteristics ; Step 6.3: The prediction network uses the Detect head pair , , Processing is performed to obtain I Category prediction label sequence and location prediction label sequences .

[0013] Furthermore, step 6.1 includes the following steps: Step 6.1.1: The initial convolutional layer Conv0 is downsampled... I After processing, the size is obtained as follows: initial convolution features of bolts I 0; Step 6.1.2: Each local feature extraction module includes: a downsampling convolution and a C2F_LSKA module; Bolt initial convolution features I 0 passes through in sequence n After processing by the local feature extraction module, a set of bolt local features is generated accordingly. ,in, Indicates the size is The u Local features of a bolt; Step 6.1.3: The fast spatial pyramid pooling module SPPF... Multi-scale contextual information fusion is performed to obtain a size of Features of reinforced bolts I SPPF ; Furthermore, step 6.2 includes the following steps: Step 6.2.1: The first feature fusion unit includes: a first upsampling module, a first stitching module, and a first C2F_DW module; The first upsampling module uses nearest neighbor interpolation to perform... I SPPF After processing, the size is obtained as follows: First upsampling feature ; The first splicing module is... and I n-1 After processing, the size is obtained as follows: First splicing feature ; The first C2F_DW module is... After processing, the size is obtained as follows: First fusion feature ; Step 6.2.2: The second feature fusion unit includes: a second upsampling module, a second stitching module, and a first C2F module; The second upsampling module uses nearest neighbor interpolation to perform... After processing, the size is obtained as follows: Second upsampling feature ; The second splicing module and I n-2 After processing, the size is obtained as follows: Second splicing feature ; The first C2F module After processing, the size is obtained as follows: Second fusion feature ; Step 6.2.3: The third feature fusion unit includes: a first downsampling module, a third stitching module, and a second C2F module; The first downsampling module uses a convolutional layer with a 3×3 kernel to... After processing, the size is obtained as follows: First downsampling feature ; The third splicing module and After processing, the size is obtained as follows: The third splicing feature ; The second C2F module After processing, the size is obtained as follows: The third fusion feature ; Step 6.2.4: The fourth feature fusion unit includes: a second downsampling module, a fourth stitching module, and a second C2F_DW module; The second downsampling module uses a convolutional layer with a 3×3 kernel to... After processing, the size is obtained as follows: Second downsampling feature ; The fourth splicing module and I SPPF After processing, the size is obtained as follows: The fourth splicing feature ; The second C2F_DW module After processing, the size is obtained as follows: The fourth fusion feature .

[0014] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing the data-enhanced steel bridge bolt loosening detection method, and the processor is configured to execute the program stored in the memory.

[0015] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the data-enhanced steel bridge bolt loosening detection method.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Compared to existing data augmentation methods, which struggle to effectively expand datasets with limited samples of detached bolts, resulting in insufficient bolt diversity and realism, this paper proposes an improved FastGAN model. This model utilizes the cross-scale feature fusion module CSFPN to enrich the low-resolution bolt feature map information, mitigating information decay during gradient propagation and effectively improving the stability of training on small samples of detached bolts as well as the quality of the generated images.

[0017] 2. Compared to existing methods for evaluating the quality of generated images: manual selection methods are inefficient when dealing with large sample sizes, and methods based on KNN, GMM, and BGMM require continuous testing to determine the optimal K value, and they lack diversity by only selecting high-quality bolt images. Therefore, this paper proposes a generated image quality evaluation method based on the HDBSCAN clustering algorithm. This method not only avoids the need for K value selection but also takes into account the requirements for high quality and diversity of bolt images, effectively achieving rapid selection of high-quality pseudo-images of detached bolts.

[0018] 3. Compared to existing target detection methods: Existing detection models often employ Transformer or self-attention mechanisms to improve detection accuracy in complex scenes, but these methods are computationally intensive and difficult to meet the lightweight requirements of loose bolt detection. Therefore, this invention proposes the YOLOV8-LSKA network, which not only improves the detection accuracy of loose bolts and reduces the false negative rate, but also effectively achieves a lightweight model design. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the specific process of the method of the present invention; Figure 2 This is a structural diagram of the cross-scale feature fusion generator of the present invention; Figure 3 This is a structural diagram of the cross-scale feature fusion module CSFPN of the present invention; Figure 4 This is a two-dimensional visualization diagram of the cluster class of the present invention; Figure 5 This is a structural diagram of the YOLOV8-LSKA network of the present invention; Figure 6 This is a structural diagram of the C2F_LSKA module of the present invention; Figure 7 This is a structural diagram of the C2F_DW module of the present invention. Detailed Implementation

[0020] In this example, a data augmentation-based method for detecting loose steel bridge bolts mainly includes: 1. Generating pseudo-images of loose bolts using an improved FASTGAN network, effectively augmenting the dataset when loose bolt samples are scarce; 2. Proposing an image quality assessment method based on the HDBSCAN clustering algorithm to select high-quality and diverse generated images; 3. Training a YOLOv8-LSKA network to achieve loose steel bridge bolt detection. Specifically, as... Figure 1 As shown, the method is performed according to the following steps: Step 1: Obtain the image set of loose bolts ,in, Indicates the size is The n Image of a loose bolt. N This represents the total number of images of detached bolts. Represents the length of the image of the detached bolt. Represents the width of the image of the detached bolt; Obtain normal bolt image set ,in, Indicates the first m A normal bolt image, M This represents the total number of normal bolt images. In this example, we take... N = 50, 512, M = 650.

[0021] Step 2: Construct an improved FASTGAN network, including a cross-scale feature fusion generator and a discriminator, and handle random noise. Z and XThe images are processed to obtain discrimination scores, which are then used to construct a loss function to train an improved FASTGAN network, resulting in a trained image generation model for loose bolts. This model is then used to generate a pseudo-image set of loose bolts. ,in, Indicates the first A fake image of a loose bolt. A This represents the total number of pseudo-images of detached bolts, taken as... A = 3000.

[0022] Step 2.1: The cross-scale feature fusion generator includes: an initialization layer, an upsampling module, a local skip activation network, a local feature fusion network, and an output convolutional module, with the following structure: Figure 2 As shown, and for random noise Z Processing is performed to obtain a high-resolution pseudo-feature map. and low-resolution pseudo-feature maps .

[0023] Step 2.1.1: Initialize the layer for random noise Z The process is performed to output an initial noise feature map of size 4×4. P 0.

[0024] Step 2.1.2: The upsampling module uses bilinear interpolation to sequentially... P 0 proceed H The next upsampling process generates the corresponding... H A set of coarse noise feature maps ,in, The first part represents the no-local-jump excitation and no-local-feature fusion. h Take a coarse noise feature map, and extract... The upsampling factor is 2. The dimensions are 64×64.

[0025] Step 2.1.3: The local skip reinforcement network includes: J A jump excitation module SLE, taking J =3; No. j The first jump excitation module SLE utilizes equation (1) to perform local jump excitation and local feature fusion on the first jump excitation module. h+j Coarse noise feature map and the j- The first local jump excitation and the first no-local feature fusion H Coarse noise feature map Processing is performed to obtain the first... j The first local jump excitation and the fusion of no local features H Coarse noise feature map Thus, the size is The The first local jump excitation and the fusion of no local features H Coarse noise feature map The dimensions are 512×512; (1) In equation (1), This indicates an upsampling operation with an upsampling factor of 2; the function... This includes downsampling operations and downsampling to 1×1, as well as Sigmoid gating operations; .

[0026] Step 2.1.4: As Figure 3 As shown, the local feature fusion network includes: K Each feature fusion module CSFPN, and ,Pick K =2; No. k The first feature fusion module CSFPN uses equation (2) to perform the first feature fusion. The first local jump excitation and the fusion of no local features H Coarse noise feature map and the The local jump excitation and the first The first local feature fusion H Coarse noise feature map Processing is performed to obtain the first... The local jump excitation and the first The first local feature fusion H Coarse noise feature map Thus, by the first K The output size of the feature fusion module CSFPN is The The local jump excitation and the first The first local feature fusion H coarse noise characteristics The dimensions are 128×128; (2) In equation (2), Down represents the downsampling operation; Align represents the channel alignment operation; + Indicates element-wise addition; function F This includes convolution operations with a 3×3 kernel, BatchNorm normalization, and GLU gating operations.

[0027] Step 2.1.5: Output convolution module After processing, the size is obtained as follows: High-resolution pseudo-feature map ; Output convolution module After processing, the size is obtained as follows: Low-resolution pseudo-feature maps .

[0028] Step 2.2: The discriminator includes a first downsampling module, a second downsampling module, and a convolutional layer, and performs... as well as , The data is processed to obtain the discrimination score of the high-resolution real image. L Discrimination score of 0 and low-resolution real images L 1 and the discrimination score of high-resolution pseudo-images l Discrimination scores for 0 and low-resolution pseudo-images l 1; Step 2.2.1: The first downsampling module... conduct Sub-sampling processing, Taking 6, we get the dimension as That is, an 8×8 high-resolution pseudo-feature map ; The first downsampling module conduct Sub-sampling processing, Taking 2, we get the size as That is, a 128×128 true feature map ; The first downsampling module conduct Sub-sampling processing, Taking 4, we get the dimension as That is, a high-resolution 8×8 true feature map .

[0029] Step 2.2.2: The second downsampling module... conduct Sub-sampling processing, Taking 4, we get the dimension as That is, an 8×8 low-resolution pseudo-feature map ; The second downsampling module conduct Sub-sampling processing, Taking 4, we get the dimension as That is, an 8×8 low-resolution true feature map .

[0030] Step 2.2.3: The convolutional layer will , Converted into high-resolution real image discrimination scores respectivelyL 0 and low-resolution real image discrimination score L 1; will P a , P b Conversion to high-resolution pseudo-image discrimination score l 0 and low-resolution pseudo-image discrimination score l 1.

[0031] Step 2.3: Calculate the discriminator loss using equations (3) and (4). L D and the loss of the cross-scale feature fusion generator L G ; (3) (4) In equations (3) and (4), This indicates a vector concatenation operation.

[0032] Step 2.4: Train the improved FASTGAN network using the Adam optimizer and calculate... L G and L D To update network parameters until L G and L D The process continues until convergence is achieved, resulting in a well-trained model for generating images of detached bolts, and a set of pseudo-images of detached bolts is generated. .

[0033] In this example, 3000 images of detached bolts were generated using FASTGAN, and compared with a set of pseudo-images of detached bolts generated by an improved FASTGAN. The FID and KID values ​​are shown in Table 1. The improved FASTGAN generated images with FID and KID values ​​improved by 15% and 33%, respectively.

[0034] Table 1. FID and KID values ​​of images generated by FASTGAN and improved FASTGAN models.

[0035] Step 3: Use the HDBSCAN clustering algorithm to... After filtering, a high-quality image set of detached bolts was obtained. ,in, Indicates the first b High-quality image of the loose bolt. B This represents the total number of high-quality images of the detached bolts.B Take 1200.

[0036] Step 3.1: Extract using InceptionV3 True feature vector ,in, express The true feature vector, N This represents the total number of true feature vectors; Extracted using InceptionV3 pseudo-feature vectors ,in, express pseudo-feature vectors A This represents the total number of pseudo-feature vectors.

[0037] Step 3.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require and Merge to obtain the merged feature vector. ,in, Indicates the first d The merged feature vectors D This represents the total number of eigenvectors after merging, and D=A+N= 3050; Step 3.3: Define the parameters of the minimum sample and the size of the minimum cluster ,Pick 15, the current number of clusters is C and initialize C =0.

[0038] Step 3.4: Calculation and The Euclidean distance between each merged feature vector is calculated, and the first one is selected. s The smallest Euclidean distance is denoted as . core distance Thus, the core distance set is obtained. ; Step 3.5: From core distance , No. The merged feature vectors core distance as well as and The maximum value among the Euclidean distances between them is selected as and mutual reachability distance Thus, the set of mutually reachable distances is obtained. .

[0039] Step 3.6: [The sentence is incomplete and requires more context to be translated accurately.] In the eigenvectors, all merged eigenvectors are considered as a vertex, and the reachability distance between any two merged eigenvectors is... Consider the weights of the edges between two vertices, and generate a standard minimum spanning tree with the goal of minimizing the sum of the edge weights; Add a self-loop edge with the core distance of the corresponding vertex to each vertex in the standard minimum spanning tree to form the extended minimum spanning tree EMST.

[0040] Step 3.7: Sort the edges in EMST in ascending order according to their weights to obtain the edge list. ,in, Indicates the first g Edge, let express The corresponding weight value, G Represents the total number of edges, and initializes the edge index. g =1, set EMST as the current cluster; Step 3.8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require If the current cluster has not split after being removed from it, proceed to step 3.10; otherwise, proceed to step 3.9.

[0041] Step 3.9: Determine if the number of vertices in the new cluster split from the current cluster is greater than or equal to the number of vertices in the new cluster. p If so, then C +1 is assigned to C And mark the newly split cluster as the first C If a cluster is split, all vertices in the new cluster will be marked as noise points; otherwise, all vertices in the split cluster will be marked as noise points. Step 3.10: Remove The next cluster will be used as the current cluster. g +1 is assigned to g Then, return to step 3.8 and execute sequentially until... g > G Finally obtained C There are 11 cluster classes in this example, and the two-dimensional visualization of the cluster classes is as follows: Figure 4 As shown, gray dots represent noise points, and red dots represent true feature vectors.

[0042] Step 3.11: Calculation The quality score is used to obtain the set of quality scores. ; (5) In equation (5), express The mass fraction; express The weight of the edge removed when it is marked as noise; express The weight of the edge removed when the cluster splits.

[0043] Step 3.12: [The rest of the text appears to be a list of steps or instructions, and doesn't form a co After sorting in descending order and removing true feature vectors, select the highest-scoring ones. B Generate feature vectors, and from Take the highest score from B The corresponding generated feature vectors B Pseudo-images of loose bolts are used to construct a high-quality image set of loose bolts. .

[0044] In this example, manual selection, KNN, GMM, and BGMM methods were used to extract images from the set of pseudo-images of loose bolts. 1200 photos were extracted and compared with a high-quality image set of loose bolts extracted by the HDBSCAN clustering algorithm. The FID and KID values ​​of the images were obtained, and the results are shown in Table 2. Compared with KNN, GMM, and BGMM methods, the HDBSCAN clustering algorithm improved the FID and KID values ​​of the images by 44.5%, 40.7%, and 40.5%, respectively, while the difference from manual selection was small, confirming the effectiveness of the proposed method.

[0045] Table 2. FID and KID values ​​of photos extracted by different generated image quality assessment methods.

[0046] Step 4: For the normal bolt dataset Augmentation is performed to construct a normal bolt training image set. ,in, Indicates the first e A normal bolt training image, E E represents the total number of normal bolt training images, with E set to 1200.

[0047] Will , Merged into a bolt training image set and order T Any bolt training image is I ,and I The size is M I × N I ,in, M I Indicates the length of the bolt training image. N I This represents the width of the bolt training image. M ITake 640, N I Take 640.

[0048] Step 5: [Regarding...] I middle s The location and category of each detected target are labeled to obtain... s Location label sequence of targets g and s A sequence of category labels for each target c In this example, the detected targets include two categories: "0: bolt" and "1: loss"; Step 6: As Figure 5 As shown, a YOLOv8-LSKA network is constructed, comprising a feature extraction network, a feature fusion network, and a prediction network, and then... I Processing is performed to obtain I Category prediction label sequence and location prediction label sequences ; Step 6.1: The feature extraction network consists of an initial convolutional layer Conv0, ... n A local feature extraction module and a fast spatial pyramid pooling module SPPF, and for I The process is performed to obtain the characteristics of the reinforced bolt. I SPPF ; Step 6.1.1: The initial convolutional layer Conv0 is downsampled... I After processing, the size is obtained as follows: initial convolution features of bolts I 0.

[0049] Step 6.1.2: Each local feature extraction module includes: a downsampling convolution and a C2F_LSKA module. Number of local feature extraction modules n Choosing option 4, the bottleneck module of C2F_LSKA integrates the large kernel separable convolutional attention module LSKA, decomposing a large 7×7 convolutional kernel into horizontal depthwise convolutions and vertical depthwise convolutions, and then using 1×1 convolutions for channel mixing, significantly reducing computation and the number of parameters while maintaining a large receptive field. Its network structure is as follows: Figure 6 As shown; Bolt initial convolution features I 0 passes through in sequence n After processing by the local feature extraction module, a set of bolt local features is generated accordingly. ,in, Indicates the size is The u Local features of a bolt.

[0050] Step 6.1.3: Fast Space Pyramid Pooling Module SPPF Multi-scale contextual information fusion is performed to obtain a size of Features of reinforced bolts I SPPF .

[0051] Step 6.2: The feature fusion network includes four feature fusion units, and... I SPPF , No. n -1 bolt local features I n-1 and the n -2 bolt local features I n-2 The process was performed to obtain the second bolt fusion feature. Third bolt fusion characteristics Fourth bolt fusion characteristics .

[0052] Step 6.2.1: The first feature fusion unit includes: a first upsampling module, a first concatenation module, and a first C2F_DW module. The bottleneck module of C2F_DW integrates an efficient channel attention module (ECA) and uses separable convolutional layers to effectively reduce model computation parameters. Its network structure is as follows: Figure 7 As shown; The first upsampling module uses nearest neighbor interpolation to perform... I SPPF After processing, the size is obtained as follows: First upsampling feature ; The first splicing module is... and I n-1 After processing, the size is obtained as follows: First splicing feature ; The first C2F_DW module is... After processing, the size is obtained as follows: First fusion feature .

[0053] Step 6.2.2: The second feature fusion unit includes: a second upsampling module, a second stitching module, and a first C2F module; The second upsampling module uses nearest neighbor interpolation to perform... After processing, the size is obtained as follows: Second upsampling feature ; The second splicing module and In-2 After processing, the size is obtained as follows: Second splicing feature ; The first C2F module After processing, the size is obtained as follows: Second fusion feature .

[0054] Step 6.2.3: The third feature fusion unit includes: a first downsampling module, a third stitching module, and a second C2F module.

[0055] The first downsampling module uses a convolutional layer with a 3×3 kernel to... After processing, the size is obtained as follows: First downsampling feature ; The third splicing module and After processing, the size is obtained as follows: The third splicing feature ; The second C2F module After processing, the size is obtained as follows: The third fusion feature .

[0056] Step 6.2.4: The fourth feature fusion unit includes a second downsampling module, a fourth stitching module, and a second C2F_DW module; The second downsampling module uses a 3×3 convolutional layer. After processing, the size is obtained as follows: Second downsampling feature ; The fourth splicing module and I SPPF After processing, the size is obtained as follows: The fourth splicing feature ; The second C2F_DW module After processing, the size is obtained as follows: The fourth fusion feature .

[0057] Step 6.3: The prediction network uses the Detect head pair. , , The fusion features of the three scales are processed to obtain... I Category prediction label sequence and location prediction label sequences .

[0058] Step 7: Through and as well as and Classification loss and bounding box loss are constructed separately, and the YOLOV8-LSKA network is iteratively trained using the SGD optimizer until the classification loss and bounding box loss converge, thus obtaining a trained bolt detection model for detecting loose bolts in bolt images.

[0059] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0060] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0061] In this example, the FASTGAN model is used to construct a dataset of loose bolts, and YOLOV8 for detecting loose bolts is used as the baseline method. The method of this invention is compared with the method of this invention in terms of F1 score, mean accuracy (MAP@0.5), regression rate (R), model parameters, model size, and computational complexity (GFLOPs).

[0062] Table 3 Comparison of the improved method and the baseline method

[0063] The comparison results are shown in Table 3. Compared with the baseline method, the present invention improves the F1 score, average accuracy and regression rate by 2%, 2.3% and 5.8% respectively, and reduces the model parameters, model size and computational complexity by 29.5%, 28.2% and 20.98% respectively.

Claims

1. A data augmentation based bolt shedding detection method for steel bridges, characterized in that, Comprising the steps of: Step 1: Obtain the loosened bolt image set wherein, represents the size of the th loosened bolt image, n N represents the total number of loosened bolt images, represents the length of the loosened bolt image, represents the width of the loosened bolt image;​ acquiring a set of normal bolt images wherein, denotes the m normal bolt images, M denotes the total number of normal bolt images; Step 2: constructing an improved FASTGAN network, including: a cross-scale feature fusion generator and a discriminator, and processing the random noise Z and X to obtain an image discrimination score, thereby constructing a loss function for training the improved FASTGAN network to obtain a trained shed bolt image generation model for generating a set of shed bolt pseudo images wherein, represents the first shed bolt pseudo image, A represents the total number of shed bolt pseudo images; Step 3: using HDBSCAN clustering algorithm to cluster the high-quality image set of the falling bolt Step 4: screening to obtain a high-quality image set of the falling bolt wherein, represents the first b high-quality image of the falling bolt, B represents the total number of high-quality images of the falling bolt; Step 4: Augmenting the normal bolt dataset Constructing the normal bolt training image set wherein, represents the e th normal bolt training image, E represents the total number of normal bolt training images; Will , Merged into a bolt training image set and order T Any bolt training image is I ,and I The size is M I × N I ,in, M I Indicates the length of the bolt training image. N I Indicates the width of the bolt training image; Step 5: label the position and category of each detection target to obtain a position label sequence of the target I and a category label sequence of the target s s g s c ;​​​​ Step 6: Construct the YOLOv8-LSKA network, which includes, in sequence: a feature extraction network, a feature fusion network, and a prediction network, and then... I Processing is performed to obtain I Category prediction label sequence and location prediction label sequences ; Step 7: based on and and and The classification loss and the bounding box loss are constructed respectively, and the YOLOV8-LSKA network is iteratively trained by an SGD optimizer until the classification loss and the bounding box loss converge, thereby obtaining a trained bolt detection model for detecting the missing bolt in the bolt image.

2. The method of claim 1, wherein, Step 2 comprises the steps of: Step 2.1: The cross-scale feature fusion generator comprises an initialization layer, an up-sampling module, a local skip excitation network, a local feature fusion network, and an output convolution module, and processes random noise Z to obtain a high-resolution pseudo feature map and a low-resolution pseudo feature map ; Step 2.2: The discriminator includes a first down-sampling module, a second down-sampling module and a convolutional layer, and processes and , to obtain a discrimination score L 0 of the high-resolution real image and a discrimination score L 1 of the low-resolution real image and a discrimination score l 0 of the high-resolution fake image and a discrimination score l 1 of the low-resolution fake image. Step 2.3: Compute the loss of the discriminator with formula (3) and formula (4) L D and the loss of the cross-scale feature fusion generator L G ; (3) (4) In formula (3) and formula (4), represents a vector concatenation operation; Step 2.4: Train the improved FASTGAN network using the Adam optimizer and calculate L G and L D to update the network parameters until L G and L D converge, thereby obtaining a trained falling bolt image generation model and generating a set of falling bolt pseudo images .

3. The method of claim 2, wherein the method is based on data augmentation for bolt loosening detection of steel bridges. Step 2.1 comprises the steps of: Step 2.1.1: The initialization layer processes the random noise Z and outputs the initial noise feature map P 0; Step 2.1.2: The up-sampling module sequentially performs the sub-sampling processing on the coarse noise feature maps by using the bilinear interpolation method, and correspondingly generates a set of coarse noise feature maps P 0 H , where H , and , wherein represents the i-th coarse noise feature map without local jump excitation and local feature fusion. h ​ Step 2.1.3: The local skip enhancement network comprises: J a skip excitation module SLE; No. j The first jump excitation module SLE utilizes equation (1) to perform local jump excitation and local feature fusion on the first jump excitation module. h+j Coarse noise feature map and the j- The first local jump excitation and the first no-local feature fusion H Coarse noise feature map Processing is performed to obtain the first... j The first local jump excitation and the fusion of no local features H Coarse noise feature map Thus, the size is The The first local jump excitation and the fusion of no local features H Coarse noise feature map ; (1) In formula (1), denotes an up-sampling operation; the function includes a down-sampling operation and a Sigmoid gating operation; ; Step 2.1.4: the local feature fusion network comprises: K a feature fusion module CSFPN, and ; No. k The first feature fusion module CSFPN uses equation (2) to perform the first feature fusion. The first local jump excitation and the fusion of no local features H Coarse noise feature map and the The local jump excitation and the first The first local feature fusion H Coarse noise feature map Processing is performed to obtain the first... The local jump excitation and the first The first local feature fusion H Coarse noise feature map Thus, by the first K The output size of the feature fusion module CSFPN is The The local jump excitation and the first The first local feature fusion H coarse noise characteristics ; (2) In formula (2), Down represents a down-sampling operation; Align represents a channel alignment operation. + represents an element-wise addition; the function F includes a convolution operation with a convolution kernel of 3x3, a BatchNorm normalization, and a GLU gating operation. Step 2.1.5: The output convolution module processes the output of step 2.1.4 to obtain a high-resolution pseudo feature map with a size of ; and Step 2.1.6: The output of step 2.1.5 is input into the output upsampling module to obtain the output of the second stage. ​ The output convolution module for After processing, the size is obtained as follows: Low-resolution pseudo-feature maps .

4. The method of claim 3, wherein the method is based on data augmentation. Step 2.2 comprises the steps of: Step 2.2.1: The first down-sampling module performs a sub-sampling process to obtain a high-resolution pseudo feature map with a size of . . . ​ The first downsampling module conduct After the second downsampling process, the size is obtained as follows: True feature map ; The first downsampling module conduct After the second downsampling process, the size is obtained as follows: High-resolution true feature map ; Step 2.2.2: The second down-sampling module performs a second down-sampling process on the low-resolution pseudo feature map to obtain a low-resolution pseudo feature map with a size of . . . . The second downsampling module conduct After the second downsampling process, the size is obtained as follows: Low-resolution true feature map ; Step 2.2.3: The convolutional layer transforms , the high-resolution real image discrimination score L 0 and the low-resolution real image discrimination score L 1, respectively; and transforms , the high-resolution fake image discrimination score l 0 and the low-resolution fake image discrimination score l 1, respectively.

5. The method of claim 4, wherein the method is based on data augmentation. Step 3 comprises the steps of: Step 3.1: Extracting real feature vectors with InceptionV3 of the real feature vectors wherein denotes of the real feature vectors, N denotes the total number of real feature vectors; extracted using InceptionV3 pseudo-feature vectors wherein denotes pseudo-feature vectors, A denotes the total number of pseudo-feature vectors; Step 3.2: merging the feature vectors and to obtain merged feature vectors wherein denotes the d merged feature vector, D denotes the total number of merged feature vectors, and D=A+N ; Step 3.3: Define parameters of the minimum sample and the size of the minimum cluster , the number of current clusters is C and initialize C = 0; Step 3.4: Calculate The Euclidean distance between each merged eigenvector in and select the s smallest Euclidean distance, denoted as The core distance , so as to obtain the core distance set ; Step 3.5: From core distance , No. The merged feature vectors core distance as well as and The maximum value among the Euclidean distances between them is selected as and mutual reachability distance Thus, the set of mutually reachable distances is obtained. ; Step 3.6: The merged eigenvectors are all regarded as a vertex, and the mutual reachable distance between any two merged eigenvectors is regarded as the weight of the edge between the two vertices. The standard minimum spanning tree is generated with the minimum sum of edge weights as the target. Step 3.6: The merged eigenvectors are all regarded as a vertex, and the mutual reachable distance between any two merged eigenvectors is regarded as the weight of the edge between the two vertices. The standard minimum spanning tree is generated with the minimum sum of edge weights as the target. Step 3.6: The merged eigenvectors are all regarded as a vertex, and the mutual reachable distance between any two merged eigenvectors Adding a self-loop edge with the core distance of the corresponding vertex as the weight for each vertex in the standard minimum spanning tree to form an extended minimum spanning tree EMST; Step 3.7: Sort the edges in EMST in ascending order according to their weights to obtain the edge list. ,in, Indicates the first g Edge, let express The corresponding weight value, G Represents the total number of edges, and initializes the edge index. g =1, set EMST as the current cluster; Step 3.8: If the current cluster is not split, then perform Step 3.10; otherwise, perform Step 3.

9. After being removed from the current cluster, if the current cluster is not split, then perform Step 3.10; otherwise, perform Step 3.

9. Step 3.9: judge whether the number of vertices in the new cluster split out by the current cluster is greater than or equal to p , if yes, assign C +1 to C , and mark the split new cluster as the C th cluster; otherwise, mark all vertices in the split new cluster as noise points; Step 3.10: Remove the cluster after the cluster as the current cluster, assign +1 to the cluster after the cluster, and return to Step 3.8 for sequential execution until Step 3.11: The final cluster class is obtained g Step 3.12: The final cluster class is obtained g Step 3.13: The final cluster class is obtained g Step 3.14: The final cluster class is obtained G Step 3.15: The final cluster class is obtained C Step 3.16: The final cluster class is obtained Step 3.11: Calculate the mass fraction of the set of mass fractions ; (5) In formula (5), denotes the mass fraction of denotes the weight of edges removed when the cluster is split; denotes the weight of edges removed when the cluster is split; Step 3.12: The generated feature vectors are ranked in descending order, and the top 10 feature vectors are selected Step 3.12: The generated feature vectors are ranked in descending order, and the top 10 feature vectors are selected B Step 3.12: The generated feature vectors are ranked in descending order, and the top 10 feature vectors are selected Step 3.12: The generated feature vectors are ranked in descending order, and the top 10 feature vectors are selected B Step 3.12: The generated feature vectors are ranked in descending order, and the top 10 feature vectors are selected .

6. The method of claim 5, wherein the method is based on data augmentation. Step 6 comprises the steps of: Step 6.1: the feature extraction network comprises an initial convolutional layer Conv0, n a local feature extraction module and a fast spatial pyramid pooling module SPPF in sequence, and processes to obtain enhanced bolt features I I SPPF ;​ Step 6.2: The feature fusion network includes four feature fusion units, and... I SPPF , No. n -1 bolt local features I n-1 and the n -2 bolt local features I n-2 The process was performed to obtain the second bolt fusion feature. Third bolt fusion characteristics Fourth bolt fusion characteristics ; Step 6.3: The prediction network processes the sequence of tokens with the Detect head to obtain a sequence of class prediction labels , , and a sequence of position prediction labels I . .​ 7. The data-augmentation-based bolt loosening detection method for steel bridges of claim 6, wherein, Step 6.1 comprises the steps of: Step 6.1.1: The initial convolution layer Conv0 processes the input image X by a down-sampling operation to obtain the bolt initial convolution feature X0 with the size of I I 0;​​ Step 6.1.2: Each local feature extraction module comprises: a down-sampling convolution and a C2F_LSKA module; Bolt initial convolution feature I 0 sequentially through n the processing of the local feature extraction module, the corresponding bolt local feature set is generated , wherein represents the i-th bolt local feature with a size of . u ​ Step 6.1.3: The fast spatial pyramid pooling module SPPF... Multi-scale contextual information fusion is performed to obtain a size of Features of reinforced bolts I SPPF。 8. The data-augmentation-based bolt loosening detection method for steel bridges of claim 7, wherein, Step 6.2 comprises the steps of: Step 6.2.1: The first feature fusion unit comprises: a first up-sampling module, a first splicing module and a first C2F_DW module; The first up-sampling module processes the first down-sampled feature map by a nearest neighbor interpolation method to obtain a first up-sampled feature map with a size of I SPPF ​​​ The first splicing module pair With I n-1 Processing, get the size of the first splicing feature ;​ The first C2F DW module pair processes the first feature map to obtain a first fusion feature with a size of ​ Step 6.2.2: The second feature fusion unit comprises: a second up-sampling module, a second splicing module and a first C2F module; The second upsampling module uses nearest neighbor interpolation to perform... After processing, the size is obtained as follows: Second upsampling feature ; The second splicing module pair with I n-2 is processed to obtain a second splicing feature with a size of ​ The first C2F module pair processes the first fused feature to obtain a second fused feature with a size of Step 6.2.3: The third feature fusion unit comprises: a first down-sampling module, a third splicing module and a second C2F module; The first downsampling module processes the first feature map through a convolution layer with a convolution kernel of 3x3 to obtain a first downsampling feature map with a size of . . ​ The third splicing module and After processing, the size is obtained as follows: The third splicing feature ; The second C2F module pair processes the third fusion feature to obtain a fourth fusion feature with a size of Step 6.2.4: The fourth feature fusion unit comprises: a second down-sampling module, a fourth splicing module and a second C2F_DW module; The second downsampling module uses a convolutional layer with a 3×3 kernel to... After processing, the size is obtained as follows: Second downsampling feature ; The fourth splicing module pair with I SPPF is processed to obtain a fourth splicing feature with a size of ; The second C2F DW module pair processes to obtain a fourth fusion feature with a size of .

9. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the data enhancement based steel bridge bolt fall-off detection method in any one of claims 1-8, and the processor is configured to execute the program stored in the memory.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to execute the steps of the data enhancement based steel bridge bolt fall-off detection method in any one of claims 1-8.

Images