Bridge structure damage identification method based on improved MobileNet-v2 model

Abstract

The invention discloses a bridge structure damage identification method based on an improved MobileNet-v2 model, belongs to the technical field of bridge structure damage identification methods, and provides the improved MobileNet-v2 model which is trained by adopting a large number of bridge structure apparent damage data sets. The method can be used for rapidly detecting six types of bridge structure apparent damage (cracks, peeling, exposed ribs, water seepage, corrosion and no damage), and is accurate and efficient. Different from human naked eye judgment which is easily influenced by subjectivity, the method has objectivity in detection, can be practically applied to equipment such as unmanned aerial vehicles and monitoring robots, and meets the engineering real-time requirement. In the later research, more other types of bridge structure apparent damage can be considered, so that the model can be better put into engineering practice, and high-precision quantitative analysis of hidden danger parts is realized.

Description

Technical Field

[0001] The present invention relates to a bridge structure damage identification method, in particular to a bridge structure damage identification method based on an improved MobileNet-v2 model, and belongs to the technical field of bridge structure damage identification methods. Background Art

[0002] To address the limitations of image processing methods in extracting high-level features, researchers have proposed algorithms based on deep learning. Zhang Tiezhi et al. proposed a convolutional neural network-based drone imaging bridge crack detection system. This system captures bridge crack images, extracts cracks, and calculates the maximum crack width. Based on extensive high-speed rail bridge test data and using the AlexNet CNN model, Lingkun Chen et al. introduced transfer learning to identify three types of surface damage in high-speed rail bridges: cracks, exposed rebar, and concrete spalling. To achieve automated and intelligent inspection of suspension bridge main cables, Feng Dongming et al. used the Faster RCNN network model and image fusion technology to accurately identify defects such as cracks, rust, and scratches on main cables. Xijun Ye et al. proposed a lightweight, improved YOLOv5 model, combined with image fusion technology, to identify three common surface defects in concrete. Fang Weihua et al. studied three common hazards in hydraulic concrete structures: cracks, spalling, and water seepage. Using an improved DeepLabv3+, they effectively segmented these defects.

[0003] However, although various deep learning neural networks can provide higher-level feature representations, they are often only targeted at specific damages. The image acquisition area is small, and the damage type and background are relatively simple. The resulting damage recognition model is not very applicable and has poor generalization ability. Therefore, a bridge structure damage recognition method based on the improved MobileNet-v2 model is designed to solve the above problems. Summary of the Invention

[0004] The main purpose of this invention is to provide a bridge structure damage identification method based on an improved MobileNet-v2 model.

[0005] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0006] The bridge structure damage identification method based on the improved MobileNet-v2 model includes the following steps:

[0007] Step 1: Collect image data of various structural damage types of bridges in service;

[0008] Step 2: Establish a data set of apparent damage of bridge structure;

[0009] Step 3: Use transfer learning to migrate the pre-trained weights and parameters to the MobileNet-v2, SqueezeNet, ShuffleNet, GoogLeNet, and ResNet18 models for comparison, and select the MobileNet-v2 model;

[0010] Step 4: Add a skip residual structure to the MobileNet-v2 model and use the Swish layer to replace the Relu6 layer to accurately classify the apparent damage of the bridge structure.

[0011] Preferably, the image data of various structural damage types of the bridge acquired in step 1 are image data of various structural damage types captured in various environments and at various angles and with complex background noise interference;

[0012] The image data sources include pictures of apparent damage to bridge structures obtained through regular bridge inspection reports, manual photography / drone inspections, and network collection.

[0013] Preferably, after step 2, before classifying the image dataset, the image must first be preprocessed. The image size must be adapted to the network input requirements, the image size must be adjusted to a uniform size, and the image must be normalized using ImageNet weights, so that the model can more easily handle images of different sizes and pixel ranges during training and prediction;

[0014] The image size is scaled to 224 pixels × 224 pixels. In addition, the image is augmented and expanded using flipping, mirroring, translation, rotation, salt and pepper noise, and contrast enhancement methods, resulting in a total of 12,000 images in the dataset.

[0015] Preferably, transfer learning uses the ImageNet image dataset as a pre-training dataset, and the trained weights and parameters are directly transferred to new classification and recognition tasks. Different network models have different characteristics.

[0016] Taking into account the task difficulty and model parameter count factors and to meet the requirements of implantation on mobile platforms, five network models, including MobileNet-v2, Squeezenet, ResNet18, Googlenet, and the classic network VGG with smaller parameters, were selected for comparison.

[0017] Preferably, Swish is used as the activation function, and its mathematical expression is as follows:

[0018] Swish(x)=xσ(βx)

[0019] Where σ is the Sigmoid function and β is a learnable parameter.

[0020] Preferably, the MobileNet v2 network includes an inverted residual and a linear bottleneck;

[0021] Inverted residual, used to increase the extraction of image features to improve accuracy;

[0022] Linear bottleneck, used to avoid information loss of nonlinear functions.

[0023] Preferably, a 1×1 convolutional jump connection similar to the residual structure is added to the linear bottleneck structure to form a jump structure;

[0024] MobileNet-v2 is used as the baseline network, using the Swish activation function and adding a skip residual structure.

[0025] Beneficial technical effects of the present invention:

[0026] The present invention provides a bridge structure damage identification method based on an improved MobileNet-v2 model. In order to realize the automatic and intelligent identification of bridge apparent damage, a method based on the improved MobileNet-v2 model is proposed. Unlike other studies, this study focuses on the apparent damage images of bridges in complex backgrounds and improves the generalization ability of the model through data enhancement. On this basis, transfer learning is used to effectively solve the problem of small data volume, which not only saves the training time of the model but also enhances the robustness of the model.

[0027] 2) The improved MobileNet-v2 network model achieved an accuracy of 95.48% in the bridge apparent damage classification task, which is 2.88% higher than the accuracy of the MobileNet-v2 model, demonstrating the effectiveness of the improvement.

[0028] 3) Through the use of visualization analysis, cluster analysis, and confidence result analysis, it can be seen that the improved MobileNet-v2 model can better extract the apparent damage features and focus on the location of damage in complex backgrounds. It can accurately identify six types of bridge apparent damage, further verifying the effectiveness of the model.

[0029] In summary, the improved MobileNet-v2 model proposed in this paper, trained on a large dataset of bridge structure apparent damage, can rapidly and accurately detect six types of bridge structure apparent damage (cracks, spalling, exposed rebar, water seepage, corrosion, and no damage). Unlike human visual judgment, which is susceptible to subjective influences, this detection is objective and can be applied to equipment such as drones and monitoring robots, meeting the real-time requirements of engineering projects. Future research will also consider additional types of bridge structure apparent damage, enabling the model to be more effectively applied in engineering practice and achieving high-precision quantitative analysis of potential hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a diagram of acquisition equipment according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention;

[0031] Figure 2 This is a partial sample image diagram of a bridge crack dataset according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention;

[0032] Figure 3 This is an example diagram of image enhancement according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention;

[0033] Figure 4 2 are example diagrams of two activation functions according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention;

[0034] Figure 5 A comparison diagram of the Bottleneck16 structure before and after improvement according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention

[0035] Among them, (a) comparison of Bottleneck16 structure before and after improvement, (b) Bottleneck structure after improvement;

[0036] Figure 6 This is a comparison diagram of model training curves according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention;

[0037] Among them, (a) accuracy comparison, (b) loss value comparison;

[0038] Figure 7 A confusion matrix comparison diagram of a preferred embodiment of a bridge structure damage identification method based on an improved MobileNet-v2 model according to the present invention;

[0039] Among them, (a) MobileNetv2, (b) improved MobileNetv2;

[0040] Figure 8 These are five bridge apparent damage image feature maps according to a preferred embodiment of the bridge structure damage identification method based on the improved MobileNet-v2 model of the present invention;

[0041] Where A: original image; B: shallow convolution feature image; C: deep convolution feature image; D: improved MobileNetv2 deep convolution heat map; E: MobileNetv2 deep convolution heat map;

[0042] Figure 9 This is a cluster analysis diagram before and after model improvement of a preferred embodiment of a bridge structure damage identification method based on an improved MobileNet-v2 model according to the present invention;

[0043] (a) Cluster analysis diagram before and after model improvement, (b) improved MobileNet-v2. DETAILED DESCRIPTION

[0044] In order to make the technical solution of the present invention more clear and specific to those skilled in the art, the present invention is further described in detail below with reference to embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0045] The bridge structure damage identification method based on the improved MobileNet-v2 model provided in this embodiment first collects a large number of image data of various structural damage types of bridges in service, captured in various environments and at various angles, with complex background noise interference. This data set establishes a dataset of bridge structure apparent damage under complex backgrounds, including various environmental interference elements and different brightness levels. Through transfer learning, the pre-trained weights and parameters are transferred to MobileNet-v2, SqueezeNet, ShuffleNet, GoogLeNet, and ResNet18 models for comparison, and the MobileNet-v2 model with better overall performance is selected. On this basis, a skip residual structure is added to the MobileNet-v2 model, and the Swish layer is used to replace the original Relu6 layer. This achieves highly accurate classification of bridge structure apparent damage, providing a foundation for the subsequent application of intelligent bridge structure damage detection equipment.

[0046] Materials and methods, data sources;

[0047] The experimental data set was collected from multiple engineering projects such as the Tangxi Avenue Phase III construction project in Foshan City and the main bridge monitoring of Sanshui Bridge. The images of apparent damage to bridge structures were obtained mainly through regular bridge inspection reports, manual photography / drone inspections, and network collection. Figure 1 shown.

[0048] The images include 6 categories: cracks, peeling, exposed reinforcement, water seepage, rust, and no disease, totaling 8645 images. The image sample examples are as follows Figure 2 shown.

[0049] Data processing, before classifying the image dataset, the image must be preprocessed first. The image size needs to adapt to the network input requirements, the image size is adjusted to a uniform size, and the image is normalized using ImageNet weights, so that the model can more easily handle images of different sizes and pixel ranges during training and prediction. The image size is scaled to 224 pixels × 224 pixels. In addition, the image is enhanced and expanded using methods such as flipping, mirroring, translation, rotation, salt and pepper noise, and contrast enhancement. A total of 12,000 images are obtained in the dataset, thereby improving the generalization ability of the model. Examples of data enhancement are as follows: Figure 3 As shown in Figure 1, the dataset is divided into training set, validation set and test set. The number of images in the dataset is shown in Table 1. The validation set is used to test the reliability of the model effect, and the test set is guaranteed not to participate in the model training process, but to evaluate the final performance of the model.

[0050] Tab.1 Number of dataset images (unit: pieces);

[0051]

[0052] Transfer Learning

[0053] Transfer learning uses the ImageNet image dataset as a pre-training dataset. The trained weights and parameters can be directly migrated for new classification and recognition tasks. Different network models have different characteristics.

[0054] Taking into account factors such as task difficulty and model parameter count, and to meet the requirements of mobile platform implantation, five network models with smaller parameter counts, namely MobileNet-v2, SqueezeNet, ShuffleNet, GoogLeNet, and ResNet18, were selected for comparison. The results in Table 2 show that the MobileNet-v2 model has higher baseline test accuracy, shorter training time, and a parameter count that better meets the lightweight requirements of the integrated system. Therefore, the MobileNet-v2 network was selected for subsequent analysis.

[0055] Table 2 Accuracy of the baseline test of the network model

[0056]

[0057] Model structure;

[0058] MobileNetv2 network;

[0059] The MobileNetv2 network is a lightweight network model proposed by the Google team. Compared to the MobileNet V1 network, it has higher accuracy and a smaller model. It continues the depthwise separable convolution used in MobileNetv1, which greatly reduces the computational speed while maintaining the accuracy of the neural network. In addition, MobileNet v2 introduces two new concepts: inverted residual and linear bottleneck. Inverted residual is mainly used to increase the extraction of image features to improve accuracy, while linear bottleneck is mainly used to avoid information loss in nonlinear functions. This makes the MobileNet v2 model not only have a good recognition rate, but also well meet the lightweight requirements of mobile devices.

[0060] This paper selects the MobileNet-v2 network as the basic framework and improves, modifies, and expands it to adapt to the application scenario of apparent damage. Its network structure is shown in Table 3.

[0061] Table 3 MobileNet-v2 structure

[0062]

[0063] Activation function, in order to further improve the accuracy of the model, this paper uses Swish as the activation function, the mathematical expression is as follows:

[0064] Swish(x)=xσ(βx)

[0065] Here, σ is the Sigmoid function, and β is a learnable parameter. The ReLU function directly truncates negative values (outputting a constant 0), which can cause neuron death. The Swish function, on the other hand, preserves negative information, preventing neuron death, while also filtering out noise through gradient decay (non-zero but close to 0). Furthermore, the ReLU function's completely monotonically increasing increments limits the model's expressiveness and makes it difficult to capture complex nonlinear relationships. However, the Swish function exhibits slight nonmonotonicity in the negative region, enhancing the model's nonlinear expressiveness. This makes it more suitable for deep networks and complex tasks and can optimize the model training process.

[0066] After multiple experiments, we found that changing the activation function in the shallow structure of the transfer learning model can improve model accuracy without significantly affecting computation time. Therefore, this paper replaces the ReLU6 layer before the bottleneck2 structure in MobileNet v2 with a Swish layer to improve the model's computational accuracy and generalization ability.

[0067] Jump residual structure;

[0068] In order to prevent the continuous convolution layers from losing important information during the feature extraction process, we add a 1×1 convolutional jump connection similar to the residual structure on the basis of the linear bottleneck structure to form a jump structure. In order not to destroy the pre-trained parameters of transfer learning, we only add jump connections to Bottleneck16. This can improve the accuracy and generalization ability of the model during operation without significantly increasing the model size. The jump residual structure before and after the improvement is shown in the figure. Figure 5 As shown in (a) and (b).

[0069] Test results and analysis, test environment;

[0070] In this experiment, the batch size is set to 32, the number of training iterations is set to 50, and the Adam optimizer is used. The initial learning rate is set to 0.0001. Before the image is input into the network training, the image is preprocessed and L2 regularized.

[0071] Table 4 Experimental environment parameters;

[0072]

[0073] Experimental results,are used to verify the effectiveness of the methods proposed in this,network;

[0074] The researchers designed ablation experiments to independently test various methods on the dataset presented in this paper. The proposed network uses MobileNet-v2 as the baseline network, employs the Swish activation function, and adds a skip residual structure. This helps the model better understand the input bridge damage images, improves its ability to capture diverse features and relationships, enhances its performance in complex tasks, and improves its generalization capabilities. The experimental results are listed in Table 4.

[0075] Table 4 Experimental results;

[0076]

[0077] As shown in Table 4, the original MobileNet-v2 model achieved a training accuracy of only 92.63%. After adding the indented bottleneck and skip residual connection structures, the MobileNet-v2 model improved its accuracy by 0.55% and 1.36%, respectively, and its F1 score by 0.55% and 1.37%, respectively. Combining the three improved methods resulted in a MobileNet-v2 model with an accuracy improvement of 2.89% and an F1 score improvement of 2.88%. This indicates that the combined improvement method significantly improves model accuracy without significantly increasing model size. Experimental results demonstrate that the improved method used in this network effectively improves the accuracy of bridge surface defect recognition while slightly increasing model size, achieving a balance between accuracy and model size.

[0078] Improved MobileNet-v2 and MobileNet-v2 Loss and Accuracy changes on the validation set are as follows Figure 6 (a) and Figure 7 As shown in (b), during the training process, the improved MobileNet-v2 model converges faster and has a larger Accuracy value, performing better under the same parameters and training conditions.

[0079] Evaluation metrics and confusion matrix analysis;

[0080] Accuracy is the ratio of correctly identified samples to the total number of samples. Recall represents the ratio of correctly predicted samples to the total number of samples in the real world. The F1-Score is a comprehensive metric that combines Precision and Recall. It is the harmonic mean of Precision and Recall and ranges from 0 to 1, with 1 representing the model's optimal output and 0 representing the model's worst output.

[0081]

[0082] Where:

[0083] TP—the number of samples that are actually positive but predicted to be positive;

[0084] FP—the number of samples that are actually negative but predicted to be positive;

[0085] FN—the number of samples that are actually positive but predicted to be negative;

[0086] TN—Number of samples that were actually negative but were predicted to be negative.

[0087] The confusion matrix is a visualization tool used to show the performance of the classification model. Introducing the confusion matrix to analyze the test set data is one of the important means to evaluate whether the model is overfitting. It is often used to evaluate the performance of the classifier. The test results show that Figure 9 The confusion matrix is shown.

[0088] according to Figure 7 As can be seen, water seepage images and non-defective images experienced the most prediction errors. This is likely due to the highly similar appearance features of water seepage and spalling images, making it difficult for the model to distinguish them. Furthermore, the improved model significantly improved the recognition of exposed rebar, with an accuracy increase of 11.2%. Furthermore, spalling images, which are easily confused, achieved higher recognition accuracy in the improved model, with a 7.8% improvement compared to the original model. Overall, the improved MobileNet-v2 model proposed in this paper demonstrated excellent classification performance on the tested bridge structure apparent damage dataset, effectively distinguishing apparent damage types and accurately classifying damaged images. Furthermore, the confusion matrix allows the accuracy, recall, and F1 score of each category in the classification model to be calculated. The results are shown in Table 5.

[0089] Table 5. Classification and recognition results of MobileNet-v2 model;

[0090]

[0091] Table 6 Classification and recognition results of the improved MobileNet-v2 model

[0092]

[0093] The classification and recognition metrics for various bridge defects on the test set are shown in Tables 3.7 and 3.8. The improved MobileNet-v2 model's recognition accuracy for each defect category is ranked as follows: leakage, cracks, exposed rebar, no defect, spalling, and corrosion. Compared to the pre-improvement model, the F1 score for all defect categories improved, with exposed rebar, corrosion, and water seepage increasing by 4.4%, 4.1%, and 1.8%, respectively, demonstrating the model's enhanced ability to represent complex defects. Furthermore, the recall rate significantly improved, with a 3.7% increase for the no defect category and a 1.9% increase for corrosion, indicating a reduction in the model's missed detection rate for negative examples. The improved model also significantly improved its overall accuracy, particularly for corrosion and exposed rebar, which saw increases of 6.2% and 4.6%, respectively, reflecting a significant reduction in the model's false positive rate. The model's overall accuracy and F1 score reached 95.52% and 95.48%, respectively. The model achieved accurate classification while maintaining a high recall of 95.53%, validating the improved algorithm's ability to mitigate missed detections in engineering applications. Although the improved scheme has effectively enhanced the feature expression capability, there is still room for optimization in metal rust discrimination and disease-free identification. The model robustness needs to be further improved through methods such as sample enhancement or enhanced feature attention.

[0094] Visual analysis of results;

[0095] This chapter combines the Gray-Map feature visualization method with the Grad-CAM attention mechanism to generate feature distribution maps and attention heat maps for result presentation, in order to explore the feature focusing mechanism of the network model in image classification tasks. Figure 8 As shown in the figure, the MobileNet-v2 model before the improvement is easily disturbed by complex environments during feature extraction, and the focus location of the disease is not accurate enough; the improved MobileNet-v2 model is more accurate in the focus location and range of bridge diseases under complex environmental interference conditions, and effectively identifies the morphological characteristics and spatial distribution characteristics of five types of damage.

[0096] Through Figure 9 Visual analysis shows that while the original MobileNet-v2 model can generate a certain clustered distribution in the bridge apparent damage classification task, the feature space of different damage types is mixed, resulting in blurred boundaries between categories. In comparison, the improved MobileNet-v2 model demonstrates superior discrimination capabilities in two-dimensional feature mapping: various damage features present a compact intra-class distribution while maintaining sufficient inter-class distance, with only a small number of boundary samples misclassified. T-SNE dimensionality reduction visualization results show that the improved MobileNet-v2 has significantly improved damage classification accuracy and feature separability. Its classification results are highly consistent with quantitative evaluation indicators, further verifying the effectiveness of the algorithm improvement scheme.

[0097] The above is only a further embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and concepts of the present invention within the scope disclosed by the present invention, which fall within the scope of protection of the present invention.

Claims

1. A bridge structure damage identification method based on an improved MobileNet-v2 model, characterized by: The steps include: Step 1: Collect image data of various structural damage types of bridges in service; Step 2: Establish a data set of apparent damage of bridge structure; Step 3: Use transfer learning to migrate the pre-trained weights and parameters to the MobileNet-v2, SqueezeNet, ShuffleNet, GoogLeNet, and ResNet18 models for comparison, and select the MobileNet-v2 model; Step 4: Add a skip residual structure to the MobileNet-v2 model and use the Swish layer to replace the Relu6 layer to accurately classify the apparent damage of the bridge structure.

2. The bridge structure damage identification method based on the improved MobileNet-v2 model according to claim 1 is characterized by: In step 1, the image data of various structural damage types of the bridge are obtained by taking the image data in various environments and at various angles and with complex background noise interference; The image data sources include pictures of apparent damage to bridge structures obtained through regular bridge inspection reports, manual photography / drone inspections, and network collection.

3. The bridge structure damage identification method based on the improved MobileNet-v2 model according to claim 2 is characterized by: After step 2, before classifying the image dataset, the images must be preprocessed. The image size must be adapted to the network input requirements, and the images must be resized to a uniform size. ImageNet weights are used for image normalization, making it easier for the model to handle images of different sizes and pixel ranges during training and prediction. The image size is scaled to 224 pixels × 224 pixels. In addition, the image is augmented and expanded using flipping, mirroring, translation, rotation, salt and pepper noise, and contrast enhancement methods, resulting in a total of 12,000 images in the dataset.

4. The bridge structure damage identification method based on the improved MobileNet-v2 model according to claim 3 is characterized by: Transfer learning uses the ImageNet image dataset as a pre-training dataset. The trained weights and parameters are directly transferred to new classification and recognition tasks. Different network models have different characteristics. Taking into account the task difficulty and model parameter count factors and to meet the requirements of implantation on mobile platforms, five network models, including MobileNet-v2, Squeezenet, ResNet18, Googlenet, and the classic network VGG with smaller parameters, were selected for comparison.

5. According to the bridge structure damage identification method based on the improved MobileNet-v2 model of claim 4, the Swish activation function is a new activation function proposed by Google. It is similar to the ReLU6 function in calculation, but has the potential to outperform ReLU6 in performance. Its mathematical expression is as follows: Swish(x)=xσ(βx) Where σ is the Sigmoid function, β is a learnable parameter, and the ReLU function directly cuts off negative values (the output is always 0), which will cause neuron death; while the Swish function retains negative value information to avoid neuron death, while filtering noise through gradient attenuation (non-zero but close to 0).

6. The bridge structure damage identification method based on the improved MobileNet-v2 model according to claim 5 is characterized by: The MobileNet-v2 network includes an inverted residual and a linear bottleneck; Inverted residual, used to increase the extraction of image features to improve accuracy; Linear bottleneck, used to avoid information loss of nonlinear functions.

7. The bridge structure damage identification method based on the improved MobileNet-v2 model according to claim 6 is characterized by: On the basis of the linear bottleneck structure, a 1×1 convolutional jump connection similar to the residual structure is added to form a jump structure; MobileNet-v2 is used as the baseline network, using the Swish activation function and adding a skip residual structure.

Images