Bridge underwater pier defect classification method based on deep learning, medium and equipment

Abstract

The invention relates to the technical field of bridge underwater pier defect detection and classification, in particular to a bridge underwater pier defect classification method based on deep learning, a medium and equipment, and aims to improve the accuracy and efficiency of defect classification. According to the technical scheme, the method comprises the following steps: S1, obtaining an underwater pier image of a bridge, and obtaining a pier defect preliminary data set; s2, classifying and marking the pier defect preliminary data set to obtain a pier defect classification data set; s3, performing data expansion on the pier defect classification data set and dividing the pier defect classification data set into a training set and a test set; s4, establishing a Res-Swinin-T model based on deep learning; s5, a hyper-parameter combination value of the Res-Swinin-T model is set, the training set is input into the Res-Swinin-T model for training until model loss convergence is achieved, and a model weight file is obtained; s6, the Res-Swinin-T model loads a model weight file, and a test set is used for testing; and S7, carrying out defect classification on the underwater piers of the bridge.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater bridge pier defect detection and classification, and specifically to an underwater bridge pier defect classification method, medium and equipment based on deep learning. Background Art

[0002] Bridges are vital infrastructure in modern transportation systems, and their safety and stability are directly related to public safety and economic development. However, due to long-term exposure to underwater environments, underwater bridge piers often face various defects such as spalling, exposed reinforcement, and exposed aggregate. If these defects are not discovered and handled in time, they may lead to a decline in bridge structural performance and even cause serious accidents such as bridge collapse.

[0003] Traditional underwater bridge pier inspection methods mainly rely on visual inspection by human divers or detection using sonar, radar and other equipment. These methods are not only costly and inefficient, but also limited by the complexity of the underwater environment and the professional skills of divers, and the accuracy and consistency of the inspection results are difficult to guarantee. In addition, traditional image processing and pattern recognition technologies have shown great limitations when dealing with problems such as poor underwater image quality, high noise, and low contrast, and it is difficult to meet the needs of high-precision, automated inspection.

[0004] In recent years, deep learning technology has made significant progress in the field of computer vision, especially in image classification, object detection and semantic segmentation. The deep learning-based bridge underwater pier defect classification method can make full use of a large amount of underwater image data, and automatically identify and classify various types of underwater pier defects by building and training a deep neural network model, thereby improving the accuracy and efficiency of detection. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and provide a method, medium and equipment for classifying underwater bridge pier defects based on deep learning to improve the accuracy and efficiency of defect classification.

[0006] The technical solution of the present invention is:

[0007] A method for classifying underwater bridge pier defects based on deep learning, comprising the following steps:

[0008] S1: Acquire underwater bridge pier images and obtain a preliminary data set of pier defects;

[0009] S2: Classify and label the preliminary data set of bridge pier defects to obtain a classification data set of bridge pier defects;

[0010] S3: Expand the data set of bridge pier defect classification and divide it into training set and test set;

[0011] S4: Establish the Res-Swin-T model based on deep learning;

[0012] S5: Set the hyperparameter combination value of the Res-Swin-T model, input the training set into the Res-Swin-T model training until the model loss converges, and obtain the model weight file;

[0013] S6: Res-Swin-T model loads the model weight file and tests it using the test set;

[0014] S7: Defect classification of underwater bridge piers.

[0015] In step S2, the classification of the preliminary data set of bridge pier defects includes exposed aggregate, mud, exposed steel bars, spalling, turbidity and normal.

[0016] In step S3, data expansion includes performing contrast enhancement, brightness enhancement, horizontal flipping, 180° rotation, Gaussian noise, and salt and pepper noise on the images of the pier defect classification data set.

[0017] In the step S3, each type of image in the pier defect classification data set is randomly divided into a training set and a test set in a ratio of 8:2.

[0018] In step S4, the Res-Swin-T model includes a block partition layer, a window encoder, a window encoder, a block fusion layer, a residual layer, a residual layer, a residual layer, a residual layer, a residual layer, a bottleneck layer, a residual layer, a residual layer, an adaptive average pooling layer, a flattening layer, and a fully connected layer, which are connected in sequence.

[0019] In step S5, setting the hyperparameter combination value includes:

[0020] The initial learning rate is 1e-5, and a linear decay strategy is used to decay the learning rate to half every 5 epochs;

[0021] The training Epoch is set to 100;

[0022] The optimizer uses Adam, and the weight decay value is set to 0;

[0023] The input image size is 224×224;

[0024] Enter the batch size as 12.

[0025] The step S7 includes: collecting images of underwater bridge piers and inputting them into the Res-Swin-T model to obtain defect classification results of the underwater bridge piers.

[0026] A computer-readable storage medium, characterized in that: a computer program is stored on the storage medium; when the computer program is executed by a processor, the deep learning-based bridge underwater pier defect classification method can be implemented.

[0027] The memory is used to store computer programs; the processor is used to implement the deep learning-based bridge underwater pier defect classification method when executing the computer program.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention adopts deep learning technology and proposes a Res-Swin-T combined model for the classification of underwater bridge pier defects. It takes advantage of Swin-T's advantages in capturing global image information and long-range dependencies, and ResNet's powerful ability in extracting fine-grained features and stably training deep networks. It can accurately extract and identify subtle features in pier images, thereby improving the accuracy of defect classification.

[0030] 2. Through data enhancement techniques (such as contrast enhancement, brightness enhancement, horizontal flip, 180° rotation, Gaussian noise and salt and pepper noise), the present invention enhances the robustness of the model to images under different environments and noise conditions, so that it can still maintain high detection performance in complex underwater environments;

[0031] 3. Since the terminal device integrates the method mentioned in the present invention, it can process and analyze underwater bridge pier images in real time, realize rapid detection and classification, provide timely feedback and suggestions for bridge maintenance, and avoid potential safety hazards;

[0032] 4. The method mentioned in the present invention is not only applicable to the defect detection of underwater bridge piers, but can also be extended to other similar underwater structure detection fields, such as docks, offshore platforms, etc., and has broad application prospects;

[0033] 5. In actual underwater inspection, existing defect recognition methods focus on discovering and locating defects, and are prone to misidentification of defects with fuzzy boundary features and irregular distribution (such as mud, turbid water, and exposed aggregates); the method mentioned in the present invention focuses on clarifying the defect type, effectively improving the accuracy and efficiency of defect classification.

[0034] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the method for classifying underwater bridge pier defects of the present invention.

[0036] Figure 2It is a schematic diagram of the classification of underwater bridge pier defects of the present invention.

[0037] Figure 3 It is a data expansion schematic diagram of the present invention.

[0038] Figure 4 It is a structural diagram of the residual network of the present invention.

[0039] Figure 5 It is a structural diagram of the window encoder of the present invention

[0040] Figure 6 It is a structural diagram of the Res-Swin-T model of the present invention.

[0041] Figure 7-1 This is one of the schematic diagrams of defect classification results of underwater bridge piers of the present invention.

[0042] Figure 7-2 This is the second schematic diagram of the defect classification result of the underwater bridge pier of the present invention.

[0043] Figure 7-3 This is the third schematic diagram of the defect classification result of the underwater bridge pier of the present invention.

[0044] Figure 7-4 This is the fourth schematic diagram of the defect classification result of the underwater bridge pier of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0046] See also Figure 1 -7. The present invention provides a technical solution: a method for classifying underwater bridge pier defects based on deep learning, comprising the following steps:

[0047] S1: Data collection and construction of preliminary dataset

[0048] Acquire underwater bridge pier images and obtain a preliminary data set of pier defects.

[0049] Underwater inspection of bridges is carried out by using underwater robots or divers carrying optical cameras to capture images of underwater bridge piers, obtain pier image data, and obtain a preliminary data set of pier defects.

[0050] Ensure the acquisition of comprehensive and diverse bridge pier image data to provide a basis for subsequent defect classification.

[0051] S2: Image Classification

[0052] The preliminary data set of bridge pier defects is classified and labeled to obtain a classification data set of bridge pier defects.

[0053] The preliminary data set of bridge pier defects was divided into six categories: exposed aggregate, mud, exposed steel, spalling, turbidity and normal, and professionals manually classified and labeled and corrected to ensure the accuracy of data labeling. Among them, exposed aggregate, mud, exposed steel, spalling and turbidity are five typical defects of underwater concrete structures of bridges, and normal is the normal situation of underwater concrete structures of bridges.

[0054] S3: Data augmentation and partitioning

[0055] The pier defect classification dataset is expanded and divided into training set and test set.

[0056] Data augmentation includes the following six categories:

[0057] ①Contrast enhancement: adjust the contrast of the image to make the details clearer;

[0058] ②Brightness enhancement: adjust image brightness to improve image visibility in low-light conditions;

[0059] ③Horizontal flip: flip the image horizontally to increase data diversity;

[0060] ④180° rotation: rotate the image 180 degrees to simulate different shooting angles;

[0061] ⑤ Gaussian noise: Add Gaussian noise to enhance the model’s robustness to noise;

[0062] ⑥Salt and pepper noise: Add salt and pepper noise to enhance the model’s robustness to noise.

[0063] After data augmentation, the pier defect classification dataset generates six enhanced versions for each image, so the total number of images of the pier defect classification dataset increases to six times the original number.

[0064] Subsequently, each category of images in the pier defect classification dataset was randomly divided into a training set and a test set in a ratio of 8:2. This step enhanced the diversity of the dataset and helped improve the generalization ability of the model.

[0065] S4: Model construction

[0066] Establish the Res-Swin-T model based on deep learning (ResNet-Swin-Transformer, underwater bridge pier defect classification model).

[0067] like Figure 6As shown in the figure, the Res-Swin-T model includes a block partition layer (Patch Partition), two window encoders (Swin-Transformer encoder), a block fusion layer (Patch Merging), five residual layers (ResBlock), a bottleneck layer (BottleNeck), two residual layers (ResBlock), an adaptive average pooling layer (AdaptiveAvgPool), a flattening layer (Flatten), and a fully connected layer (Linear).

[0068] The principle of the Res-Swin-T model is as follows:

[0069] (1) Block partitioning layer: Patch segmentation and mapping; the input image is segmented into several patches of fixed size (such as 16×16), each patch is flattened and mapped to a high-dimensional vector space (such as 256 dimensions);

[0070] (2) Window encoder: Input the patch vector to two encoders, use the multi-head self-attention mechanism to extract global features, and merge adjacent patches through the block fusion layer to reduce the size of the feature map;

[0071] (3) Residual layer and bottleneck layer: Connect multiple residual layers and bottleneck layers after the encoder to enhance the model's expression and feature extraction capabilities;

[0072] (4) Adaptive average pooling layer: The feature map is input into the adaptive average pooling layer, and the feature map size of each channel is adjusted to a fixed output size (e.g., 7×7);

[0073] (5) Flattening layer and fully connected layer: The feature map is flattened and input into the fully connected layer for feature fusion, and finally the classification result is output.

[0074] The Res-Swin-T model combines the ResNet residual network ( Figure 4 ) and Swin-Transformer encoder ( Figure 5 ). The model extracts image features through a feature extraction part based on a window encoder and a feature extraction part based on a residual network, and then achieves the classification task by fusing the features of these two parts. This combination takes advantage of the advantages of the window encoder in capturing global image information and long-range dependencies and the powerful ability of the residual network in extracting fine-grained features and stabilizing the training of deep networks, thereby improving the overall performance and robustness of the model.

[0075] Residual networks, such as Figure 4 As shown in , it includes the residual layer (ResBlock) and the bottleneck layer (BottleNeck). Figure 6 As shown, the residual network includes five residual layers (ResBlock), a bottleneck layer (BottleNeck), and two residual layers (ResBlock) connected in sequence.

[0076] The residual layer includes: the input features are sequentially subjected to convolution (3×3Conv), batch normalization (BatchNorm), activation function (ReLU), convolution (3×3Conv), batch normalization (BatchNorm) and activation function (ReLU), and the input features are then added to the output of the second activation function (ReLU) and output.

[0077] The bottleneck layer includes: the input features are sequentially subjected to convolution (1×1Conv), batch normalization (BatchNorm), activation function (ReLU), convolution (3×3Conv), batch normalization (BatchNorm), convolution (1×1Conv) and activation function (ReLU), and the input features are then added to the output of the second activation function (ReLU) and output.

[0078] Window encoder (Swin-Transformer encoder), such as Figure 5 As shown, it includes a window multi-head self-attention module encoder (W-MSA encoder) and a sliding window multi-head self-attention encoder (SW-MSA encoder). Among them:

[0079] The window multi-head self-attention encoder includes: input feature Z l-1 After layer normalization (LayerNorm) and window multi-head self-attention (W-MSA) in turn, the output features of the window multi-head self-attention (W-MSA) and the input features Z l-1 Add to get the output features Output characteristics After layer normalization (LayerNorm) and multi-layer perceptron (MLP), the output features of the multi-layer perceptron (MLP) and the output features Add to get the output feature Z l .

[0080] The sliding window multi-head self-attention encoder includes: output feature Z l After layer normalization (LayerNorm) and sliding window multi-head self-attention (SW-MSA) in turn, the output features of the sliding window multi-head self-attention (SW-MSA) and the output features of the window multi-head self-attention Z l Add to get the output features Output characteristics After layer normalization (LayerNorm) and multi-layer perceptron (MLP), the output features of the multi-layer perceptron (MLP) and the output features Add to get the output feature Z l+1 .

[0081] S5: Model training

[0082] Set the hyperparameter combination values ​​of the Res-Swin-T model, input the training set into the Res-Swin-T model training until the model loss converges, and obtain the model weight file;

[0083] Set the hyperparameter combination values ​​as follows:

[0084] The initial learning rate is 1e-5, and a linear decay strategy is used to decay the learning rate to half every 5 epochs; the training Epoch is set to 100; the input batch size is 12; the optimizer is Adam, and the weight decay value is set to 0; the input image size is 224×224; the cross entropy loss function is used, and the loss value is monitored during training until convergence, and the best model weight file is saved.

[0085] S6: Model testing

[0086] The Res-Swin-T model loads the weight file and uses the test set to test the Res-Swin-T model.

[0087] S7: Classification of underwater bridge pier defects

[0088] The images of underwater bridge piers are collected and input into the Res-Swin-T model to obtain the defect classification results of underwater bridge piers.

[0089] Example

[0090] Underwater robots or divers were used to inspect the underwater piers of thirty-seven actual wading bridges.

[0091] During the image capture process, different shooting ranges, shooting distances, and shooting angles were set to increase the diversity of the dataset samples. A total of 2,510 images of underwater concrete structure defects of bridges were collected.

[0092] We invited experienced bridge inspection engineers to classify defect images and divided the types of underwater concrete structure defects into five typical defects: exposed aggregate, mud, exposed steel bars, spalling and muddy water. We also considered the images of underwater bridge piers under normal conditions. Typical images are as follows: Figure 2 .

[0093] In order to improve the generalization ability of the model, the above classified data sets are expanded. The specific method is as follows: each image is processed using six enhancement methods: contrast enhancement, brightness enhancement, horizontal flip, 180-degree rotation, salt and pepper noise, and Gaussian noise, so that each image generates six enhanced versions. Typical enhanced images are as follows: Figure 3 .

[0094] The specific parameter settings of each enhancement method are as follows:

[0095] Contrast enhancement: randomly adjust the saturation between 1.1-1.5;

[0096] Brightness enhancement: randomly adjust the chroma between 1.1-1.3;

[0097] Horizontal flip: flip the image horizontally;

[0098] 180 degree rotation: rotate 180 degrees clockwise;

[0099] Salt and pepper noise: The probability of adding white noise pixels is 0.02, and the probability of adding black noise pixels is also 0.02;

[0100] Gaussian noise: The added noise has a mean of 0.1 and a standard deviation of 0.1.

[0101] The original dataset was greatly expanded through these data augmentation methods, and 15,060 images were finally obtained. The expanded dataset was divided into a training set and a test set in a ratio of 8:2, containing 12,048 images and 3,012 images respectively. The hyperparameter combination values ​​of the Res-Swin-T model were set, and the Res-Swin-T model with the training set as input was trained until the model loss converged, and the model weight file was obtained.

[0102] The Res-Swin-T model loads the model weight file and is tested using the test set. The model performance is evaluated using indicators such as confusion matrix, accuracy, recall, and F1 score to verify the accuracy and robustness of the model. A typical visualization image is shown in Figure 7.

[0103] Confusion Matrix:

[0104] Prediction value = Positive example Prediction = Counterexample True value = positive example TP FN True value = counterexample FP TN

[0105] Accuracy: Accuracy: Recall: F1 score:

[0106] Model performance

[0107]

[0108] The above method is embedded in a terminal device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, the above method is used to classify the underwater pier defects of actual bridges; after many experiments, the recognition accuracy of the device in actual detection is consistent with the test set results, proving that it has high practicality and reliability.

[0109] Through the detailed description of the above embodiments, the present invention successfully realizes the classification of underwater bridge pier defects based on deep learning, has high classification accuracy and practical application value, and can effectively improve the efficiency and accuracy of bridge detection.

[0110] In addition, in other embodiments, a computer-readable storage medium may be provided, on which a computer program is stored. When the computer program is executed by a processor, the method for classifying underwater bridge pier defects based on deep learning as described in S1 to S7 above can be implemented.

[0111] In addition, in other embodiments, a deep learning-based underwater bridge pier defect classification device may be provided, including a memory and a processor;

[0112] The memory is used to store computer programs;

[0113] The processor is used to implement the underwater bridge pier defect classification method based on deep learning as described in S1 to S7 above when executing the computer program.

[0114] It should be noted that the above-mentioned memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a neural network processor (NPU), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Of course, the device should also have necessary components for realizing program operation, such as power supply, communication bus, etc.

[0115] The preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

Claims

1. A method for classifying underwater bridge pier defects based on deep learning, comprising the following steps: S1: Acquire underwater bridge pier images and obtain a preliminary data set of pier defects; S2: Classify and label the preliminary data set of bridge pier defects to obtain a classification data set of bridge pier defects; S3: Expand the data set of bridge pier defect classification and divide it into training set and test set; S4: Establish the Res-Swin-T model based on deep learning; S5: Set the hyperparameter combination value of the Res-Swin-T model, input the training set into the Res-Swin-T model training until the model loss converges, and obtain the model weight file; S6: Res-Swin-T model loads the model weight file and tests it using the test set; S7: Defect classification of underwater bridge piers.

2. The method for classifying underwater bridge pier defects based on deep learning according to claim 1 is characterized by: In step S2, the classification of the preliminary data set of bridge pier defects includes exposed aggregate, mud, exposed steel bars, spalling, turbidity and normal.

3. The method for classifying underwater bridge pier defects based on deep learning according to claim 1 is characterized by: In step S3, data expansion includes performing contrast enhancement, brightness enhancement, horizontal flipping, 180° rotation, Gaussian noise, and salt and pepper noise on the images of the pier defect classification data set.

4. The method for classifying underwater bridge pier defects based on deep learning according to claim 3 is characterized by: In the step S3, each type of image in the pier defect classification data set is randomly divided into a training set and a test set in a ratio of 8:

2.

5. The method for classifying underwater bridge pier defects based on deep learning according to claim 1, characterized in that: In step S4, the Res-Swin-T model includes a block partition layer, a window encoder, a window encoder, a block fusion layer, a residual layer, a residual layer, a residual layer, a residual layer, a residual layer, a bottleneck layer, a residual layer, a residual layer, an adaptive average pooling layer, a flattening layer, and a fully connected layer, which are connected in sequence.

6. The method for classifying underwater bridge pier defects based on deep learning according to claim 1 is characterized by: In step S5, setting the hyperparameter combination value includes: The initial learning rate is 1e-5, and a linear decay strategy is used to decay the learning rate to half every 5 epochs; The training Epoch is set to 100; The optimizer uses Adam, and the weight decay value is set to 0; The input image size is 224×224; Enter the batch size as 12.

7. The method for classifying underwater bridge pier defects based on deep learning according to claim 1 is characterized by: The step S7 includes: collecting images of underwater bridge piers and inputting them into the Res-Swin-T model to obtain defect classification results of the underwater bridge piers.

8. A computer-readable storage medium, characterized in that: A computer program is stored on the storage medium; when the computer program is executed by the processor, it can implement the deep learning-based bridge underwater pier defect classification method as described in any one of claims 1 to 7.

9. A deep learning-based underwater bridge pier defect classification device, comprising a memory and a processor; characterized in that: The memory is used to store computer programs; the processor is used to implement the deep learning-based bridge underwater pier defect classification method as described in any one of claims 1 to 7 when executing the computer program.

Images