Pixel-level recognition method and system for bridge surface defects based on instance segmentation

Abstract

The present invention discloses a method and system for pixel-level identification of bridge surface defects based on instance segmentation. The method comprises constructing training datasets for each bridge part, each bridge component, and each defect category; constructing a part recognition model, a component recognition model, and a defect recognition model based on an improved VGG16 model, thereby obtaining a three-level pre-classification model; constructing a pixel-level recognition model for each defect category based on an improved Mask RCNN model; and then training the three-level pre-classification model and the pixel-level recognition model using the corresponding training datasets, performing online image recognition, and obtaining the number, area, and contour shape of defects in the pre-classified bridge surface image. The present invention can achieve pixel-level recognition of bridge surface images while balancing high accuracy and efficiency, providing an objective basis for further determining the severity of defects.

Description

Technical Field

[0001] The present invention relates to the field of bridge structure appearance information detection, and in particular to a method and system for pixel-level recognition of bridge appearance defects based on instance segmentation. Background Art

[0002] In recent years, many scholars at home and abroad have carried out research on new methods for non-destructive detection of bridge defects, among which the identification of typical defects in bridge surface images is of great significance. However, the existing bridge detection system has the following problems in extracting surface information: (1) The mixed recognition performance of typical defects is poor, and atypical defects are easily ignored. (2) The extraction of surface defect information is imperfect, and the pixel-level recognition accuracy of defects needs to be improved. (3) The input image is not pre-classified, and the defect classification and instance segmentation are completed at the same time, which increases the computational cost. The binary segmentation model needs to be connected in series to complete the judgment of all defect states, which will reduce the recognition efficiency. The performance of the instance segmentation network is also easily affected by interference such as atypical defect images. (4) The input image is not screened, the instance segmentation efficiency is low, and the workload of instance segmentation of typical surface defects is too large. (5) The model training idea is single and the application is limited. Summary of the Invention

[0003] To address the shortcomings of the existing technology, the present invention proposes a pixel-level identification method and system for bridge surface defects based on instance segmentation. The specific technical solution is as follows:

[0004] A pixel-level recognition method for bridge apparent defects based on instance segmentation, the method comprising:

[0005] Step 1: Obtain bridge surface images and annotate each image with the bridge location, the specific bridge component where the defect is located, and the defect category. Classify the images according to the bridge location to construct a training dataset for each bridge location. Then, further classify the images of each bridge location according to the specific bridge component where the defect is located to construct a training dataset for each bridge component in each bridge location. Finally, classify the images of each bridge component according to the defect category to construct a training dataset for each defect category for each bridge component in each bridge location.

[0006] Step 2: Construct an improved VGG16 model, that is, modify the number of neurons in the three fully connected layers of the VGG16 model to 1024, 512, and the number of recognition categories. Then, use the training data set of each bridge part to train the improved VGG16 model to obtain a part recognition model corresponding to each bridge part; then use the training data set of each bridge component to train the part recognition model of the corresponding bridge part to obtain a component recognition model of the corresponding component; finally, use the training data set of each disease category of each bridge component of each bridge part to train the component recognition model of the corresponding component to obtain a disease recognition model of each disease category of the corresponding bridge component; thus, a three-level pre-classification model is obtained;

[0007] Step 3: Construct an improved Mask RCNN model, using the FPN structure based on Resnet101 as the backbone network, superimposing all mid- and high-level features on the low-level for fusion, which can identify small and difficult-to-distinguish information in the disease category; the backbone network generates a feature map based on the input bridge appearance image, and then the feature map is input into the region proposal network RPN to generate the target candidate region ROI, and then the feature map and the corresponding target candidate region are processed by ROI Align to obtain a feature region of fixed size; finally, full convolution and full connection calculations are performed on each target candidate region ROI to complete mask detection, disease classification, and bounding box regression respectively; and the bridge appearance image of each disease category is used as the training set to train the improved Mask RCNN model to obtain a pixel-level recognition model corresponding to each disease category;

[0008] Step 4: Input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image, and then input the bridge appearance image to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image; finally, input the bridge appearance image to be predicted into the defect recognition model corresponding to the component, and output the defect category of the bridge appearance image to be predicted; input the bridge appearance image to be predicted into the pixel-level recognition model corresponding to the defect category, and the model outputs the number, area and contour shape of the defects in the bridge appearance image to be predicted.

[0009] Furthermore, when obtaining bridge surface images to construct a training dataset, the small amount of data obtained is first manually classified and calibrated to form a small training dataset. The improved VGG16 model is then quickly trained using the small training dataset, and the obtained three-level pre-classification model is used to predict unclassified bridge surface images. Manual verification is completed based on the classification results, and the manually verified images are added to the small training dataset to complete the semi-automatic expansion of the original dataset.

[0010] Furthermore, there are five types of pixel-level recognition models, namely, a pitting recognition model, a hole recognition model, a crack recognition model, an exposed reinforcement recognition model, and a concrete peeling recognition model.

[0011] Furthermore, when the improved VGG16 model is trained, the convolution process uses the ReLU function as the activation function, and keeps the weights of some convolutional layers close to the input end of the model unchanged, that is, freezes these convolutional layers, so as to retain some learned features and prevent parameter overfitting; the weights of some convolutional layers close to the output end are kept updated, that is, unfreezes these convolutional layers, so that the model has stronger learning ability and continuously learns new features.

[0012] Furthermore, the improved VGG16 model has a total of 5 convolution blocks, three convolution blocks close to the model input end are frozen, and two convolution blocks close to the model output end are unfrozen; the convolution blocks are separated by max-pooling connections.

[0013] Furthermore, after completing the semi-automatic expansion of the original data set, the existing bridge surface images are enhanced, and the training data set is constructed using the enhanced images.

[0014] A bridge apparent defect pixel-level recognition system based on instance segmentation is used to implement a bridge apparent defect pixel-level recognition method. The system includes:

[0015] The bridge image upload module is used as an image upload port to collect the bridge surface image;

[0016] A three-level pre-classification module, including a location recognition model, a component recognition model, and a defect recognition model, is used to gradually identify the captured location of the bridge surface image, the component where the defect is located, and the defect category;

[0017] The pixel-level disease recognition module is used to identify the number, area, and contour shape of specific types of diseases;

[0018] The classifier is used to classify and associate the bridge surface image according to the recognition results of the previous level model during the training and verification process of the part recognition model, component recognition model, disease recognition model, and pixel-level recognition model, and call the corresponding next level model for recognition to complete the transmission of the bridge surface image between models at all levels.

[0019] A database is used to store characteristic images of bridge surface information.

[0020] Furthermore, the system also includes a data enhancer for performing transformation and enhancement operations on the bridge appearance image to expand the number of bridge appearance images.

[0021] The beneficial effects of the present invention are as follows:

[0022] (1) Combining VGG16 classification with Mask RCNN segmentation and recognition, the input bridge surface image is pre-classified during the recognition process, which can reduce the interference of atypical disease images, reduce the workload of typical disease instance segmentation, improve the work efficiency of the Mask-RCNN model, and improve the efficiency of bridge inspection.

[0023] (2) If there is no three-level pre-classification of diseases, it is necessary to train a multi-segmentation model to classify and segment multiple diseases at the same time, or train multiple binary segmentation models and connect them in series to perform multiple binary recognitions on each image. Among them, the multi-segmentation model has lower accuracy and the latter has lower recognition efficiency. The disease category is obtained by combining the VGG16 three-level pre-screening results, and the pixel-level recognition model with the disease background of this category can be directly matched. After the image is directed into the recognition model, the segmentation network framework is optimized to improve the accuracy of the trained Mask RCNN model.

[0024] (3) Realize pixel-level mixed recognition of bridge surface defects, obtain the contours, locations, areas and other information of various defects in the same image, realize digital twin of bridge defect information, and provide an objective basis for judging the operating status of the bridge.

[0025] (4) The three-level pre-classification model and pixel-level recognition model have strong generalization capabilities. They can continuously improve and update the three-level pre-classification model and pixel-level recognition model of bridge surface information in the later application process, expand the training data in real time, and improve the recognition ability of atypical diseases.

[0026] (5) The recognition method and system of the present invention have strong practical application capabilities and can analyze defect information in large-scale images with rich interference factors taken during on-site bridge inspections, with high robustness. They overcome the shortcomings of traditional dataset images, such as small size, uniform background, and low differentiation from defects, and are convenient for use with equipment such as drones to achieve high-precision recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the pixel-level recognition method for bridge surface defects based on instance segmentation;

[0028] Figure 2 Flowchart for joint recognition of improved VGG16 and improved Mask R-CNN;

[0029] Figure 3 The overall framework diagram for building a three-level pre-classification model based on the improved VGG16;

[0030] Figure 4 This is a comparison chart of accuracy under different Epochs.

[0031] Figure 5It is the improved Mask R-CNN model network structure;

[0032] Figure 6 Schematic diagram of IOU calculation for evaluation index;

[0033] Figure 7 Examples of pixel-level identification of typical defects; (a)-(d): honeycombed surface; (e)-(h): holes; (i)-(l): concrete peeling; (m)-(p): cracks; (q)-(t): exposed reinforcement. DETAILED DESCRIPTION

[0034] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] like Figure 1 As shown, the bridge apparent defect pixel-level recognition method based on instance segmentation of the present invention includes the following steps:

[0036] S1: Construct a bridge surface information detection dataset;

[0037] Each bridge image collected on-site is manually classified based on its damage characteristics to generate a small preliminary dataset. This dataset allows for rapid training of a modified VGG16 model. By adjusting hyperparameters and network architecture, a more optimal model based on this small dataset is obtained. This initial model is then used to train unclassified images. After manually verifying the classification results, the newly added data is added to the existing dataset to generate a semi-automatically expanded original dataset.

[0038] Bridge defects include, but are not limited to, cracks, breakage, exposed reinforcement corrosion, and honeycombing. The calibrated Mask RCNN dataset also includes images without defects. Table 1 lists the bridge locations, components, and the corresponding defect catalog for each component. The bridge surface information detection dataset constructed in step S1 was randomly split into a 5:1 ratio: the former serving as the training set for model training and the latter as the test set for model quality evaluation. Ultimately, training datasets were obtained for each bridge location, each bridge component within each bridge location, and each damage category within each bridge component within each bridge location.

[0039] Table 1 Classification of bridge surface information

[0040]

[0041]

[0042] More importantly, the present invention performs data enhancement on the training set, including performing angular rotation, translation, shear transformation, scaling, and horizontal flipping operations on the image, thereby reducing the overfitting level of the recognition results.

[0043] S2: A three-level pre-classification model is constructed based on the improved VGG16 model;

[0044] like Figure 2 and Figure 3 As shown in Figure 2, the improved VGG16 model is improved based on the VGG16 model as the core, and the convolutional neural network model is constructed using the "Tensorflow+Python" system framework. The model is composed of 5 convolution blocks and three layers of full connection. The specific convolution layers are separated by pooling layers. Table 2 lists the structural layers of the convolutional neural network.

[0045] Table 2 VGG neural network structure

[0046]

[0047] VGG has five blocks, each consisting of 2 to 3 convolutional layers stacked with a pooling layer. The convolution kernel size used in the convolution layer is 3×3. The convolution calculation formula for the convolution layer neuron is:

[0048]

[0049] Where X is a two-dimensional vector with the region (M, N), ω ij is the convolution kernel, b is the output feature additional bias term, σ is the activation function, and this model uses the ReLU function.

[0050] The pre-classification model consists of three levels: the first level is a bridge part identification model, the second level is a bridge component identification model, and the third level is a defect identification model for each defect category. To improve model efficiency, the first-level classification network is trained and optimized first, and the second and third-level networks are trained based on the first-level model. This is because the number of categories increases and the classification difficulty increases from the first to the third level. Furthermore, the second and third levels classify information based on the first level, making the overall classification tasks similar. Under the same experimental conditions, fine-tuning the model showed a certain degree of improvement in performance.

[0051] To improve model accuracy, we investigated two strategies for unfreezing the top convolutional and pooling layers in the pre-trained VGG dataset. Compared to the original VGG results, the validation accuracy of the model significantly improved from 49% to 69.4% when convolutional block 5 was unfrozen, and reached 95.5% when convolutional blocks 4 and 5 were unfrozen.

[0052] When the number of neurons remains unchanged, unfreezing convolutional blocks 4 and 5 achieves approximately 0.25 higher training and validation accuracy than unfreezing block 5 alone. Changing the number of neurons in the two dense layers also slightly alters model performance; when the number of neurons in the dense layer is 1024 + 256, validation accuracy reaches its highest, reaching 96%. The results show that the improved VGG algorithm improves validation accuracy by approximately 45% compared to the original VGG algorithm on the bridge surface defect classification task. See Table 3 for detailed data.

[0053] Therefore, the number of neurons in the first two dense layers of the model was changed from 4096 to 1024 and 256 to speed up the model training.

[0054] Table 3 Comparison of original and improved VGG performance

[0055]

[0056] To improve the model training accuracy, some convolutional layers at the bottom of the model are frozen to retain some learned features and prevent overfitting caused by too many parameters; at the same time, the head convolutional layers are unfrozen to give the model stronger learning ability and continuously learn new features.

[0057] Epoch represents the round of data set training. The present invention comprehensively determines its optimization direction based on the data set size, GPU performance and model performance. Appropriately increasing Epoch can improve accuracy and make the model training process more stable; but too large Epoch may lead to overfitting or invalid training because the model has reached the optimal level. Analysis of the training results shows that since there are only three primary classifications, the overall convergence is faster, and a higher accuracy can be achieved by taking Epoch of 100. The secondary component classification model has a large number of classifications, namely 11 categories (bridge deck system), 7 categories (substructure) and 5 categories (superstructure). When Epoch = 100, the model oscillates greatly, the accuracy is about 0.8 and it is not stable, indicating that the network as a whole is still underfitting. When Epoch = 150 for the bridge deck system and superstructure, and Epoch = 200 for the substructure, the learning curve is more stable, and the accuracy after stabilization is improved by about 10%. Table 4 gives the hyperparameter settings of the improved VGG16 model. Figure 4 This is a comparison chart of accuracy under different Epochs.

[0058] Table 4 VGG model hyperparameter settings

[0059] Hyperparameters value Epoch 30-200 Initial learning rate <![CDATA[1×10 -4 ~4×10 -6 ]]> Decay 0.8 / 0.9 Waiting step length 5 (epoch) Batch 128 / 256

[0060] The bridge surface information image and its corresponding three-level information are used as the input data and output data of the model. The number of training iterations is set to 100-200 times, and the training result parameters are saved. At the same time, the results are output in the following format for different recognition results: "part-component-disease".

[0061] Images of different categories in the validation set are selected to verify the effectiveness of the convolutional neural network model. The model recognition results are compared with the actual classification. The recognition accuracy must reach more than 80%, otherwise the parameters and hyperparameters of the model are adjusted until the error meets the requirements.

[0062] After the model training is completed, the surface information image of the bridge to be inspected is input into the three-level pre-classification model of bridge defects with updated parameters to determine the final type of defect.

[0063] S3: Build a pixel-level recognition model for each disease category based on the improved Mask RCNN model;

[0064] This example considers three key factors: their commonness, significant impact on bridge service condition, and weighted importance in bridge service condition assessment. Five defects, namely exposed rebar, cracks, holes, spalling, and honeycombing, were selected as targets for more in-depth identification. The aforementioned three-level pre-classification process generated defect classification information, and images containing typical and significant defects were extracted and distinguished from the large number of original input images, serving as the foundation for this work.

[0065] The backbone network of Mask RCNN has two classic structures. Considering the problem of small target detection contained in bridge defects, the FPN structure is adopted, and the FPN based on Resnet101 is selected as the backbone network.

[0066] ResNet is a structure that ensures that the performance of deep networks is comparable to that of shallow networks when building them. To address the performance degradation of deep networks, the deep network is divided into several blocks, and shortcut connections with identity mapping are added to each block. If the input feature of each block is x, the feature originally required to be learned by the layer is the target function H(x). In the residual network, the target function is divided into two parts: H(x) = F(x) + x, so that the network's action approximates the residual function F(x), and x is added by the shortcut connection. The output of the network layer is then:

[0067] Output of layer l: x l+1 =x l +F(x l ,w l ), x l For the l-th layer input, the l+1-th layer output is: x l+2 =x l+1 +F(x l+1 ,w l+1 )=x l +F(x l ,w l )+F(x l+1 ,w l+1 )

[0068] Therefore, the output of the Lth layer can be expressed as,

[0069] Using the chain rule, we can calculate the gradient:

[0070]

[0071] While ensuring the overall gradient is non-zero according to the above formula, the introduction of the identity mapping prevents the loss of shallow parameter gradients due to the chain rule during gradient calculation. Furthermore, when adding a network layer does not improve performance, the residual F(x) = 0, and the output feature x equals the input feature x. This is equivalent to not adding a layer, ensuring that network performance does not degrade.

[0072] After different numbers of convolution and pooling operations, the Resnet network will obtain feature maps of different sizes. The traditional Resnet design only uses the top-level feature map for prediction, but since more detailed information is contained in the ignored shallow feature map, such a design is not conducive to the detection of small objects. Figure 4 As shown in the figure, after adding FPN, the Resnet network uses the traditional Resnet design. First, in Conv5, that is, after detecting the C5 layer, the corresponding feature map P5 is generated to complete the upsampling. This feature map P5 is then fused with Conv4 to complete the detection and generate a new feature map P4. This process is repeated to obtain P3 and P2. If the original network architecture is not modified, the Resnet network will output the P2 feature map as the final recognition result.

[0073] like Figure 5 As shown in the figure, FPN simultaneously utilizes the four feature maps generated by the last layer of the ResNet network in five different convolutional stages. The original feature maps P2 to P5 are further convolved to generate four new feature maps P2 to P5. The newly generated feature map P5 is resized to generate P6. Re-recognition of the five newly generated feature maps P2 to P6 yields the final output feature map. This method combines high-level semantic features with low-level detail information. Using FPN to fuse feature maps at different levels enables the network to adapt to the information required to identify small objects such as cracks and exposed rebar.

[0074] The input bridge defect image is fed through the Resnet101-FPN backbone network. The extracted image features are then fed into the Region Proposal Network (RPN), generating candidate target regions (ROIs). The feature map of the original image is then aligned with the ROIs. ROIAlign is then used to generate fixed-size feature regions. Finally, full convolution and full connectivity are performed on each candidate target region to complete the three tasks of mask classification (class) and bounding box regression (box). The output of Mask RCNN includes three prediction tasks, and its loss function can be expressed as:

[0075] L=L cls +L box +L mask

[0076] Among them, L cls Indicates the classification loss value of Bbox, L box Represents the regression loss value of Bbox, that is, the detection error; L mask Represents the loss value of the mask part, that is, the error of the semantic segmentation branch.

[0077] FPN only adds a small computational cost, so Mask R-CNN performs object detection faster overall. The data input to Mask R-CNN in this invention is the five types of defects classified by VGG classification, which reduces the input of images with other defects and those without defects. This reduces the workload of Mask R-CNN by pre-screening the input, improving its pixel recognition efficiency, and also reduces the interference of other defects.

[0078] The training optimizer uses Adam, which combines the advantages of the AdaGrad and RMSProp optimization algorithms. Its formula is:

[0079]

[0080] m t =β1m t-1 +(1-β1)g t ,

[0081]

[0082]

[0083] Where t is the time step, g t is the gradient value of the new round, f(θ) is the random objective function value of parameter θ, θ0 is the initial parameter vector, m t and v tare the first and second matrix vectors respectively, β1=0.9 and β2=0.999 are the exponential decay rates of moment estimates, are the first and second matrix vectors for bias correction, and α is the learning rate.

[0084] In order to prevent overfitting, the training adopts the L2 regularization method, that is, the regularization term is introduced into the original loss function. The formula is:

[0085]

[0086] Where L represents the loss function, is the original loss function, W k，l Represents each element in the weight matrix, and λ is the regularization coefficient.

[0087] In this field, models are usually evaluated using three metrics: IOU (Intersection over Union), precision, and recall. IOU represents the ratio of the overlap area between the true value region and the predicted region to the union area. Its formula can be expressed as:

[0088]

[0089] Among them, IOU bb The middle area refers to the area surrounded by the frame, which is used to evaluate the performance of the model in the target detection task, and the IOU m The area in IOU refers to the mask area, which is used to evaluate the performance of the model in the instance segmentation task. bb 、IOU m The overlapping area and the union area are as follows Figure 6 shown.

[0090] Accuracy, also known as precision, refers to the proportion of positive samples in all samples predicted to be positive samples. The formula is:

[0091]

[0092] The regression value is also called the recall rate, which refers to the proportion of all positive samples that are correctly identified as positive samples. The formula is:

[0093]

[0094] Unrelated detections that are not defect type A but are identified as defect type A are considered false positives; detections that are defect type A but are not correctly identified are considered false negatives. Detections that are correctly identified are considered true positives. Because instance segmentation tasks are more demanding than classification tasks, the criteria for correct identification are more stringent. Correct identification is considered correct only when the detected region matches the defect type of the ground-truth region and the IoU between the two regions is at least 0.5.

[0095] After obtaining the precision and recall values, the PR curve is drawn with the two as the vertical and horizontal coordinates respectively. The area enclosed by the curve and the coordinate axis is the AP (average precision) value of a certain type of disease. The average precision obtained by using IoUbb = 0.5 as the judgment standard is recorded as AP bb , and IoU m =0.5, then the corresponding indicator AP m The average of the AP of all categories is the mAP value, which can be expressed as:

[0096]

[0097] Among them, n is the total number of categories, AP i Indicates the AP value of the i-th category (AP bb Corresponding mAP bb , AP m Corresponding mAP m )

[0098] The present invention adopts mAP (mAP bb and mAP m ) (mean average precision) is used as an indicator to judge the accuracy of model recognition. The recognition accuracy must exceed 80%. Otherwise, the model parameters and hyperparameters should be adjusted, or the network layers should be fine-tuned, and some convolutional layers should be frozen and unfrozen until the error meets the requirements.

[0099] S4: Input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image, and then input the bridge appearance image to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image; finally, input the bridge appearance image to be predicted into the defect recognition model corresponding to the component, and output the defect category of the bridge appearance image to be predicted; input the bridge appearance image to be predicted into the pixel-level recognition model corresponding to the defect category, and the model outputs the number, area and contour shape of the defects in the bridge appearance image to be predicted.

[0100] The present invention provides a pixel-level recognition system for apparent bridge defects based on instance segmentation, which is used to implement a method for pixel-level recognition of apparent bridge defects. The system comprises:

[0101] The bridge image upload module is used as an image upload port to collect the bridge surface image;

[0102] A three-level pre-classification module, including a location recognition model, a component recognition model, and a defect recognition model, is used to gradually identify the captured location of the bridge surface image, the component where the defect is located, and the defect category;

[0103] The pixel-level disease recognition module is used to identify the number, area, and contour shape of specific types of diseases;

[0104] The classifier is used to classify and associate the bridge surface image according to the recognition results of the previous level model during the training and verification process of the part recognition model, component recognition model, disease recognition model, and pixel-level recognition model, and call the corresponding next level model for recognition to complete the transmission of the bridge surface image between models at all levels.

[0105] A database is used to store characteristic images of bridge surface information.

[0106] The system also includes a data enhancer, which is used to perform transformation and enhancement operations on the bridge appearance image to expand the number of bridge appearance images.

[0107] Example

[0108] In this example, 7,200 images of bridge inspections, including those classified as disease-free, were collected on-site. 1,200 of these images were randomly sampled and manually classified to obtain a small preliminary dataset. This small preliminary dataset was used to rapidly train a VGG16 model. After adjusting the number of neurons and hyperparameters in the VGG16 model, the model's structure was optimized, resulting in a superior model based on this small dataset.

[0109] The optimized VGG16 model was used to train 6,000 unclassified images. After manual verification of the results, the new data was added to the existing dataset to complete the construction of the original dataset. This original dataset was used to obtain information on the image location, component type, and disease type of each bridge inspection image in the dataset. Finally, multiple classified data files were used to construct the initial dataset for bridge surface information detection.

[0110] The completed initial dataset for bridge surface information detection was randomly divided into a training set and a validation set, resulting in two data modules: a training set containing 6,000 images and a validation set containing 1,200 images. A VGG16 model with a modified network architecture was trained using the training set, and the validation set was used to evaluate the model's training quality. After training, the model achieved an accuracy of over 90% for each of the three defect classifications, with minimal overfitting. This resulted in a fully trained and usable three-level pre-classification model. This three-level pre-classification model was used to process a total of 14,842 bridge surface images that did not have a "no-defect" classification. The resulting images included 3,352 images of exposed rebar, 5,916 images of cracks, 1,558 images of holes, 2,740 images of spalling, and 1,276 images of honeycombing. All images were stored at their original size (2048 × 1,536 dpi) without compression.

[0111] Each category of images was used as input data for the improved Mask RCNN model for model training with 250 iterations. The training results were saved to obtain a pixel-level recognition model for bridge surface defects.

[0112] The remaining images were used to verify the effectiveness of the model. The effective defect recognition accuracy must exceed 80%. Otherwise, the model parameters and hyperparameters were adjusted until the recognition rate met the requirements. The hyperparameter values ​​set in this example are detailed in Table 5. After training, the average defect recognition accuracy for the training set reached 89%, and the average defect recognition accuracy for the validation set reached 81%, meeting the requirements.

[0113] Table 5 Example model hyperparameter settings

[0114] Hyperparameters value Learning rate <![CDATA[10 -5 ]]> Init_Epoch 0 Epoch 50 / 100 Batch 1 Steps per Epoch 250 Validation steps 25

[0115] In the pixel-level disease detection task, the recognition results are displayed on the image after the model prediction, including the generated light-colored prediction mask, the bounding box of the identified object, the class label, and the confidence score. When two or more objects overlap in the image, in order to distinguish different instances of the same object, the output mask and bounding box will be distinguished in different styles (such as Figure 7 (a)(h)).

[0116] As can be seen from the examples, the pixel-level recognition model can effectively identify five typical defect images: honeycomb surface, holes, concrete peeling, cracks, and exposed steel bar corrosion. By judging whether a single pixel belongs to an instance or background, the number of defects, their location in the image, as well as their outline, shape, and pixel area are further obtained. For the same type of defect on different components, although the background features and the defect features are somewhat different, the model can still effectively identify them. For example Figure 7 (c) and (d) are the honeycomb surface of the platform cap beam and the bridge deck pavement respectively. Figure 7 (i) (j) Concrete delamination at the curb and beam slab, respectively. The model also performs well for large-area defects (such as pitting) with irregular shapes or even large-area defects surrounded by the background (such as cracks and exposed rebar), as well as small-scale defects (such as cracks and exposed rebar).

[0117] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A pixel-level recognition method for bridge surface defects based on instance segmentation, characterized in that: The method includes: Step 1: Obtain bridge surface images and annotate each image with the bridge location, the specific bridge component where the defect is located, and the defect category. Classify the images according to the bridge location to construct a training dataset for each bridge location. Then, further classify the images of each bridge location according to the specific bridge component where the defect is located to construct a training dataset for each bridge component in each bridge location. Finally, classify the images of each bridge component according to the defect category to construct a training dataset for each defect category for each bridge component in each bridge location. Step 2: Construct an improved VGG16 model, that is, modify the number of neurons in the three fully connected layers of the VGG16 model to 1024, 512, and the number of recognition categories. Then, use the training data set of each bridge part to train the improved VGG16 model to obtain a part recognition model corresponding to each bridge part; then use the training data set of each bridge component to train the part recognition model of the corresponding bridge part to obtain a component recognition model of the corresponding component; finally, use the training data set of each disease category of each bridge component of each bridge part to train the component recognition model of the corresponding component to obtain a disease recognition model of each disease category of the corresponding bridge component; thus, a three-level pre-classification model is obtained; Step 3: Construct an improved Mask RCNN model, using the FPN structure based on Resnet101 as the backbone network, superimposing all mid- and high-level features on the low-level for fusion, which can identify small and difficult-to-distinguish information in the disease category; the backbone network generates a feature map based on the input bridge appearance image, and then the feature map is input into the region proposal network RPN to generate the target candidate region ROI, and then the feature map and the corresponding target candidate region are processed by ROI Align to obtain a feature region of fixed size; finally, full convolution and full connection calculations are performed on each target candidate region ROI to complete mask detection, disease classification, and bounding box regression respectively; and the bridge appearance image of each disease category is used as the training set to train the improved Mask RCNN model to obtain a pixel-level recognition model corresponding to each disease category; Step 4: Input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image, and then input the bridge appearance image to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image; finally, input the bridge appearance image to be predicted into the defect recognition model corresponding to the component, and output the defect category of the bridge appearance image to be predicted; input the bridge appearance image to be predicted into the pixel-level recognition model corresponding to the defect category, and the model outputs the number, area and contour shape of the defects in the bridge appearance image to be predicted.

2. The pixel-level recognition method for bridge apparent defects based on instance segmentation according to claim 1 is characterized in that: When acquiring bridge surface images to construct a training dataset, the small amount of data obtained is first manually classified and calibrated to form a small training dataset. The improved VGG16 model is then quickly trained using the small training dataset, and the resulting three-level pre-classification model is used to predict unclassified bridge surface images. Manual verification is then completed based on the classification results, and the manually verified images are added to the small training dataset, completing the semi-automatic expansion of the original dataset.

3. The pixel-level recognition method for bridge apparent defects based on instance segmentation according to claim 1 is characterized in that: There are five types of pixel-level recognition models, namely, a pitting recognition model, a hole recognition model, a crack recognition model, an exposed reinforcement recognition model, and a concrete peeling recognition model.

4. The pixel-level recognition method for bridge apparent defects based on instance segmentation according to claim 1 is characterized in that: During training of the improved VGG16 model, the convolution process uses the ReLU function as the activation function, and keeps the weights of some convolutional layers close to the input end of the model unchanged, that is, freezes these convolutional layers, so as to retain some learned features and prevent parameter overfitting; the weights of some convolutional layers close to the output end are kept updated, that is, unfreezes these convolutional layers, so that the model has stronger learning ability and continuously learns new features.

5. The pixel-level recognition method for bridge apparent defects based on instance segmentation according to claim 4 is characterized in that: The improved VGG16 model has a total of 5 convolution blocks, three convolution blocks close to the model input end are frozen, and two convolution blocks close to the model output end are unfrozen; the convolution blocks are separated by max-pooling connections.

6. The pixel-level recognition method for bridge apparent defects based on instance segmentation according to claim 2 is characterized in that: After completing the semi-automatic expansion of the original data set, the existing bridge surface images are enhanced and the training data set is constructed using the enhanced images.

7. A pixel-level recognition system for bridge surface defects based on instance segmentation, characterized by: The system is used to implement the pixel-level identification method for bridge surface defects according to any one of claims 1 to 6, and the system comprises: The bridge image upload module is used as an image upload port to collect the bridge surface image; A three-level pre-classification module, including a location recognition model, a component recognition model, and a defect recognition model, is used to gradually identify the captured location of the bridge surface image, the component where the defect is located, and the defect category; The pixel-level disease recognition module is used to identify the number, area, and contour shape of specific types of diseases; The classifier is used to classify and associate the bridge surface image based on the recognition results of the previous level model during the training and verification of the part recognition model, component recognition model, disease recognition model, and pixel-level recognition model, and call the corresponding next level model for recognition, completing the transmission of the bridge surface image between the models at each level; A database is used to store characteristic images of bridge surface information.

8. The bridge apparent disease pixel-level recognition system according to claim 7 is characterized in that: The system also includes a data enhancer, which is used to perform transformation and enhancement operations on the bridge appearance image to expand the number of bridge appearance images.

Images