Bridge disease identification method and system
By using YOLO image segmentation model and scale conversion technology, the existing bridge disease identification methods are solved, and efficient and accurate bridge disease identification is achieved, which is suitable for a fast-responsive bridge monitoring system.
Patent Information
- Application Number
- CN202411835049.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-18
AI Technical Summary
The existing bridge disease identification methods are inefficient, costly, unstable in recognition effects, and have problems such as high identification error and complex data processing procedures.
Using the YOLO image segmentation model, the model is trained to generate the output results of the disease segmentation mask, bounding box coordinates, confidence and category probability by collecting and labeling typical bridge disease disease data sets, and accurately mark the actual boundary range of the disease through scale conversion.
It significantly improves the speed and accuracy of bridge disease recognition, can process a large number of monitoring images in a short time, realize real-time or near-real-time disease recognition, reduces hardware resource requirements, and improves the overall coverage of disease recognition.
Smart Images
Figure CN119851084A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of bridge detection, and in particular to a bridge defect identification method and system. Background Art
[0002] With the rapid advancement of my country's transportation infrastructure construction, the safety of bridges as key transportation hubs has become increasingly important. In order to cope with the growing demand for scheduled bridge inspections and daily inspections and maintenance, as well as the ever-increasing requirements for safety and reliability, it is urgent to innovate traditional inspection methods.
[0003] At present, bridge structure inspection faces two major challenges: one is the low efficiency, strong subjectivity and great safety hazards of traditional manual inspection methods; the other is the high cost, unstable recognition effect and low efficiency of computer graphics and multimodal automatic recognition technology, which need to be overcome. The three mainstream bridge defect recognition methods are mainly the following: manual visual interpretation, which is inefficient, time-consuming and costly, and the manual inspection method is greatly affected by factors such as the environment and the professional and technical literacy of the inspectors, and the inspection results are uncertain; the deep learning convolutional neural network model requires a large amount of labeled data and high image quality, and is still susceptible to lighting and shooting angles (image correction is required), high recognition errors, low degree of technical integration, and complex data processing procedures; the multimodal fusion hybrid feature model extracts features of different dimensions through various means, and realizes defect recognition through feature fusion, but there are problems such as a single identifiable defect, a complex recognition process, low recognition accuracy, and high software and hardware costs.
[0004] Therefore, it is necessary to provide a bridge defect identification method and system to solve the problems in the prior art. Summary of the invention
[0005] In view of this, the present invention proposes a bridge defect identification method and system, aiming to solve the problems in the prior art.
[0006] In one aspect, the present invention provides a bridge disease identification method, comprising:
[0007] Collect the original typical bridge disease data set, use the general image annotation tool to annotate the disease images, mark the disease category, location and shape information, and use OBB to represent the disease area;
[0008] Divide the labeled data set into training set, validation set and test set, and make sure that diseases of different categories are evenly distributed in the training set, validation set and test set;
[0009] The training set is input into the YOLO image segmentation model for model training, the verification set is input into the trained model for model evaluation, and the test set is input into the trained model for actual model verification;
[0010] Set the scale of the disease image;
[0011] Input the bridge damage image and send it to the YOLO image segmentation model backbone network to generate the output results of the damage segmentation mask, damage bounding box coordinates, damage confidence and damage category probability;
[0012] The actual boundary range of the disease is converted by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the diseased area;
[0013] The bridge damage information is marked, wherein the bridge damage information includes damage category, confidence level, segmentation mask, actual length of the damage area, and actual area of the damage area.
[0014] Furthermore, the general image annotation tool includes:
[0015] LabelImg and VGG Image Annotator.
[0016] Furthermore, the loss function of the YOLO image segmentation model is:
[0017]
[0018] Among them, k represents the total number of pixels, y i represents the label (0 or 1) of the i-th pixel in the true mask, p i represents the probability that the model predicts that the pixel belongs to the target category, w i Represents the weight for each pixel or disease category.
[0019] Furthermore, when the validation set is input into the trained model to evaluate the model, the following steps are performed:
[0020] After the YOLO image segmentation model training is completed, the performance of the YOLO image segmentation model is evaluated by target detection and segmentation mask.
[0021] Furthermore, when the target detection evaluation and segmentation mask evaluation are performed on the performance of the YOLO image segmentation model, the target detection evaluation index includes Precision, Recall, mAP, mAP50 and mAP50-95; wherein,
[0022]
[0023] In the above formula, Precision represents the proportion of the bounding boxes predicted by the model that are correctly marked as a certain category, TP represents true positive examples, that is, the bounding boxes correctly predicted by the model, and FP represents false positive examples, that is, the bounding boxes incorrectly marked as a certain category by the model;
[0024]
[0025] In the above formula, Recall represents the proportion of bounding boxes of all actually existing categories that the model successfully finds, TP represents true positive examples, that is, bounding boxes correctly predicted by the model, and FN represents false negative examples, that is, actual bounding boxes that the model failed to correctly detect;
[0026] mAP is the average of APs of each category. AP is the integral under the Precision-Recall curve, which reflects the recall ability of the model at different precision levels.
[0027] mAP50 uses an IoU threshold of 0.50 for matching when calculating AP. Only when the IoU between the predicted bounding box and the true bounding box is greater than or equal to 0.5, the prediction is considered a true example.
[0028] mAP50-95 is to calculate the AP when the IoU threshold increases from 0.50 to 0.95 in steps of 0.05, and take the average value;
[0029] Segmentation mask evaluation indices include: Mask Precision, Mask Recall, Mean Average Precision for Masks, mAP50, and mAP50-95.
[0030] Furthermore, the setting of the scale of the disease image includes:
[0031] During the process of collecting bridge disease images, the shooting distance and angle of the bridge disease images remain unchanged;
[0032] Select a known length of ground feature or standard line on the bridge defect image to draw a straight line, and the system automatically obtains the number of pixels of the marked straight line length ScalePixels ;
[0033] Enter the actual length of the line. Input , length conversion is performed according to the number of scale pixels and actual distance of the mark, and pixel information is converted into actual length and area information;
[0034] Calculate the scale conversion factor Factor based on the number of pixels of the line and the actual length of the input Scale , the formula is as follows:
[0035]
[0036] In the above formula, length Input Indicates the actual length of the input, length ScalePixels The number of pixels representing the line.
[0037] Furthermore, when the input bridge damage image is sent to the YOLO image segmentation model backbone network to generate the damage segmentation mask, damage bounding box coordinates, damage confidence and damage category probability output results, it includes:
[0038] In the YOLO image segmentation model backbone network, the bridge damage image is divided into two branches: a target detection branch and a segmentation branch;
[0039] The object detection branch generates bounding box features, which include predictions of bounding box coordinates, objectness scores, confidence, and class probability predictions to determine the location and type of the disease;
[0040] The segmentation branch generates a segmentation feature map, which is processed by the Focal Softmax function to obtain a pixel-level segmentation mask to distinguish between diseased areas and non-diseased areas;
[0041] The results of the object detection branch and the segmentation branch are combined to generate the output results of disease segmentation mask, disease bounding box coordinates, disease confidence and disease category probability.
[0042] Furthermore, the actual boundary range conversion of the disease is realized by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the disease region, including:
[0043] The categories involved in the actual length statistics of the disease include: horizontal cracks, oblique cracks and vertical cracks. The actual length of the disease is calculated by the following formula: Actual :
[0044] Length Actual =length mask ·Factor scale ;
[0045] In the above formula, Length Actual Indicates the actual length of the disease, length mask Indicates the distance pixel of the target segmentation mask, Factor scale Represents the scale conversion factor.
[0046] Furthermore, when the actual boundary range of the disease is converted by scale conversion and the number of disease pixels is converted into the actual length of the disease or the actual area of the diseased region, it also includes:
[0047] The types of defects involving the actual area of the defect area include: network cracks, exposed rebars with cavities, exposed rebars with rust and expansion, damaged exposed rebars, honeycombs, pitting, damage and exposed anchor heads. The defect area is calculated by the following formula: Actual :
[0048]
[0049] In the above formula, Area Actual Indicates the actual area of the diseased area, Area mask represents the number of pixels of the target segmentation mask, Represents the square of the scale conversion factor.
[0050] Compared with the prior art, the beneficial effect of the present invention is that the YOLO image segmentation model adopted by the present invention can simultaneously predict the location and category of the target (disease) in a forward propagation process, greatly reducing the steps of regional proposal generation and subsequent classification / border regression in the traditional two-stage detection method (such as the R-CNN series), and significantly improving the detection speed. At the same time, the accurate boundary of the disease and the actual impact range (actual length or area of the disease) are finally marked, which is particularly important for the bridge monitoring system that requires rapid response. A large number of monitoring images can be processed in a short time to achieve real-time or near real-time disease identification. At the same time, the present invention has a lightweight application and a simple model structure, without complex pre-processing or post-processing steps, which simplifies the overall detection process. In addition, its model structure is relatively simple and has a small number of parameters, which is convenient for training and is conducive to deployment on edge devices, reducing hardware resource requirements and adapting to the lightweight trend of field detection equipment. There may be a situation where multiple types of diseases exist simultaneously on the surface of the bridge, forming a dense target distribution. The YOLO image segmentation model can better handle this target-intensive scene through a gridded prediction method, avoid missed detection, and improve the overall coverage of disease identification. The YOLO image segmentation model can not only locate the location of the disease, but also accurately segment the disease area and provide more detailed information on the disease morphology. This integrated detection and segmentation marking capability helps to assess the extent of the disease, accurately mark the actual length and area of the disease, and greatly improve the efficiency of disease detection and identification.
[0051] On the other hand, the present application also provides a bridge disease identification system, comprising:
[0052] The acquisition unit is used to collect the original bridge typical disease data set, use the general image annotation tool to accurately annotate the disease image, mark the disease category, location and shape information, and use OBB to represent the disease area;
[0053] The classification unit is used to divide the labeled data set into a training set, a validation set, and a test set, and to make the diseases of each category evenly distributed in the training set, the validation set, and the test set;
[0054] A model generation unit is used to input the training set into a YOLO image segmentation model for model training, input the verification set into the trained model for model evaluation, and input the test set into the trained model for actual model verification;
[0055] A scale setting unit, used to set the scale of the disease image;
[0056] The result output unit is used to input the bridge disease image and send it to the YOLO image segmentation model backbone network to generate the output results of the disease segmentation mask, disease boundary box coordinates, disease confidence and disease category probability;
[0057] A conversion unit, used to convert the actual boundary range of the disease through scale conversion, and convert the number of disease pixels into the actual length of the disease or the actual area of the disease region;
[0058] The disease marking unit is used to mark the bridge disease information, wherein the bridge disease information includes the disease category, confidence, segmentation mask, actual length of the diseased area, and actual area of the diseased area.
[0059] It is understandable that the bridge defect identification method and system provided in the present application have the same beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0061] Figure 1 A flow chart of a bridge defect identification method provided by an embodiment of the present invention;
[0062] Figure 2 A functional block diagram of a bridge defect identification system provided by an embodiment of the present invention;
[0063] Figure 3 A schematic diagram of classifying and labeling data using an OBB method provided by an embodiment of the present invention;
[0064] Figure 4 A schematic diagram of a scale setting provided in an embodiment of the present invention;
[0065] Figure 5This is a display diagram of the disease identification effect provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0066] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0067] In some embodiments of the present application, see Figure 1 As shown, this embodiment provides a bridge disease identification method, comprising the following steps:
[0068] S100, collect the original bridge typical disease data set, use the general image annotation tool to annotate the disease image, mark the disease category, location and shape information, and use OBB to represent the disease area;
[0069] S200, dividing the labeled data set into a training set, a validation set, and a test set, and making the diseases of each category evenly distributed in the training set, the validation set, and the test set;
[0070] S300, inputting the training set into a YOLO image segmentation model for model training, inputting the verification set into the trained model for model evaluation, and inputting the test set into the trained model for actual model verification;
[0071] S400, setting the scale of the disease image;
[0072] S500, input a bridge disease image, send it to the YOLO image segmentation model backbone network, and generate output results of disease segmentation mask, disease bounding box coordinates, disease confidence and disease category probability;
[0073] S600, converting the actual boundary range of the disease by scale conversion, converting the number of disease pixels into the actual length of the disease or the actual area of the disease region;
[0074] S700: Marking bridge damage information, wherein the bridge damage information includes damage category, confidence level, segmentation mask, actual length of the damage area, and actual area of the damage area.
[0075] It is understandable that the YOLO image segmentation model used in the present invention can simultaneously predict the location and category of the target (disease) during a forward propagation process, greatly reducing the steps of regional proposal generation and subsequent classification / border regression in the traditional two-stage detection method (such as the R-CNN series), and significantly improving the detection speed. At the same time, the accurate boundary of the disease and the actual impact range (actual length or area of the disease) are finally marked, which is particularly important for the bridge monitoring system that requires a fast response. A large number of monitoring images can be processed in a short time to achieve real-time or near real-time disease identification. At the same time, the present invention has a lightweight application and a simple model structure, without complex pre-processing or post-processing steps, which simplifies the overall detection process. In addition, its model structure is relatively simple and has a small number of parameters, which is convenient for training and is conducive to deployment on edge devices, reducing hardware resource requirements and adapting to the lightweight trend of field detection equipment. There may be a situation where multiple types of diseases exist simultaneously on the surface of the bridge, forming a dense target distribution. The YOLO image segmentation model can better handle this target-intensive scene through a gridded prediction method, avoid missed detection, and improve the overall coverage of disease identification. The YOLO image segmentation model can not only locate the location of the disease, but also accurately segment the disease area and provide more detailed information on the disease morphology. This integrated detection and segmentation marking capability helps to assess the extent of the disease, accurately mark the actual length and area of the disease, and greatly improve the efficiency of disease detection and identification.
[0076] In some embodiments of the present application, the general image annotation tool includes:
[0077] LabelImg and VGG Image Annotator.
[0078] In some embodiments of the present application, the loss function of the YOLO image segmentation model is:
[0079]
[0080] Among them, k represents the total number of pixels, y i represents the label (0 or 1) of the i-th pixel in the true mask, p i represents the probability that the model predicts that the pixel belongs to the target category, w i Represents the weight for each pixel or disease category.
[0081] In some embodiments of the present application, when the validation set is input into the trained model to evaluate the model, the process includes:
[0082] After the YOLO image segmentation model training is completed, the performance of the YOLO image segmentation model is evaluated by target detection and segmentation mask.
[0083] In some embodiments of the present application, when the target detection evaluation and segmentation mask evaluation are performed on the performance of the YOLO image segmentation model, the target detection evaluation index includes Precision, Recall, mAP, mAP50 and mAP50-95; wherein,
[0084]
[0085] In the above formula, Precision represents the proportion of the bounding boxes predicted by the model that are correctly marked as a certain category, TP represents true positive examples, that is, the bounding boxes correctly predicted by the model, and FP represents false positive examples, that is, the bounding boxes incorrectly marked as a certain category by the model;
[0086]
[0087] In the above formula, Recall represents the proportion of bounding boxes of all actually existing categories that the model successfully finds, TP represents true positive examples, that is, bounding boxes correctly predicted by the model, and FN represents false negative examples, that is, actual bounding boxes that the model failed to correctly detect;
[0088] mAP is the average of APs of each category. AP is the integral under the Precision-Recall curve, which reflects the recall ability of the model at different precision levels.
[0089] mAP50 uses an IoU threshold of 0.50 for matching when calculating AP. Only when the IoU between the predicted bounding box and the true bounding box is greater than or equal to 0.5, the prediction is considered a true example.
[0090] mAP50-95 is to calculate the AP when the IoU threshold increases from 0.50 to 0.95 in steps of 0.05, and take the average value;
[0091] Segmentation mask evaluation indices include: Mask Precision, Mask Recall, Mean Average Precision for Masks, mAP50, and mAP50-95.
[0092] Specifically, Mask Precision and Mask Recall in the segmentation mask evaluation index: In the instance segmentation task, the concepts of Precision and Recall are also applicable, except that they are for segmentation masks instead of bounding boxes. Precision measures the proportion of correctly classified pixels in the pixel mask predicted by the model; Recall measures the proportion of all actual category pixels successfully identified by the model. The calculation method is similar to Precision and Recall in object detection, except that the bounding box is replaced by a pixel-level mask match.
[0093] Mean Average Precision for Masks in Segmentation Mask Evaluation Index: Similar to mAP in object detection, mAP in instance segmentation is also calculated based on AP. However, the AP here is based on pixel-level matching, that is, comparing the IoU between the predicted mask and the true mask. mAP measures the average segmentation performance of the model on different categories.
[0094] mAP50 and mAP50-95 in the segmentation mask evaluation index: The definition is the same as that in object detection, except that the mAP calculation here is based on the segmentation mask instead of the bounding box. mAP50 and mAP50-95 reflect the average performance of the model on the segmentation mask at an IoU threshold of 0.50 and in the range from 0.50 to 0.95, respectively.
[0095] In some embodiments of the present application, the setting of the scale of the disease image includes:
[0096] During the process of collecting bridge disease images, the shooting distance and angle of the bridge disease images remain unchanged;
[0097] Select a known length of ground feature or standard line on the bridge defect image to draw a straight line, and the system automatically obtains the number of pixels of the marked straight line length ScalePixels ;
[0098] Enter the actual length of the line. Input , length conversion is performed according to the number of scale pixels and actual distance of the mark, and pixel information is converted into actual length and area information;
[0099] Calculate the scale conversion factor Factor based on the number of pixels of the line and the actual length of the input Scale , the formula is as follows:
[0100]
[0101] In the above formula, length Input Indicates the actual length of the input, length ScalePixels The number of pixels representing the line.
[0102] In some embodiments of the present application, when the input bridge damage image is sent to the YOLO image segmentation model backbone network to generate the damage segmentation mask, damage bounding box coordinates, damage confidence and damage category probability output results, it includes:
[0103] In the YOLO image segmentation model backbone network, the bridge damage image is divided into two branches: a target detection branch and a segmentation branch;
[0104] The object detection branch generates bounding box features, which include predictions of bounding box coordinates, objectness scores, confidence, and class probability predictions to determine the location and type of the disease;
[0105] The segmentation branch generates a segmentation feature map, which is processed by the Focal Softmax function to obtain a pixel-level segmentation mask to distinguish between diseased areas and non-diseased areas;
[0106] The results of the object detection branch and the segmentation branch are combined to generate the output results of disease segmentation mask, disease bounding box coordinates, disease confidence and disease category probability.
[0107] It is understandable that although the focus of image segmentation is on pixel-level classification, as a target detection model, the original bounding box prediction and confidence calculation parts of YOLOv8 are still important because they provide the location information of the target and help locate the specific location of the disease in the image.
[0108] Each grid cell predicts several bounding boxes, each of which is composed of a center coordinate offset (tx,ty), a width-to-height ratio (tw,th), and an absolute value of width and height (B w ,B h ). The actual bounding box coordinates B can be calculated using the following formula:
[0109] B x =σ(tx)+c x ;
[0110] B y =σ(ty)+c y ;
[0111] B w =p w e tw ;
[0112] B h =p h e th ;
[0113] Among them, c x 、c y is the grid cell center coordinate, p w and p h is the width and height of the grid unit, and σ is the activation function (usually the sigmoid function) used to map the predicted value to the (0, 1) interval.
[0114] Each bounding box predicts a confidence score, which indicates the degree to which the bounding box contains the object and the predicted box overlaps with the true box. The confidence score usually consists of two parts: the probability that the object exists in the bounding box (the objectness score) and a measure of the accuracy of the bounding box prediction (the localization error).
[0115] Confidence=Pr (Object) IoU (B,G) ;
[0116] Among them, Pr (Object) It is output by the sigmoid function, indicating whether there is a target in the grid unit; IoU (B,G) It is the intersection over union of the predicted box B and the true box G, which measures the prediction accuracy.
[0117] In the image segmentation model, a branch is added to predict the segmentation mask of the object to which each grid cell belongs. This branch contains an additional convolutional layer or decoder structure to generate a segmentation feature map of the same size as the input image. For each pixel, the segmentation feature map outputs a vector with a dimension equal to the number of target categories + 1 (background class). Through the Focal Softmax function, these values are converted into a category probability distribution, corresponding to the probability of the pixel belonging to each category. Finally, the category corresponding to the maximum probability is selected as the segmentation label of the pixel.
[0118] Focal Softmax is used to reduce the model's attention to easy-to-classify samples and increase the learning of difficult-to-classify samples, which is particularly useful for distinguishing similar or slightly different disease types.
[0119]
[0120] Among them, z j is the “preference” or confidence of the jth category, in the neural network z j Usually it is the activation value of the neuron corresponding to category j in the network output layer; Yes j exponential function, which means that the output of the Softmax function is always positive; γ is a hyperparameter that controls the degree of focus on difficult-to-classify samples, p j and p m Both use softmax to calculate the probability of a certain category.
[0121] In some embodiments of the present application, the actual boundary range conversion of the disease is realized by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the disease region, including:
[0122] The categories involved in the actual length statistics of the disease include: horizontal cracks, oblique cracks and vertical cracks. The actual length of the disease is calculated by the following formula: Actual :
[0123] Length Actual=length mask ·Factor scale ;
[0124] In the above formula, Length Actual Indicates the actual length of the disease, length mask Indicates the distance pixel of the target segmentation mask, Factor scale Represents the scale conversion factor.
[0125] In some embodiments of the present application, the actual boundary range conversion of the disease is realized by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the disease region, which also includes:
[0126] The types of defects involving the actual area of the defect area include: network cracks, exposed rebars with cavities, exposed rebars with rust and expansion, damaged exposed rebars, honeycombs, pitting, damage and exposed anchor heads. The defect area is calculated by the following formula: Actual :
[0127]
[0128] In the above formula, Area Actual Indicates the actual area of the diseased area, Area mask represents the number of pixels of the target segmentation mask, Represents the square of the scale conversion factor.
[0129] On the other hand, see Figure 2 As shown, the present application also provides a bridge defect identification system for applying the above bridge defect identification method, including:
[0130] The acquisition unit is used to collect the original bridge typical disease data set, use the general image annotation tool to accurately annotate the disease image, mark the disease category, location and shape information, and use OBB to represent the disease area;
[0131] The classification unit is used to divide the labeled data set into a training set, a validation set, and a test set, and to make the diseases of each category evenly distributed in the training set, the validation set, and the test set;
[0132] A model generation unit is used to input the training set into a YOLO image segmentation model for model training, input the verification set into the trained model for model evaluation, and input the test set into the trained model for actual model verification;
[0133] A scale setting unit, used to set the scale of the disease image;
[0134] The result output unit is used to input the bridge disease image and send it to the YOLO image segmentation model backbone network to generate the output results of the disease segmentation mask, disease boundary box coordinates, disease confidence and disease category probability;
[0135] A conversion unit, used to convert the actual boundary range of the disease through scale conversion, and convert the number of disease pixels into the actual length of the disease or the actual area of the disease region;
[0136] The disease marking unit is used to mark the bridge disease information, wherein the bridge disease information includes the disease category, confidence, segmentation mask, actual length of the diseased area, and actual area of the diseased area.
[0137] Specifically, in order to ensure the feasibility of disease detection, the present invention merges some diseases. A large number of disease images (more than 1500 images) were sorted from the historical data of bridge inspection. After sorting, adjusting and merging, there are 11 types of diseases, including:
[0138] Cracks, including horizontal cracks, oblique cracks, vertical cracks, and reticular cracks;
[0139] Exposed rebar, including damaged exposed rebar, rusted and swollen exposed rebar, and hollow exposed rebar;
[0140] Others include damage, honeycomb, pockmarks, and exposed anchor heads.
[0141] After the disease category is determined, the disease category, shape, and location of all images need to be accurately labeled.
[0142] Disease identification model training:
[0143] Considering the universality and scalability of data annotation, the present invention uses the image annotation tool X-Anylabeling to perform Yolo format data annotation (other general image annotation tools are also acceptable), and adopts OBB (Oriented Bounding Box) to record category labeled data, which is convenient for subsequent image recognition and image segmentation operations. Figure 3 .
[0144] In order to ensure the image segmentation modeling effect, the disease sample library requires that various disease images be as balanced as possible and the disease characteristics are obvious. At the same time, combined with the existing historical data, a total of 1320 labeled images were sorted out, including 11 disease categories and 1567 labeled spots. The statistical information of the labeled data is shown in Table 1.
[0145] Table 1
[0146]
[0147] Through actual disease data verification, the present invention further verifies the bridge disease recognition model based on Yolo image segmentation. The model evaluation results are shown in Table 2.
[0148] Table 2
[0149]
[0150]
[0151] Note: Precision means accuracy, Recall means recall rate, and mAP is the average value of AP of each category (meanAverage Precision);
[0152] mAP50 refers to the use of an IoU threshold of 0.50 for matching when calculating AP; mAP50 reflects the performance of the model under relatively loose matching conditions, and mAP50-95 requires the model to maintain good performance under a higher IoU standard. Therefore, it is a more stringent evaluation indicator that can better reflect the overall performance of the model under different accuracy requirements.
[0153] mAP50-95 refers to the AP under the IoU threshold of 0.50-0.95 (usually increasing in steps of 0.05) when calculating AP, and taking the average value of each category;
[0154] The following main analysis conclusions can be drawn from the table:
[0155] The overall evaluation results of the model show that the target detection mAP50 is 0.632 and the instance segmentation mAP50 is 0.638. Compared with the image segmentation evaluation results of the COCO dataset officially released by ultralytics / yolov8, it is found that the model achieves good results in both target detection and instance segmentation.
[0156] From the evaluation results of various defects (mAP50), it can be seen that the recognition effects of the five types of defects, namely oblique cracks, mesh cracks, vertical cracks, horizontal cracks and damage, are very good; the recognition effects of the four types of defects, namely rusted and swollen exposed rebar, damaged exposed rebar, honeycomb and pockmarked surface, are second; the effects of hollow exposed rebar and exposed anchor head are poor.
[0157] From the evaluation results of Precision and Recall, it can be seen that various types of cracks (oblique cracks, mesh cracks, vertical cracks, horizontal cracks) and damage diseases can be effectively detected and rarely misjudgment occurs; various types of exposed rebar diseases (rusted and swollen exposed rebar, damaged exposed rebar, hollow exposed rebar) have a certain degree of missed detection and misjudgment; honeycomb diseases can basically be detected but misjudgment occurs frequently; pitting diseases are prone to missed detection; and there are many cases of missed detection and misjudgment in the identification of exposed anchor heads.
[0158] In summary, the defect recognition scheme based on YOLO image segmentation technology proposed in this invention fully meets the requirements of actual engineering applications for defect detection systems by virtue of its real-time and efficient defect recognition capability, flexible and extensive adaptability to defect types, and convenience in reducing pre-processing steps. This method provides a scientific and practical solution for the intelligent identification of bridge concrete defects and other structural defects.
[0159] Scale settings:
[0160] First, load the disease image through the system interface, draw a straight scale at the standard line position on the image (if the length is known, use the line on your finger to draw the scale), and enter the actual length of the scale (for example, set the length unit: cm), such as Figure 4 The system automatically calculates the conversion factor. scale .
[0161] Disease identification marks:
[0162] The method of the present invention is actually verified on the disease test set (264 pictures, 330 disease marks), and the disease recognition effect is shown as follows Figure 5 As shown, Figure 5 From left to right, from top to bottom: 1. Honeycomb FW, confidence level 0.86, actual area of disease 34.65cm 2 2. Web-like cracks WZLF, confidence level 0.81, actual area of disease 56.45 cm 2 ; 3. Rusted and swollen exposed reinforcement XZLJ, confidence level 0.84, actual area of disease 21.76cm 2 ; 4. Web-like cracks WZLF, confidence level 0.74, actual area of disease 53.11cm 2 .
[0163] The categories that mark the actual length of the defects are: horizontal cracks, oblique cracks, and vertical cracks.
[0164] The categories that mark the actual area of defects include: mesh cracks, exposed rebar due to cavities, exposed rebar due to rust and expansion, damaged exposed rebar, honeycomb, pockmarks, breakage, and exposed anchor heads.
[0165] The disease identification results are marked with the following information: disease type, confidence, bounding box, actual range, actual length or area.
[0166] The recognition results show that the bridge defect recognition method has extremely high recognition efficiency and accuracy in identifying the exact location of common bridge defects, with low omissions and repeated recognitions. It can greatly improve the current work efficiency of bridge defect recognition and has great value for promotion and application.
[0167] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0168] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A bridge disease identification method, characterized in that: include: Collect the original typical bridge disease data set, use the general image annotation tool to annotate the disease images, mark the disease category, location and shape information, and use OBB to represent the disease area; Divide the labeled data set into training set, validation set and test set, and make sure that diseases of different categories are evenly distributed in the training set, validation set and test set; The training set is input into the YOLO image segmentation model for model training, the verification set is input into the trained model for model evaluation, and the test set is input into the trained model for actual model verification; Set the scale of the disease image; Input the bridge damage image and send it to the YOLO image segmentation model backbone network to generate the output results of the damage segmentation mask, damage bounding box coordinates, damage confidence and damage category probability; The actual boundary range of the disease is converted by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the diseased area; The bridge damage information is marked, wherein the bridge damage information includes damage category, confidence level, segmentation mask, actual length of the damage area, and actual area of the damage area.
2. The bridge disease identification method according to claim 1, characterized in that: The general image annotation tool comprises: Label Img and VGG Image Annotator.
3. The bridge disease identification method according to claim 2, characterized in that: The loss function of the YOLO image segmentation model is: Among them, k represents the total number of pixels, y i represents the label (0 or 1) of the i-th pixel in the true mask, p i represents the probability that the model predicts that the pixel belongs to the target category, w i Represents the weight for each pixel or disease category.
4. The bridge disease identification method according to claim 3 is characterized in that: When the validation set is input into the trained model to evaluate the model, it includes: After the YOLO image segmentation model training is completed, the performance of the YOLO image segmentation model is evaluated by target detection and segmentation mask.
5. The bridge disease identification method according to claim 4, characterized in that: When the target detection evaluation and segmentation mask evaluation are performed on the performance of the YOLO image segmentation model, the target detection evaluation index includes Precision, Recall, mAP, mAP50 and mAP50-95; wherein, In the above formula, Precision represents the proportion of the bounding boxes predicted by the model that are correctly marked as a certain category, TP represents true positive examples, that is, the bounding boxes correctly predicted by the model, and FP represents false positive examples, that is, the bounding boxes incorrectly marked as a certain category by the model; In the above formula, Recal l represents the proportion of bounding boxes of all actually existing categories that the model successfully finds, TP represents true positive examples, that is, bounding boxes correctly predicted by the model, and FN represents false negative examples, that is, actual bounding boxes that the model failed to correctly detect; mAP is the average value of AP in each category. AP is the integral under the Precision-Recal l curve, which reflects the recall ability of the model at different precision levels. mAP50 uses an IoU threshold of 0.50 for matching when calculating AP. Only when the IoU between the predicted bounding box and the true bounding box is greater than or equal to 0.5, the prediction is considered a true example. mAP50-95 is to calculate the AP when the IoU threshold increases from 0.50 to 0.95 in steps of 0.05, and take the average value; Segmentation mask evaluation indices include: Mask Precision, Mask Recall, Mean Average Precision for Masks, mAP50, and mAP50-95.
6. The bridge defect identification method according to claim 5, characterized in that: The setting of the scale of the disease image includes: During the process of collecting bridge disease images, the shooting distance and angle of the bridge disease images remain unchanged; Select a known length of ground feature or standard line on the bridge defect image to draw a straight line, and the system automatically obtains the number of pixels of the marked straight line length ScalePixels ; Enter the actual length of the line. Input , length conversion is performed according to the number of scale pixels and actual distance of the mark, and pixel information is converted into actual length and area information; Calculate the scale conversion factor Factor based on the number of pixels of the line and the actual length of the input Scale , the formula is as follows: In the above formula, length Input Indicates the actual length of the input, length ScalePixels The number of pixels representing the line.
7. The bridge defect identification method according to claim 6, characterized in that: The input bridge damage image is sent to the YOLO image segmentation model backbone network to generate the damage segmentation mask, damage bounding box coordinates, damage confidence and damage category probability output results, including: In the YOLO image segmentation model backbone network, the bridge damage image is divided into two branches: a target detection branch and a segmentation branch; The object detection branch generates bounding box features, which include predictions of bounding box coordinates, objectness scores, confidence, and class probability predictions to determine the location and type of the disease; The segmentation branch generates a segmentation feature map, which is processed by the Focal Softmax function to obtain a pixel-level segmentation mask to distinguish between diseased areas and non-diseased areas; The results of the object detection branch and the segmentation branch are combined to generate the output results of disease segmentation mask, disease bounding box coordinates, disease confidence and disease category probability.
8. The bridge disease identification method according to claim 7, characterized in that: The actual boundary range conversion of the disease is realized by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the disease region, including: The categories involved in the actual length statistics of the disease include: horizontal cracks, oblique cracks and vertical cracks. The actual length of the disease is calculated by the following formula: Actual : Length Actual =length mask ·Factor scale ; In the above formula, Length Actual Indicates the actual length of the disease, length mask Indicates the distance pixel of the target segmentation mask, Factor scale Represents the scale conversion factor.
9. The bridge disease identification method according to claim 8, characterized in that: When the actual boundary range of the disease is converted by scale conversion, and the number of disease pixels is converted into the actual length of the disease or the actual area of the diseased area, it also includes: The types of defects involving the actual area of the defect area include: network cracks, exposed rebars with cavities, exposed rebars with rust and expansion, damaged exposed rebars, honeycombs, pitting, damage and exposed anchor heads. The defect area is calculated by the following formula: Actual : In the above formula, Area Actual Indicates the actual area of the diseased area, Area mask represents the number of pixels of the target segmentation mask, Represents the square of the scale conversion factor.
10. A bridge defect identification system, used for applying the bridge defect identification method according to any one of claims 1 to 9, characterized in that: include: The acquisition unit is used to collect the original bridge typical disease data set, use the general image annotation tool to accurately annotate the disease image, mark the disease category, location and shape information, and use OBB to represent the disease area; The classification unit is used to divide the labeled data set into a training set, a validation set, and a test set, and to make the diseases of each category evenly distributed in the training set, the validation set, and the test set; A model generation unit is used to input the training set into a YOLO image segmentation model for model training, input the verification set into the trained model for model evaluation, and input the test set into the trained model for actual model verification; A scale setting unit, used to set the scale of the disease image; The result output unit is used to input the bridge disease image and send it to the YOLO image segmentation model backbone network to generate the output results of the disease segmentation mask, disease boundary box coordinates, disease confidence and disease category probability; A conversion unit, used to convert the actual boundary range of the disease through scale conversion, and convert the number of disease pixels into the actual length of the disease or the actual area of the disease region; The disease marking unit is used to mark the bridge disease information, wherein the bridge disease information includes the disease category, confidence, segmentation mask, actual length of the diseased area, and actual area of the diseased area.