Dam apparent defect intelligent identification method

Abstract

The invention provides a dam apparent defect intelligent identification method, and belongs to the field of computer vision target detection. In order to solve the problem that dike defect recognition of a general target detection model is not accurate enough, a global attention mechanism and a boundary intersection-to-union ratio loss function are introduced into a YOLOv7 network, and the effects of improving the dike defect detection precision, reducing the omission ratio and accurately distinguishing two types of defects of dike apparent cracks and apparent pits are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision target detection, and in particular to a method for intelligently identifying apparent defects in dams. Background Art

[0002] China has an extremely large levee system. During the flood season, leakage, piping, collapse, and even breaches are very likely to occur in weak or hidden danger locations of the levees. Apparent defects such as cracks and pits in the levees are usually external manifestations of the development of hidden dangers and diseases inside the levees. Therefore, efficient and accurate identification of apparent defects in levees is a key task in the inspection of levees and flood season inspections. Traditional identification of apparent defects in levees is mainly done manually.

[0003] In recent years, thanks to the rapid development of technologies such as drones and computer vision, the integration of the two for detecting defects in engineering structures has become a research hotspot. YOLOv7 is an advanced real-time target detection model consisting of three main parts: backbone, neck, and detection head. These parts work together to achieve efficient feature extraction and accurate target detection. However, the YOLOv7 network is designed for target detection of general objects, and its direct application in the embankment defect recognition scenario based on drone images will face many challenges. For example, affected by weather conditions and drone flight morphology and altitude, the original YOLOv7 network does not pay enough attention to small defect targets, and there are many missed detections and false detections. How to develop a target detection model suitable for embankment engineering is still an unresolved problem.

[0004] Therefore, it is urgent to provide a solution to improve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent recognition method for dam surface defects, which can solve the problem that the recognition of dam defects by general target detection models is not accurate enough.

[0006] The present invention provides a method for intelligently identifying apparent defects of a dam, comprising the following steps:

[0007] After acquiring and preprocessing the dam defect images, a training set is constructed;

[0008] The initial model was obtained by integrating the global attention mechanism into the ELAN-2 layer and the ELAN-3 layer of the YOLOv7 network backbone and replacing the localization loss function in the YOLOv7 network detection head with the boundary intersection-over-union loss function.

[0009] After training the initial model based on the training set, a dam surface defect recognition model is obtained;

[0010] Based on the dam apparent defect recognition model, the dam defect image is recognized to obtain the recognition result. The intelligent recognition method provided by the present invention, by introducing the global attention mechanism and the boundary intersection-over-union loss function into the YOLOv7 network, achieves the effect of improving the dam defect detection accuracy, reducing the missed detection rate, and accurately distinguishing the two types of defects of the dam apparent cracks and apparent pits.

[0011] Optionally, when acquiring the dam defect image, the dam defect image is a high-definition image with cracks and pits.

[0012] Optionally, when preprocessing the dam defect image, after the dam defect image is cleaned, data annotation is performed on the dam defect position and the dam defect type on the dam defect image.

[0013] Optionally, when the dam defect image is cleaned, image data in the dam defect image that cannot be used to identify the dam defect position and / or the dam defect type is removed.

[0014] Optionally, after the initial model is obtained, the initial model includes four parts: an input end, a backbone, a neck and a detection end.

[0015] Optionally, when training the initial model based on the training set, iterative training is performed based on the loss function value in the initial model. When the loss functions in the initial model converge and the sum of the loss function values ​​is minimized, a dam apparent defect recognition model is obtained and the weight data in the dam apparent defect recognition model is saved; the loss function includes a boundary intersection-over-union loss function, a bounding box prediction loss function, and a type prediction loss function.

[0016] Optionally, when the initial model is trained based on the training set, the boundary intersection-over-union loss function in the initial model is used to calculate the vector angle between the defect prediction position and the defect annotation position, and the angle cost, distance cost, shape cost and intersection-over-union cost are set as penalty indicators to adjust the weight of the initial model. Optionally, when the dam apparent defect recognition model is obtained after the initial model is trained based on the training set, the training set is randomly divided into training data and verification data, the initial model is trained based on the training data, and the performance of the trained initial model and hyper-parameter adjustment are performed based on the verification data.

[0017] Optionally, when the performance of the trained initial model is evaluated based on the verification data, the evaluation indicators include precision, recall, the harmonic mean of precision and recall, single-type average precision and multi-type average precision.

[0018] Optionally, after the dam defect image is identified based on the dam apparent defect identification model to obtain the identification result, the identification result includes the apparent defect type and the apparent defect position. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Construct a process map for the levee apparent defect training set;

[0020] Figure 2 Schematic diagram of angle, distance and shape cost for boundary intersection-over-union loss function;

[0021] Figure 3 The network structure diagram of YOLOv7 and GS-YOLOv7;

[0022] Figure 4 This is the network structure diagram of the key modules of GS-YOLOv7;

[0023] Figure 5 This is a graph showing the change in loss function and evaluation index during training;

[0024] Figure 6 YOLOv7 performance radar chart and parameter bar chart for applying different improvement strategies;

[0025] Figure 7 Detect model performance radar charts and parameter bar charts for different targets;

[0026] Figure 8 The detection results of YOLOv7 and GS-YOLOv7 are shown in Figure 2.

[0027] Fig. 9 Grad-CAM visualization of the model’s reasoning graph. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the usual meanings understood by people with general skills in the field to which the present invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. The present invention provides a method for intelligent identification of apparent defects in dams, comprising the following steps:

[0029] S1, after obtaining and preprocessing the dam defect images, construct a training set;

[0030] S2. Integrate the global attention mechanism into the ELAN-2 and ELAN-3 layers of the YOLOv7 network backbone

[0031] (Global Attention Mechanism, GAM), and replace the positioning loss function in the YOLOv7 network detection head with the boundary intersection loss function (Scylla Intersection over Union, SIoU) to obtain the initial model (GS-YOLOv7);

[0032] S3, after training the initial model based on the training set, a dam surface defect recognition model is obtained;

[0033] S4, identifying the dam defect image based on the dam apparent defect recognition model to obtain a recognition result. In fact, when executing step S1, if Figure 1 As shown, a drone equipped with a gimbal camera is controlled to take aerial photos, and two types of high-definition dam defect image data, namely, apparent cracks on the dam and apparent pits on the dam, are obtained by flying along the dam; the dam defect image data are cleaned, and image data in the dam defect image that cannot be used to identify the dam defect position and / or the dam defect type are removed; labels are a prerequisite for building a semi-supervised learning model, and the obtained image data itself does not have a label file. The label platform Makesense AI is used to annotate the dam defect position and dam defect type on the dam defect image, generate the corresponding .txt label file, and construct a training data set;

[0034] In fact, when executing step S2, a global attention mechanism is introduced into the YOLOv7 network. The structure of GAM is similar to that of the Convolutional Block Attention Module (CBAM), which is composed of a channel attention module (CAM) and a spatial attention module (SAM) in series. In order to solve the problem of cross-dimensional information loss caused by ignoring the interaction between channels and spaces in CBAM, GAM redesigns CAM and SAM to amplify the interactive features of global cross-dimensional channels and spaces, thereby improving the network's attention to defects and its ability to capture global information.

[0035] From the analysis of the YOLOv7 network structure, the size of the feature map of each layer gradually decreases along the path of the backbone network, and the secondary semantic information will also be gradually diluted; it can be seen that the effect of improving network performance by increasing the attention weight at the back of the backbone network is limited; from the perspective of the position of adding the GAM module, multiple comparative experiments were conducted in actual operation, that is, the combination of adding GAM modules at different positions, and finally it was found that the integration of the GAM module after the ELAN-1 and ELAN-2 of the network backbone can obtain the best detection effect and give full play to the role of the GAM module. In fact, the structures of ELAN-1, ELAN-2, ELAN-3 and ELAN-4 are the same;

[0036] In the YOLOv7 network, the localization loss function of the original three-scale detection head in the detection end is replaced by the complete intersection over Union (CIoU) loss function to the boundary intersection over Union (IoU) loss function. The boundary intersection over Union loss function considers the vector angle between the predicted box and the true value box, and sets the angle cost, distance cost, shape cost and intersection over Union (IoU) cost as penalty indicators to adjust the weight of the initial model. The calculation formula is as follows: (1) Angle cost Λ

[0037]

[0038] in, and They are the horizontal and vertical coordinates of the center point of the prediction box respectively; and The ratio is the horizontal and vertical coordinates of the center point of the true value box, such as Figure 2 (a)

[0039] shown.

[0040] (2) Distance cost Δ

[0041]

[0042] Where: cw 2 and c h2 are the width and height of the minimum bounding rectangle that can contain both the prediction box and the true value box, respectively. Figure 2 (b) as shown.

[0043] (3) Shape cost Ω

[0044]

[0045] Where: θ is the control coefficient, It determines the attention paid to the shape cost. The geometric meaning of |w-wgt| and |h-hgt| is as follows Figure 2 (c) as shown.

[0046] Finally, by combining equations (2) and (3), we can get the calculation formula of the boundary intersection-over-union loss function:

[0047]

[0048] Where: BI BGT| is the intersection of the predicted box and the true value box, and |BUBGT| is the union of the predicted box and the true value box.

[0049] like Figure 3 As shown in Figure 1, the GS-YOLOv7 network consists of four parts: input end, backbone, neck and detection end, and the inference end is the application effect; Figure 4 As shown in the figure, the backbone of the GS-YOLOv7 network consists of CBS layer, ELAN layer and MP layer. The CBS layer includes ordinary convolution, batch normalization and sigmoid linear unit (SiLU) activation function, and has three different forms of structure: CBS-1, CBS-2, CBS-3. The ELAN layer is one of the highlights of the YOLOv7 network improvement. It greatly improves the learning ability of the network by controlling the gradient path, allowing the network to learn more diverse features. The ELAN layer has two forms of structure: ELAN and ELAN-W. The MP layer adopts a two-branch structure, adding a maximum pooling layer on the basis of the CBS convolution layer, and then using the Concat splicing operation to perform feature fusion on the branch structure. The MP layer has two forms of structure, named MP-1 layer and MP-2 layer, the difference between the two is the number of channels.

[0050] In fact, when executing step S3, the training set is randomly divided into training data and verification data, the initial model is trained based on the training data, and the performance of the trained initial model and hyper-parameter adjustment are performed based on the verification data;

[0051] When the initial model is trained based on the training set, iterative training is performed based on the loss function value in the initial model. When the loss functions in the initial model converge and the sum of the loss function values ​​is minimized, a dam apparent defect recognition model is obtained and weight data in the dam apparent defect recognition model is saved; the loss function includes a boundary intersection loss function, a boundary box prediction loss function, and a type prediction loss function;

[0052] When the performance of the trained initial model is evaluated based on the verification data, the evaluation indicators include precision, recall, and the harmonic mean of precision and recall (F 1 -score), single-type average precision (Average Precision, AP) and multi-type average precision (Mean Average Precision, mAP), the calculation formula is as follows:

[0053] 1) Precision and Recall

[0054]

[0055] In the formula, TP is the number of defective samples that are correctly identified; FP is the number of non-defective samples that are incorrectly identified as defective; FN is the number of missing defective samples. The higher the precision, the higher the accuracy of the detection result, and the higher the recall, the fewer targets are missed during the detection process.

[0056] 2) F 1 -score

[0057]

[0058] Precision and recall are two mutually constrained indicators. Using either indicator alone cannot accurately evaluate network performance. The harmonic mean of precision and recall takes both indicators into account and can measure network performance more scientifically. The larger the value, the better the model performance.

[0059] 3) AP and mAP

[0060] The single-type average precision is the area enclosed by the PR curve drawn with R as the horizontal coordinate and P as the vertical coordinate and the coordinate axis.

[0061]

[0062] The single-type average precision represents the average precision of a category. When measuring the average precision of all categories in the network, the multi-type average precision should be used.

[0063]

[0064] Where N is the number of categories. It should be noted that mAP@0.5 represents the calculated value when IoU is equal to 0.5, and mAP@0.5:0.95 represents the average calculated value when IoU is 0.5 to 0.95 (with a step size of 0.05). The larger the multi-type average precision, the better the network performance;

[0065] When training the YOLOv7 model, YOLOv7+GAM model, YOLOv7+SIoU model, and GS-YOLOv7 model, the loss functions Box_Loss, Cla_Loss, Obj_Loss, and the evaluation indicators mAP@0.5 and mAP@0.5:0.95 change with the number of model iterations as follows:

[0066] like Figure 5 As shown in (a), the loss function Box_Loss curve gradually shows a downward trend with the increase of iteration number and stabilizes after the 200th iteration. From the performance of Box_Loss on the training set, it can be seen that the network convergence speed slows down after adding the GAM module. Analysis shows that this is caused by the increase of network complexity, but thanks to the addition of the boundary intersection loss function, the network convergence speed can be partially restored.

[0067] Figure 5 The loss function Cla_Loss curve in (b) converges quickly and converges to a basically stable state after the 100th iteration. At the same time, Cla_Loss loss has a good performance in both the training set and the validation set, which shows that the characteristics of the two types of defects are quite different, and the network has the ability to accurately classify the two types of defects.

[0068] Figure 5 The performance of the Obj_Loss curve in (c) on the training set is Figure 5 The similarity in (a) further illustrates that the addition of boundary intersection-over-union loss function enhances the network’s target localization capability.

[0069] Figure 5 As shown in (d), although the mAP value of the network after adding the GAM module increases more slowly than the original YOLOv7 network, the mAP value of the combined two improved methods surpasses other networks during the network convergence process, and finally achieves the highest mAP@0.5 and mAP@0.5:0.95. This once again proves the effectiveness of the improvement strategy, and the above-mentioned improvement method can improve the detection performance of the network.

[0070] To verify the effectiveness of introducing GAM and SIoU into the YOLOv7 network at the same time, Figure 6 As shown, the detection performance of the YOLOv7 model, YOLOv7+GAM model, YOLOv7+SIoU model and GS-YOLOv7 model with the optimal weights is evaluated. The experimental results are shown in the following table:

[0071] Table 1 Ablation experiment results of adding improved modules

[0072]

[0073] Note: In the table, YOLOv7 is used as the baseline network for comparison, and those with better performance indicators are shown in bold.

[0074] By analyzing the experimental results in Table 1, we can see that compared with the original YOLOv7 network, whether adding the GAM module, SIoU module alone, or combining the two modules, these improvement strategies have enhanced the detection performance of the network to varying degrees. On the one hand, the biggest benefit of introducing the GAM module is the improvement of the detection accuracy of the model, which is due to the fact that the GAM module effectively improves the network's attention to defects and enhances the network's ability to capture global information; on the other hand, the benefits brought by the introduction of the SIoU module are mainly to reduce the network's missed detection rate and improve the F 1 -score;

[0075] Combining the above improvement strategies, the two modules are combined into the original YOLOv7 network to develop a dam defect target detection network suitable for drone aerial images. This network enhances the feature extraction capability of the backbone network, improves the bounding box positioning capability of the detection end, and has excellent performance on the dam defect dataset: Compared with the original YOLOv7 network, the method proposed in this patent has a significant improvement in Precision, Recall, and F. 1 -score increased by 2.4%, 1.5% and 1.9% respectively; mAP@0.5 and mAP@0.5:0.95 increased by 1.0% and 1.7% respectively;

[0076] Comprehensive analysis shows that compared with other target detection networks, GAM and SIoU are introduced into the YOLOv7 network at the same time, and it is determined that the GS-YOLOv7 network has better detection performance in dam defect identification;

[0077] To further verify the superiority of the GS-YOLOv7 network, Figure 7 As shown, the detection performance of FasterRCNN, YOLOv5, YOLOR, YOLOX, YOLOv7 and GS-YOLOv7 with the optimal weights is evaluated. The experimental results are shown in the following table:

[0078] Table 2 Comparative test results of different target detection networks

[0079]

[0080] Note: In the table, YOLOv7 is used as the baseline network for comparison, and those with better performance indicators are shown in bold.

[0081] By analyzing the experimental results in Table 2, it can be seen that the GS-YOLOv7 network shows many advantages compared with other target detection networks. Compared with the original YOLOv7 network, all performance indicators of the GS-YOLOv7 network have been greatly improved, but the number of model parameters has only increased by 17.4M. Although the recall rate Recall and mAP@0.5 of the Faster RCNN network have increased by 14.7% and 6.8% respectively, and are even better than GS-YOLOv7 in some indicators, its model parameters are too large, which is not conducive to later deployment on mobile devices. The performance of the YOLOv5 network in all aspects is not as good as the original YOLOv7 network, and it is slightly insufficient in the needs of embankment defect detection. The YOLOR network and the YOLOX network have better performance than GS-YOLOv7 in recall rate Recall and precision Precision, respectively, but they lack significant disadvantages in other aspects. For example, the precision of YOLOR is low, the recall rate Recall of YOLOX is too low, only 0.34, and the comprehensive index F of the two is also low. 1 -score is also lower than GS-YOLOv7. According to Table 1, radar charts and parameter bar charts of different target detection networks are drawn to intuitively show the performance differences of various networks, such as Figure 7 Comprehensive analysis shows that compared with other target detection networks, the GS-YOLOv7 network has better detection performance on the levee defect dataset. At the same time, the number of parameters of the improved model does not increase much, which is conducive to later deployment on mobile devices.

[0082] In fact, when executing step S4, if Figure 8 As shown, based on the dam apparent defect recognition model, the dam defect image is recognized to obtain the recognition result, and the recognition result includes the apparent defect type and the apparent defect position. Compared with the original YOLOv7 network, the GS-YOLOv7 network gives a more accurate prediction box position and can accurately locate the defect. At the same time, both pits and cracks have obtained higher confidence values, and the network's recognition ability for defects has been greatly improved. This shows that the recognition accuracy of the target detection model of the invention is better than that of the detection model, thereby verifying the rationality and scientificity of the invention;

[0083] like Fig. 9As shown in the figure, the network is visualized using Gradient-weighted Class Activation Mapping (Grad-CAM). Compared with the original YOLOv7 network, the area of ​​interest of the GS-YOLOv7 network is significantly reduced, and the area of ​​interest is concentrated on the hardened road surface of the embankment. It is worth noting that the original YOLOv7 network also allocated more attention to crack-free defect areas such as the sky and the embankment slope protection. This problem has been improved in the GS-YOLOv7 network, further proving the effectiveness of the improved strategy.

[0084] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A method for intelligently identifying apparent defects of dams, characterized by: After acquiring and preprocessing the dam defect images, a training set is constructed; The initial model was obtained by integrating the global attention mechanism into the ELAN-2 layer and the ELAN-3 layer of the YOLOv7 network backbone and replacing the localization loss function in the YOLOv7 network detection head with the boundary intersection-over-union loss function. After the initial model is trained based on the training set, a dam apparent defect recognition model is obtained; based on the dam apparent defect recognition model, the dam defect image is recognized to obtain a recognition result.

2. The method for intelligently identifying apparent defects of dams according to claim 1, characterized in that: When acquiring the dam defect image, the dam defect image is a high-definition image with cracks and pits.

3. The method for intelligently identifying apparent defects of dams according to claim 1, characterized in that: When preprocessing the dam defect image, after the dam defect image is cleaned, data annotation is performed on the dam defect position and the dam defect type on the dam defect image.

4. The method for intelligently identifying apparent defects of dams according to claim 3, characterized in that: When the dam defect image is cleaned, image data in the dam defect image that cannot be used to identify the dam defect position and / or the dam defect type is removed.

5. The method for intelligently identifying apparent defects of dams according to claim 1, characterized in that: After the initial model is obtained, the initial model includes four parts: an input end, a backbone, a neck and a detection end.

6. The method for intelligently identifying apparent defects of dams according to claim 1, characterized in that: When training the initial model based on the training set, iterative training is performed based on the loss function value in the initial model. When the loss functions in the initial model converge and the sum of the loss function values ​​is minimized, a dam apparent defect recognition model is obtained and the weight data in the dam apparent defect recognition model is saved. The loss function includes a boundary intersection-union ratio loss function, a bounding box prediction loss function, and a type prediction loss function.

7. The method for intelligently identifying apparent defects of dams according to claim 6, characterized in that: When the initial model is trained based on the training set, the boundary intersection-over-union loss function in the initial model is used to calculate the vector angle between the defect prediction position and the defect annotation position, and the angle cost, distance cost, shape cost and intersection-over-union cost are set as penalty indicators to adjust the weight of the initial model.

8. The method for intelligently identifying apparent defects of dams according to claim 1, characterized in that: When the dam apparent defect recognition model is obtained after the initial model is trained based on the training set, the training set is randomly divided into training data and verification data, the initial model is trained based on the training data, and the performance of the trained initial model and hyper-parameter adjustment are performed based on the verification data.

9. The method for intelligently identifying apparent defects of dams according to claim 8, characterized in that: When the performance of the trained initial model is evaluated based on the verification data, the evaluation indicators include precision, recall, the harmonic mean of precision and recall, single-type average precision and multi-type average precision.

10. The method for intelligently identifying apparent defects of dams according to claim 1, characterized in that: After the dam defect image is identified based on the dam apparent defect identification model to obtain the identification result, the identification result includes the apparent defect type and the apparent defect position.

Images