Great wall damaged area detection method based on aerial photography

Abstract

The invention relates to a great wall damaged area detection method based on aerial photography, and belongs to the technical field of image detection, and the method comprises the steps: collecting great wall data through employing an unmanned aerial vehicle technology, constructing a marking data set of a great wall damaged area, and carrying out the data enhancement of the marking data set; a YOLOv5s detection and recognition network is improved, a CIoULoss loss function is introduced, a P2 small target detection layer is added, the target recognition accuracy of a small target is improved, in addition, a clustered anchor frame is calculated, and a prior frame more suitable for a Great Wall scene is obtained; the data set is sent to an improved algorithm network for model training, and the overall performance and generalization ability of the model in a test set are evaluated; and sending a to-be-detected picture into the model, and checking a picture presentation effect. According to the method, the unmanned aerial vehicle shooting technology and deep learning are combined, the sizes and positions of collapse, falling and crack targets of the Great Wall can be accurately detected, and the maintenance cost of the Great Wall is reduced while the working efficiency of Great Wall damage detection is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image detection, and in particular, relates to a method for detecting damaged areas of the Great Wall based on aerial photography, electronic equipment and storage medium. Background Art

[0002] In order to protect the Great Wall, it is particularly important to detect and repair these damaged areas in a timely manner.

[0003] At present, there are mainly the following technical solutions for detecting damaged areas of the Great Wall:

[0004] 1. Manual inspection: Professional maintenance personnel regularly inspect the Great Wall to find and record damaged areas, or use image capture equipment to take simple bird's-eye views of the Great Wall, and detect damaged areas through manual or semi-automatic image analysis. This method improves coverage and detection efficiency, but still relies on manual post-processing and has high environmental requirements. Manual inspection is still the main detection method for damaged areas of the Great Wall. However, this method is easily limited by human subjective factors, and the accuracy of detection is difficult to guarantee. In addition, human visual fatigue makes detection prone to false detection and misdetection. It is difficult to work in harsh environments, and the cost is high, time-consuming and labor-intensive, and inefficient. In addition, it is difficult to achieve large-scale and high-frequency detection due to the professional level of personnel.

[0005] 2. Traditional image processing method: First, the image data is collected and processed using traditional image processing techniques, such as edge detection and texture analysis, to identify damaged areas. This method improves the degree of automation to a certain extent, but it has poor adaptability to complex backgrounds and variable damage forms, limited detection accuracy, and low robustness.

[0006] 3. Detection method based on deep learning: Use deep learning models, such as SSD and YOLO series, to train and detect images taken by drones of the Great Wall. This type of method has significant advantages in detection accuracy and automation, but it requires a large amount of labeled data, which requires a large amount of data labeling work, and has high model training and deployment costs. There is still room for optimization in the specific application of detecting damaged areas of the Great Wall.

[0007] In view of the problems existing in the above technologies, it is urgent to adopt an image recognition method using deep learning technology to perform intelligent detection of the Great Wall UAV visual images, overcome the shortcomings of manual detection in the detection task, reduce the interference of unstable factors in manual judgment, and realize reliable, stable and low-cost detection and identification of damaged areas of the Great Wall. Summary of the invention

[0008] To solve the above problems, the present invention proposes a method, electronic device and storage medium for detecting damaged areas of the Great Wall based on aerial photography, which combines drone photography technology with deep learning, and uses a deep learning model to identify damaged areas using the collected video data. It can accurately detect targets and determine their size and position, reduce the interference of unstable factors in manual judgment, and thus improve the detection accuracy of damaged areas in drone images of the Great Wall.

[0009] A first aspect of the present invention provides a method for detecting damaged areas of the Great Wall based on aerial photography, comprising:

[0010] S1, build a dataset of damaged areas of the Great Wall;

[0011] S2, data augmentation for the labeled Great Wall damaged area dataset;

[0012] S3, build an improved YOLOv5s detection and recognition network;

[0013] S4, calculate the clustered anchor boxes and identify objects suitable for the Great Wall scene;

[0014] S5, sending the training set to the improved YOLOv5s detection and recognition network for model training, and evaluating it through the validation set, adjusting the parameters of the model according to the evaluation results, and obtaining a trained detection and recognition model;

[0015] S6, sending the test set to the trained detection and recognition model for effect evaluation to evaluate the final overall performance and generalization ability of the model;

[0016] S7, sending the image to be detected to the trained detection and recognition model, evaluating the final overall performance and generalization ability of the model, and verifying the image presentation effect.

[0017] Furthermore, the S1, constructing a Great Wall damaged area dataset, specifically includes:

[0018] The high-definition camera of the drone is used to collect video data of the Great Wall. The video data is screened, and the image data of the damaged Great Wall is screened out. The labeled data set of the damaged area of ​​the Great Wall is obtained by annotation. The labeled data set of the damaged area of ​​the Great Wall is divided into training set, verification set and test set according to proportion.

[0019] Furthermore, the S2 performs data enhancement on the labeled Great Wall damaged area dataset, including:

[0020] Eight pictures are randomly read from the data set each time, and the eight pictures are flipped, scaled, and color gamut changed respectively. After the operation is completed, a stitching center is selected from the eight pictures, and the images are placed in sequence and stitched again to generate a new picture. A part of the data set is selected for cropping, flipping, and translation operations to further augment the data.

[0021] Furthermore, the S3 constructs an improved YOLOv5s detection and recognition network, including:

[0022] S3.1, replace the GIoU_Loss loss function of the original YOLOv5s network with the CIoU_Loss loss function;

[0023] S3.2, add small target detection layer.

[0024] Furthermore, the S3.1: replacing the GIoU_Loss loss function of the original YOLOv5s network with the CIoU_Loss loss function includes:

[0025] Replace the GIoU_Loss loss function with the CIoU_Loss loss function as the loss function for target box regression to calculate the positioning loss;

[0026] CIoU_Loss loss function L CIoU The expression is as follows:

[0027]

[0028] Among them, α is the weight; v is the similarity of the aspect ratio; b represents the center point of the prediction box, b gt represents the center point of the real box; ρ is the Euclidean distance between the center points of the predicted box and the real box; c represents the slant distance of the minimum bounding box that contains both the predicted box and the real box; IoU represents the intersection over union ratio of the predicted box and the real box; A is the predicted box, B is the real box, ∩ is the intersection operation, ∪ is the union operation; h is the height of the predicted box; w is the width of the predicted box; h gt is the height of the real frame; w gt is the actual box width.

[0029] Furthermore, the S3.2, adding a small target detection layer, includes:

[0030] S3.2.1: In the neck network layer, delete the original connection between the first Concat module and the first CSP module;

[0031] S3.2.2: In the neck network layer, add the second CSP module, the first CBS module, the upsampling module, the second Concat module, the third CSP module, the second CBS module and the third Concat module, and connect the output of the previous module to the input of the next module in sequence;

[0032] The output of the second CSP module is reduced to 64*80*80 by the first CBS module, and then input into the upsampling module for upsampling to obtain a 64*160*160 feature map.

[0033] S3.2.3: In the neck network layer, connect the output of the first Concat module to the input of the second CSP module; connect the output of the first CBS module to the input of the third Concat module; connect the output of the third Concat module to the input of the first CSP module;

[0034] S3.2.4: Setting a P2 feature module in the backbone network layer, and adding a connection to the P2 feature module to the input end of the second Concat module in the neck network layer;

[0035] S3.2.5: Add a P2 detection head in the head network layer; connect the output end of the third CSP module to the input end of the P2 detection head; output the P2 detection head result from the P2 detection head.

[0036] Furthermore, the S4, calculating the clustered anchor frames and identifying the objects suitable for the Great Wall scene, includes:

[0037] S4.1, collect all anchor frames, and randomly select a sample anchor frame x0 from the anchor frame set as the initial existing cluster center anchor frame C0;

[0038] S4.2, calculate the i-th sample anchor box x i With the existing cluster center anchor box C i-1 The distance D(x i );

[0039] S4.3: According to D(x i )Calculate the point x i The probability of becoming the cluster center anchor box P(x i );

[0040] S4.4: Filter the probability values ​​of all points and take the point with the highest probability value as the next cluster center anchor box C i ;

[0041] S4.5: Repeat steps S4.2 to S4.4 until the position of the cluster center anchor box does not change, then output the clustered anchor box.

[0042] Furthermore, in S6, the test set is fed into the trained detection and recognition model to evaluate the final overall performance and generalization ability of the model, including:

[0043] The trained detection and recognition model is compared with the original YOLOv5s network model on the test set, and the precision formula, recall formula, average precision AP formula, average precision mAP formula, and F1 Score formula are used as evaluation indicators to compare and evaluate the effects of the YOLOv5s network model before and after improvement.

[0044]

[0045] Among them, Precision represents the probability of correct detection among all detected targets; Recall represents the probability of correct identification among all positive samples; F1 Score is the harmonic mean of precision and recall; TP is the number of correctly classified positive samples, that is, predicted to be positive samples, and actually positive samples; FP is the number of negative samples that are mistakenly marked as positive samples, that is, actually negative samples are predicted to be positive samples; FN refers to the number of positive samples that are mistakenly marked as negative samples, that is, actually positive samples are predicted to be negative samples; N is the total number of detection categories; R is the Recall value; APi is the AP value calculated for each category i.

[0046] According to a second aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described in the first aspect of the present invention.

[0047] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program can be loaded by a processor to execute any one of the methods according to the first aspect of the present invention.

[0048] The advantages of the present invention compared to the prior art are:

[0049] The present invention uses a high-definition camera of a drone to collect data of the Great Wall, uses a data enhancement method to expand the data set, incorporates CIoU_LOSS into the loss function in the improved YOLOv5s detection and recognition network, adds a small target detection layer to the network detection head, and re-clusteres the data to adjust the detection network prior frame, so that the damaged area of ​​the Great Wall can be identified and located more accurately and efficiently.

[0050] The present invention optimizes the model structure and training strategy according to the characteristics of the detection of damaged areas of the Great Wall, combines drone shooting technology with deep learning, and uses the collected video data to identify damaged areas through a deep learning model. It can accurately detect targets and determine their sizes and positions, reduce the interference of unstable factors in manual judgment, thereby improving the detection accuracy of damaged areas in drone images of the Great Wall and reducing the cost of maintenance of the Great Wall.

[0051] Compared with traditional image processing algorithms, the proposed method has higher accuracy and can effectively detect three categories of the Great Wall, namely, collapse, falling off, and cracks. Compared with the original YOLOv5s, the entire network has improved the accuracy of detecting damaged areas of the Great Wall, especially the detection ability of small areas and damaged areas with complex backgrounds. Data enhancement and multi-dataset acquisition enhance the generalization ability and robustness of the model, enabling it to work stably in different environments and lighting conditions.

[0052] The present invention realizes efficient and automated detection of damaged areas of the Great Wall, reduces reliance on manual inspections, and improves detection efficiency; it provides real-time detection capabilities, can promptly discover and report damaged areas, and contributes to the timely maintenance and protection of the Great Wall. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of a method provided by an embodiment of the present invention.

[0054] Figure 2 It is a schematic diagram of an improved YOLOv5s detection and recognition network structure provided by an embodiment of the present invention.

[0055] Figure 3 It is a schematic diagram of the output effect of an improved YOLOv5s detection and recognition network provided by an embodiment of the present invention.

[0056] Figure 4 It is a training indicator result diagram provided by an embodiment of the present invention.

[0057] Figure 5 This is a detection effect diagram provided by an embodiment of the present invention.

[0058] Figure 6 This is a detection effect diagram provided by an embodiment of the present invention.

[0059] Figure 7 This is a detection effect diagram provided by an embodiment of the present invention.

[0060] Figure 8 It is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0062] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0063] First, the following concepts are introduced.

[0064] (1) YOLO: You Only Look Once, a series of target detection algorithms, including YOLOv1, YOLOv2.... In the deep learning network, the YOLOv5 algorithm is currently a popular target detection algorithm. YOLOv5s is one of the network models in the YOLOv5 series. As an improved version of the series, YOLOv5s has the advantages of lightweight, fast speed and high accuracy, and is very suitable for real-time detection in resource-constrained environments.

[0065] (2) Mosaic-8: A data augmentation method that combines eight images by random scaling, random cropping, and random arrangement.

[0066] (III) YOLOv5s original model structure: YOLOv5s mainly consists of four parts: input end (iput), backbone network layer, neck network layer and head network layer.

[0067] 1. The backbone network layer is used to aggregate and form convolutional neural networks of image features at different image granularities. The backbone network layer is mainly composed of CBS module (i.e. ConBNSiLU module), CSP module (i.e. C3 module) and SPPF module.

[0068] (1) CBS module, which encapsulates three functions: ordinary convolution (Conv2d), Batch Normalization and activation function SiLU.

[0069] (2) CSP module: It consists of three CBS modules and several BottleNeck (bottleneck network) modules.

[0070] The backbone network layer uses CSP1_x to represent the CSP module, and the neck network layer uses CSP2_x to represent the CSP module.

[0071] (3) SPPF module, which stands for Spatial Pyramid Pooling Fast.

[0072] 2. The neck network layer is a series of network layers that mix and combine image features and pass the image features to the prediction layer. It is mainly composed of FPN (Feature Pyramid Network) and PANet (Path Aggregation Network).

[0073] (1) FPN, through top-down deconvolution and bottom-up skip connection, fuses feature maps of different scales together to form a feature pyramid, allowing the network to focus on features of different scales at the same time.

[0074] (2) PANet, based on FPN, introduces a bottom-up path. After top-down feature fusion, bottom-up feature fusion is performed again, so that the bottom-level position information can also be passed to the deep layer, thereby enhancing the positioning capability at multiple scales.

[0075] 3. The head network layer is used to predict image features, generate bounding anchor boxes, and merge predicted categories. The main part of the original design of the head network layer is three detection heads, which expand the number of channels of feature maps of different scales obtained by the neck network layer module through 1*1 convolution. The three detection heads are the P3 detection head of 80*80 (small targets), the P4 detection head of 40*40 (medium targets), and the P5 detection head of 20*20 (large targets). For example, when the input is a 640*640 image, the three detection heads P3 to P5 output feature maps of three scales of 80*80, 40*40, and 20*20 respectively.

[0076] Method Embodiment

[0077] The present invention mainly solves the problems existing in the existing Great Wall damaged area detection technology, such as time-consuming and labor-intensive human inspections, high costs, and the shortcomings of low accuracy and weak generalization ability of traditional image processing under complex backgrounds. To this end, the present invention provides an embodiment of a Great Wall damaged area detection method based on aerial photography, which improves detection accuracy and efficiency, reduces model training and deployment costs, makes the model more generalizable, and realizes efficient, accurate, and automated detection of the Great Wall damaged area.

[0078] A method for detecting damaged areas of the Great Wall based on aerial photography, the framework process is shown in the attached figure. Figure 1 As shown, the specific technical solution includes:

[0079] S1: Construct a dataset of damaged areas of the Great Wall.

[0080] Specifically, it includes: using a high-definition camera of a drone to collect video data of the Great Wall, screening the video data, screening out image data of the damaged Great Wall, annotating them to obtain a labeled data set of damaged areas of the Great Wall, and dividing the labeled data set of damaged areas of the Great Wall into a training set, a verification set, and a test set in proportion.

[0081] For example, the image data of the Great Wall damage obtained by screening are divided into three categories, namely collapse, wall falling off and cracks, with a total of several images. The semi-automatic tool X-AnyLabeling is used to annotate the image data to produce a data set with collapse, wall falling off and crack labels. The labeled data set is divided into a training set, a validation set and a test set in a ratio of 8:1:1.

[0082] Using drones’ high-definition cameras to capture data on damage to the Great Wall reduces reliance on manual labor and improves detection efficiency.

[0083] S2: Data augmentation for the labeled Great Wall damaged area dataset.

[0084] Specifically, eight pictures are randomly read from the data set each time, and operations such as flipping, scaling, and color gamut changes are performed on the eight pictures. After the operations are completed, a stitching center is selected for the eight pictures, and the images are placed in sequence and stitched again to generate a new picture. A part of the data set, such as 10% of the data, is selected for cropping, flipping, and translation operations to further augment the data.

[0085] For example, due to the small amount of data, Mosaic-8 is used for data enhancement to enrich the data set, and small sample targets are added to improve the network training speed. When performing normalization operations, 8 images can be calculated at one time, which can effectively reduce the model's demand for memory; at the same time, the image data set is further cropped, flipped, translated, and other data operations are performed, in which some random noise is reasonably introduced to enhance the network model's ability to distinguish small target samples in the image and improve the model's generalization ability. Based on Mosaic-8, random cropping, rotation, and translation, these data enhancement technologies increase the diversity of training data and improve the generalization ability of the model.

[0086] S3: Build an improved YOLOv5s detection and recognition network.

[0087] To address the problem of low accuracy of YOLOv5s detection, especially low accuracy of small target detection, the following operations are performed on the original YOLOv5s network:

[0088] S3.1: Replace the GIoU_Loss loss function of the original YOLOv5s network with the CIoU_Loss loss function.

[0089] The original YOLOv5s network algorithm uses the GIoU_Loss loss function, and GIoU_Loss can be a distance metric of the loss function, and can also directly meet the measurement requirements of the basic loss distance function. At the same time, because GIoU_Loss also has a strong scale invariance, the expression is as follows:

[0090]

[0091] Among them, A C It is the minimum enclosed area of ​​the two boxes, IoU represents the intersection and union ratio of the predicted box and the real box, and U is the union of the two boxes.

[0092] However, when using GIoU_Loss, when there is inclusion between two detection boxes, GIoU_Loss will degenerate into ordinary IoU_Loss (ordinary intersection and union Loss function). In addition, GIoU_Loss requires many iterations to converge. Considering the determination of GIoU (generalized intersection on union), the present invention replaces the GIoU_Loss loss function with the CIoU_Loss loss function as the loss function of the target box regression to calculate the positioning loss.

[0093] CIoU_Loss loss function L CIoU The expression is as follows:

[0094]

[0095]

[0096] Among them, α is the weight, which is a balance parameter; v is the similarity of the aspect ratio; b represents the center point of the prediction box, b gt represents the center point of the real box; ρ is the Euclidean distance between the center points of the predicted box and the real box; c represents the slant distance of the minimum bounding box that contains both the predicted box and the real box; IoU represents the intersection over union ratio of the predicted box and the real box; A is the predicted box, B is the real box, ∩ is the intersection operation, ∪ is the union operation; h is the height of the predicted box; w is the width of the predicted box; h gt is the height of the real frame; w gt is the actual box width.

[0097] CIoU_Loss can directly minimize the distance between the center point of the predicted box and the real box to accelerate convergence. At the same time, it also adds distance Loss (loss function, a key concept in machine learning and deep learning, used to measure the difference or error between the model prediction and the actual target) that can detect different scales of the real box. By adding the length and width Loss, the entire predicted box will be more consistent with the real box.

[0098] It should be noted that the real frame here is a frame made according to the real position of the target in the image, so that the predicted frame can be used for loss calculation with the real frame. The target refers to the object to be detected, such as collapse, cracks, and detachment. The target frame has no specific context. The target frame may be a real frame, a predicted frame, or other frames.

[0099] S3.2: Add small target detection layer.

[0100] Since the original YOLOv5s model uses the P3 feature layer for feature extraction for model learning, and has only three detection heads, namely the 80*80 (small target) P3 detection head, the 40*40 (medium target) P4 detection head, and the 20*20 (large target) P5 detection head. However, the target sizes of the Great Wall cracks and wall detachment are too small, and the features that can be extracted by the original model after three samplings are still limited, which is not enough to improve the recognition accuracy. Therefore, the original YOLOv5s model is difficult to effectively extract small-sized damage features, and there is a problem of difficulty in detecting small targets. In order to achieve small target detection, this embodiment makes corresponding improvements to the original YOLOv5s. The improved model network architecture is shown in the attached figure. Figure 2 shown.

[0101] This embodiment makes the following improvements on the original YOLOv5s model:

[0102] S3.2.1: In the neck network layer, delete the connection between the original first Concat module [Concat(256*80*80)] and the first CSP module [CSP2_1(128*80*80)].

[0103] S3.2.2: In the neck network layer, add the following modules and connect the output of the previous module to the input of the next module in sequence: second CSP module, first CBS module, upsampling module, second Concat module, third CSP module, second CBS module, third Concat module.

[0104] Among them, the second CSP module is CSP2_1 (256*80*80) module,

[0105] The first CBS module is a CBS (64*80*80) module.

[0106] The upsampling module is the Upsample (64*160*160) module.

[0107] The second Concat module is the Conat(128*160*160) module.

[0108] The third CSP module is CSP2_1 (64*160*160) module,

[0109] The second CBS module is a CBS (64*80*80) module.

[0110] The third Concat module is a Conat (128*80*80) module.

[0111] In the neck network layer, the output of the second CSP module is reduced to 64*80*80 through the first CBS module and input into the upsampling module for upsampling to obtain a 64*160*160 feature map.

[0112] S3.2.3: In the neck network layer, connect the output of the first Concat module to the input of the second CSP module; connect the output of the first CBS module to the input of the third Concat module; connect the output of the third Concat module to the input of the first CSP module.

[0113] S3.2.4: A P2 feature module is set in the backbone network layer, and the P2 feature module is newly connected to the input end of the second Concat module in the neck network layer.

[0114] Specifically, the CSP1_1 (64*160*160) module in the backbone network layer is used as a newly added feature module, namely, the P2 feature module.

[0115] The newly added P2 feature module is equivalent to downsampling the original input image only twice in the backbone network layer, a total of 4 times, to obtain the P2 feature map. This feature map has a small receptive field and relatively rich position information, which can improve the detection effect of small-sized targets such as cracks, and provide subsequent networks with shallow feature information containing richer targets.

[0116] For example, the original image is 640*640, and after being input into the improved model, the P2 feature module outputs a P2 feature map with a large resolution of 160*160.

[0117] S3.2.5: Add a P2 detection head in the head network layer; connect the output end of the third CSP module to the input end of the P2 detection head; output the P2 detection head result from the P2 detection head.

[0118] The resolution of the P2 detection head is 160×160 pixels. The P2 detection head is generated by a low-level, high-resolution feature map (a feature map represents the image features extracted from the input image through convolution operations in a convolutional neural network) and is more sensitive to tiny objects.

[0119] The downsampling multiplier of YOLOv5s is large, and the deeper feature map makes it difficult to learn the features of small targets. Therefore, the P2 detection head for small target detection is added to detect the shallower feature map. The results of the P2 detection head are used for model training, which enhances the model's feature extraction ability for small areas, improves the model's ability to recognize small targets, and improves the detection accuracy of small targets in the damaged area of ​​the Great Wall.

[0120] After adding the P2 detection head, the two P2 layer features obtained from top to bottom and bottom to top in the neck network layer are fused with the same-scale features (160×160) in the backbone network through the concat module. The output feature is the fusion result of the three input features. This enables the P2 layer detection head to detect small targets quickly and effectively.

[0121] The main part of the original design of the head network layer is 3 detection heads. The present invention sets 4 detection heads, P2 detection head plus the original P3, P4, P5 detection heads. The 4-head structure can alleviate the negative impact of drastic target scale changes and scale variance, corresponding to the 4 different sizes of feature maps obtained in the neck network layer module. For these 4 feature maps, the improved YOLOv5s presets 4 anchor frames with different aspect ratios on each grid to predict and regress targets. The added P2 detection head is for the underlying features, which is generated by low-level, high-resolution feature maps. It is more sensitive to small targets and improves the model's detection ability for small targets. After adding the P2 detection head, although the calculation and storage costs are increased, the detection performance of small objects has been greatly improved. At the same time, the input 640*640 picture can be downsampled and reduced in dimension, and output to the original detection head through the improved neck network layer to achieve the results of the three detection heads P3, P4, and P5.

[0122] The output effect diagram of adding P2 detection head is as shown in the attached figure. Figure 3 shown.

[0123] S4: Calculate the clustered anchor boxes and identify objects suitable for the Great Wall scene.

[0124] In this embodiment, the size of the clustered anchor box is calculated based on the improved K-means++ clustering algorithm to accommodate small targets. The specific solution is:

[0125] S4.1: Collect all anchor boxes and randomly select a sample anchor box x0 from the anchor box set as the initial existing cluster center anchor box C0.

[0126] That is, C0 = x0, where x0∈anchor box set.

[0127] S4.2: Calculate the i-th sample anchor box x i With the existing cluster center anchor box C i-1 The distance D(x i ):

[0128] D(x i )=1-IoU(x i , C i-1 ).

[0129] Where i>0; x i ∈ anchor box set; IoU(x i , C i-1 ) is the sample x in the set i To the existing cluster center C i-1 The intersection-over-combination ratio is used to measure each sample anchor box x i and the cluster center anchor box C i-1 The overlap degree between the sample anchor box and the cluster center anchor box are both coordinate values.

[0130] S4.3: According to D(x i )Calculate the point x i The probability of becoming the cluster center anchor box P(x i ).

[0131] P(x i ) is calculated as:

[0132] Among them, N is the number of sample anchor boxes, sample anchor box x i ∈ the set of anchor boxes.

[0133] S4.4: Filter the probability values ​​of all points and take the point with the highest probability value as the next cluster center anchor box C i .

[0134] S4.5: Repeat steps S4.2 to S4.4 until the position of the cluster center anchor box does not change, then output the clustered anchor box.

[0135] The K-means++ clustering algorithm before the improvement uses the Euclidean distance, while the present invention uses 1-IoU as the distance. After this improvement, when processing tasks involving bounding boxes, such as object detection and segmentation, the clustering process can be made more in line with actual application requirements:

[0136] 1) It has all the properties of distance metrics: non-negativity, uncertainty, symmetry, and triangle inequality;

[0137] 2) It is scale-invariant and insensitive to scale, which means that the similarity between any two boxes A and B has nothing to do with their spatial scale;

[0138] 3) It is not only used to determine positive and negative samples, but also to evaluate the distance between the predicted box and the true box.

[0139] S5: Send the training set to the improved YOLOv5s detection and recognition network for model training, and evaluate it through the validation set. Adjust the model parameters according to the evaluation results to obtain a trained detection and recognition model.

[0140] The training set is used to train the model so that it can learn the patterns and features in the data. Through the training set, the parameters of the model are adjusted so that the model can make accurate predictions on the given input data.

[0141] The validation set is used to tune the model hyperparameters and select the best model architecture. During the training process, the validation set is used regularly to evaluate the performance of the model, and the best hyperparameters (such as learning rate, regularization parameters, etc.) are selected through the validation error to prevent overfitting. The validation set is executed together with the training set and is generally not operated separately.

[0142] Send the prepared training set to the improved YOLOv5s detection and recognition network, perform model training on the server, set training parameters, including rounds, batch-size, image size, learning rate and other parameters, and visualize the training results after multiple training. For example, during the model training process, set the parameter epcoh to 320, batch-size to 16, image size to 640*640, and select SGD optimization. During model training, the learning rate is preheated using Warmup training. During the Warmup stage, the learning rate of the bias layer decreases from 0.1 to 0.01, and the learning rate of other parameters increases from 0 to 0.01. After the Warmup is completed, the cosine annealing learning algorithm is used to update the learning rate, and the loss function is used to calculate the error. The model parameters are optimized through the back propagation algorithm to minimize the error. After multiple parameter adjustment and training, the final training indicator results are as shown in the attached figure. Figure 4 shown.

[0143] S6: Send the test set to the trained detection and recognition model to evaluate the final performance and generalization ability of the model.

[0144] After the model is trained and tuned, the test set is used for evaluation and the test error (such as accuracy, precision, recall, F1 value, etc.) is calculated to measure the generalization ability of the model. This step is independent of the training and validation process and is the evaluation standard for the final model.

[0145] The test set is used to finally evaluate the performance of the model and provide performance indicators of the model in real scenarios.

[0146] The trained detection and recognition model was compared with the original YOLOv5s network model on the test set, and the precision (Precision), recall rate (Recall), average precision (AP), mean average precision (mAP), and F1 value (F1Score) were used as evaluation indicators to compare and evaluate the effects of the YOLOv5s network model before and after improvement.

[0147]

[0148] Among them, Precision is a measure of accuracy, which indicates the probability of correct detection among all detected targets; Recall is a measure of coverage, which indicates the probability of correct identification among all positive samples; F1 Score is the harmonic mean of precision and recall, with a minimum of 0, indicating the worst; and a maximum of 1, indicating the best. TP (True Positive) is the number of correctly classified positive samples, that is, samples predicted as positive and actually positive; FP (False Positive) is the number of negative samples that are incorrectly marked as positive, that is, samples that are actually negative but predicted as positive; TN (True Negative) refers to the number of correctly classified negative samples, that is, samples predicted as negative and actually negative; FN (False Negative) refers to the number of positive samples that are incorrectly marked as negative, that is, samples that are actually positive but predicted as negative; N is the total number of detection categories; R is the Recall value; APi is the AP value calculated for each category i. As shown in the above formula, the average precision AP is the area enclosed by plotting with Precision and Recall as the coordinate axes. Mean Average Precision (mAP) is the average of the average precision (AP) of all target categories. It is used to describe the detection performance of the model for all target detection categories. The larger the mAP value, the better the detection effect of the algorithm and the higher the recognition accuracy.

[0149] Specifically, the test set is input into the trained detection and recognition model to detect the damaged area of ​​the Great Wall, and the confidence and IoU values ​​are set.

[0150] Confidence is the degree of trust the model has in the detection result, indicating the probability that a certain target is contained in the detection box. Adjusting the confidence parameter can help filter out high-confidence detection results.

[0151] IoU is the intersection and union ratio of the predicted box and the true box. Adjusting the IoU value can help determine the accuracy of the detection results.

[0152] For example, the test set is tested by the trained detection and recognition model, and the confidence level is set to 0.3 and the IoU value is set to 0.35. The detection effect diagram of the improved detection network is shown in the attached figure. Figures 5 to 7 shown.

[0153] The comparison data of the YOLOv5s model before and after improvement is as follows:

[0154] Table 1 Performance comparison results of YOLOv5s and improved YOLOv5s algorithms

[0155]

[0156] By comparing the results, we can see that:

[0157] 1) The Precision value of the improved model is larger, indicating that the improved model has significantly improved accuracy, indicating that the proportion of correctly detected targets is higher and false positives are reduced.

[0158] 2) The Recall of the improved model is larger, which means that the recall rate of the improved model has also been improved, indicating that the proportion of correctly detected targets among all actual targets is higher, reducing missed targets.

[0159] 3) The F1Score of the improved model is larger, which means that the improved model also performs better in this indicator, indicating that the improved model has better performance when considering precision and recall rate.

[0160] 4) The mAP value of the improved model has been significantly improved, indicating that the improved model has improved detection accuracy in all categories.

[0161] S7: Send the image to be detected into the trained detection and recognition model to check the image presentation effect.

[0162] Device Embodiment

[0163] According to one aspect of an embodiment of the present invention, an electronic device is provided.

[0164] Figure 81 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device may include one or more (only one is shown in the figure) processors 102 (the processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the electronic device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only and does not limit the structure of the above terminal device. Figure 8 More or fewer components as shown, or with Figure 8 Equivalent functions or comparisons shown Figure 8 A different configuration with more features is shown.

[0165] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a method for detecting damaged areas of the Great Wall based on aerial photography in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal device via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the switching device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0167] Readable storage medium embodiment

[0168] According to one aspect of an embodiment of the present invention, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can be loaded by a processor to execute the method described in the first aspect of the present invention.

[0169] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for detecting damaged areas of the Great Wall based on aerial photography, characterized in that: include: S1, build a dataset of damaged areas of the Great Wall; S2, data augmentation for the labeled Great Wall damaged area dataset; S3, build an improved YOLOv5s detection and recognition network; S4, calculate the clustered anchor boxes and identify objects suitable for the Great Wall scene; S5, sending the training set to the improved YOLOv5s detection and recognition network for model training, and evaluating it through the validation set, adjusting the parameters of the model according to the evaluation results, and obtaining a trained detection and recognition model; S6, sending the test set to the trained detection and recognition model to evaluate the final overall performance and generalization ability of the model; S7, sending the image to be detected to the trained detection and recognition model to test the image presentation effect.

2. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 1, characterized in that: S1, constructing a dataset of damaged areas of the Great Wall, specifically includes: The high-definition camera of the drone is used to collect video data of the Great Wall. The video data is screened, and the image data of the damaged Great Wall is screened out. The labeled data set of the damaged area of ​​the Great Wall is obtained by annotation. The labeled data set of the damaged area of ​​the Great Wall is divided into training set, verification set and test set according to proportion.

3. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 2, characterized in that: S2, data enhancement is performed on the labeled Great Wall damaged area dataset, including: Eight pictures are randomly read from the data set each time, and the eight pictures are flipped, scaled, and color gamut changed respectively. After the operation is completed, a stitching center is selected from the eight pictures, and the images are placed in sequence and stitched again to generate a new picture. A part of the data set is selected for cropping, flipping, and translation operations to further augment the data.

4. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 3 is characterized in that: S3, building an improved YOLOv5s detection and recognition network, includes: S3.1, replace the GIoU_Loss loss function of the original YOLOv5s network with the CIoU_Loss loss function; S3.2, add small target detection layer.

5. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 4, characterized in that: S3.1: Replace the GIoU_Loss loss function of the original YOLOv5s network with the CIoU_Loss loss function, including: Replace the GIoU_Loss loss function with the CIoU_Loss loss function as the loss function for target box regression to calculate the positioning loss; CIoU_Loss loss function L CIoU The expression is as follows: Among them, α is the weight; v is the similarity of the aspect ratio; b represents the center point of the prediction box, b gt represents the center point of the real box; ρ is the Euclidean distance between the center points of the predicted box and the real box; c represents the slant distance of the minimum bounding box that contains both the predicted box and the real box; IoU represents the intersection over union ratio of the predicted box and the real box; A is the predicted box, B is the real box, ∩ is the intersection operation, ∪ is the union operation; h is the height of the predicted box; w is the width of the predicted box; h gt is the height of the real frame; w gt is the actual box width.

6. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 5, characterized in that: S3.2 adds a small target detection layer, including: S3.2.1: In the neck network layer, delete the original connection between the first Concat module and the first CSP module; S3.2.2: In the neck network layer, add the second CSP module, the first CBS module, the upsampling module, the second Concat module, the third CSP module, the second CBS module and the third Concat module, and connect the output of the previous module to the input of the next module in sequence; The output of the second CSP module is reduced to 64*80*80 by the first CBS module, and then input into the upsampling module for upsampling to obtain a 64*160*160 feature map. S3.2.3: In the neck network layer, connect the output of the first Concat module to the input of the second CSP module; connect the output of the first CBS module to the input of the third Concat module; connect the output of the third Concat module to the input of the first CSP module; S3.2.4: Setting a P2 feature module in the backbone network layer, and adding a connection to the P2 feature module to the input end of the second Concat module in the neck network layer; S3.2.5: Add a P2 detection head in the head network layer; connect the output end of the third CSP module to the input end of the P2 detection head; output the P2 detection head result from the P2 detection head.

7. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 6, characterized in that: The S4 calculates the clustered anchor frames and identifies objects suitable for the Great Wall scene, including: S4.1, collect all anchor frames, and randomly select a sample anchor frame x0 from the anchor frame set as the initial existing cluster center anchor frame C0; S4.2, calculate the i-th sample anchor box x i With the existing cluster center anchor box C i-1 The distance D(x i ); S4.3: According to D(x i )Calculate the point x i The probability of becoming the cluster center anchor box P(x i ); S4.4: Filter the probability values ​​of all points and take the point with the highest probability value as the next cluster center anchor box C i ; S4.5: Repeat steps S4.2 to S4.4 until the position of the cluster center anchor box does not change, then output the clustered anchor box.

8. The method for detecting damaged areas of the Great Wall based on aerial photography according to claim 7, characterized in that: S6, send the test set to the trained detection and recognition model to evaluate the final overall performance and generalization ability of the model, including: The trained detection and recognition model is compared with the original YOLOv5s network model on the test set, and the precision formula, recall formula, average precision AP formula, average precision mAP formula, and F1 value F1Score formula are used as evaluation indicators to compare and evaluate the effects of the YOLOv5s network model before and after improvement. Among them, Precision represents the probability of correct detection among all detected targets; Recall represents the probability of correct identification among all positive samples; F1 Score is the harmonic mean of precision and recall; TP is the number of correctly classified positive samples, that is, predicted to be positive samples, and actually positive samples; FP is the number of negative samples that are mistakenly marked as positive samples, that is, actually negative samples are predicted to be positive samples; FN refers to the number of positive samples that are mistakenly marked as negative samples, that is, actually positive samples are predicted to be negative samples; N is the total number of detection categories; R is the Recall value; APi is the AP value calculated for each category i.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be loaded by a processor to execute the method according to any one of claims 1 to 8.

Images