Deep learning network unmanned aerial vehicle visual angle dam crack detection method, system and device and medium

Abstract

The invention discloses a deep learning network unmanned aerial vehicle visual angle dam crack detection method, system and device and a medium, and belongs to the technical field of computer vision, and the method comprises the steps: collecting a dam surface image through an unmanned aerial vehicle, and carrying out the preprocessing of the collected image; constructing a target detection network fused with an attention mechanism, and training the target detection network; inputting a to-be-detected image into the trained network, outputting bounding box coordinates of a crack target, and meanwhile, quantifying a trained model and deploying the model to a back-end server; and the unmanned aerial vehicle transmits a real-time shot picture to the server, and the rear-end server analyzes the picture and inputs the picture into the model for real-time detection. The method has the advantages of high safety and high detection efficiency, has the advantage in the detection speed of an algorithm, does not need frame extraction for an unmanned aerial vehicle picture, is low in omission ratio and high in detection precision for a scene in which the unmanned aerial vehicle flies quickly, and further enhances the anti-interference capability of unmanned aerial vehicle detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a deep learning network unmanned aerial vehicle perspective dam crack detection method, system, device and medium. BACKGROUND

[0002] The current dam crack detection technology still has many bottlenecks in practical application, and it is difficult to meet the engineering requirements of high precision and strong adaptability. The specific defects are as follows: It relies on artificial climbing or scaffolding inspection. For a dam with a height of more than 100 meters, the overall detection takes a long time, and the hidden area of the dam body is difficult to cover, resulting in a detection blind area and low efficiency. The detection result is significantly affected by the skill level of the personnel. Manual recording of artificial inspection has many missed and false detections, and cannot meet the fine evaluation requirements. It is difficult to work in harsh conditions such as flood season and winter icing period. The emergency detection response is lagging in emergency situations, and there is a risk of falling in high-altitude operations.

[0003] The technology based on Sobel, Canny and other edge detection algorithms has limitations. The anti-interference ability is very weak. The interference such as moss, water stains and repair marks on the surface of the dam will produce similar gray changes as cracks, resulting in a high false detection rate. In strong light or backlight scenes, the algorithm is almost ineffective, the scale adaptability is poor, only the obvious cracks with wide width can be identified, the feature extraction ability for slender and net-like micro-cracks is insufficient, and the image resolution difference cannot be handled. The conventional detection method based on deep learning has shortcomings. Some models with attention modules cannot focus on the spatial position and scale features of the cracks at the same time, and the positioning accuracy of the cross cracks and bifurcated cracks is insufficient. The existing algorithm takes a long time to process a single unmanned aerial vehicle image. In order to ensure the real-time performance of the unmanned aerial vehicle inspection, the frame extraction strategy is usually used. For small crack targets, the frame extraction strategy is likely to cause a large number of missed detections and cannot realize real-time warning on site. SUMMARY

[0004] In view of the above problems, the present application is proposed.

[0005] Therefore, the present application aims to solve the following key technical problems: break through the limitations of traditional manual detection, such as low efficiency, incomplete coverage, precision depending on subjective experience and high risk, realize the automation and efficiency of dam crack detection, and improve the detection ability of hidden areas and micro-cracks, and reduce the detection error. Solve the problems of weak anti-interference ability, poor scale adaptability and lack of intelligent decision-making ability based on traditional image processing technology, enhance the suppression ability of complex interference on the surface of the dam, and improve the recognition effect of cracks of different scales.

[0006] Solve the problems of general model adaptability, attention mechanism design defects, data dependence and generalization dilemma in conventional deep learning detection technology, improve the model's extraction ability of high similarity features between dam cracks and concrete background, optimize the attention mechanism to improve the positioning accuracy of complex cracks, reduce the dependence on large-scale labeled data, and enhance the generalization performance of the model in different dam types and environments. Break through the bottlenecks of image quality constraints, multi-source data fusion defects and real-time deficiencies in unmanned aerial vehicle inspection supporting technology, improve the image quality under different shooting distances to reduce the measurement error of crack geometric parameters, realize effective mapping of two-dimensional detection results and three-dimensional dam model to support structure safety evaluation, and improve the algorithm processing speed to meet the real-time early warning demand on site.

[0007] To solve the above technical problems, the present application provides the following technical scheme: a kind of deep learning network unmanned aerial vehicle perspective dam crack detection method, it includes, collect dam surface image by unmanned aerial vehicle, and the collected image is pretreated;The target detection network of fusion attention mechanism is constructed, the pretreated image is divided into training set, verification set and test set, and the target detection network is trained;The image to be detected is input into the trained network, and the boundary box coordinates of the crack target are output, and the trained model is quantized and deployed to the back-end server;Unmanned aerial vehicle transmits real-time shooting picture to server, and the back-end server analyzes the picture and inputs the model for real-time detection.

[0008] As a preferred scheme of the deep learning network unmanned aerial vehicle perspective dam crack detection method described in the application, wherein: the collected dam surface image includes, the GPS coordinates and attitude parameters recorded by the unmanned aerial vehicle when collecting the dam surface image, and the distant shooting and the close shooting are adopted; When confirming the distant shooting, set the distance between the unmanned aerial vehicle and the dam and the adjacent image overlap rate, and obtain the low-resolution panoramic image of the dam body; When confirming the close shooting, set the distance between the unmanned aerial vehicle and the dam, and shoot the detail image.

[0009] As a preferred scheme of the deep learning network unmanned aerial vehicle perspective dam crack detection method described in the application, wherein: the pretreatment includes, after enlarging the low-resolution image of distant shooting, denoising by filtering; The detail image shot by close shooting is denoised, and the two types of processed images are cropped.

[0010] As a preferred scheme of the deep learning network unmanned aerial vehicle perspective dam crack detection method described in the application, wherein: the training of the target detection network includes, according to the attention mechanism of convolutional neural network, constructing the target detection network of fusion attention mechanism; The preprocessed image is divided into a training set, a verification set and a test set, and the constructed network is trained by data enhancement training such as rotation, scaling and horizontal flip, and the network parameters are optimized until the model converges.

[0011] As a preferred scheme of the deep learning network dam crack detection method from the perspective of a UAV according to the application, the real-time detection comprises inputting the image to be detected into the trained network, the division ratio of the training set, the verification set and the test set is 7:2:1, the initial learning rate is set to 0.001, and the Adam optimizer is used to output the boundary box coordinates of the crack target. The boundary box is subjected to non-maximum suppression processing, the threshold of the non-maximum suppression is set to 0.25, when the intersection over union IoU of two boundary boxes is greater than 0.25, the boundary box with the highest confidence is retained, the remaining redundant boxes and the containing relationship boxes are removed, and finally the crack detection result is obtained; based on the GPS coordinates and the attitude parameters, a three-dimensional model of the dam is constructed through oblique photography technology, and the crack boundary box is mapped to the three-dimensional model to calculate the length and the maximum width of the crack. The trained image is subjected to onnx quantization, the FLOAT32 model is quantized to the INT8 model, and the onnx quantized image is deployed on the backend server. The UAV carries out on-site inspection work, transmits the real-time shooting pictures to the push stream server through the 5G module, generates a fixed rtmp push stream address, the backend server analyzes the fixed rtmp push stream address, inputs each frame of picture into the algorithm model, generates a detection result, and outputs the detection boundary box, the detection category and the confidence of the crack.

[0012] The preferred technical scheme in the embodiment of the application has the following beneficial effects: through the multi-distance shooting of the UAV combined with the GPS coordinates and the attitude parameters, panoramic and detailed images of the dam body can be obtained, comprehensive coverage and accurate positioning are realized, and rich and geographical reference data basis is provided for subsequent detection; comprehensive coverage and multi-scale image acquisition of the dam surface are realized, and the integrity and accuracy of crack detection are ensured.

[0013] As a preferred scheme of the deep learning network dam crack detection method from the perspective of a UAV according to the application, the preprocessing further comprises inputting a low-quality image with a resolution of , wherein is the height, is the width, is the number of channels, and shallow features are obtained through shallow feature extraction Deep feature extraction is inputted, long-distance feature dependencies are captured through residual SwinTransformer blocks, and deep features are outputted, and a reconstruction module is fused and The feature information is used to output a high-quality restored image through upsampling and convolution operations. ; The shallow feature extraction assumes low-quality images. The pixel value Let i, j, and k be the coordinates of a pixel in the height, width, and channel dimensions, respectively. The weights of the 3×3 convolution kernel are... It contains d weight matrices with height a, width b, and number of channels c, where d is the shallow feature layer. The number of channel dimensions, shallow features The calculation is as follows: ,in, For the first The bias term for each output channel. The function is defined as ; The deep feature extraction consists of N cascaded residual SwinTransformer blocks and a final 3×3 convolutional layer. The core of the SwinTransformer block is a window attention mechanism, which divides the feature map into non-overlapping windows, with the window size set to... Self-attention computation is performed only within each window, dividing the feature map of the input residual SwinTransformer block into multiple non-overlapping windows if the feature map resolution is not reduced. Integer division, using zero-padding to ensure consistent window size; For each feature within a window, first assemble and flatten it into a sequence of length [length missing]. Then, the association weight between each feature pixel and the remaining pixels in the window is calculated using a multi-head attention mechanism: Concat is the merge operation. For the number of attention heads, To output the projection matrix, the attention head formula is as follows: in, , The first The query, key, and value matrix of the size. The dimension of the hidden layer is M, where M is the mask matrix to avoid interference between windows.

[0014] The beneficial effects of the preferred technical solution in the embodiments of the present invention are as follows: by fusing shallow and deep feature extraction and using attention mechanism to reconstruct high-quality images, the details and clarity of low-resolution images are effectively improved and noise interference is reduced; image quality is significantly improved and the visibility of crack features is enhanced, thereby improving the input data quality of the detection model.

[0015] As a preferred scheme of the deep learning network unmanned aerial vehicle perspective dam crack detection method, wherein: the target detection network of the fusion attention mechanism comprises: the size of the input image is unified by scaling or padding, the pixel point value is normalized to [0, 1] by a normalization method, and the standardization is carried out according to the statistical mean and standard deviation of the training set: Wherein The pixel value of the scaled image in the coordinate And the channel , The normalized pixel output value is The coordinate value is; The mosaic data enhancement method is used, 4 training images are randomly selected, and a new image is spliced by random scaling, rotation, mirroring and cropping to expand the dam crack sample; the anchor box driving strategy of the YOLO series is used, the anchor box suitable for different size targets is defined in advance, 9 kinds of anchor boxes of different sizes are obtained by clustering, and the anchor box suitable for different size crack targets is obtained. During training, the intersection over union IoU of the target box and the 9 kinds of anchor boxes is calculated, if IoU is greater than or equal to 0.25, the current anchor box is used as the positive sample anchor box of the current target, and participates in the prediction loss calculation, otherwise it is a negative sample anchor box, and only the background confidence loss is calculated.

[0016] The preferred technical scheme in the embodiment of the application has the following beneficial effects: the mosaic data enhancement and anchor box driving strategy are used, the training sample is diversified, the anchor box matching is optimized, the adaptability and detection robustness of the model to different size cracks are improved, the generalization ability and detection precision of the model are enhanced, and the change of crack size and shape is effectively coped with.

[0017] Another object of the application is to provide a deep learning network unmanned aerial vehicle perspective dam crack detection system.

[0018] To solve the above technical problems, the application provides the following technical scheme: a deep learning network unmanned aerial vehicle perspective dam crack detection system, comprising: a collection module, a training module, a processing module and a detection module. The collection module collects dam surface images by an unmanned aerial vehicle and pre-processes the collected images; The training module constructs a target detection network with a fusion attention mechanism, divides the pre-processed images into a training set, a validation set and a test set, and trains the target detection network; The processing module inputs the to-be-detected image into the trained network, outputs the boundary box coordinates of the crack target, quantizes the trained model and deploys it to a backend server; The detection module, the unmanned aerial vehicle transmits a real-time picture to a server, and a back-end server analyzes the picture and inputs a model for real-time detection.

[0019] The application provides a computer device, including a memory and a processor, the memory stores a computer program, characterized in that the processor executes the computer program to realize the steps of the deep learning network unmanned aerial vehicle perspective dam crack detection method.

[0020] The application provides a computer readable storage medium, which stores a computer program, characterized in that the computer program is executed by a processor to realize the steps of the deep learning network unmanned aerial vehicle perspective dam crack detection method.

[0021] The application has the advantages that the deep learning network unmanned aerial vehicle perspective dam crack detection method based on the attention mechanism has the advantages of high safety and high detection efficiency compared with the existing artificial dam inspection, and has the advantage in the detection speed of the algorithm, and due to the advantage in the detection speed, the unmanned aerial vehicle picture does not need to be framed, the missing detection rate is low and the detection accuracy is high for the scene of rapid flight of the unmanned aerial vehicle, and the anti-interference capability of the unmanned aerial vehicle detection is further enhanced due to the use of the sample expansion and sample oversampling technologies. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced, and obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the application is shown in the overall flowchart.

[0024] Figure 2 A data set construction flowchart of the deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the application is shown.

[0025] Figure 3 An algorithm training flowchart of the deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the application is shown.

[0026] Figure 4 A model quantization flowchart of the deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the application is shown.

[0027] Figure 5A deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the present application provides an unmanned aerial vehicle detection flowchart.

[0028] Figure 6 An algorithm model training process structure diagram of a deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the present application.

[0029] Figure 7 A channel attention mechanism of a deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the present application.

[0030] Figure 8 A RepConv module structure of a deep learning network unmanned aerial vehicle perspective dam crack detection method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0032] Embodiment 1, refer to Figures 1-8 For an embodiment of the present application, the embodiment provides a deep learning network unmanned aerial vehicle perspective dam crack detection method, comprising: S100, collecting dam surface images through an unmanned aerial vehicle, and pre-processing the collected images; S200, constructing a target detection network fused with an attention mechanism, dividing the pre-processed images into a training set, a verification set and a test set, and training the target detection network; S300, inputting a to-be-detected image into the trained network, outputting the boundary box coordinates of the crack target, and quantifying and deploying the trained model to a backend server; the unmanned aerial vehicle transmits real-time shooting pictures to the server, the backend server parses the pictures and inputs the model for real-time detection; It should be noted that the existing dam crack detection method mainly relies on manual inspection, and has the problems of low efficiency, high risk, strong subjectivity and difficulty in comprehensive coverage. The automatic method based on traditional image processing or early deep learning model often has low detection accuracy, high false detection and missed detection rates, and is difficult to meet the demand of real-time inspection under the unmanned aerial vehicle perspective due to factors such as complex image background, variable crack size, poor lighting conditions and low quality of remote transmission images.

[0033] Therefore, in order to solve the above problems, through the steps of S100-S300, an end-to-end automatic detection scheme from data acquisition, network optimization to engineering deployment is realized. The method ensures the comprehensiveness of the data through multi-scale image acquisition, enhances the feature extraction capability of the micro cracks by using the deep learning network with the fusion attention mechanism, and finally solves the problem of dam crack detection in complex environment through model quantization and streaming media transmission technology.

[0034] Embodiment 2, refer to Figures 1-8 In an embodiment of the present application, the embodiment provides a deep learning network unmanned aerial vehicle perspective dam crack detection method, comprising: In the embodiment of the present application, in S100, the dam surface image is collected by the unmanned aerial vehicle, and the collected image is preprocessed, including the following steps S101-S102: In the embodiment of the present application, S101, the dam surface image is collected by the unmanned aerial vehicle, including the following steps A1-A3: A1, two ways of far shooting and close shooting are adopted: A2, according to the actual site disposal experience and repeated test confirmation, when shooting from a distance, the unmanned aerial vehicle is 5-10m away from the dam, the adjacent image overlap rate is 60%-70%, and the dam body panoramic image is obtained; A3, when shooting close, the unmanned aerial vehicle is 1-3m away from the dam, and high-definition detail images are shot for suspected crack areas.

[0035] In an optional embodiment, the dam surface image collected in S101 can be adjusted in shooting distance and overlap rate, and two ways of far shooting and close shooting are adopted: When shooting from a distance, the unmanned aerial vehicle is 8-12m away from the dam, the adjacent image overlap rate is 50%-60%, and the dam body panoramic image is obtained; when shooting close, the unmanned aerial vehicle is 2-4m away from the dam, and high-definition detail images are shot for suspected crack areas; but due to the increase of shooting distance and the decrease of overlap rate, the image resolution is reduced and the panoramic splicing is incomplete, which affects the accuracy of crack detection.

[0036] In another optional embodiment, the dam surface image collected in S101 can also change the shooting order and parameters, adopt the way of shooting close first and then shooting far, collect details in suspected crack areas first, and then shoot panoramas, when shooting close, the unmanned aerial vehicle is 1-2m away from the dam, and high-definition detail images are shot, when shooting far, the unmanned aerial vehicle is 6-9m away from the dam, the adjacent image overlap rate is 55%-65%, and the dam body panoramic image is obtained; but due to the change of shooting order, the dam surface cannot be fully covered, resulting in missing crack areas, and the unstable overlap rate reduces the quality of panoramic image.

[0037] In an embodiment of the present application, S102, the collected images are preprocessed, including the following steps B1-B3: B1, for the low-resolution image taken from a distance, enlarge it by 4 times using a super-resolution network (such as SwinIR), and then denoise it by bilateral filtering; wherein the super-resolution network is SwinIR, which realizes image detail reconstruction through a Transformer structure.

[0038] B2, for the high-definition image taken from a close distance, use mean filtering and wavelet denoising processing; both images are cropped into 640x640 pixel blocks with an overlap rate of 20% between blocks.

[0039] B3, 2, the cropped image blocks are subjected to random brightness and contrast adjustment for data enhancement, with an enhancement amplitude of ±15%.

[0040] In an optional embodiment, the preprocessing in S102 can be, for the low-resolution image taken from a distance, enlarge it by 4 times using a bicubic interpolation algorithm, and then denoise it using Gaussian filtering; for the high-definition image taken from a close distance, use median filtering and Fourier transform denoising processing; then both types of images are cropped into 640x640 pixel blocks with an overlap rate of 20% between blocks; the cropped image blocks are subjected to random rotation and horizontal flipping for data enhancement, with a rotation angle range of ±10 degrees.

[0041] In another optional embodiment, the preprocessing in S102 can also be, for the low-resolution image taken from a distance, enlarge it by 4 times using EDSR (enhanced deep super-resolution network), and then denoise it using non-local mean processing. Wherein EDSR is a super-resolution method based on convolutional neural network, which can improve the resolution of the image; for the high-definition image taken from a close distance, use guided filtering and wavelet threshold denoising processing; then both types of images are cropped into 640x640 pixel blocks with an overlap rate of 20% between blocks; the cropped image blocks are subjected to random color jittering (including brightness, saturation and hue fine tuning) for data enhancement, with an adjustment amplitude of ±10%.

[0042] Furthermore, the input is a low-quality image with a resolution of , wherein is the height, is the width, is the number of channels, such as an RGB image , which is subjected to a shallow feature extraction module to obtain shallow features The input deep feature extraction module captures long-distance feature dependencies through a residual SwinTransformer block (RSTB) and outputs deep features ; finally, the reconstruction module fuses and The feature information is used to output a high-quality restored image through operations such as upsampling and convolution. (The resolution is adjusted according to the task requirements, such as in super-resolution tasks.) resolution is (This refers to the over-scoring factor). The specific steps are as follows: The shallow feature extraction: Use a single The convolutional layer is the core component, and the number of convolutional kernels is set to... (Adjust according to task requirements; typical values ​​are 64 or 128), with a stride of 1 and padding of 1, ensuring that the resolution of the feature map after convolution is consistent with the low-quality input image. A ReLU activation function is applied after the convolutional layer to introduce non-linear transformation capabilities, enhancing the module's ability to express complex image features and avoiding the linear bottleneck problem in the feature extraction process.

[0043] Set low quality image The pixel value Let i, j, and k be the coordinates of a pixel in the height, width, and channel dimensions, respectively. The weights of the 3×3 convolution kernel are... It contains d weight matrices with height a, width b, and number of channels c, where d is the shallow feature layer. The number of channel dimensions, shallow features The calculation is as follows: ,in, For the first The bias term for each output channel. The function is defined as ; The deep feature extraction: As the core module of the algorithm, it is responsible for capturing long-range feature dependencies in the image (such as cross-regional texture associations and structural consistency). At the same time, it alleviates the gradient vanishing problem of deep networks through residual connections, improving the depth and accuracy of feature extraction. This module consists of N cascaded residual Swin Transformer blocks (RSTB) and a final 3×3 convolutional layer.

[0044] Each RSTB contains M cascaded Swing Transformer layers (M is typically 2 or 4), one multilayer perceptron (MLP), and two residual connections.

[0045] The core of the Swin Transformer layer is a window multi-head self-attention (W-MSA) mechanism, which divides the feature map into non-overlapping windows (the window size is set to , and the typical value is 8x8 or 16x16), and only performs self-attention calculation within each window, which greatly reduces the computational complexity. The feature map of the input RSTB is divided into multiple non-overlapping windows. If the feature map resolution cannot be divided evenly, zero padding is used to ensure uniform window size.

[0046] For the features within each window, the features are first assembled and flattened into a sequence with a length of , and then the correlation weight of each feature pixel with the remaining pixels in the window is calculated through the multi-head attention mechanism: , where Concat is the merging operation, is the number of attention heads, is the output projection matrix, and the attention head formula is as follows: wherein, , are the query, key, and value matrices of the i-th head, is the dimension of the hidden layer, and M is the mask matrix to avoid interference between windows; After N RSTBs are connected in series, the output feature map needs to pass through a 3x3 convolution layer to adjust the feature channel from to the same number of channels as the shallow layer feature , to obtain the deep layer feature . Finally, according to the specific image restoration task, through feature fusion, upsampling, convolution, etc., the shallow layer feature and the deep layer feature are converted into a high-quality image by a high-quality image reconstruction module, to realize the conversion of a low-quality image to a high-quality image.

[0047] In the embodiments of the present application, the target detection network with fusion attention mechanism is constructed in S200, the preprocessed image is divided into a training set, a validation set and a test set, and the target detection network is trained, including the following steps S201-S202: S201, constructing a target detection network with fusion attention mechanism: taking a YOLO series network (such as YOLOv7) as a basic framework, embedding a channel attention module in the neck (Neck) of the feature extraction network, wherein the channel attention sub-module generates channel weights by fusing global average pooling and global maximum pooling.

[0048] ​​The feature extraction network of the YOLO series network adopts CSPDarknet, the neck thereof adopts a PANet structure for multi-scale feature fusion, and an SC-SAM attention module is embedded in a feature fusion layer of the PANet.

[0049] The specific implementation of the channel attention sub-module is that global average pooling and global maximum pooling are respectively performed on an input feature map to obtain two 1x1xC feature vectors, the channel weights are generated by a Sigmoid activation function after fusion through a full connection layer, and the channel weights are multiplied with the original feature map.

[0050] The specific implementation of the spatial attention sub-module is that average pooling and maximum pooling are performed on the feature map processed by the channel attention in the channel dimension to obtain two HxWx1 feature maps, the spatial weights are generated by a 3x3 convolution and a Sigmoid activation function after the HxWx2 feature map is obtained by splicing, and the spatial weights are multiplied with the input feature map.

[0051] Specifically, the main steps of the target detection network principle process of the fusion attention mechanism are as follows: Preprocessing: The original image is converted into standardized data meeting the input requirements of the model, and the model generalization capability is improved through data enhancement. The size of the input image is unified through scaling or padding, and the pixel point value is normalized to [0, 1] through a normalization method, and the standardized value is obtained according to the statistical mean and standard deviation of the training set: wherein is the pixel value of the scaled image at the coordinate and the channel , is the normalized pixel output value, is the coordinate value; Data enhancement: The mosaic data enhancement method is used, four training images are randomly selected, and a new image is spliced by randomly scaling, rotating, mirroring and cropping. In this way, the sample in the specific scene of the dam crack is expanded, and the small target detection capability of the crack is improved.

[0052] Anchor box initialization: The anchor box driving strategy of the YOLO series is used, and anchor boxes suitable for different size targets are defined in advance. Due to the large difference in sample size of the pictures collected by the unmanned aerial vehicle, nine anchor boxes of different sizes are obtained by clustering, which are suitable for different size crack targets. During training, the intersection over union (IoU) of the target box and the nine anchor boxes is calculated, if IoU≥0.25, the anchor box is used as the positive sample anchor box of the target, and participates in the subsequent prediction loss calculation; otherwise, it is a negative sample anchor box, and only the background confidence loss is calculated.

[0053] Backbone feature extraction: The backbone feature extraction network is composed of residual blocks of convolutional layers, pooling layers, and activation layers, and performs down-sampling and step-by-step extraction of image features at each level through layer-by-layer combination; The convolutional layer uses the Focus network structure, which specifically operates on every other pixel in a picture to obtain a value. At this time, four independent feature layers are obtained, which are then stacked. At this time, the width and height information is concentrated into the channel information, and the input channel is expanded by four times. In order to meet the detection efficiency of the model, the activation function is selected as the relu activation function, which directly sets the value less than 0 to 0. The formula of the relu activation function is as follows: Neck feature enhancement: The neck feature enhancement module is based on the backbone feature extraction module and adopts the feature pyramid technology to extract three different scale feature layer information during feature extraction, which are located at the middle layer, the middle lower layer, and the bottom layer. Each layer is up-sampled by bilinear interpolation and spliced in the channel dimension with the corresponding feature layer of the same size. The spliced features are extracted by a convolution block, then down-sampled, and then merged in the channel dimension with the processed feature layer of the same size.

[0054] Channel attention: The convolution block in the above pyramid technology includes a channel attention module. Attention mechanisms have been widely used in deep learning, and their role is to selectively enhance feature extraction. In order to further improve the learning ability of the network, the algorithm model is improved, and an attention mechanism module is also added. In the embodiment of the present application, the attention mechanism of the convolutional neural network adopts a channel attention mechanism, including the following steps C1-C2: C1, the channel attention mechanism extracts features according to the channel dimension of the feature map to obtain a 1x1xchannel number weight matrix, and then multiplies the two matrices to obtain a weighted feature map. C2, since the weight matrix can be learned, the deep network is updated through continuous iteration, and the network can selectively strengthen or weaken the features in some channels. Such a module is effective for strengthening the classification task, so the channel attention mechanism is added to the present application to strengthen the feature selection ability of the network.

[0055] In an optional embodiment, the attention mechanism of the convolutional neural network of S201 can adopt a spatial attention mechanism In another optional embodiment, the attention mechanism of the convolutional neural network of S201 can adopt a hybrid attention mechanism that combines the spatial attention mechanism and the channel attention module.

[0056] It is worth noting that the reason why the spatial attention mechanism or the hybrid attention mechanism is not selected in the present application is that the spatial attention mechanism is used to enhance the rotation invariance ability of the network, and can play an excellent role for general object detection tasks or rotation object detection tasks. However, for special object detection tasks such as dam cracks, which usually have a fixed direction, there is no need to add a spatial attention mechanism. The hybrid attention mechanism, which combines the advantages of the channel attention mechanism and the spatial attention mechanism, has been widely used in various networks in recent years, but the problem of increased parameter quantity generated thereby can cause the network to fail to achieve real-time detection capability, and therefore is not selected.

[0057] Head detection: The present application adds a RepConv module in front of the three detection heads, respectively, and realizes model reparameterization in the same convolutional layer by combining 3*3 convolution and 1*1 convolution; The complex convolution path therein will be omitted in the inference process, and only the 1*1 convolution part is retained to participate in prediction, so that the model obtains higher precision without any burden of parameter quantity increase.

[0058] To ensure the speed of the detection task, the present application selects to directly output prediction for multi-scale feature maps, and learns key features when constructing a loss function: The size of the target contained in the detection frame is compared with the actual labeled frame, and the higher the overlap between the two represents the better prediction of the network; The position information of the target in the picture, if all the target positions are successfully detected in the prediction, zero false detection and zero missed detection will be calculated; Finally, the category to which the detected target belongs is judged whether it is consistent with the true label: The CIoU loss function when the mask is true is used to represent the size loss function of the prediction frame; The position loss function of the target is a cross-entropy loss function, The loss value when the target is correctly judged, The loss value when the target is incorrectly judged; The category loss function is made by using the BCE loss function (binary classification cross-entropy loss function); are weight coefficients of three loss functions, and should satisfy that the sum is 1, and in a normal case, in order to make the detection network pay more attention to the detection accuracy and position information, three weight hyperparameters can be set as 0.25, 0.5 and 0.25.

[0059] In order to accelerate the loss function calculation process, the application uses the design of the mask to design the loss function: the input size of the model is set to 640*640, and three feature sizes of output will be generated, which are 80*80, 40*40 and 20*20 respectively; For example, taking the 80*80 output feature as an example, a Boolean value matrix mask of the same size will be generated, and the grid where the bounding box appears will be recorded as True, and vice versa as False. The mask matrix with True Boolean value is calculated according to the anchor And , wherein is the statistical number of the mask matrix with True Boolean value. The target class loss function is calculated , and there is a hyperparameter , which is a weight parameter and has a value range of (0.5-1), which determines the degree to which the model pays attention to the mask with True.

[0060] S202, divide the preprocessed image into a training set, a validation set and a test set, and train the constructed network by using rotation, scaling and horizontal flip for data enhancement, and optimize the network parameters until the model converges; In the embodiment of the application, the image to be detected in S300 is input into the trained network, and the bounding box coordinates of the crack target are output, and the trained model is quantized and deployed to the backend server; the unmanned aerial vehicle transmits the real-time captured picture to the server, and the backend server analyzes the picture and inputs the model for real-time detection, including the following steps S301-S302: S301, input the image to be detected into the trained network, the division ratio of the training set, the validation set and the test set is 7:2:1, the initial learning rate is set to 0.001, and the Adam optimizer is used to output the bounding box coordinates of the crack target; The bounding box is subjected to non-maximum suppression processing, the threshold value of non-maximum suppression is set to 0.25, when the intersection over union IoU of two bounding boxes is greater than 0.25, the bounding box with the highest confidence is retained, and the remaining redundant boxes and the included relationship boxes are removed, to obtain the final crack detection result; based on the GPS coordinates and the attitude parameters, a three-dimensional model of the dam is constructed through oblique photography technology, and the crack bounding box is mapped to the three-dimensional model, and the length and the maximum width of the crack are calculated. In an embodiment of the present application, S302, quantizing the trained image, comprises the following steps D1-D2: D1, quantizing the trained image in onnx (open neural network exchange format) quantizes the FLOAT32 model to the INT8 model; D2, deploying the onnx-quantized image on the backend server; In an optional embodiment, the quantization in S302 can be using TensorRT quantization, converting the trained target detection network model to ONNX format, using the Python API or command line tool of TensorRT, importing the ONNX model, and selecting the INT8 precision mode. Generate quantization parameters through the calibration process (use part of the images in the validation set to calculate the activation value distribution) to optimize the model inference speed; deploy the constructed TensorRT engine file to the backend server, integrate it into the inference pipeline, and process the real-time pictures transmitted by the unmanned aerial vehicle.

[0061] In another optional embodiment, the quantization in S302 can also be using PyTorch static quantization, loading the trained model in PyTorch and setting it to evaluation mode, using the torch.quantization.prepare and torch.quantization.convert APIs of PyTorch for static quantization. First, calibrate the model with the validation set images to collect activation value statistics; quantize the model weights and activation values from FLOAT32 to INT8, convert the quantized model to TorchScript format, and deploy it to the backend server for real-time crack detection inference.

[0062] The unmanned aerial vehicle carries out on-site inspection work, transmits the real-time shooting pictures to the streaming server through the 5G module, generates a fixed rtmp streaming address, the backend server parses the fixed rtmp streaming address, inputs each frame of picture into the algorithm model, generates a detection result, and outputs the detection bounding box and detection category, confidence of the crack.

[0063] Embodiment 3 is an embodiment of the present application, and the above is a schematic scheme of a deep learning network unmanned aerial vehicle perspective dam crack detection method. It should be noted that the technical scheme of a deep learning network unmanned aerial vehicle perspective dam crack detection system belongs to the same concept as the technical scheme of the above-mentioned deep learning network unmanned aerial vehicle perspective dam crack detection method. The technical scheme of the deep learning network unmanned aerial vehicle perspective dam crack detection system in this embodiment is not described in detail, and the description of the technical scheme of the deep learning network unmanned aerial vehicle perspective dam crack detection method can be referred to.

[0064] The embodiment provides a deep learning network unmanned aerial vehicle perspective dam crack detection system, which comprises a collection module, a training module, a processing module and a detection module. The collection module collects dam surface images through an unmanned aerial vehicle and pre-processes the collected images. The training module constructs a target detection network fused with an attention mechanism, divides the pre-processed images into a training set, a verification set and a test set, and trains the target detection network. The processing module inputs a to-be-detected image into the trained network, outputs the boundary box coordinates of a crack target, simultaneously quantizes the trained model and deploys the model to a backend server. The detection module transmits a real-time shooting picture to the server by the unmanned aerial vehicle, and the backend server analyzes the picture and inputs the model for real-time detection.

[0065] The embodiment also provides an electronic device suitable for a deep learning network unmanned aerial vehicle perspective dam crack detection method, which comprises a memory and a processor.

[0066] The embodiment also provides a storage medium having a computer program stored thereon, and the program is executed by a processor to realize the deep learning network unmanned aerial vehicle perspective dam crack detection method.

[0067] The storage medium provided by the embodiment and the deep learning network unmanned aerial vehicle perspective dam crack detection method provided by the above embodiment belong to the same inventive concept, and the technical details not described in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by software and necessary general hardware, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH, a hard disk or an optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method of each embodiment of the present application.

[0069] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A method for detecting dam crack by deep learning network unmanned aerial vehicle perspective, characterized in that: The application relates to a method for detecting cracks on a dam surface based on a UAV (unmanned aerial vehicle) and a server. The method comprises the following steps: collecting dam surface images by a UAV and preprocessing the collected images; a target detection network is constructed by fusing an attention mechanism, the preprocessed images are divided into a training set, a verification set and a test set, and the target detection network is trained; the trained network is input with an image to be detected, and the coordinates of a bounding box of a crack target are output, and the trained model is quantized and deployed to a backend server; the UAV transmits real-time shooting pictures to the server, and the backend server analyzes the pictures and inputs the model for real-time detection.

2. The method for detecting dam cracks from the perspective of a drone using a deep learning network as described in claim 1, characterized in that: The collected dam surface images include GPS coordinates and attitude parameters recorded simultaneously when the dam surface images are collected by the UAV, and the images are shot from a distance and from close up; when shooting from a distance is confirmed, the distance between the UAV and the dam and the adjacent image overlap rate are set, and a low-resolution panoramic image of the dam body is obtained; when shooting from close up is confirmed, the distance between the UAV and the dam is set, and a detail image is shot. 3.The method of claim 2, wherein the method further comprises: determining a crack region of the dam based on the crack probability map. The preprocessing includes the following steps: after the low-resolution image shot from a distance is enlarged, the image is denoised by filtering; the detail image shot from close up is denoised, and the two types of processed images are cropped. 4.The method of claim 3, wherein the method further comprises: The training of the target detection network includes the following steps: according to the attention mechanism of a convolutional neural network, a target detection network is constructed by fusing an attention mechanism; the preprocessed images are divided into a training set, a verification set and a test set, the constructed network is trained by adopting rotation, scaling and horizontal flip for data enhancement, and network parameters are optimized until the model converges. 5.The method of claim 4, wherein the method further comprises: The real-time detection includes the following steps: the trained network is input with an image to be detected, the division ratio of the training set, the verification set and the test set is 7:2:1, the initial learning rate is set to 0.001, the Adam optimizer is adopted, and the coordinates of a bounding box of a crack target are output; non-maximum suppression is performed on the bounding box, the threshold of the non-maximum suppression is set to 0.25, when the intersection over union (IoU) of two bounding boxes is greater than 0.25, the bounding box with the highest confidence is retained, the remaining redundant boxes and the containing relationship boxes are removed, and finally, the crack detection result is obtained; based on the GPS coordinates and the attitude parameters, a three-dimensional dam model is constructed by using oblique photography technology, the crack bounding box is mapped to the three-dimensional model, and the length and the maximum width of the crack are calculated; the trained image is quantized by onnx, the FLOAT32 model is quantized to the INT8 model, and the onnx-quantized image is deployed on the backend server; the UAV carries out on-site inspection work, the real-time shooting pictures are transmitted to the streaming server through a 5G module, a fixed rtmp streaming address is generated, the backend server analyzes the fixed rtmp streaming address, each frame of picture is input into the algorithm model, the detection result is generated, and the detection bounding box, the detection category and the confidence of the crack are output. 6.The method of claim 5, wherein the method further comprises: The preprocessing further comprises inputting a low-quality image , a resolution of , wherein is a height, is a width, is a number of channels, shallow feature extraction is performed to obtain shallow features , , deep feature extraction is performed, long-distance feature dependencies are captured through a residual SwinTransformer block, and deep features are output , a reconstruction module fuses feature information of and , and outputs a high-quality recovered image through upsampling and convolution operations ; ​ The shallow feature extraction assumes low-quality images. The pixel value Let i, j, and k be the coordinates of a pixel in the height, width, and channel dimensions, respectively. The weights of the 3×3 convolution kernel are... It contains d weight matrices with height a, width b, and number of channels c, where d is the shallow feature layer. The number of channel dimensions, shallow features The calculation is as follows: ,in, For the first The bias term for each output channel. The function is defined as ; The deep feature extraction is composed of N serial residual SwinTransformer blocks and 1 final 3x3 convolution layer; the core of the SwinTransformer block is a window attention mechanism, which divides the feature map into non-overlapping windows, and the window size is set to , and only self-attention calculation is performed inside each window, the feature map input into the residual SwinTransformer block is divided into multiple non-overlapping windows, and if the feature map resolution is not divisible, the window size is ensured to be consistent by zero padding; ​ For each feature within a window, first assemble it into a sequence with length Then, the correlation weight of each feature pixel and the remaining pixels in the window is calculated by the multi-head attention mechanism: where Concat is the merging operation, is the number of attention heads, is the output projection matrix, and the attention head formula is as follows: wherein, , are the first head query, key, value matrices, respectively, dimension of the hidden layer, M is a mask matrix to avoid window interference.

7. The method for detecting dam cracks from a UAV perspective using a deep learning network as described in claim 6, characterized in that: The construction of the target detection network by fusing an attention mechanism includes the following steps: the size of the input image is unified by scaling or padding, the pixel point value is normalized to [0, 1] by a normalization method, and the statistical mean and the standard deviation of the training set are used for standardization. wherein is the scaled image in coordinates and channel pixel value, is the normalized pixel output value, is the coordinate value; Using the mosaic data augmentation method, 4 training images are randomly selected, and a new image is spliced by random scaling, rotation, mirroring and cropping to expand the dam crack sample; using the anchor box driving strategy of the YOLO series, the anchor box suitable for different size targets is defined in advance, and 9 different size anchor boxes are obtained by clustering, which adapt to different size crack targets. During training, the intersection over union IoU of the target box and the 9 anchor boxes is calculated. If IoU is greater than or equal to 0.25, the current anchor box is the positive sample anchor box of the current target and participates in the prediction loss calculation, otherwise it is the negative sample anchor box and only the background confidence loss is calculated.

8. A deep learning network unmanned aerial vehicle perspective dam crack detection system, applying a deep learning network unmanned aerial vehicle perspective dam crack detection method according to any one of claims 1-7, characterized in that, It comprises: a collection module, a training module, a processing module and a detection module; The collection module collects the dam surface image by the unmanned aerial vehicle, and pre-processes the collected image; The training module constructs a target detection network with a fusion attention mechanism, divides the pre-processed image into a training set, a verification set and a test set, and trains the target detection network; The processing module inputs the to-be-detected image into the trained network, outputs the boundary box coordinates of the crack target, quantizes the trained model and deploys it to the backend server; The detection module transmits the real-time shooting picture to the server by the unmanned aerial vehicle, and the backend server analyzes the picture and inputs the model for real-time detection. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to realize the steps of the deep learning network unmanned aerial vehicle perspective dam crack detection method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the deep learning network unmanned aerial vehicle perspective dam crack detection method in any one of claims 1 to 7.

Images