Method, device, equipment and medium for detecting surface damage of concrete building
By improving the target detection model and combining a lightweight skeleton network and depthwise separable convolution, the problems of slow detection speed and insufficient extraction of multiple damage features in existing models are solved, achieving high-precision and efficient damage detection of concrete buildings, especially with good applicability in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGSHA UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2023-11-17
- Publication Date
- 2026-05-22
AI Technical Summary
Existing deep learning models have a huge number of parameters in concrete building damage detection, and the detection speed cannot meet the real-time requirements. They also lack the ability to extract multiple damage features, have poor applicability and generalization ability, and perform poorly, especially in complex environments.
An improved object detection model is adopted, including a backbone network, a spatial pyramid pooling module, a bidirectional pyramid feature fusion layer, and a YOLO Head. Through transfer learning and image enhancement techniques, a lightweight YOLOv4 architecture is constructed. Fusion-MBConv Block and Inverted Residual Block are introduced, and depthwise separable convolution and optimized activation functions are combined to improve feature extraction capabilities and detection accuracy.
It improves the accuracy and speed of damage detection in concrete structures, enhances the applicability of the model in complex environments, and is suitable for real-time multi-damage detection.
Smart Images
Figure CN117522826B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, equipment and medium for detecting surface damage of concrete buildings. Background Technology
[0002] During long-term service, concrete structures can suffer varying degrees of damage to their surfaces due to factors such as temperature, humidity, and load. This damage can include cracks, concrete spalling, and exposed rebar, leading to structural performance degradation or even failure. Therefore, it is essential to promptly detect this damage and implement appropriate repair and reinforcement measures.
[0003] Currently, deep learning methods have shown great application potential in damage identification in civil engineering and infrastructure. However, existing models typically have a large number of parameters, which severely limits their detection speed and prevents them from meeting the real-time requirements of practical applications. Most existing methods focus on identifying a limited number of damage types, such as cracks, exposed rebar, and concrete spalling, resulting in insufficient feature extraction capabilities for multiple damage types and low accuracy in detecting damage in concrete structures. Furthermore, existing research on concrete structure damage images is mostly based on close-up shots, with each image containing only a small number of damages. There is also a lack of research on damage detection under complex environments such as low lighting and severe weather, leading to limited applicability and poor generalization ability of the damage identification models. Summary of the Invention
[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.
[0005] The main objective of this invention is to provide a method, apparatus, equipment, and medium for detecting surface damage in concrete structures, which can improve the ability to extract damage features from models, reduce the computational load of models, and improve the accuracy, speed, and generalization ability of model damage detection.
[0006] To achieve the above objectives, a first aspect of the present invention provides a method for detecting surface damage of concrete structures, the method comprising:
[0007] Acquire images of the surface of a concrete building to be inspected;
[0008] A target detection model is constructed, which includes a backbone network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The backbone network includes cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks.
[0009] The image to be detected is input into the skeleton network to obtain a multi-scale feature map output by the skeleton network; the feature map output by the last layer of the skeleton network is input into the spatial pyramid pooling module to obtain a feature map output by the spatial pyramid pooling module; the feature map output by the spatial pyramid pooling module and the remaining feature maps of different scales output by the intermediate layers of the skeleton network are input into the bidirectional pyramid feature fusion layer to obtain a feature map output by the bidirectional pyramid feature fusion layer; the feature map output by the bidirectional pyramid feature fusion layer is input into the YOLO Head to obtain the damage detection result of the concrete building surface output by the YOLO Head.
[0010] In some embodiments, the object detection model is trained in the following manner:
[0011] Select the training images with labeled surface damage on concrete buildings;
[0012] Construct an initial detection model based on YOLOv4; wherein, the initial detection model includes a backbone network based on the YOLOv4 architecture, the spatial pyramid pooling module, the bidirectional pyramid feature fusion layer, and the YOLO Head;
[0013] An intermediate detection model is generated based on the initial detection model; wherein the intermediate detection model has the same spatial pyramid pooling module, bidirectional pyramid feature fusion layer and YOLOHead as the initial detection model, and the backbone network of the intermediate detection model is MonileNetv3;
[0014] The intermediate detection model is trained using the image to be trained to obtain the trained intermediate detection model;
[0015] The target detection model is generated based on the intermediate detection model after training;
[0016] The target detection model is trained using the images to be trained until training ends.
[0017] In some embodiments, the convolutional module following each fusion layer in the bidirectional pyramid feature fusion layer sequentially includes 3 cascaded standard convolutional layers and 2 depthwise separable convolutional layers;
[0018] The skeleton network and the spatial pyramid pooling module are further connected by cascaded two layers of standard convolutions and one layer of depthwise separable convolutions; the step of inputting the feature map output from the last layer of the skeleton network into the spatial pyramid pooling module includes:
[0019] The feature map output from the last layer of the skeleton network is input into the cascaded two standard convolutional layers and one depthwise separable convolutional layer, and the output feature map is input into the spatial pyramid pooling module.
[0020] The bidirectional pyramid feature fusion layer and the YOLO Head further include one depthwise separable convolution layer; the step of inputting the feature map output by the bidirectional pyramid feature fusion layer into the YOLO Head includes:
[0021] The feature map output from the bidirectional pyramid feature fusion layer is input into a 1-layer depthwise separable convolution, and the output feature map is input into the YOLO Head.
[0022] In some embodiments, after selecting the training image of the labeled concrete building surface damage, the concrete building surface damage detection method further includes:
[0023] The training image is subjected to a first enhancement transformation and / or a second enhancement transformation; wherein the first enhancement transformation of the training image includes at least two enhancement transformations among motion blur, cloud / fog, fog / frost, heavy rain / snow, rain, snow spots, and noise enhancement transformation; the second enhancement transformation of the training image includes one enhancement transformation among halving or doubling the contrast of the training image, increasing brightness, and enhancing color saturation.
[0024] In some embodiments, the activation function of each convolution in the object detection model is a combination of Hard-swish and SiLU activation functions.
[0025] In some embodiments, the attention mechanism in the Fusion-MBConv Block and the Inverted Residual Block includes one of the following attention mechanisms: SENet, CANet, and DANet.
[0026] In some embodiments, the concrete structure includes a concrete bridge; damage to the surface of the concrete structure includes at least one of concrete degradation, honeycombing, concrete spalling, exposed rebar, erosion, voids, and cracks.
[0027] A second aspect of this invention provides a device for detecting surface damage to concrete structures, the device comprising:
[0028] The image acquisition unit is used to acquire the image of the concrete building surface to be detected.
[0029] The model building unit is used to build an object detection model, which includes a skeleton network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The skeleton network includes cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks.
[0030] The damage detection unit is used to input the image to be detected into the skeleton network to obtain a multi-scale feature map output by the skeleton network; input the feature map output by the last layer of the skeleton network into the spatial pyramid pooling module to obtain a feature map output by the spatial pyramid pooling module; input the feature map output by the spatial pyramid pooling module and the remaining feature maps of different scales output by the middle layers of the skeleton network into the bidirectional pyramid feature fusion layer to obtain a feature map output by the bidirectional pyramid feature fusion layer; and input the feature map output by the bidirectional pyramid feature fusion layer into the YOLO Head to obtain the damage detection result of the concrete building surface output by the YOLO Head.
[0031] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the above-described method for detecting surface damage of a concrete building.
[0032] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described method for detecting surface damage in concrete structures.
[0033] One embodiment of this application provides a method for detecting surface damage in concrete structures. The applicable target detection model includes a skeleton network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The skeleton network sequentially includes cascaded convolutional layers, six Fusion-MBConv Blocks, and nine Inverted Residual Blocks. Unlike the skeleton network of the YOLOv4 network, the target detection model provided in this embodiment constructs a novel skeleton network through sequentially cascaded convolutional layers, six Fusion-MBConv Blocks, and nine Inverted Residual Blocks. The introduction of Fusion-MBConv Block and Inverted Residual Block structures solves the problem of low feature extraction efficiency in the original YOLOv4 skeleton network, reducing network computation and improving the detection accuracy of the target detection model.
[0034] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a schematic flowchart of a method for detecting surface damage of a concrete building according to an embodiment of this application;
[0037] Figure 2 This is a schematic diagram of the structure of Fusion-MBConv Block and Inverted ResidualBlock provided in one embodiment of this application;
[0038] Figure 3 This is a schematic diagram of the transfer training process of an object detection model provided in one embodiment of this application;
[0039] Figure 4 This is a data diagram of seven types of surface damage on concrete bridges provided in one embodiment of this application;
[0040] Figure 5 This is a schematic diagram of the process for image amplification of surface damage on a concrete bridge under complex conditions, provided in one embodiment of this application.
[0041] Figure 6 This is a schematic diagram of the architecture of a target detection model provided in one embodiment of this application;
[0042] Figure 7 This is a schematic diagram of transfer learning for a target detection model of surface damage on a concrete bridge provided in one embodiment of this application;
[0043] Figure 8 This is a flowchart illustrating the detection process of bridge surface damage using a target detection model provided in one embodiment of this application.
[0044] Figure 9 This is a Precision-Recall curve of an object detection model provided in one embodiment of this application;
[0045] Figure 10 This is the result of identifying surface damage on a concrete bridge using a target detection model under different fields of view, provided in one embodiment of this application.
[0046] Figure 11 This is the result of a target detection model in a complex environment provided in one embodiment of this application identifying surface damage on a concrete bridge;
[0047] Figure 12 This is a schematic diagram of the structure of a device for detecting surface damage of a concrete building according to an embodiment of this application;
[0048] Figure 13 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0050] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0052] During long-term service, concrete structures can suffer varying degrees of damage to their surfaces due to factors such as temperature, humidity, and load. This damage can include cracks, concrete spalling, and exposed rebar, leading to structural performance degradation or even failure. Therefore, it is essential to promptly detect this damage and implement appropriate repair and reinforcement measures.
[0053] Currently, deep learning methods have shown great application potential in damage identification in civil engineering and infrastructure. However, existing models typically have a large number of parameters, which severely limits their detection speed and prevents them from meeting the real-time requirements of practical applications. Most existing methods focus on identifying a limited number of damage types, such as cracks, exposed rebar, and concrete spalling, resulting in insufficient feature extraction capabilities for multiple damage types and low accuracy in detecting damage in concrete structures. Furthermore, existing research on concrete structure damage images is mostly based on close-up shots, with each image containing only a small number of damages. There is also a lack of research on damage detection under complex environments such as low lighting and severe weather, leading to limited applicability and poor generalization ability of the damage identification models.
[0054] To address the aforementioned technical deficiencies, referring to... Figure 1 One embodiment of this application provides a method for detecting surface damage to concrete structures, the method comprising steps S110-S130:
[0055] Step S110: Obtain the image of the concrete building surface to be inspected.
[0056] Step S120: Construct an object detection model. The object detection model includes a skeleton network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLOHead based on the YOLOv4 architecture. The skeleton network includes cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 InvertedResidual Blocks.
[0057] Step S130: Input the image to be detected into the skeleton network to obtain multi-scale feature maps output by the skeleton network; input the feature map output by the last layer of the skeleton network into the spatial pyramid pooling module to obtain the feature map output by the spatial pyramid pooling module; input the feature map output by the spatial pyramid pooling module and the other feature maps of different scales output by the intermediate layers of the skeleton network into the bidirectional pyramid feature fusion layer to obtain the feature map output by the bidirectional pyramid feature fusion layer; input the feature map output by the bidirectional pyramid feature fusion layer into the YOLO Head to obtain the damage detection results of the concrete building surface output by the YOLO Head.
[0058] In this embodiment, concrete structures include, but are not limited to, concrete industrial buildings (such as production workshops), concrete residential buildings (such as houses, apartments, etc.), and concrete transportation buildings (bridges, highways, etc.).
[0059] This embodiment aims to study damage detection in concrete structures, thereby achieving early detection and early warning. Taking bridges as an example, damage includes, but is not limited to, concrete splitting, honeycombing, concrete spalling, exposed rebar, erosion, holes, and cracks.
[0060] In step S110, the first step is to acquire the image of the concrete building surface to be inspected. The image to be inspected is the image used to input into the target detection model for building damage detection. The image can be obtained by taking a picture with a camera, or by cropping from a video, or by obtaining it from the Internet; no specific limitation is made here.
[0061] In step S120, an object detection model is constructed. The object detection model is used to detect surface damage of concrete buildings from the image to be detected. In this embodiment, the object detection model includes a skeleton network, a spatial pyramid pooling module based on the YOLOv4 architecture (released in 2020, it is a widely used deep learning-based object detection algorithm), a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The skeleton network includes cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks.
[0062] First, we introduce the YOLOv4 network architecture. The YOLOv4 network consists of a backbone network (CSPDarkNet53), a Spatial Pyramid Pooling (SPP) module, a Bidirectional Pyramid Feature Fusion (PANet) layer, and a YOLO Head. The YOLOv4 backbone network differs from the backbone network of this object detection model, while the Spatial Pyramid Pooling module, the Bidirectional Pyramid Feature Fusion (PANet) layer, and the YOLO Head are the same as those in this object detection model. The architecture of the YOLOv4 network is common knowledge in the field and will not be elaborated upon here.
[0063] Secondly, unlike the YOLOv4 network, the object detection model's backbone network is a redesigned network, specifically including cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks. The YOLOv4 network's backbone network is CSPDarkNet53. The backbone network of this embodiment's object detection model is obtained through structural improvements as follows: First, CSPDarkNet53 was replaced with MobileNetv3 (proposed by the Google team in 2019); then, because the depthwise separable convolutions in the shallow layers of MobileNetv3 were not fully utilized, resulting in insufficient feature extraction capabilities, MobileNetv3 was further improved by replacing the shallow layers with 6 Fusion-MBConv Blocks, forming a backbone network composed of convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks.
[0064] like Figure 2 As shown, Figure 2 The structures of the Fusion-MBConv Block and the Inverted Residual Block are shown. In some embodiments of this application, the Fusion-MBConv Block uses SENet as the attention mechanism by default, but in some cases, the attention mechanism can be adjusted, for example, using CANet and DANet as the attention mechanism.
[0065] In step S130, the data processing flow of the object detection model is similar to that of the YOLOv4 network; that is, the processing flow of the spatial pyramid pooling module, the bidirectional pyramid feature fusion layer, and the YOLO Head is the same. Furthermore, like the YOLOv4 network's skeleton network, the object detection model's skeleton network also outputs three feature maps at different scales, and the network structure corresponding to the input of these three feature maps is also the same as the YOLOv4 network's processing flow. However, unlike the YOLOv4 network's data processing flow, the object detection model has redesigned its skeleton network. The skeleton network in this application includes convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks. Therefore, the generated feature maps are different from those generated by the YOLOv4 skeleton network. The object detection model's skeleton network has a stronger ability to extract damage features from concrete structures, resulting in higher accuracy of the generated feature maps.
[0066] The existing YOLOv4 skeleton network is Darknet-53, which has a complex structure and a huge number of parameters, making it unsuitable for high-precision real-time detection of surface damage on concrete bridges. Therefore, this method improves the YOLOv4 skeleton network using an object detection model. This model's skeleton network consists of cascaded convolutional layers, six Fusion-MBConv Blocks, and nine Inverted Residual Blocks. Unlike the YOLOv4 skeleton network, the object detection model's skeleton network introduces a Fusion-MBConv Block structure. While retaining the advantages of the YOLOv4 framework, the object detection model addresses the low feature extraction efficiency of the original YOLOv4 skeleton network, improving its damage feature extraction capabilities. This not only reduces network computation but also enhances the detection accuracy of the object detection model.
[0067] Reference Figure 3 In some embodiments, the target detection model is obtained through transfer learning, specifically including steps S210-S260:
[0068] Step S210: Select the training image of the concrete building surface damage that has been labeled.
[0069] Step S220: Construct an initial detection model based on YOLOv4; wherein, the initial detection model includes a backbone network based on the YOLOv4 architecture, a spatial pyramid pooling module, a bidirectional pyramid feature fusion layer, and a YOLO Head.
[0070] Step S230: Generate an intermediate detection model based on the initial detection model; wherein the intermediate detection model has the same spatial pyramid pooling module, bidirectional pyramid feature fusion layer and YOLO Head as the initial detection model, and the backbone network of the intermediate detection model is MonileNetv3.
[0071] Step S240: Train the intermediate detection model using the image to be trained to obtain the trained intermediate detection model.
[0072] Step S250: Generate an object detection model based on the trained intermediate detection model.
[0073] Step S260: Train the target detection model using the images to be trained until training ends.
[0074] In step S210, a training set is first selected, i.e., multiple labeled training images are chosen, with each image labeled with the surface damage category of the concrete building. In step S220, an initial detection model is generated directly using the YOLOv4 network. The initial detection model includes pre-trained weights from Darknet-53. In step S230, an intermediate detection model is generated based on the initial detection model, replacing the skeleton network in the initial detection model with MonileNetv3, forming an intermediate detection model composed of the skeleton network MonileNetv3, a spatial pyramid pooling module, a bidirectional pyramid feature fusion layer, and a YOLO Head. Then, in step S240, the intermediate detection model is trained using the training images to obtain a trained intermediate detection model, which now possesses prior knowledge of the training images. In step S250, a target detection model is generated based on the trained intermediate detection model, i.e., the shallow network in the skeleton network MonileNetv3 of the intermediate detection model is replaced with 6 Fusion-MBConv Blocks. Then in step S260, the target detection model is trained using the image to be trained until training ends, and a trained target detection model is obtained.
[0075] Directly training the object detection model can lead to overly random weights in the backbone network, resulting in poor feature extraction. Therefore, this embodiment employs a three-stage object detection model: generating an initial detection model with pre-trained Darknet-53 weights; generating an intermediate detection model based on the initial model; training the intermediate model; generating the object detection model based on the intermediate model; and finally training the object detection model. By using the pre-trained weights of the initial and intermediate detection models for transfer learning training, the object detection model inherits prior knowledge from both models, thereby improving its feature extraction performance.
[0076] In some embodiments, the convolutional module following each fusion layer in the bidirectional pyramid feature fusion layer sequentially includes 3 cascaded standard convolutional layers and 2 depthwise separable convolutional layers.
[0077] Between the skeleton network and the spatial pyramid pooling module, there are also cascaded 2 layers of standard convolutions and 1 layer of depthwise separable convolutions; step S130, inputting the feature map output from the last layer of the skeleton network into the spatial pyramid pooling module, includes:
[0078] Step S310: Input the feature map output from the last layer of the skeleton network into a cascaded 2-layer standard convolution and 1-layer depthwise separable convolution, and input the output feature map into the spatial pyramid pooling module.
[0079] The bidirectional pyramid feature fusion layer and the YOLO Head also include one depthwise separable convolution layer; step S130 inputs the feature map output by the bidirectional pyramid feature fusion layer into the YOLO Head, including step S320:
[0080] Step S320: Input the feature map output from the bidirectional pyramid feature fusion layer into a 1-layer depthwise separable convolution, and input the output feature map into the YOLO Head.
[0081] In this embodiment, depthwise separable convolution is used instead of standard convolution in the target detection model. This approach can improve feature fusion efficiency and reduce network computation parameters, thereby improving detection efficiency.
[0082] In some embodiments, after selecting training images of labeled concrete building surface damage, the method for detecting concrete building surface damage further includes:
[0083] Step S410: Perform a first enhancement transformation and / or a second enhancement transformation on the training image; wherein the first enhancement transformation on the training image includes at least two enhancement transformations among motion blur, cloud / fog, fog / frost, heavy rain / snow, rain, snow spots, and noise enhancement transformations; the second enhancement transformation on the training image includes one of enhancement transformations among reducing the contrast of the training image to half or twice its original value, increasing brightness, and enhancing color saturation.
[0084] In this embodiment, image enhancement is performed on the target detection model before training it with the target image, which can improve the accuracy of the target detection model training.
[0085] In some embodiments, the activation function for each convolution in the object detection model is a combination of Hard-swish and SiLU activation functions. By combining activation functions, the feature extraction capability of the object detection model can be improved.
[0086] In some embodiments, taking a concrete bridge as an example, a method for detecting surface damage on a concrete bridge is described in detail. This method includes the following steps:
[0087] Step S510: Establish a dataset of surface damage images of concrete bridges.
[0088] On-site data collection of surface damage on concrete bridges was conducted using drones, wall-climbing robots, and mobile devices such as smartphones. Simultaneously, based on the types of surface damage collected on-site, relevant, high-resolution surface damage images were collected from the internet to supplement the database. The collected surface damage data included seven common types of concrete bridge damage: concrete separation, honeycombing, concrete spalling, exposed rebar, erosion, voids, and cracks. Figure 4 As shown.
[0089] The "Labelimg" data annotation tool was used to annotate image samples of a dataset of surface damage on concrete bridges to create a database that can be used for model training. During the annotation process, individual damages in the image samples were labeled and saved according to the Pascal VOC dataset format, resulting in a dataset of multiple types of surface damage on concrete bridges. Each image sample yielded an .xml format annotation file containing information on the type, location, and shape of the surface damage.
[0090] Step S520: Simulate concrete surface damage images under complex environments.
[0091] Reference Figure 5Surface damage images under complex environments are usually difficult to collect. In this embodiment, the "Imgaug" data augmentation toolkit is used to simulate concrete bridge surface damage samples under severe weather conditions such as fog, rain, and snow, as well as complex environments such as motion blur, color changes, and noise interference, to augment the Pascal VOC dataset of concrete bridge surface damage.
[0092] Randomly select 2-4 enhancement transformations from motion blur, cloud and fog, fog / frost, heavy rain / snow, rain, snow spots and noise enhancement transformations. At the same time, randomly select 1 transformation from contrast to half or twice the original, brightness increase and color saturation enhancement transformations. Then set an appropriate enhancement factor to enhance the Pascal VOC dataset.
[0093] The damage dataset is obtained by operating on the Pascal VOC dataset in step S520.
[0094] It should be noted that during the training process of the neural network, the model is divided into a training set, a validation set, and a test set. This embodiment only shows the training process of the model using the training set. The process of validating the model using the validation set and testing the model using the test set is not shown.
[0095] Step S530: Establish a target detection model for surface damage on concrete bridges and train it.
[0096] Reference Figure 6 Using the YOLOv4 network as the initial detection model, the Darknet-53 backbone network of YOLOv4 is deemed insufficient for high-precision real-time detection of surface damage on concrete bridges due to its complex structure and large number of parameters. Therefore, to address this deficiency, the YOLOv4 backbone network CSPDarkNet53 is replaced with MonileNetv3, resulting in an intermediate detection model. Since the depthwise separable convolutions in the shallow layers of MonileNetv3 are not fully utilized, leading to insufficient feature extraction capabilities, the shallow layers of the intermediate detection model's MonileNetv3 backbone network are replaced with six Fusion-MBConv Blocks, resulting in the target detection model. While retaining the advantages of the basic YOLOv4 framework, the target detection model, through improvements in the backbone network and activation function combination, achieves a more lightweight network structure and stronger damage feature extraction capabilities, thus improving detection accuracy and speed.
[0097] Reference Figure 7To ensure that the target detection model for concrete bridge surface damage has sufficient generalization ability and detection accuracy, this embodiment employs a transfer learning method during the training process of the target detection model. Specifically, it uses the pre-trained weights of the initial prediction model and intermediate detection models in stages to perform transfer learning on the target detection model, enabling the target detection model to inherit the prior knowledge of both the initial prediction model and the intermediate detection model. This includes the following:
[0098] First, an initial detection model with pre-trained weights from Darknet-53 is used to generate an intermediate detection model. Since YOLOv4 is an open-source object detection network, the pre-trained weights can be downloaded from GitHub. The intermediate detection model is trained using image samples from the PascalVOC dataset to give it prior knowledge of the PascalVOC dataset. Then, the trained intermediate detection model is replaced with an object detection model, and the object detection model is trained using the PascalVOC dataset to give it prior knowledge of the PascalVOC dataset. Finally, the object detection model is fine-tuned using a damage dataset.
[0099] The process of training an object detection model using a damage dataset includes:
[0100] Multi-scale feature extraction is performed using the backbone network of the object detection model. The network parameters of the backbone network are shown in Table 1. In the table, AM represents the attention mechanism. Each layer of the backbone network uses the SENet (Squeeze-and-Excitation Network) channel attention mechanism. The attention mechanism can be changed to different types of attention mechanisms such as CANet and DANet, depending on the characteristics of the network structure and the type of detection task. NL represents the non-linear activation function, HS represents the Hard-Swish activation function, and RE represents the ReLU activation function. FMBConv 4 and IRBlock 6 represent that the channels of the first PW Conv 1×1 of the Fused-MBConv 3×3 and the Inverted Residual Block are expanded by 4 times and 6 times, respectively, and so on. SE0.25 means that the number of channels of the first fully connected layer in the SENet module is 0.25 times the number of channels of the feature matrix input to the FMBConv module and the IRBlock module. SE0.25 is used in each layer of the detection model.
[0101]
[0102]
[0103] Table 1
[0104] In object detection models, stages 1-6 are referred to as shallow networks.
[0105] Damaged image samples are processed by the skeleton network of the target detection model, which outputs three feature maps at different scales. The feature map output from the last layer of the skeleton network is input into the spatial pyramid pooling module, where local and global features are fused, greatly increasing the receptive field, enriching the expressive power of the feature maps, and improving the network's detection accuracy. The feature maps at different scales are further converged into a bidirectional pyramid feature fusion layer. In this layer, convolution and upsampling operations are used to concatenate and accumulate feature information of the same scale according to different weights, resulting in fused feature information.
[0106] To improve feature fusion efficiency and reduce network computation parameters, depthwise separable convolutions are used instead of standard convolutions in the network, thereby reducing network computation and saving detection time (please refer to the following for the modifications). Figure 6 ). CB-NL represents a standard convolutional block, and DSCB-NL represents a convolutional block with depthwise separable convolution.
[0107] The impact of different activation functions on the network's feature extraction capability was also analyzed, including ReLU, ReLU6, LeakyReLU, SiLU, and H-Swish activation functions, which are defined as follows:
[0108]
[0109]
[0110]
[0111]
[0112]
[0113] By studying different combinations of activation functions, the optimal network structure parameters are determined.
[0114] Combining Hard-swish and SiLU maximizes the network's feature extraction capability and improves the model's detection speed. Therefore, Hard-swish and SiLU combined activation functions are used in the convolutional blocks of the object detection model.
[0115] Feature maps of different scales are input into the YOLO Head, where bounding box and class predictions for surface damage are performed in the detection output layer. This includes: based on feature fusion, the feature map is divided into three different grid sizes—13×13, 26×26, and 52×52—to detect small, medium, and large targets, respectively. A set of convolutional layers outputs the feature map, where each cell corresponds to a region of the input image. For each cell, the location information of the bounding box (such as the center coordinates, width, and height) is predicted through regression. Simultaneously, multi-class classification prediction is performed on each cell using the softmax activation function, ultimately obtaining the location and class information of the damage in the image.
[0116] Step S540: Apply the trained target detection model in practice.
[0117] Reference Figures 9 to 11 The figure shows the Precision-Recall curves of the object detection model. As can be seen from the figure, the intermediate detection model uses MobileNetv3 as its backbone network and achieves a significant reduction in network parameters to only 11.43 MB through the use of depthwise separable convolutions. This represents an 82.13% reduction compared to the initial detection model. However, due to the limited performance of depthwise separable convolutions in shallow networks, the intermediate detection model exhibits lower accuracy in identifying multiple damages compared to the initial detection model.
[0118] The target detection model improves upon the intermediate MobileNetv3 detection model by replacing the shallow network of MobileNetv3 with six Fusion-MBConv cores, effectively enhancing the multi-damage identification accuracy of the target detection model. The target detection model combining Hard-Swish and SiLU activation functions achieves mAP and F1 scores of 77.32% and 71.95%, respectively, surpassing advanced detection models such as YOLOv5-s, YOLOv7-tiny, and YOLOv8-. Furthermore, the target detection model achieves an FPS of 85 f / s, making it suitable for real-time multi-damage detection of concrete bridges.
[0119] The target detection model performs excellently in identifying multiple damages on concrete bridge surfaces under different field-of-view scenarios (small, medium, and large), outperforming YOLOv8-s. By simulating real-world scenarios such as motion blur, fog, rain / snow, noise interference, and color changes, the applicability of the target detection model for identifying multiple damages on concrete bridge surfaces in complex environments was verified.
[0120] In summary, this embodiment acquires surface damage images of concrete bridges to be detected using a mobile device to establish a dataset. The "Imgaug" data augmentation toolkit is then used to augment the samples in the dataset, simulating surface defects on concrete bridges under adverse weather conditions such as fog, rain, and snow, as well as complex environments including motion blur, color changes, and noise interference. This establishes a rich database of surface damage on concrete bridges under complex environments, thereby improving the applicability of the target detection model. The processed images are then input into the trained target detection model for damage feature extraction, damage localization, and classification prediction, yielding surface damage detection results for the concrete bridge images. Notably, the skeleton network of the target detection model incorporates the Fused-MBConv structure, addressing the low feature extraction efficiency of the shallow MobileNetv3 network, reducing network computation while improving detection accuracy.
[0121] Furthermore, the target detection model provided in this embodiment uses depthwise separable convolutions instead of standard convolutions in the network, reducing model parameters and improving detection speed. An optimization strategy was employed to select the activation function in the network, enhancing its feature extraction capabilities and effectively improving the detection accuracy of surface damage on concrete bridges.
[0122] Reference Figure 12 One embodiment of this application provides a device for detecting surface damage of concrete structures. The device 1000 includes an image acquisition unit 1100, a model building unit 1200, and a damage detection unit 1300.
[0123] The image acquisition unit 1100 is used to acquire the image to be detected on the surface of the concrete building.
[0124] The model building unit 1200 is used to build an object detection model. The object detection model includes a skeleton network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The skeleton network includes cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks.
[0125] The damage detection unit 1300 is used to input the image to be detected into the skeleton network to obtain multi-scale feature maps output by the skeleton network; input the feature map output by the last layer of the skeleton network into the spatial pyramid pooling module to obtain the feature map output by the spatial pyramid pooling module; input the feature map output by the spatial pyramid pooling module and the other feature maps of different scales output by the intermediate layers of the skeleton network into the bidirectional pyramid feature fusion layer to obtain the feature map output by the bidirectional pyramid feature fusion layer; input the feature map output by the bidirectional pyramid feature fusion layer into YOLO Head to obtain the damage detection results of the concrete building surface output by YOLO Head.
[0126] It should be noted that the concrete building surface damage detection device provided in this embodiment is based on the same inventive concept as the concrete building surface damage detection method embodiment described above. Therefore, the relevant content of the concrete building surface damage detection method embodiment described above is also applicable to this concrete building surface damage detection device embodiment, and will not be described in detail here.
[0127] like Figure 13 This application also provides an electronic device, which includes:
[0128] At least one memory;
[0129] At least one processor;
[0130] At least one program;
[0131] The program is stored in memory, and the processor executes at least one program to implement the method for detecting surface damage of concrete structures described above.
[0132] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.
[0133] The electronic devices according to embodiments of this application will now be described in detail.
[0134] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.
[0135] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the concrete building surface damage detection method of the embodiments of this invention.
[0136] The input / output interface 1800 is used to implement information input and output.
[0137] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0138] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);
[0139] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.
[0140] This invention also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described method for detecting surface damage to concrete structures.
[0141] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0142] The embodiments described in this invention are intended to more clearly illustrate the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.
[0143] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0145] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0146] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0147] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0148] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0152] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.
Claims
1. A method for detecting surface damage in concrete structures, characterized in that, The method for detecting surface damage to concrete structures includes: Acquire images of the surface of a concrete building to be inspected; An object detection model is constructed, comprising a backbone network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The backbone network sequentially comprises cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks; the Fusion-MBConv Blocks are fused inverted residual convolutional blocks. The image to be detected is input into the skeleton network to obtain a multi-scale feature map output by the skeleton network; the feature map output by the last layer of the skeleton network is input into the spatial pyramid pooling module to obtain a feature map output by the spatial pyramid pooling module; the feature map output by the spatial pyramid pooling module and the remaining feature maps of different scales output by the intermediate layers of the skeleton network are input into the bidirectional pyramid feature fusion layer to obtain a feature map output by the bidirectional pyramid feature fusion layer; the feature map output by the bidirectional pyramid feature fusion layer is input into the YOLO Head to obtain the damage detection result of the concrete building surface output by the YOLO Head. The target detection model is trained in the following manner: Select the training images with labeled surface damage on concrete buildings; Construct an initial detection model based on YOLOv4; wherein, the initial detection model includes a backbone network based on the YOLOv4 architecture, the spatial pyramid pooling module, the bidirectional pyramid feature fusion layer, and the YOLO Head; An intermediate detection model is generated based on the initial detection model with pre-trained weights; wherein the intermediate detection model has the same spatial pyramid pooling module, bidirectional pyramid feature fusion layer and YOLO Head as the initial detection model, and the backbone network of the intermediate detection model is MobileNetv3; The intermediate detection model is trained using the image to be trained to obtain the trained intermediate detection model; The target detection model is generated based on the intermediate detection model after training; The target detection model is trained using the images to be trained until training ends.
2. The method for detecting surface damage of concrete structures according to claim 1, characterized in that: The convolutional module following each fusion layer in the bidirectional pyramid feature fusion layer sequentially includes 3 cascaded standard convolutional layers and 2 depthwise separable convolutional layers; The skeleton network and the spatial pyramid pooling module are further connected by cascaded two layers of standard convolutions and one layer of depthwise separable convolutions; the step of inputting the feature map output from the last layer of the skeleton network into the spatial pyramid pooling module includes: The feature map output from the last layer of the skeleton network is input into the cascaded two standard convolutional layers and one depthwise separable convolutional layer, and the output feature map is input into the spatial pyramid pooling module. The bidirectional pyramid feature fusion layer and the YOLO Head further include one depthwise separable convolution layer; the step of inputting the feature map output by the bidirectional pyramid feature fusion layer into the YOLO Head includes: The feature map output from the bidirectional pyramid feature fusion layer is input into a 1-layer depthwise separable convolution, and the output feature map is input into the YOLO Head.
3. The method for detecting surface damage of concrete structures according to claim 1, characterized in that, After selecting the training image of the labeled concrete building surface damage, the method for detecting concrete building surface damage further includes: The training image is subjected to a first enhancement transformation and / or a second enhancement transformation; wherein the first enhancement transformation of the training image includes at least two enhancement transformations among motion blur, cloud / fog, fog / frost, heavy rain / snow, rain, snow spots, and noise enhancement transformation; the second enhancement transformation of the training image includes one enhancement transformation among halving or doubling the contrast of the training image, increasing brightness, and enhancing color saturation.
4. The method for detecting surface damage of concrete structures according to claim 1 or 2, characterized in that, The activation function for each convolution in the target detection model is a combination of Hard-swish and SiLU activation functions.
5. The method for detecting surface damage of concrete structures according to any one of claims 1 to 2, characterized in that, The attention mechanisms in the Fusion-MBConv Block and Inverted Residual Block include one of the following: SENet, CANet, and DANet attention mechanisms.
6. The method for detecting surface damage of concrete structures according to claim 1, characterized in that, The concrete structure includes a concrete bridge; damage to the surface of the concrete structure includes at least one of concrete splitting, honeycombing, concrete spalling, exposed rebar, erosion, holes, and cracks.
7. A device for detecting surface damage to concrete buildings, characterized in that, The device for detecting surface damage to concrete structures includes: The image acquisition unit is used to acquire the image of the concrete building surface to be detected. The model building unit is used to build an object detection model, which includes a skeleton network, a spatial pyramid pooling module based on the YOLOv4 architecture, a bidirectional pyramid feature fusion layer based on the YOLOv4 architecture, and a YOLO Head based on the YOLOv4 architecture. The skeleton network includes cascaded convolutional layers, 6 Fusion-MBConv Blocks, and 9 Inverted Residual Blocks. The Fusion-MBConv Blocks are fused inverted residual convolutional blocks. The damage detection unit is used to input the image to be detected into the skeleton network to obtain a multi-scale feature map output by the skeleton network; input the feature map output by the last layer of the skeleton network into the spatial pyramid pooling module to obtain a feature map output by the spatial pyramid pooling module; input the feature map output by the spatial pyramid pooling module and the remaining feature maps of different scales output by the intermediate layers of the skeleton network into the bidirectional pyramid feature fusion layer to obtain a feature map output by the bidirectional pyramid feature fusion layer; and input the feature map output by the bidirectional pyramid feature fusion layer into the YOLO Head to obtain the damage detection result of the concrete building surface output by the YOLO Head. The target detection model is trained in the following manner: Select the training images with labeled surface damage on concrete buildings; Construct an initial detection model based on YOLOv4; wherein, the initial detection model includes a backbone network based on the YOLOv4 architecture, the spatial pyramid pooling module, the bidirectional pyramid feature fusion layer, and the YOLO Head; An intermediate detection model is generated based on the initial detection model with pre-trained weights; wherein the intermediate detection model has the same spatial pyramid pooling module, bidirectional pyramid feature fusion layer and YOLO Head as the initial detection model, and the backbone network of the intermediate detection model is MobileNetv3; The intermediate detection model is trained using the image to be trained to obtain the trained intermediate detection model; The target detection model is generated based on the intermediate detection model that has been trained; The target detection model is trained using the images to be trained until training ends.
8. An electronic device, characterized in that, include: At least one control processor and a memory for communicatively connecting to the at least one control processor; The memory stores instructions that can be executed by the at least one control processor to enable the at least one control processor to perform the method for detecting surface damage of concrete structures according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method for detecting surface damage of concrete structures as described in any one of claims 1 to 6.