Concrete building peeling defect detection method, device and system and storage medium

Abstract

The invention discloses a concrete building peeling defect detection method, device and system, and a storage medium, and belongs to the technical field of bridge defect detection, and the method comprises the steps: dividing a concrete building peeling image into a training set and a test set; according to the training set, training a PL-YOLO11 model; carrying out structured pruning on the trained PL-YOLO11 model, and then carrying out precision fine tuning training on the model after the structured pruning, so as to obtain a PL-YOLO11 Pruned model; carrying out model format conversion on the PL-YOLO11Pruned model; and deploying the PL-YOLO11Pruned model after model format conversion to an RKNN hardware platform, performing multi-thread real-time detection on a test set, and when a concrete building is detected to have a peeling phenomenon, marking a defect position and displaying a detection confidence coefficient. By adopting the technical scheme provided by the invention, rapid and accurate detection of the building spalling defect is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of bridge defect detection, and particularly relates to a method and device, system, and storage medium for detecting spalling defects in concrete buildings. Background Art

[0002] Concrete buildings are exposed to the natural environment for a long time. Spalling is usually caused by the weakening of the bonding force between concrete and steel bars due to the expansion effect of steel bar corrosion, which in turn causes the concrete surface to fall off. If not repaired in time, it will lead to the exposure of internal steel bars and accelerate corrosion, and may seriously cause structural failure, reduce the structural bearing capacity, and threaten the safety of concrete building structures.

[0003] However, there are many problems with existing detection methods. Especially for the method of detecting spalling in concrete buildings based on deep learning, the real-time performance is not high and the efficiency is low. Usually, data needs to be transmitted to a remote server for processing, and the detection results cannot be quickly fed back to guide the maintenance work of concrete buildings. Secondly, the generalization ability of the model is not high. These methods cannot identify spalling defects under different background features because the constructed dataset has a single background, and the deep learning model cannot learn the spalling features of buildings with rich background features in reality. At the same time, there are obvious deficiencies in the feature extraction of concrete building spalling, and it is difficult to effectively identify the fine features of concrete building spalling, and the accuracy and reliability of the detection results are limited. Finally, most of the existing models have complex structures and consume a large amount of computing resources, and it is difficult to meet the requirements of lightweight and low power consumption of detection equipment in this special outdoor environment of concrete buildings. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device, system, and storage medium for detecting spalling defects in concrete buildings, which can accurately extract the features of concrete building spalling to achieve rapid and accurate detection of building spalling defects.

[0005] To achieve the above object, the present invention adopts the following technical solutions: A method for detecting spalling defects in concrete buildings includes: Step S1, preprocessing the obtained spalling image of the concrete building; Step S2, dividing the preprocessed spalling image of the concrete building into a training set and a test set; Step S3: Train the PL-YOLO11 model according to the training set. In the PL-YOLO11 model, replace the C3k2 module of the backbone network of the YOLO11 model with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the PKINet for remote sensing image object detection. Replace the convolution of its output channels with multi-scale convolution to obtain the OD-PKI module, and then integrate it into C3k2 to get the C3k2_ODPKIModule module. Replace the LADH detection head with the YOLO detection head. Step S4: Perform structured pruning on the trained PL-YOLO11 model, and then perform fine-tuning training on the accuracy of the pruned model to obtain the PL-YOLO11_Pruned model. Step S5: Perform model format conversion on the PL-YOLO11_Pruned model. Step S6: Deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start a thread pool to perform multi-threaded real-time detection on the test set. When it is detected that the concrete building has spalling, mark the defect location and display the detection confidence.

[0006] Preferably, in step S4, the pruning operation on specific modules in the trained PL-YOLO11 model includes: pruning OD-PKI in the C3k2_ODPKIModule module, pruning Adown, and pruning the LADH detection head. Among them, for the pruning of the LADH detection head, the pruned parts include the Reg branch and the Cls branch of the P3, P4, and P5 layers.

[0007] Preferably, in step S5, the PL-YOLO11_Pruned model is converted into an ONNX model through the first model conversion; the ONNX model is converted into an RKNN model through the second model conversion. Among them, during the second model conversion, the ONNX model is transplanted to the Linux system, the model is quantized to convert floating-point storage into integer storage, and then the model is encapsulated into an RKNN model suitable for deployment to edge devices.

[0008] The present invention also provides a device for detecting spalling defects of concrete buildings, including: A first processing module for preprocessing the acquired spalling image of the concrete building. A second processing module for dividing the preprocessed spalling image of the concrete building into a training set and a test set. The third processing module is used to train the PL-YOLO11 model according to the training set; among them, in the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. The convolution of its output channels is replaced with multi-scale convolution to obtain the OD-PKI module, and then it is integrated into C3k2 to obtain the C3k2_ODPKIModule module; the LADH detection head is replaced with the YOLO detection head; The fourth processing module is used to perform structured pruning on the trained PL-YOLO11 model, and then perform fine-tuning training on the accuracy of the model after structured pruning to obtain the PL-YOLO11_Pruned model; The fifth processing module is used to perform model format conversion on the PL-YOLO11_Pruned model; The sixth processing module is used to deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start a thread pool to perform multi-threaded real-time detection on the test set. When it is detected that there is a peeling phenomenon in the concrete building, mark the defect location and display the detection confidence.

[0009] Preferably, the pruning operation of specific modules in the trained PL-YOLO11 model by the fourth processing module includes: pruning the OD-PKI in the C3k2_ODPKIModule module and pruning the LADH detection head; among them, for the pruning of the LADH detection head, the pruned parts include the Reg branch and the Cls branch of the P3, P4, and P5 layers.

[0010] Preferably, the fifth processing module converts the PL-YOLO11_Pruned model into an ONNX model through the first model conversion; and converts the ONNX model into an RKNN model through the second model conversion.

[0011] The present invention also provides a concrete building peeling defect detection system, including: a memory and a processor. A computer program is stored on the memory and run by the processor. When the computer program is run by the processor, it executes the concrete building peeling defect detection method.

[0012] The present invention also provides a storage medium, on which a computer program is stored. When the computer program runs, it executes the concrete building peeling defect detection method.

[0013] The present invention has the following technical effects: 1. Strong background adaptability: The present invention collects pictures of concrete building spalling under different backgrounds and uses various methods for data augmentation, including diverse background information. During the model training process, it is exposed to a wider range of visual effects, enabling the model to learn more extensive background features. Compared with the prior art, the dataset of the present invention has a richer and more diverse background, can better simulate complex background conditions in real scenarios, and significantly improves the robustness of the model to complex backgrounds.

[0014] 2. Precise feature extraction ability: The present invention optimizes the YOLO11 algorithm. By improving the feature extraction layer and network structure of the algorithm, the present invention can more effectively capture the detailed features of the spalling area, thereby significantly improving the accuracy and reliability of the detection results.

[0015] 3. Lightweight and high efficiency: The present invention obtains the lightweight PL - YOLO11 model by improving the YOLO11 model. On this basis, lightweight operations such as structured pruning and quantization are also carried out, enabling it to significantly reduce the complexity and computational resource consumption of the model while maintaining high - efficiency detection performance, and being more suitable for deployment and operation on edge devices.

[0016] 4. Real - time and convenience: Based on edge intelligence technology, the present invention adopts a multi - thread inference mechanism to achieve real - time detection of concrete building spalling by the lightweight model. The detection device can be directly deployed at the concrete building site without relying on a remote server, significantly shortening the detection cycle and quickly providing detection results to guide the maintenance work of concrete buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0018] Figure 1 It is a schematic flow chart of the method for detecting concrete building spalling defects of the present invention; Figure 2 It is a network structure diagram of the unimproved YOLO11 model; Figure 3 Based on the present invention Figure 2 The designed network structure diagram of the PL - YOLO11 model; Figure 4 It is a network structure diagram of the unimproved PKI Module module; Figure 5 Based on the present invention Figure 4Schematic diagram of the improved OD-PKI module structure of the design; Figure 6 Schematic diagram of the LADH detection head structure; Figure 7 Schematic diagram of the workflow for multi-threaded inference on the RKNN hardware platform. Specific implementation manners

[0019] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0020] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the drawings and specific implementation manners.

[0021] Embodiment 1: As Figure 1 shown, the embodiment of the present invention provides a method for detecting spalling defects in concrete buildings, including: Step S1, preprocess the obtained spalling images of concrete buildings; Step S2, divide the preprocessed spalling images of concrete buildings into a training set and a test set; Step S3, train the PL-YOLO11 model according to the training set; Step S4, perform structured pruning on the trained PL-YOLO11 model, and then perform fine-tuning training on the accuracy of the model after structured pruning to obtain the PL-YOLO11_Pruned model; Step S5, perform two model format conversions on the PL-YOLO11_Pruned model; Step S6, deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start a thread pool to perform multi-threaded real-time detection on the test set, and when it is detected that the concrete building has a spalling phenomenon, mark the defect position and display the detection confidence.

[0022] As an implementation manner of the embodiment of the present invention, in step S1, the spalling images of concrete buildings include: spalling images of concrete buildings under various different backgrounds such as concrete cross-river bridges, elevated highway bridges, and arch bridges.

[0023] As an implementation manner of the embodiment of the present invention, in step S1, preprocessing the obtained spalling images of concrete buildings includes: Step S11: Use the LabelImg tool to label the spalling targets in the concrete building spalling images to obtain the first bridge image dataset containing target categories and target locations; Step S12: Perform data augmentation on the first bridge image dataset to obtain the second bridge image dataset; among them, the data augmentation uses randomly adjusting brightness and contrast, randomly adding solar flares, randomly adding fog, randomly adding shadows, randomly blurring, randomly adjusting hue and saturation to augment the first bridge image dataset.

[0024] As an implementation manner of the embodiment of the present invention, in step S2, the second bridge image dataset is divided into a training set and a test set.

[0025] As an implementation manner of the embodiment of the present invention, in step S3, the YOLO11 model is improved to obtain the PL-YOLO11 model, as Figure 2 The YOLO11 model, the improvement method is: replace the C3k2 module in the backbone network with the C3k2_ODPKIModule module, and the C3k2_ODPKIModule module is based on the PKI Module in PKINet, and the structure of the PKI Module is as Figure 4 shown. The feature map after downsampling needs to go through a 3x3 DWConv, and then is divided into five branches: performing DWConv of 5x5, 7x7, 9x9, 11x11 respectively and identity mapping (Identity). After the element-wise addition operation of the outputs of each branch, it is then convolved by 1x1 convolution, and finally the output is provided for subsequent processing. The improved structure of the PKI Module is as Figure 5 shown, and other modules remain unchanged. The feature map after downsampling needs to enter a 3x3 DWConv, and then is divided into five branches of DWConv and identity mapping. After the element-wise addition operation of the outputs of each branch, it goes through the replaced multi-scale convolution and finally outputs for processing. The improved structure of the PKI Module is named the OD-PKI module, and this module is incorporated into C3k2 to obtain C3k2_ODPKIModule; then replace the YOLO detection head with the more powerful LADH detection head, and the structure of the LADH detection head is as Figure 6 , the network structure of the LADH detection head receives the features output by the neck, and its dimension is H×W×C. The features are first reduced in dimension by 1×1 convolution, and then the features are extracted by 3×3 DWConv. Then it is divided into three branches: one outputs the classification result (Cls), one outputs the IoU prediction (IoU), and one outputs the bounding box regression result (Reg). Finally, these results are integrated and output for the target detection task. Combining the two improvements, the PL-YOLO11 model is obtained, and its network model structure is as Figure 3, The input data of the Backbone part passes through multiple layers of Conv, C3k2_ODPKIModule, etc. in sequence, and finally reaches the C2PSA and is input to the neck. In the Neck, the features of different levels of the Backbone are fused: the features of the 4th layer C3k2_ODPKIModule of the Backbone are input to the 15th layer Concat for feature fusion, and the features of the 6th layer C3k2_ODPKIModule are input to the 12th layer Concat for feature fusion. In addition, the features output by C2PSA also participate in subsequent processing through the 11th layer Upsample operation. After the 15th layer Concat, it is connected to the 16th layer C3k2 and input to the LADH_Head1 of the Detection head; the 18th layer Concat is processed by the 19th layer C3k2 and then input to the LADH_Head2; the 21st layer Concat is processed by the 22nd layer C3k2 and then input to the LADH_Head3. During the training process, the features have to go through the feature processing of Conv convolution and upsampling. Each component collaborates through modules such as feature fusion, upsampling, and C3k2 to achieve multi-scale object detection and improve the recognition and localization capabilities for different objects.

[0026] Furthermore, in the training of the PL-YOLO11 model, the data path setting points to a configuration file containing information about bridge-related categories, which is used to specify the training and validation data sets. When training, the sizes of the bridge images in the input training set are uniformly 640×640 pixels, and a total of 3000 rounds of iteration are performed. It is set that if the accuracy does not improve within 100 rounds, the training will be automatically terminated, and 32 images are processed in each round. The optimizer uses Stochastic Gradient Descent (SGD).

[0027] As an implementation manner of the embodiment of the present invention, in step S4, the trained PL-YOLO11 model is structurally pruned, aiming to reduce the complexity and the number of parameters of the model by removing unimportant weights or channels in the model, so as to improve the inference efficiency and running speed of the trained PL-YOLO11 model. The specific process is as follows.

[0028] First, load the trained PL-YOLO11 model and determine the pruning threshold through the `get_threshold` method of the `PRUNE` class. This threshold is based on the set pruning rate, and through experiments, it is most appropriate to set it to 0.6. The pruning rate is used to determine which weights or channels can be removed. Then, perform pruning operations on specific modules in the trained PL-YOLO11 model. The pruning operations include: pruning the OD-PKI in the C3k2_ODPKIModule module, pruning the LADH detection head, and then resetting the gradients of the model parameters for precision fine-tuning training to obtain the PL-YOLO11_Pruned model. Among them, for the pruning of the LADH detection head, such as Figure 6 , the pruned parts include the Reg and Cls branches of the P3, P4, and P5 layers. P3, P4, and P5 correspond to the feature maps of different levels of the three detection heads. Reg is the regression prediction output, and Cls is the classification prediction output. Prune the convolutional modules of these two branches layer by layer, remove the redundant channels with little impact on the results, and adjust the input channels of the subsequent modules to ensure the coherence of the network structure. The whole process not only reduces the computational burden of the model but also, through a fine pruning strategy, retains the key features and performance of the model as much as possible.

[0029] As an implementation manner of the embodiment of the present invention, in step S5, the PL-YOLO11_Pruned model is converted twice. The PL-YOLO11_Pruned model is converted into an ONNX model through the first model conversion; the ONNX model is converted into an RKNN model through the second model conversion; among them, during the second model conversion, the ONNX model is transplanted to the Linux system, the model is quantized to convert the floating-point storage into integer storage, and then the model is packaged into an RKNN model suitable for deployment on edge devices.

[0030] As an implementation manner of the embodiment of the present invention, in step S6, in order to analyze the concrete building spalling images captured in real time, multi-threading work is enabled, such as Figure 7 shown, thread A is the thread for reading video frames; thread pool B is enabled to set multiple threads to load, infer, and post-process the PL-YOLO11_Pruned model after model format conversion. To avoid the performance overhead caused by frequent creation and destruction of threads, a certain number of threads are pre-created and placed in a pool for unified management. In this way, tasks can be scheduled efficiently, ensuring that each thread can quickly respond to new tasks when idle, improving the inference speed and response ability of the model when processing consecutive video frames; finally, another thread C is enabled to push the inferred video frames and save the results to the corresponding files.

[0031] Embodiment 2: An embodiment of the present invention also provides a detection device for spalling defects in concrete buildings, including: A first processing module for preprocessing the obtained spalling images of concrete buildings; A second processing module for dividing the preprocessed spalling images of concrete buildings into a training set and a test set; A third processing module for training the PL-YOLO11 model according to the training set; wherein, in the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. The convolution of its output channels is replaced with a multi-scale convolution to obtain the OD-PKI module, and then the C3k2 is incorporated to obtain the C3k2_ODPKIModule module; the LADH detection head is replaced with the YOLO detection head; A fourth processing module for performing structured pruning on the trained PL-YOLO11 model, and then performing fine-tuning training on the accuracy of the structured pruned model to obtain the PL-YOLO11_Pruned model; A fifth processing module for performing model format conversion on the PL-YOLO11_Pruned model; A sixth processing module for deploying the PL-YOLO11_Pruned model with the converted model format to the RKNN hardware platform, starting a thread pool to perform multi-threaded real-time detection on the test set. When it is detected that spalling occurs in the concrete building, the defect position is marked and the detection confidence is displayed.

[0032] As an implementation manner of an embodiment of the present invention, the fourth processing module performs pruning operations on specific modules in the trained PL-YOLO11 model, including: pruning the OD-PKI in the C3k2_ODPKIModule module and pruning the LADH detection head; wherein, for the pruning of the LADH detection head, the pruned part includes the Reg branch and the Cls branch of the P3, P4, and P5 layers.

[0033] As an implementation manner of an embodiment of the present invention, the fifth processing module converts the PL-YOLO11_Pruned model into an ONNX model through the first model conversion; and converts the ONNX model into an RKNN model through the second model conversion.

[0034] Example 3: The present invention also provides a detection system for spalling defects of concrete buildings, comprising: a memory and a processor, wherein a computer program run by the processor is stored on the memory, and the computer program executes a method for detecting spalling defects of concrete buildings when being run by the processor.

[0035] Embodiment 4: The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes a method for detecting spalling defects of concrete buildings when running.

[0036] The above embodiments are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for detecting spalling defects in concrete buildings, characterized in that, Including: Step S1: Preprocess the obtained spalling image of a concrete building; Step S2: Divide the preprocessed spalling image of a concrete building into a training set and a test set; Step S3: Train the PL-YOLO11 model according to the training set; among them, in the PL-YOLO11 model, replace the C3k2 module of the backbone network of the YOLO11 model with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. Replace the convolution of its output channels with multi-scale convolution to obtain the OD-PKI module, and then integrate it into C3k2 to obtain the C3k2_ODPKIModule module; replace the LADH detection head with the YOLO detection head; Step S4: Perform structured pruning on the trained PL-YOLO11 model, and then perform fine-tuning training on the accuracy of the pruned model to obtain the PL-YOLO11_Pruned model; Step S5: Perform model format conversion on the PL-YOLO11_Pruned model; Step S6: Deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start a thread pool to perform multi-threaded real-time detection on the test set. When it is detected that spalling occurs in the concrete building, mark the defect location and display the detection confidence.

2. The method for detecting spalling defects in concrete buildings according to claim 1, characterized in that, In step S4, the pruning operation on specific modules in the trained PL-YOLO11 model includes: pruning OD-PKI in the C3k2_ODPKIModule module, pruning Adown, and pruning the LADH detection head; among them, for the pruning of the LADH detection head, the pruned parts include the Reg branch and the Cls branch of the P3, P4, and P5 layers.

3. The method for detecting spalling defects of concrete buildings according to claim 2, wherein In step S5, the PL-YOLO11_Pruned model is converted into an ONNX model through the first model conversion; the ONNX model is converted into an RKNN model through the second model conversion; among them, in the second model conversion, the ONNX model is transplanted to the Linux system, the model is quantized to convert floating-point storage into integer storage, and then the model is packaged into an RKNN model suitable for deployment to edge devices.

4. A device for detecting spalling defects in concrete buildings, characterized in that, Including: The first processing module is used to preprocess the obtained spalling image of a concrete building; The second processing module is used to divide the preprocessed spalling image of a concrete building into a training set and a test set; The third processing module is used to train the PL-YOLO11 model according to the training set; in the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. The convolution of its output channels is replaced with multi-scale convolution to obtain the OD-PKI module, and then the C3k2 is incorporated to obtain the C3k2_ODPKIModule module; the LADH detection head is replaced with the YOLO detection head. The fourth processing module is used to perform structured pruning on the trained PL-YOLO11 model, and then perform fine-tuning training on the accuracy of the pruned model to obtain the PL-YOLO11_Pruned model. The fifth processing module is used to perform model format conversion on the PL-YOLO11_Pruned model. The sixth processing module is used to deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start a thread pool to perform multi-threaded real-time detection on the test set. When it detects that the concrete building has peeling phenomenon, it marks the defect position and displays the detection confidence.

5. The concrete building spalling defect detection device according to claim 4, characterized in that, The pruning operations performed by the fourth processing module on specific modules in the trained PL-YOLO11 model include: pruning the OD-PKI in the C3k2_ODPKIModule module and pruning the LADH detection head; among them, for the pruning of the LADH detection head, the pruned parts include the Reg and Cls branches of the P3, P4, and P5 layers.

6. The concrete building spalling defect detection device according to claim 5, characterized in that The fifth processing module converts the PL-YOLO11_Pruned model into an ONNX model through the first model conversion; and converts the ONNX model into an RKNN model through the second model conversion.

7. A detection system for spalling defects in concrete buildings, characterized in that, Including: A memory and a processor, where a computer program is stored on the memory and run by the processor. When the computer program is run by the processor, it executes the concrete building peeling defect detection method according to any one of claims 1-3.

8. A storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program runs, it executes the concrete building peeling defect detection method according to any one of claims 1-3.

Images