Concrete building spalling defect detection method, device, system, and storage medium

Abstract

The present invention discloses a method, device, system, and storage medium for detecting spalling defects in concrete buildings, belonging to the technical field of bridge defect detection. The method comprises the following steps: dividing concrete building spalling images into a training set and a test set; training a PL-YOLO11 model based on the training set; performing structured pruning on the trained PL-YOLO11 model, and then fine-tuning the accuracy of the pruned model to obtain a PL-YOLO11_Pruned model; converting the PL-YOLO11_Pruned model to a new format; deploying the converted PL-YOLO11_Pruned model on an RKNN hardware platform, performing multi-threaded real-time detection on the test set, and marking the defect location and displaying the detection confidence level when spalling is detected in the concrete building. The technical solution of the present invention enables rapid and accurate detection of building spalling defects.

Description

Technical Field

[0001] The present invention belongs to the technical field of bridge defect detection, and in particular relates to a method and device, a system, and a storage medium for detecting spalling defects in concrete buildings. Background Art

[0002] Concrete buildings are exposed to the elements for long periods of time. Spalling is often caused by the expansion of steel bars due to corrosion, which weakens the bond between the concrete and the steel, leading to surface spalling. If not repaired promptly, this can expose the internal steel and accelerate corrosion. In severe cases, this can lead to structural failure, reduce the load-bearing capacity, and threaten the safety of concrete buildings.

[0003] However, existing detection methods have many problems, especially those based on deep learning for concrete building spalling detection. These methods are not very real-time and inefficient, often requiring data transmission to a remote server for processing. This prevents rapid feedback of detection results to guide concrete building maintenance. Secondly, the models are not very generalizable, and these methods are unable to identify spalling defects under different background features. This is because the constructed datasets have a single background, and deep learning models are unable to learn the characteristics of building spalling that have rich background features in reality. At the same time, there are significant deficiencies in feature extraction for concrete building spalling, making it difficult to effectively identify the subtle features of concrete building spalling, limiting the accuracy and reliability of the detection results. Finally, most existing models are complex in structure and consume a lot of computing resources, making it difficult to adapt to the requirements of lightweight and low-power detection equipment in the special outdoor environment of concrete buildings. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device, system, and storage medium for detecting spalling defects in concrete buildings, which can accurately extract the characteristics of spalling in concrete buildings to achieve rapid and accurate detection of spalling defects in buildings.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for detecting spalling defects in concrete buildings, comprising:

[0007] Step S1, pre-processing the acquired concrete building spalling image;

[0008] Step S2: dividing the pre-processed concrete building spalling images into a training set and a test set;

[0009] Step S3, training the PL-YOLO11 model according to the training set; wherein, in the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet, and its output channel convolution is replaced with multi-scale convolution to obtain the OD-PKI module, which is then integrated into C3k2 to obtain the C3k2_ODPKIModule module; the LADH detection head replaces the YOLO detection head;

[0010] Step S4: perform structured pruning on the trained PL-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the PL-YOLO11_Pruned model;

[0011] Step S5: convert the PL-YOLO11_Pruned model into a new model format;

[0012] Step S6: Deploy the converted PL-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when spalling of concrete buildings is detected, mark the defect location and display the detection confidence.

[0013] Preferably, in step S4, pruning a specific module in the trained PL-YOLO11 model includes: pruning OD-PKI in the C3k2_ODPKIModule module, pruning Adown, and pruning the LADH detection head; wherein, for the pruning of the LADH detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers.

[0014] Preferably, in step S5, the PL-YOLO11_Pruned model is converted into an ONNX model through a first model conversion; the ONNX model is converted into an RKNN model through a second model conversion; wherein, during the second model conversion, the ONNX model is ported to the Linux system, the model is quantized to convert floating-point storage into integer storage, and then the model is encapsulated into an RKNN model suitable for deployment on edge devices.

[0015] The present invention also provides a concrete building spalling defect detection device, comprising:

[0016] A first processing module is used to pre-process the acquired concrete building spalling image;

[0017] The second processing module is used to divide the pre-processed concrete building spalling images into a training set and a test set;

[0018] The third processing module is used to train the PL-YOLO11 model based on the training set. In the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. The convolution of its output channel is replaced with multi-scale convolution to obtain the OD-PKI module, which is then integrated into C3k2 to obtain the C3k2_ODPKIModule module. The LADH detection head replaces the YOLO detection head.

[0019] The fourth processing module is used to perform structured pruning on the trained PL-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the PL-YOLO11_Pruned model;

[0020] The fifth processing module is used to convert the PL-YOLO11_Pruned model into a model format;

[0021] The sixth processing module is used to deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when spalling of concrete buildings is detected, mark the defect location and display the detection confidence.

[0022] Preferably, the fourth processing module performs pruning operations on specific modules in the trained PL-YOLO11 model, including: pruning the OD-PKI in the C3k2_ODPKIModule module and pruning the LADH detection head; wherein, for the pruning of the LADH detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers.

[0023] Preferably, the fifth processing module converts the PL-YOLO11_Pruned model into an ONNX model through a first model conversion; and converts the ONNX model into an RKNN model through a second model conversion.

[0024] The present invention also provides a concrete building spalling defect detection system, comprising: a memory and a processor, wherein the memory stores a computer program run by the processor, and the computer program executes a concrete building spalling defect detection method when run by the processor.

[0025] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is run, the method for detecting spalling defects in concrete buildings is executed.

[0026] The present invention has the following technical effects:

[0027] 1. Strong background adaptability. This method collects images of concrete spalling in different backgrounds and uses multiple methods to enhance the data. This includes diverse background information, exposing the model to a wider range of visual effects during training, enabling the model to learn a wider range of background features. Compared with existing technologies, this dataset is richer and more diverse in background, better simulating the complex background conditions found in real-world scenarios and significantly improving the model's robustness to complex backgrounds.

[0028] 2. Accurate feature extraction capability. The present invention optimizes the YOLO11 algorithm. By improving the feature extraction layer and network structure of the algorithm, the present invention can more effectively capture the detailed features of the peeling area, thereby significantly improving the accuracy and reliability of the detection results.

[0029] 3. Lightweight and efficient. The present invention improves the YOLO11 model to obtain a lightweight PL-YOLO11 model. On this basis, it also performs lightweight operations such as structured pruning and quantization, which greatly reduces the complexity of the model and the consumption of computing resources while maintaining efficient detection performance, making it more suitable for deployment and operation on edge devices.

[0030] 4. Real-time and convenient performance. Based on edge intelligence technology and a multi-threaded reasoning mechanism, this invention enables real-time detection of concrete spalling using a lightweight model. The detection equipment can be deployed directly on-site, eliminating the need for remote servers. This significantly shortens the detection cycle and provides rapid feedback on test results to guide concrete building maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0032] Figure 1 Schematic diagram of the process of the concrete building spalling defect detection method of the present invention;

[0033] Figure 2 This is the network structure diagram of the unimproved YOLO11 model;

[0034] Figure 3 The present invention is based on Figure 2 Schematic diagram of the designed PL-YOLO11 model network structure;

[0035] Figure 4 This is a schematic diagram of the network structure of the unmodified PKI Module;

[0036] Figure 5 The present invention is based on Figure 4 Schematic diagram of the designed improved OD-PKI module structure;

[0037] Figure 6 This is a schematic diagram of the LADH detection head structure;

[0038] Figure 7 Schematic diagram of the workflow for multi-threaded inference on the RKNN hardware platform. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Example 1:

[0042] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting spalling defects in concrete buildings, comprising:

[0043] Step S1, pre-processing the acquired concrete building spalling image;

[0044] Step S2: dividing the pre-processed concrete building spalling images into a training set and a test set;

[0045] Step S3: train the PL-YOLO11 model based on the training set;

[0046] Step S4: perform structured pruning on the trained PL-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the PL-YOLO11_Pruned model;

[0047] Step S5: Perform two model format conversions on the PL-YOLO11_Pruned model;

[0048] Step S6: Deploy the converted PL-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when spalling of concrete buildings is detected, mark the defect location and display the detection confidence.

[0049] As an implementation of the embodiment of the present invention, in step S1, the concrete building spalling images include concrete building spalling images under various backgrounds, such as a concrete river bridge, an elevated highway bridge, and an arch bridge.

[0050] As an implementation of an embodiment of the present invention, in step S1, preprocessing the acquired concrete building spalling image includes:

[0051] Step S11: Use the LabelImg tool to label the spalling targets in the concrete building spalling image to obtain a first bridge image dataset containing target categories and target locations;

[0052] Step S12: performing data enhancement on the first bridge image dataset to obtain a second bridge image dataset; wherein the data enhancement is performed by randomly adjusting brightness and contrast, randomly adding sun flares, randomly adding fog, randomly adding shadows, randomly blurring, and randomly adjusting hue and saturation to enhance the first bridge image dataset.

[0053] As an implementation of the embodiment of the present invention, in step S2, the second bridge image dataset is divided into a training set and a test set.

[0054] As an implementation method of an embodiment of the present invention, in step S3, the YOLO11 model is improved to obtain the PL-YOLO11 model, such as Figure 2 For the YOLO11 model, the improvement method is: replace the C3k2 module in the backbone network with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in PKINet. The PKI Module structure is as follows: Figure 4 As shown in Figure 1, the downsampled feature map needs to go through a 3x3 DWConv, and then be divided into five branches: 5x5, 7x7, 9x9, 11x11 DWConv and identity mapping. After the output of each branch is element-wise added, it undergoes a 1x1 convolution and the final output is used for subsequent processing. The improved PKI Module structure is shown in Figure 1. Figure 5As shown in the figure, the other modules remain unchanged. The downsampled feature map needs to be fed into a 3x3 DWConv, which is then divided into five branches of DWConv and identity mapping. After the element-wise addition operation, the output of each branch undergoes a replaced multi-scale convolution before being output. The improved PKI module structure is named OD-PKI module. This module is integrated into C3k2 to obtain C3k2_ODPKIModule. The YOLO detection head is then replaced by the LADH detection head with stronger extraction capabilities. The LADH detection head structure is shown as follows: Figure 6 The LADH detection head's network structure receives features output from the neck, whose dimensions are H×W×C. These features are first reduced in dimension by a 1×1 convolution, and then extracted by a 3×3 DWConv. The network then splits into three branches: one outputting the classification result (Cls), one outputting the IoU prediction (IoU), and one outputting the bounding box regression result (Reg). These results are then integrated and output for the object detection task. Combining these two improvements, the PL-YOLO11 model is obtained, whose network model structure is shown below. Figure 3 Backbone input data is processed sequentially through multiple layers of Conv and C3k2_ODPKIModule before being fed into the neck by C2PSA. Within the neck, features from different Backbone layers are fused: features from the Backbone's fourth layer, C3k2_ODPKIModule, are fed into the 15th layer of Concat for feature fusion, and features from the 6th layer of C3k2_ODPKIModule are fed into the 12th layer of Concat for feature fusion. Furthermore, features output by C2PSA undergo 11 layers of Upsample for subsequent processing. The 15th layer of Concat is followed by the 16th layer of C3k2 and fed into LADH_Head1 of the Detection head. The 18th layer of Concat is processed by the 19th layer of C3k2 and then fed into LADH_Head2. The 21st layer of Concat is processed by the 22nd layer of C3k2 and then fed into LADH_Head3. During the training process, all features must undergo Conv convolution and upsampling feature processing. Each component collaborates through modules such as feature fusion, upsampling, and C3k2 to achieve multi-scale target detection and improve the ability to identify and locate different targets.

[0055] Furthermore, when training the PL-YOLO11 model, the data path is set to point to a configuration file containing information about bridge categories, which specifies the training and validation datasets. The bridge images in the training set are uniformly sized at 640×640 pixels, and training is performed for 3000 epochs. Training is automatically terminated if accuracy does not improve within 100 epochs, with 32 images processed per epoch. Stochastic gradient descent (SGD) is used as the optimizer.

[0056] As an implementation method of an embodiment of the present invention, in step S4, structured pruning is performed on the trained PL-YOLO11 model to reduce the complexity and number of parameters of the model by removing unimportant weights or channels in the model, thereby improving the inference efficiency and running speed of the trained PL-YOLO11 model. The specific process is as follows.

[0057] First, load the trained PL-YOLO11 model and determine the pruning threshold through the `get_threshold` method of the `PRUNE` class. The threshold is based on the set pruning rate. After experiments, it is most appropriate to set it to 0.6. The pruning rate is used to determine which weights or channels can be removed. Then, prune the specific modules in the trained PL-YOLO11 model. The pruning operation includes: pruning the OD-PKI in the C3k2_ODPKIModule module, pruning the LADH detection head, and then resetting the gradient of the model parameters for precision fine-tuning training to obtain the PL-YOLO11_Pruned model. Among them, for the pruning of the LADH detection head, such as Figure 6 The pruning process involves the Reg and Cls branches of the P3, P4, and P5 layers. P3, P4, and P5 correspond to feature maps at different levels of the three detection heads. Reg represents the regression prediction output, while Cls represents the classification prediction output. The convolutional modules of these two branches are pruned layer by layer, removing redundant channels that have little impact on the results. The input channels of subsequent modules are also adjusted to ensure the coherence of the network structure. This entire process not only reduces the model's computational burden but also, through a sophisticated pruning strategy, preserves the model's key features and performance to the greatest extent possible.

[0058] As an implementation method of an embodiment of the present invention, in step S5, the PL-YOLO11_Pruned model is converted twice. The first model conversion converts the PL-YOLO11_Pruned model into an ONNX model; the second model conversion converts the ONNX model into an RKNN model. In the second model conversion, the ONNX model is ported to a Linux system, the model is quantized to convert floating-point storage into integer storage, and the model is then encapsulated into an RKNN model suitable for deployment on edge devices.

[0059] As an implementation method of the embodiment of the present invention, in step S6, in order to realize the analysis of the concrete building spalling image captured in real time, multi-threaded work is started, such as Figure 7 As shown, thread A reads video frames; thread pool B is started and multiple threads are configured to load, infer, and post-process the PL-YOLO11_Pruned model after model format conversion. To avoid the performance overhead of frequent thread creation and destruction, a certain number of threads are pre-created and placed in a pool for unified management. This approach enables efficient task scheduling, ensuring that each thread can quickly respond to new tasks when idle, improving the model's inference speed and responsiveness when processing continuous video frames. Finally, thread C is started to stream the inferred video frames and save the results to the corresponding file.

[0060] Example 2:

[0061] The embodiment of the present invention further provides a device for detecting spalling defects in concrete buildings, comprising:

[0062] A first processing module is used to pre-process the acquired concrete building spalling image;

[0063] The second processing module is used to divide the pre-processed concrete building spalling images into a training set and a test set;

[0064] The third processing module is used to train the PL-YOLO11 model based on the training set. In the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. The convolution of its output channel is replaced with multi-scale convolution to obtain the OD-PKI module, which is then integrated into C3k2 to obtain the C3k2_ODPKIModule module. The LADH detection head replaces the YOLO detection head.

[0065] The fourth processing module is used to perform structured pruning on the trained PL-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the PL-YOLO11_Pruned model;

[0066] The fifth processing module is used to convert the PL-YOLO11_Pruned model into a model format;

[0067] The sixth processing module is used to deploy the PL-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when spalling of concrete buildings is detected, mark the defect location and display the detection confidence.

[0068] As an implementation method of an embodiment of the present invention, the fourth processing module performs a pruning operation on a specific module in the trained PL-YOLO11 model, including: pruning the OD-PKI in the C3k2_ODPKIModule module and pruning the LADH detection head; wherein, for the pruning of the LADH detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers.

[0069] As an implementation method of the embodiment of the present invention, the fifth processing module converts the PL-YOLO11_Pruned model into the ONNX model through the first model conversion; and converts the ONNX model into the RKNN model through the second model conversion.

[0070] Example 3:

[0071] The present invention also provides a concrete building spalling defect detection system, comprising: a memory and a processor, wherein the memory stores a computer program run by the processor, and the computer program executes a concrete building spalling defect detection method when run by the processor.

[0072] Example 4:

[0073] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is run, the method for detecting spalling defects in concrete buildings is executed.

[0074] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for detecting spalling defects in concrete buildings, characterized in that: include: Step S1, pre-processing the acquired concrete building spalling image; Step S2: dividing the pre-processed concrete building spalling images into a training set and a test set; Step S3, training the PL-YOLO11 model according to the training set; wherein, in the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet, and its output channel convolution is replaced with multi-scale convolution to obtain the OD-PKI module, which is then integrated into C3k2 to obtain the C3k2_ODPKIModule module; the LADH detection head replaces the YOLO detection head; Step S4: perform structured pruning on the trained PL-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the PL-YOLO11_Pruned model; Step S5: convert the PL-YOLO11_Pruned model into a new model format; Step S6: Deploy the converted PL-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when spalling of concrete buildings is detected, mark the defect location and display the detection confidence level; In step S4, pruning specific modules in the trained PL-YOLO11 model includes pruning the OD-PKI in the C3k2_ODPKIModule module, pruning Adown, and pruning the LADH detection head. The pruning of the LADH detection head includes the Reg branch and Cls branch of the P3, P4, and P5 layers. In step S5, the PL-YOLO11_Pruned model is converted into an ONNX model through the first model conversion; and the ONNX model is converted into an RKNN model through the second model conversion. During the second model conversion, the ONNX model is ported to the Linux system, the model is quantized to convert floating-point storage into integer storage, and then the model is encapsulated into an RKNN model suitable for deployment on edge devices.

2. A device for detecting spalling defects in concrete buildings, characterized in that: include: A first processing module is used to pre-process the acquired concrete building spalling image; The second processing module is used to divide the pre-processed concrete building spalling images into a training set and a test set; The third processing module is used to train the PL-YOLO11 model based on the training set. In the PL-YOLO11 model, the C3k2 module of the backbone network of the YOLO11 model is replaced with the C3k2_ODPKIModule module. The C3k2_ODPKIModule module is based on the PKI Module in the remote sensing image target detection PKINet. The convolution of its output channel is replaced with multi-scale convolution to obtain the OD-PKI module, which is then integrated into C3k2 to obtain the C3k2_ODPKIModule module. The LADH detection head replaces the YOLO detection head. The fourth processing module is used to perform structured pruning on the trained PL-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the PL-YOLO11_Pruned model; The fifth processing module is used to convert the PL-YOLO11_Pruned model into a model format; The sixth processing module is used to deploy the converted PL-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when spalling of concrete buildings is detected, mark the defect location and display the detection confidence level; The fourth processing module prunes specific modules in the trained PL-YOLO11 model, including pruning the OD-PKI in the C3k2_ODPKIModule module and pruning the LADH detection head. The pruning of the LADH detection head includes the Reg and Cls branches of the P3, P4, and P5 layers. The fifth processing module converts the PL-YOLO11_Pruned model into an ONNX model through the first model conversion; and converts the ONNX model into an RKNN model through the second model conversion.

3. A concrete building spalling defect detection system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program to be executed by the processor, and when the computer program is executed by the processor, the method for detecting spalling defects in concrete buildings according to claim 1 is executed.

4. A storage medium, characterized in that The storage medium stores a computer program, which executes the concrete building spalling defect detection method according to claim 1 when running.

Images