Concrete bridge efflorescence defect detection method, device, system, and storage medium

Abstract

The present invention discloses a method, device, system, and storage medium for detecting efflorescence defects in concrete bridges. These methods, which fall within the technical field of bridge defect detection, include dividing concrete bridge efflorescence images into a training set and a test set; training an ADSN-YOLO11 model based on the training set; performing structured pruning on the trained ADSN-YOLO11 model and fine-tuning the pruned model to obtain an ADSN-YOLO11_Pruned model; converting the ADSN-YOLO11_Pruned model to a new format; and deploying the converted ADSN-YOLO11_Pruned model on an RKNN hardware platform. Multi-threaded real-time detection is then performed on the test set. When efflorescence is detected, the defect location is marked and the detection confidence level is displayed. The technical solution of the present invention enables rapid and accurate detection of bridge efflorescence defects.

Description

Technical Field

[0001] The present invention belongs to the technical field of bridge defect detection, and in particular relates to a method and device, a system, and a storage medium for detecting efflorescence defects in concrete bridges. Background Art

[0002] Concrete bridges, exposed to the elements for extended periods, are susceptible to rain, temperature and humidity fluctuations, and freeze-thaw cycles, leading to efflorescence. If not promptly repaired, efflorescence not only affects the appearance but can also cause concrete surface spalling, reducing the concrete's density and durability, and threatening the safety of the bridge structure.

[0003] However, existing detection methods have many problems, especially those based on deep learning for concrete bridge efflorescence detection. These methods are not very real-time and inefficient, often requiring data to be transmitted to a remote server for processing, and are unable to provide rapid feedback on detection results to guide bridge maintenance. Secondly, the models are not very generalizable, and these methods are unable to identify efflorescence defects under different background features. This is because the constructed datasets have a single background, and deep learning models are unable to learn the characteristics of efflorescence on bridges with rich background features in reality. At the same time, there are significant deficiencies in feature extraction for concrete bridge efflorescence, making it difficult to effectively identify the subtle features of concrete bridge efflorescence, limiting the accuracy and reliability of the detection results. Finally, most existing models are complex in structure and consume a lot of computing resources, making it difficult to adapt to the requirements of lightweight and low-power detection equipment in the special outdoor environment of bridges. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device, system, and storage medium for detecting efflorescence defects in concrete bridges, so as to accurately extract the characteristics of efflorescence in concrete bridges and realize rapid and accurate detection of efflorescence defects in bridges.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for detecting efflorescence defects in concrete bridges, comprising:

[0007] Step S1, preprocessing the acquired concrete bridge efflorescence image;

[0008] Step S2: dividing the pre-processed concrete bridge efflorescence images into a training set and a test set;

[0009] Step S3: Train the ADSN-YOLO11 model according to the training set. In the ADSN-YOLO11 model, the convolution block of the YOLO11 model is replaced with the Adown module. The ADSN-YOLO11 model adopts the OD2SN neck structure, which is based on the SlimNeck neural network neck. The two convolution blocks of the input channel in the VoVGSCSP module are replaced with the ODConv module.

[0010] Step S4: perform structured pruning on the trained ADSN-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the ADSN-YOLO11_Pruned model;

[0011] Step S5: Convert the ADSN-YOLO11_Pruned model to a new model format.

[0012] Step S6: Deploy the converted ADSN-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when efflorescence of concrete bridges is detected, mark the defect location and display the detection confidence.

[0013] Preferably, in step S4, pruning operations are performed on specific modules in the trained ADSN-YOLO11 model, including: pruning Bottleneck in the C3k2 module, pruning Adown, pruning GSConv in the OD2SN neck structure, specifying pruning of convolution kernels between different modules, and pruning the detection head; wherein, for the pruning of the detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers.

[0014] Preferably, in step S5, the ADSN-YOLO11_Pruned model is converted into an ONNX model through a first model conversion; and the ONNX model is converted into an RKNN model through a second model conversion; wherein, during the second model conversion, the ONNX model is ported to the Linux system, the model is quantized to convert the floating-point storage into integer storage, and then the model is encapsulated into an RKNN model suitable for deployment on edge devices.

[0015] The present invention also provides a device for detecting efflorescence defects in concrete bridges, comprising:

[0016] The first processing module is used to pre-process the acquired concrete bridge efflorescence image;

[0017] The second processing module is used to divide the pre-processed concrete bridge alkali-fading images into a training set and a test set;

[0018] The third processing module is used to train the ADSN-YOLO11 model based on the training set. In the ADSN-YOLO11 model, the convolution block of the YOLO11 model is replaced with the Adown module. The ADSN-YOLO11 model adopts the OD2SN neck structure, which is based on the SlimNeck neural network neck. The two convolution blocks of the input channel in the VoVGSCSP module are replaced with the ODConv module.

[0019] The fourth processing module is used to perform structured pruning on the trained ADSN-YOLO11 model, and then fine-tune the accuracy of the structured pruned model to obtain the ADSN-YOLO11_Pruned model;

[0020] The fifth processing module is used to convert the ADSN-YOLO11_Pruned model into a model format;

[0021] The sixth processing module is used to deploy the ADSN-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when alkali efflorescence is detected in concrete bridges, mark the defect location and display the detection confidence.

[0022] Preferably, the fourth processing module performs pruning operations on specific modules in the trained ADSN-YOLO11 model, including: pruning Bottleneck in the C3k2 module, pruning Adown, pruning GSConv in the OD2SN neck structure, specifying pruning of convolution kernels between different modules, and pruning the detection head; wherein, for the pruning of the detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers.

[0023] Preferably, the fifth processing module converts the ADSN-YOLO11_Pruned model into an ONNX model through a first model conversion; and converts the ONNX model into an RKNN model through a second model conversion.

[0024] The present invention also provides a concrete bridge efflorescence defect detection system, comprising: a memory and a processor, wherein the memory stores a computer program run by the processor, and the computer program executes a concrete bridge efflorescence defect detection method when run by the processor.

[0025] The present invention also provides a storage medium on which a computer program is stored. When the computer program is run, the method for detecting efflorescence defects in concrete bridges is executed.

[0026] Compared with the prior art, the present invention has the following technical effects:

[0027] 1. Strong Background Adaptability: This method collects images of concrete bridge efflorescence under different backgrounds and uses multiple methods to enhance the data. This data includes diverse background information, exposing the model to a wider range of visual effects during training, enabling the model to learn a wider range of background features. Compared to existing technologies, this dataset is richer and more diverse in background, better simulating the complex background conditions found in real-world scenarios and significantly improving the model's robustness to complex backgrounds.

[0028] 2. Accurate feature extraction capability: This paper optimizes the YOLO11 algorithm. By improving the algorithm's feature extraction layer and network structure, it can more effectively capture the detailed features of the efflorescence area, thereby significantly improving the accuracy and reliability of the detection results.

[0029] 3. Lightweight and efficient: The present invention improves the YOLO11 model to obtain a lightweight ADSN-YOLO11 model. On this basis, it also performs lightweight operations such as structured pruning and quantization, which greatly reduces the complexity and computing resource consumption of the model while maintaining efficient detection performance, making it more suitable for deployment and operation on edge devices.

[0030] 4. Real-time and convenience: Based on edge intelligence technology and adopting a multi-threaded reasoning mechanism, this invention realizes real-time detection of efflorescence in concrete bridges using a lightweight model. It can be directly deployed at the bridge site without relying on a remote server, significantly shortening the detection cycle and providing rapid feedback of detection results to guide bridge maintenance work. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0032] Figure 1 Schematic diagram of the process of the method for detecting efflorescence defects in concrete bridges according to the present invention;

[0033] Figure 2 This is the network structure diagram of the unimproved YOLO11 model;

[0034] Figure 3 The present invention is based on Figure 2 Schematic diagram of the designed ADSN-YOLO11 model network structure;

[0035] Figure 4 This is a schematic diagram of the network structure of the Adown module;

[0036] Figure 5 This is the network structure diagram of the VoVGSCSP module in the unimproved SlimNeck neck structure;

[0037] Figure 6 The present invention is based on Figure 5 Designed improved VoVGSCSP module;

[0038] Figure 7 Schematic diagram of the detection head structure;

[0039] Figure 8 Schematic diagram of the workflow for multi-threaded inference on the RKNN hardware platform. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example 1:

[0043] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting efflorescence defects in concrete bridges, comprising:

[0044] Step S1, preprocessing the acquired concrete bridge efflorescence image;

[0045] Step S2: dividing the pre-processed concrete bridge efflorescence images into a training set and a test set;

[0046] Step S3: train the ADSN-YOLO11 model based on the training set;

[0047] Step S4: perform structured pruning on the trained ADSN-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the ADSN-YOLO11_Pruned model;

[0048] Step S5: Perform two model format conversions on the ADSN-YOLO11_Pruned model;

[0049] Step S6: Deploy the converted ADSN-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when efflorescence of concrete bridges is detected, mark the defect location and display the detection confidence.

[0050] As an implementation of the embodiment of the present invention, in step S1, the concrete bridge efflorescence image includes concrete bridge efflorescence images under various backgrounds, such as a concrete river-spanning bridge, an elevated highway bridge, and an arch bridge.

[0051] As an implementation of an embodiment of the present invention, in step S1, preprocessing the acquired concrete bridge efflorescence image includes:

[0052] Step S11: Use the LabelImg tool to label the efflorescence targets in the concrete bridge efflorescence image to obtain a first bridge image dataset containing target categories and target locations;

[0053] Step S12: performing data enhancement on the first bridge image dataset to obtain a second bridge image dataset; wherein the data enhancement is performed by randomly adjusting brightness and contrast, randomly adding sun flares, randomly adding fog, randomly adding shadows, randomly blurring, and randomly adjusting hue and saturation to enhance the first bridge image dataset;

[0054] As an implementation of the embodiment of the present invention, in step S2, the second bridge image dataset is divided into a training set and a test set.

[0055] As an implementation method of the embodiment of the present invention, in step S3, the YOLO11 model is improved to obtain the ADSN-YOLO11 model, such as Figure 2 The improvement method is to replace the convolution block of the YOLO11 model with the Adown module for enhancing the feature extraction capability of concrete bridge alkali and lightweight. The network structure of the Adown module is as follows: Figure 4, the feature map output from the C3k2 module first passes through the AvgPool2d operation, and then is split into two parts by Split, with the number of channels in each part halved. One part directly enters the first CBS module; the other part first passes through the MaxPool2d operation, and then enters the second CBS module. Finally, the outputs of the two parts are merged in the Concat operation to restore the original number of channels. This structure combines the average pooling and maximum pooling operations, enhances the diversity and robustness of feature extraction, realizes the downsampling of feature maps, reduces the number of parameters and calculations, and improves the detection accuracy of the model. The ADSN-YOLO11 model adopts the OD2SN neck structure. The OD2SN neck structure is based on the SlimNeck neural network neck. The two convolution blocks of the input channels in the VoVGSCSP module are replaced by ODConv blocks with stronger extraction capabilities to enhance the extraction capabilities of pan-alkali features. The VoVGSCSP network structure is as follows Figure 5 The feature map obtained by Concat fusion enters the first convolutional layer for feature extraction, and the output feature map is input into the GS bottleneck module for further feature processing. At the same time, the fused feature map also enters the second convolutional layer through another path for feature extraction. The two feature map outputs after processing by the GS bottleneck module and the second convolutional layer are input into the Concat operation for feature splicing. The spliced ​​feature map is finally passed through the third convolutional layer for feature fusion to obtain the final output feature map. The optimized VoVGSCSP network structure is shown below. Figure 6 , the feature map obtained by Concat feature fusion enters the first ODConv layer for feature extraction, and the output feature map is input into the GS bottleneck module for further feature processing. At the same time, the fused feature map also enters the second ODConv layer through another path for feature extraction. The two feature map outputs processed by the GS bottleneck module and the second ODConv layer are input into the Concat operation for feature splicing. The spliced ​​feature map is finally passed through the Conv layer for feature fusion to obtain the final output feature map. This structure enhances the diversity of feature extraction and the expression ability of the network through parallel ODConv paths and GS bottleneck modules. The Concat operation effectively fuses the feature information of different paths, which helps to improve the performance and accuracy of the model when processing complex images. Combining the two improvements, the ADSN-YOLO11 model is obtained, and its network model structure is as follows Figure 3The image to be detected enters the Backbone and is processed sequentially through multiple layers of Conv, C3k2, and ADown before finally being fed into the neck by C2PSA. In the OD2SN neck structure, features at different levels are fused and processed: some features are directly output by the Backbone's 4th layer, fused with other processed features in the 15th Concat layer, processed by VoVGSCSP, and then fed into Head 1 of the detection head structure. Features output by C2PSA are upsampled by the 11th Upsample layer, fused with features from the Backbone's 6th layer in the 12th Concat layer, processed by the 13th VoVGSCSP layer, fused with other features in the 18th Concat layer, connected to Head 2 by the 19th VoVGSCSP layer. Another branch passes through a GSConv layer and is fused with another Backbone output feature in the 21st Concat layer. Finally, it is connected to Head 3 by the 22nd VoVGSCSP layer. This structure achieves multi-scale target detection through multiple feature fusions, upsampling, and processing of specific modules GSConv and VoVGSCSP. The components work closely together to improve the accuracy and efficiency of detection and ensure the effective recognition and positioning of targets of different sizes.

[0056] Furthermore, when training the ADSN-YOLO11 model, the data path is set to point to a configuration file containing information about bridge categories, which specifies the training and validation datasets. The bridge images in the training set are uniformly sized at 640×640 pixels, and training is repeated for 3000 rounds. Training is automatically terminated if accuracy does not improve within 100 rounds, with 32 images processed per round. Stochastic gradient descent (SGD) is used as the optimizer.

[0057] As an implementation method of an embodiment of the present invention, in step S4, structured pruning is performed on the trained ADSN-YOLO11 model to reduce the complexity and number of parameters of the model by removing unimportant weights or channels in the model, thereby improving the inference efficiency and running speed of the trained ADSN-YOLO11 model. The specific process is as follows.

[0058] First, load the trained ADSN-YOLO11 model and determine the pruning threshold through the `get_threshold` method of the `PRUNE` class. The threshold is based on the set pruning rate. After experiments, it is most appropriate to set it to 0.6. The pruning rate is used to determine which weights or channels can be removed. Then, prune specific modules in the trained ADSN-YOLO11 model. The pruning operations include: pruning Bottleneck in the C3k2 module, pruning Adown, pruning GSConv in the OD2SN neck structure, pruning the detection head, and then resetting the gradient of the model parameters for precision fine-tuning training to obtain the ADSN-YOLO11_Pruned model. Among them, for the pruning of the detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers, such as Figure 7 As shown, P3, P4, and P5 correspond to feature maps at different levels of the three detection heads. Reg represents the regression prediction output, and Cls represents the classification prediction output. The convolutional modules in these two branches are pruned layer by layer, removing redundant channels that have little impact on the results. The input channels of subsequent modules are then adjusted to ensure the coherence of the network structure. This entire process not only reduces the model's computational burden but also, through a sophisticated pruning strategy, preserves the model's key features and performance to the greatest extent possible.

[0059] As an implementation method of an embodiment of the present invention, in step S5, the ADSN-YOLO11_Pruned model is converted twice. The first model conversion converts the ADSN-YOLO11_Pruned model into an ONNX model; the second model conversion converts the ONNX model into an RKNN model. In the second model conversion, the ONNX model is ported to a Linux system, the model is quantized to convert floating-point storage into integer storage, and the model is then encapsulated into an RKNN model suitable for deployment on edge devices.

[0060] As an implementation method of the embodiment of the present invention, in step S6, in order to realize the analysis of the real-time captured concrete bridge image, multi-threaded work is started, such as Figure 8 As shown, thread A reads in video frames; thread pool B is started and multiple threads are configured to load, infer, and post-process the converted ADSN-YOLO11_Pruned model. To avoid the performance overhead of frequent thread creation and destruction, a certain number of threads are pre-created and placed in a pool for unified management. This approach enables efficient task scheduling, ensuring that each thread can quickly respond to new tasks when idle, improving the model's inference speed and responsiveness when processing continuous video frames. Finally, thread C is started to stream the inferred video frames and save the results to the corresponding file.

[0061] Example 2:

[0062] An embodiment of the present invention further provides a device for detecting efflorescence defects in concrete bridges, comprising:

[0063] The first processing module is used to pre-process the acquired concrete bridge efflorescence image;

[0064] The second processing module is used to divide the pre-processed concrete bridge alkali-fading images into a training set and a test set;

[0065] The third processing module is used to train the ADSN-YOLO11 model based on the training set. In the ADSN-YOLO11 model, the convolution block of the YOLO11 model is replaced with the Adown module. The ADSN-YOLO11 model adopts the OD2SN neck structure, which is based on the SlimNeck neural network neck. The two convolution blocks of the input channel in the VoVGSCSP module are replaced with the ODConv module.

[0066] The fourth processing module is used to perform structured pruning on the trained ADSN-YOLO11 model, and then fine-tune the accuracy of the structured pruned model to obtain the ADSN-YOLO11_Pruned model;

[0067] The fifth processing module is used to convert the ADSN-YOLO11_Pruned model into a model format;

[0068] The sixth processing module is used to deploy the ADSN-YOLO11_Pruned model after model format conversion to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and when alkali efflorescence is detected in concrete bridges, mark the defect location and display the detection confidence.

[0069] As an implementation method of an embodiment of the present invention, the fourth processing module performs pruning operations on specific modules in the trained ADSN-YOLO11 model, including: pruning Bottleneck in the C3k2 module, pruning Adown, pruning GSConv in the OD2SN neck structure, specifying pruning of convolution kernels between different modules, and pruning the detection head; wherein, for the pruning of the detection head, the pruned part includes the Reg branch and Cls branch of the P3, P4, and P5 layers.

[0070] As an implementation manner of an embodiment of the present invention, the fifth processing module converts the ADSN-YOLO11_Pruned model into an ONNX model through a first model conversion; and converts the ONNX model into an RKNN model through a second model conversion.

[0071] Example 3:

[0072] The present invention also provides a concrete bridge efflorescence defect detection system, comprising: a memory and a processor, wherein the memory stores a computer program run by the processor, and the computer program executes a concrete bridge efflorescence defect detection method when run by the processor.

[0073] Example 4:

[0074] The present invention also provides a storage medium on which a computer program is stored. When the computer program is run, the method for detecting efflorescence defects in concrete bridges is executed.

[0075] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for detecting efflorescence defects in concrete bridges, characterized in that: include: Step S1, preprocessing the acquired concrete bridge efflorescence image; Step S2: dividing the pre-processed concrete bridge efflorescence images into a training set and a test set; Step S3: Train the ADSN-YOLO11 model according to the training set. In the ADSN-YOLO11 model, the convolution block of the YOLO11 model is replaced with the Adown module. The ADSN-YOLO11 model adopts the OD2SN neck structure, which is based on the SlimNeck neural network neck. The two convolution blocks of the input channel in the VoVGSCSP module are replaced with the ODConv module. Step S4: perform structured pruning on the trained ADSN-YOLO11 model, and then perform precision fine-tuning training on the structured pruned model to obtain the ADSN-YOLO11_Pruned model; Step S5: Convert the ADSN-YOLO11_Pruned model to a new model format. Step S6: Deploy the converted ADSN-YOLO11_Pruned model to the RKNN hardware platform, enable the thread pool to perform multi-threaded real-time detection on the test set, and indicate the defect location and display the detection confidence level when efflorescence of concrete bridges is detected. In step S4, pruning specific modules in the trained ADSN-YOLO11 model includes pruning the Bottleneck in the C3k2 module, pruning Adown, pruning the GSConv in the OD2SN neck structure, and pruning the detection head. The pruning of the detection head includes the Reg branch and Cls branch of the P3, P4, and P5 layers. In step S5, the ADSN-YOLO11_Pruned model is converted into an ONNX model through the first model conversion; and the ONNX model is converted into an RKNN model through the second model conversion. During the second model conversion, the ONNX model is ported to the Linux system, the model is quantized to convert floating-point storage into integer storage, and then the model is encapsulated into an RKNN model suitable for deployment on edge devices.

2. A device for detecting efflorescence defects in concrete bridges, characterized in that: include: The first processing module is used to pre-process the acquired concrete bridge efflorescence image; The second processing module is used to divide the pre-processed concrete bridge alkali-fading images into a training set and a test set; The third processing module is used to train the ADSN-YOLO11 model based on the training set. In the ADSN-YOLO11 model, the convolution block of the YOLO11 model is replaced with the Adown module. The ADSN-YOLO11 model adopts the OD2SN neck structure, which is based on the SlimNeck neural network neck. The two convolution blocks of the input channel in the VoVGSCSP module are replaced with the ODConv module. The fourth processing module is used to perform structured pruning on the trained ADSN-YOLO11 model, and then fine-tune the accuracy of the structured pruned model to obtain the ADSN-YOLO11_Pruned model; The fifth processing module is used to convert the ADSN-YOLO11_Pruned model into a model format; The sixth processing module is used to deploy the converted ADSN-YOLO11_Pruned model to the RKNN hardware platform, start the thread pool to perform multi-threaded real-time detection on the test set, and mark the defect location and display the detection confidence when efflorescence of concrete bridges is detected; The fourth processing module prunes specific modules in the trained ADSN-YOLO11 model, including: pruning Bottleneck in the C3k2 module, pruning Adown, pruning GSConv in the OD2SN neck structure, and pruning the detection head. Among them, the pruning of the detection head includes the Reg branch and Cls branch of the P3, P4, and P5 layers. The fifth processing module converts the ADSN-YOLO11_Pruned model into an ONNX model through the first model conversion; and converts the ONNX model into an RKNN model through the second model conversion.

3. A concrete bridge efflorescence defect detection system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the method for detecting efflorescence defects in concrete bridges according to claim 1 is executed.

4. A storage medium, characterized in that The storage medium stores a computer program, which executes the method for detecting efflorescence defects in concrete bridges according to claim 1 when the computer program is run.

Images