A pavement disease detection method and system based on Mask R-CNN and transfer learning

Abstract

This invention discloses a pavement defect detection method and system based on Mask R-CNN and transfer learning. The method comprises: collecting pavement defect images to form a dataset; annotating the dataset with defects using the Labelme annotation tool; expanding the annotated road defect dataset using an optimized Stable Diffusion 3.5 Large model; extracting multiscale features from the expanded road defect dataset using multiple backbone feature extraction architectures; fusing the extracted multiscale features using a scale-complementary feature fusion network (SCFN); and constructing an optimal detection model based on Mask R-CNN based on the fused multiscale features. Finally, the optimal detection model is used to quantify the defect images to be detected, completing defect detection in complex road environments. This invention effectively solves the problems of multi-type and multi-target pavement defect detection and pixel quantization in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of pavement defect detection, and specifically relates to a pavement defect detection method and system based on Mask R-CNN and transfer learning. Background Art

[0002] Pavement cracks (such as transverse cracks, longitudinal cracks, and network cracks) and potholes are key factors affecting road safety and durability. These defects not only threaten vehicle safety but also accelerate the degradation of road structures, significantly shortening highway service life. Therefore, rapid and accurate detection of pavement defects is crucial for ensuring traffic safety and optimizing maintenance decisions. In recent years, artificial intelligence technologies (such as deep learning) have demonstrated significant advantages in pavement defect detection, but their practical application still faces many challenges.

[0003] Convolutional neural networks are deep learning algorithms that require extensive training data to improve their performance. However, the amount of available pavement defect data is limited, and there are no large, publicly available standard datasets. Therefore, correctly augmenting pavement defect images is a prerequisite for achieving optimal CNN performance. With the increasing demands for pavement maintenance, single-model defect classification, defect location, and pixel segmentation are no longer sufficient for precise maintenance. Furthermore, independent detection using multiple models is time-consuming and prone to multiple errors. Therefore, achieving multifunctional pavement defect detection is crucial. Currently, road backgrounds are becoming increasingly complex, while most current research focuses on cracks within a single background. Therefore, overcoming the impact of complex backgrounds and achieving rapid and accurate defect detection is a prerequisite for road maintenance. Current methods for road defect detection involve continuous image capture or direct recording of road videos, processing them into frames, and then directly feeding these frames into the model for detection. This approach can cause a single defect to appear repeatedly across multiple images or frames, leading to duplicate calculations in the model and impacting maintenance decisions. In general, there is an urgent need to propose a road surface damage detection method based on deep learning to achieve fast and accurate damage detection. Summary of the Invention

[0004] To solve the problem of how to quickly and accurately detect pavement defects in complex backgrounds, the present invention provides a pavement defect detection method and system based on Mask R-CNN and transfer learning, which effectively solves the problems of multi-type and multi-target pavement defect detection and pixel quantization in complex backgrounds.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A pavement damage detection method based on Mask R-CNN and transfer learning, the method comprising:

[0007] Step 1: Collect pavement disease images to form a data set;

[0008] Step 2: Use the Labelme annotation tool to label the disease in the dataset;

[0009] Step 3: Use the optimized Stable Diffusion 3.5 Large model to expand the annotated road damage dataset;

[0010] Step 4: Multi-scale features are extracted from the expanded road damage dataset using a variety of backbone feature extraction architectures. The extracted multi-scale features are fused using the Scale Complementary Feature Fusion Network (SCFN). Based on the fused multi-scale features, an optimal detection model based on Mask R-CNN is constructed.

[0011] In step 5, the optimal detection model based on Mask R-CNN is used to quantify the defect images to be detected and complete defect detection in complex road backgrounds.

[0012] Preferably, in step 3, the annotated road damage dataset is expanded using the optimized Stable Diffusion 3.5 Large model, including:

[0013] Use VAE-encoder to extract visual features of the input image;

[0014] Dynamically adjusting generation parameters and cue word embedding vectors based on the visual features to generate synthetic images with structural diversity and style variations;

[0015] Based on the synthesized image, a domain prompting strategy based on the physical morphology of road damage is introduced to generate content that is more in line with the evolution characteristics of actual cracks and potholes;

[0016] Build an automated screening mechanism and use the trained convolutional neural network to detect the semantic consistency and feature fidelity of synthetic images to eliminate noisy and forged images.

[0017] Preferably, in step 4, the multiple backbone feature extraction architectures include: selecting two deep convolutional networks, ResNet-50 and ResNet-101, as the backbone structures, and respectively combining FPN (Pyramid Feature Network), C4 (Fourth Stage Convolution Output), and DC5 (Five Stage Dilated Convolution Structure) to construct feature extraction modules, and designing six differentiated feature paths;

[0018] Among them, the C4 structure extracts mid-level features from ResNet Stage 4; the DC5 architecture introduces dilated convolution in Stage 5 to expand the receptive field to enhance global semantic expression; the FPN structure uses cross-layer lateral connections and upsampling mechanisms to fuse semantic information of different scales.

[0019] Preferably, in step 4, fusing the extracted multi-scale features based on a scale complementary feature fusion network includes:

[0020] Extract features with texture resolution that meets preset requirements from the middle and shallow layers of the backbone network;

[0021] Obtaining abstract semantic representations with global perception capabilities from deep layers;

[0022] At the same time, the multi-scale semantic features constructed in the FPN module are integrated to cover target information of different sizes.

[0023] Preferably, in step 4, based on the fused multi-scale features, an optimal detection model based on Mask R-CNN is constructed, including:

[0024] ResNet50 / 101 is used as the backbone network, and eight feature combinations are constructed with FPN, C4, DC5, and SCFN respectively. These are then substituted into the Mask R-CNN framework to form a detection model based on Mask R-CNN.

[0025] Adjust the detection model hyperparameters to obtain the optimal detection model based on Mask R-CNN.

[0026] The present invention also provides a pavement disease detection system based on Mask R-CNN and transfer learning, the system is used to implement the above method, the system includes: an acquisition module, a labeling module, an expansion module, a construction module and a detection module;

[0027] The acquisition module is used to collect road surface disease images to form a data set;

[0028] The labeling module is used to label the data set with diseases using the Labelme labeling tool;

[0029] The expansion module is used to expand the annotated road damage dataset using the optimized Stable Diffusion 3.5 Large model;

[0030] The construction module is used to extract multi-scale features from the expanded road damage dataset using multiple backbone feature extraction architectures, fuse the extracted multi-scale features based on a scale complementary feature fusion network (SCFN), and construct an optimal detection model based on Mask R-CNN based on the fused multi-scale features;

[0031] The detection module is used to quantify the disease images to be detected using the optimal detection model based on Mask R-CNN to complete disease detection in complex road backgrounds.

[0032] Preferably, the expansion module includes: an extraction unit, an adjustment unit, an enhancement unit and a screening unit;

[0033] The extraction unit is used to extract visual features of the input image using the VAE-encoder;

[0034] The adjustment unit is used to dynamically adjust the generation parameters and the prompt word embedding vector according to the visual features to generate a synthetic image with structural diversity and style changes;

[0035] The enhancement unit is configured to introduce a domain prompting strategy based on the physical morphology of road damage according to the synthesized image, so as to generate content that is more in line with the evolution characteristics of actual cracks and potholes;

[0036] The screening unit is used to build an automated screening mechanism, use a trained convolutional neural network to detect semantic consistency and feature fidelity of the synthesized image, and eliminate noisy images and forged images.

[0037] Preferably, in the construction module, the multiple backbone feature extraction architectures include: selecting two deep convolutional networks, ResNet-50 and ResNet-101, as the backbone structure, and respectively combining FPN (Pyramid Feature Network), C4 (Fourth Stage Convolution Output), and DC5 (Five Stage Dilated Convolution Structure) to construct feature extraction modules, and designing six differentiated feature paths;

[0038] Among them, the C4 structure extracts mid-level features from ResNet Stage 4; the DC5 architecture introduces dilated convolution in Stage 5 to expand the receptive field to enhance global semantic expression; the FPN structure uses cross-layer lateral connections and upsampling mechanisms to fuse semantic information of different scales.

[0039] Preferably, the construction module includes: a resolution feature unit, a semantic representation unit and a fusion unit;

[0040] The resolution feature unit is used to extract features with texture resolution that meets preset requirements from the middle and shallow layers of the backbone network;

[0041] The semantic representation unit is used to obtain an abstract semantic representation with global perception capability from a deep layer;

[0042] The fusion unit is used to simultaneously fuse the multi-scale semantic features constructed in the FPN module to cover target information of different sizes.

[0043] Preferably, in the construction module, an optimal detection model based on Mask R-CNN is constructed based on the fused multi-scale features, including:

[0044] ResNet50 / 101 is used as the backbone network, and eight feature combinations are constructed with FPN, C4, DC5, and SCFN respectively. These are then substituted into the Mask R-CNN framework to form a detection model based on Mask R-CNN.

[0045] Adjust the detection model hyperparameters to obtain the optimal detection model based on Mask R-CNN.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. This paper introduces the improved Stable Diffusion 3.5 Large model into the road damage data augmentation process for the first time, serving as the core sample generation engine. Unlike traditional geometric transformation augmentation methods (such as rotation, mirroring, and cropping), this method utilizes an explicitly guided image-to-image generation module, combined with prompt embedding and image parameter control mechanisms, to generate crack and pothole images with significant structural differences and diverse visual styles, significantly improving the dataset's coverage of damage diversity and scene complexity.

[0048] 2. This paper uses a transfer learning approach to compare the performance of Faster R-CNN and Mask R-CNN algorithms for pavement defect recognition in complex backgrounds. The results show that Mask R-CNN outperforms Faster R-CNN in pavement defect detection. By adjusting the hyperparameters of the Mask R-CNN model, an optimal detection model was obtained. This model was used to perform quantitative verification on defect images to be detected, demonstrating its effectiveness in defect detection in complex road environments. This method solves the problems of multi-type and multi-target pavement defect detection and pixel quantization in complex backgrounds, demonstrates high reliability, and will make a positive contribution to the field of intelligent pavement defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 Schematic diagram of some pavement damage images in the data set of an embodiment of the present invention;

[0051] Figure 2 Schematic diagram showing images of partially labeled transverse cracks, longitudinal cracks, network cracks, and pits in an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the Stable Diffusion image expansion process based on adaptive prompt words and parameters according to an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of a scale-complementary feature fusion network according to an embodiment of the present invention;

[0054] Figure 5 This is a flow chart of a pavement damage detection method based on Mask R-CNN and transfer learning according to an embodiment of the present invention;

[0055] Figure 6 Figure 1 shows a performance comparison of Faster R-CNN and Mask R-CNN under six different backbone architectures, including (a) AP diagram and (b) test time diagram.

[0056] Figure 7 Schematic diagram of the optimal detection model based on Mask R-CNN in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Example 1

[0060] The present invention discloses a pavement disease detection method based on Mask R-CNN and transfer learning, which belongs to the field of pavement disease detection. Pavement diseases can cause serious harm to vehicle driving safety and also affect the service life of roads. Therefore, timely and accurate detection of pavement diseases such as cracks and potholes has very important engineering significance. With the continuous improvement of pavement maintenance requirements, single disease classification, disease location and disease pixel segmentation functions can no longer meet the needs of precise maintenance, and multiple models are time-consuming and prone to multiple errors. Therefore, it is very important to realize multifunctional pavement disease detection. In addition, the current road background is becoming more and more complex, and most of the research objects today are still the study of cracks under a single background. It is still difficult to overcome the influence of complex background and realize fast and accurate pavement disease detection. Therefore, the present invention adopts the Stable Diffusion 3.5 Large diffusion model, and through adaptive prompt guidance and input parameters (such as noise level, redraw amplitude, noise type, etc.), batch generates synthetic images containing multiple types of defects (horizontal cracks, longitudinal cracks, network cracks and potholes) and complex backgrounds (shadows, low light, different weather conditions), significantly expanding the size of the dataset and scene diversity. It also uses the transfer learning method to compare the performance of the Faster R-CNN and Mask R-CNN algorithms for road defect recognition under complex backgrounds, proving that Mask R-CNN outperforms Faster R-CNN in road defect detection. By adjusting the hyperparameters of the better Mask R-CNN model, the optimal detection model is obtained. The optimal model is used to quantize the defect images to be detected, verifying the effectiveness of the model in defect detection under complex road backgrounds. This method can effectively solve the problems of multi-type and multi-target road defect detection and pixel quantization in complex backgrounds. The specific implementation process is as follows:

[0061] like Figure 5 As shown, this embodiment provides a pavement damage detection method based on Mask R-CNN and transfer learning, including the following implementation steps:

[0062] Step 1: Collect pavement damage images to form a dataset.

[0063] The dataset used is composed of pavement disease images collected by vehicle-mounted cameras. The dataset contains 900 RGB pavement disease images with an image resolution of 600×600, including different types of pavement diseases such as transverse cracks, longitudinal cracks, network cracks and potholes (such as Figure 1 ).

[0064] Step 2: Use the Labelme annotation tool to label the disease in the dataset.

[0065] For subsequent research needs, it is necessary to perform polygon annotation on road defects and generate json format files. This is done because target detection algorithms (such as Faster R-CNN) and instance segmentation algorithms (such as Mask R-CNN, BlendMask) are needed to identify and segment road defects. In this process, we use the open source image annotation tool Labelme to create and edit image annotation data. Labelme supports multiple annotation types such as points, lines, rectangles, and polygons, and can save image data and annotation information in json format. This makes data processing and analysis more convenient. During the annotation process, we first load the image folder, select the image to be annotated, and then manually annotate the boundaries of each defect. After the annotation is completed, Labelme will generate a corresponding json file, which contains information such as the name of the annotated image, the disease label, the polygon coordinates of the disease edge, and the size of the annotated image (such as Figure 2 In this invention, the network migration models are all pre-trained based on the COCO dataset, and all algorithms are written in Python.

[0066] Step 3: Use the Stable Diffusion 3.5 Large model to expand the road disease dataset.

[0067] like Figure 4 As shown, the road damage sample generation and enhancement process is based on the sample library constructed in step 1. Clearly labeled real-world images are selected, covering a variety of typical road damage morphologies, including transverse cracks, longitudinal cracks, network cracks, and potholes. On the locally deployed Stable Diffusion platform, this study introduced the StableDiffusion 3.5 Large model, fine-tuned from the bridge damage dataset, and combined it with the Image-to-Image generation module to expand the sample set.

[0068] To improve the matching degree between the generated image and the target disease morphology, the system designed a set of adaptive generation mechanisms based on image content: first, the VAE-encoder extracts the texture, edge density and other visual features of the input image, and the question prompt word extracts the text vector through the Bert model. Then, the image feature vector and the text vector are aligned. The alignment strategy uses the cross-modal attention mechanism, and then the generation parameters and the prompt word embedding vector (such as Figure 3, thereby generating synthetic images with structural diversity and stylistic variation. Furthermore, to further enhance the quality and relevance of generated images, a domain hinting strategy based on the physical morphology of road damage is introduced, ensuring that the generated content more closely reflects the evolutionary characteristics of actual cracks and potholes. Specifically, using crack images as an example, physical constraints are added through loss calculation during data augmentation, ensuring that crack length and width are coordinated, and depth and shadow color are proportional.

[0069] During the data cleaning phase, a deep automated screening mechanism was constructed. Specifically, trained convolutional neural networks were used to check the semantic consistency and feature fidelity of synthetic images, eliminating noisy and fabricated images. Finally, manual review was performed to ensure that the images met training requirements for visual integrity and defect expression, providing high-quality input for subsequent model training.

[0070] Step 4: Compare the performance of Faster R-CNN and Mask R-CNN by migrating eight different backbone network structures.

[0071] To achieve high-precision detection of pavement defects at varying scales, this paper systematically evaluates detection performance based on multiple backbone feature extraction architectures. Specifically, two commonly used deep convolutional networks, ResNet-50 and ResNet-101, were selected as the backbone architectures. Feature extraction modules were constructed by combining FPN (Pyramid Feature Network), C4 (fourth-stage convolution output), and DC5 (fifth-stage dilated convolution structure), along with an SCFN network feature extraction module. This resulted in the design of eight differentiated feature paths.

[0072] Among these structures, the C4 structure extracts mid-level features from ResNet Stage 4 and is suitable for detecting medium-sized targets; the DC5 architecture introduces dilated convolution in Stage 5 to expand the receptive field to enhance global semantic expression; the FPN structure uses cross-layer lateral connections and upsampling mechanisms to fuse semantic information of different scales, significantly enhancing the model's robustness to small-sized targets and complex background interference.

[0073] In order to more effectively integrate feature information from different levels, such as Figure 4 As shown in Figure 2, this paper proposes a Scale-Complementary Feature Fusion Network (SCFN) to improve the robustness and accuracy of detection models for diseased objects of varying sizes. This network architecture employs a dual-pathway feature acquisition mechanism, focusing on shallow texture details and deep semantic structure information, respectively, enabling joint modeling of small and large objects.

[0074] Specifically, the proposed SCFN network extracts high-resolution features with rich texture details from the middle and shallow layers of the backbone network (such as the C4 structure), and obtains abstract semantic representations with more global perception capabilities from the deep layers (such as the DC5 structure), while integrating the multi-scale semantic features constructed in the FPN module to comprehensively cover target information of different sizes.

[0075] To achieve efficient fusion of multi-scale features, the feature maps output by each branch are first processed in a unified spatial dimension (for example, resolution matching is achieved through bilinear interpolation). Next, a multi-channel attention mechanism (such as an improved EMA module) is introduced to dynamically model the importance of channels at different scales, guiding the network to flexibly adjust the fusion weights of features at each scale based on the context of the current input image, thereby improving the perception of salient areas.

[0076] Among them, the improved EMA module is as follows. The module inputs a tensor, denoted as , where X is a multi-channel feature map containing multi-scale information generated by multiple feature extraction modules (including C4 structure, DC5 structure and FPN module). The process is as follows:

[0077] First, the input features are channel compressed by using a 1×1 convolution to reduce the number of channels from down to (r is the dimension reduction scaling factor, r is usually set to [4, 8, 16]). Then, in the spatial dimension ( ) is normalized to standardize the feature distribution and improve stability. Then it enters the multi-scale feature extraction stage. The features after dimensionality reduction are sent to three parallel branches respectively: the first branch uses 7×7 depthwise convolution (Depthwise Convolution) for processing. The second branch uses 5×5 depthwise convolution. The third branch uses 3×3 depthwise convolution. Each branch extracts features of different receptive fields and effectively perceives multi-scale spatial information. Subsequently, the outputs of the three branches are concatenated (Concat) in the channel dimension to form a comprehensive multi-scale feature. The concatenated features are mixed through a 1×1 convolution and LayerNorm is applied again for normalization. Finally, a 1×1 convolution is used to expand the channels and restore the number of channels of the feature to the initial value. , which is output as the final result of the improved EMA module. Regarding object detection framework selection, this paper introduces Faster R-CNN and Mask R-CNN, respectively. The former utilizes a region proposal network (RPN) in conjunction with regression and classification heads to construct a detection pipeline; the latter adds a mask prediction branch to achieve fine-scale instance segmentation. Each set of structures (such as FPN / C4 / DC5 / SCFN) is uniformly connected to the RPN and RoI Align modules to achieve candidate box generation and feature alignment.

[0078] Combining ResNet50 / 101 as the backbone network, eight feature combinations were constructed with FPN, C4, and DC5, as well as the proposed SCFN. These were then incorporated into the Faster R-CNN and Mask R-CNN frameworks, resulting in a total of sixteen model architectures (as shown in Table 1). Model naming, such as "F-R50-FPN," indicates a Faster R-CNN + ResNet-50 + FPN architecture. All models employed transfer learning to load pre-trained COCO weights and were fine-tuned on the Stable Diffusion 3.5 Large amplified disease image dataset using the proposed adaptive cue word. Finally, the performance of each model was compared and analyzed using multiple metrics, including accuracy, average precision (AP), and inference time, to select the optimal solution.

[0079] Table 1 Backbone structures of Faster R-CNN and Mask R-CNN

[0080]

[0081] To evaluate the model's detection performance, the study selected AP as the core evaluation metric. AP calculates precision and recall at different thresholds, obtaining the area under the curve to measure the model's average performance across all categories. Precision represents the proportion of positive samples predicted by the model that are actually positive, while recall represents the proportion of actual positive samples correctly identified by the model. The calculation formulas for both are as follows:

[0082]

[0083]

[0084]

[0085] Where T P(True Positive): Indicates the number of samples predicted by the model as positive and actually positive, that is, the number of positive samples correctly predicted by the model as positive. P (False Positive): Indicates the number of samples that the model predicts as positive but are actually negative, that is, the number of negative samples that the model mistakenly predicts as positive. N (False Negative): Indicates the number of samples that the model predicts as negative but are actually positive, that is, the number of positive samples that the model mistakenly predicts as negative. In this paper, the Faster R-CNN and Mask R-CNN algorithms are comprehensively evaluated using the image average test time and AP. Figure 6 The experimental results show that Mask R-CNN performs better than Faster R-CNN in overall performance.

[0086] Step 5: Adjust the hyperparameters of the better model to obtain the optimal detection model. Use the model to quantify the defect images to be detected and test its effect on defect detection in complex road backgrounds.

[0087] In the specific implementation process of this invention, in order to evaluate the adaptability of different detection algorithms in the task of road defect identification, the average inference time and average precision (AP) were selected as evaluation criteria. The performance of multiple typical detection algorithms, including representative methods such as Faster R-CNN and Mask R-CNN, was tested.

[0088] like Figure 6 As shown in the figure, by uniformly evaluating the above models under the same dataset and hardware environment, and based on the comprehensive performance of the two indicators, in the subsequent optimization design, Mask R-CNN was determined as the overall framework. Subsequently, the ResNet-101 backbone network and the SCFN feature extraction module will be selected as the basic detection model.

[0089] On this basis, to further improve the accuracy and adaptability of disease detection, we performed hyperparameter tuning on the selected model, focusing on the impact of the learning rate on detection performance. Relying on a standard training process, we set multiple learning rates (including but not limited to 0.01, 0.001, and 0.0001) for model training to explore their specific effects on model fitting ability and detection stability.

[0090] Finally, based on the comprehensive training performance, the Mask R-CNN (M-R101-SCFN structure) model with a learning rate of 0.001 was selected as the target detection model. This model was used as the core framework to complete the diseased area extraction and subsequent quantitative analysis processing, such as Figure 7 shown.

[0091] Example 2

[0092] The present invention also provides a pavement disease detection system based on Mask R-CNN and transfer learning, the system is used to implement the above method, the system includes: an acquisition module, a labeling module, an expansion module, a construction module and a detection module;

[0093] An acquisition module is used to collect pavement disease images to form a data set;

[0094] The labeling module is used to label the disease of the dataset using the Labelme labeling tool;

[0095] The expansion module is used to expand the annotated road damage dataset using the optimized Stable Diffusion 3.5 Large model;

[0096] A construction module is used to extract multi-scale features from the expanded road damage dataset using multiple backbone feature extraction architectures, fuse the extracted multi-scale features based on the Scale Complementary Feature Fusion Network (SCFN), and construct an optimal detection model based on Mask R-CNN based on the fused multi-scale features;

[0097] The detection module is used to quantify the defect images to be detected using the optimal detection model based on Mask R-CNN, and complete defect detection in complex road backgrounds.

[0098] In this embodiment, the expansion module includes: an extraction unit, an adjustment unit, an enhancement unit, and a screening unit;

[0099] An extraction unit, used to extract visual features of the input image using a VAE-encoder;

[0100] An adjustment unit, configured to dynamically adjust generation parameters and cue word embedding vectors based on the visual features to generate a synthetic image with structural diversity and style variations;

[0101] an enhancement unit, configured to introduce a domain prompting strategy based on the physical morphology of road damage according to the synthesized image, so as to generate content that is more in line with the evolution characteristics of actual cracks and potholes;

[0102] The screening unit is used to build an automated screening mechanism, using the trained convolutional neural network to detect the semantic consistency and feature fidelity of synthetic images, and to eliminate noisy images and forged images.

[0103] In this embodiment, the construction module includes multiple backbone feature extraction architectures: selecting two deep convolutional networks, ResNet-50 and ResNet-101, as the backbone structure, and combining FPN (Pyramid Feature Network), C4 (fourth stage convolution output), and DC5 (fifth stage dilated convolution structure) to construct feature extraction modules, designing six differentiated feature paths;

[0104] Among them, the C4 structure extracts mid-level features from ResNet Stage 4; the DC5 architecture introduces dilated convolution in Stage 5 to expand the receptive field to enhance global semantic expression; the FPN structure uses cross-layer lateral connections and upsampling mechanisms to fuse semantic information of different scales.

[0105] In this embodiment, the building blocks include: a resolution feature unit, a semantic representation unit, and a fusion unit;

[0106] The resolution feature unit is used to extract features with texture resolution that meets preset requirements from the middle and shallow layers of the backbone network;

[0107] Semantic representation unit, used to obtain abstract semantic representation with global perception capabilities from deep layers;

[0108] The fusion unit is used to simultaneously fuse the multi-scale semantic features constructed in the FPN module to cover target information of different sizes.

[0109] In this embodiment, in the construction module, an optimal detection model based on Mask R-CNN is constructed based on the fused multi-scale features, including:

[0110] ResNet50 / 101 is used as the backbone network, and eight feature combinations are constructed with FPN, C4, DC5, and SCFN respectively. These are then substituted into the Mask R-CNN framework to form a detection model based on Mask R-CNN.

[0111] Adjust the detection model hyperparameters to obtain the optimal detection model based on Mask R-CNN.

[0112] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A pavement damage detection method based on Mask R-CNN and transfer learning, characterized in that: The method comprises: Step 1: Collect pavement disease images to form a data set; Step 2: Use the Labelme annotation tool to label the disease in the dataset; Step 3: Use the optimized Stable Diffusion 3.5 Large model to expand the annotated road damage dataset; Step 4: Multi-scale features are extracted from the expanded road damage dataset using a variety of backbone feature extraction architectures. The extracted multi-scale features are fused using the Scale Complementary Feature Fusion Network (SCFN). Based on the fused multi-scale features, an optimal detection model based on Mask R-CNN is constructed. Step 5: Quantify the defect images to be detected using the optimal detection model based on Mask R-CNN to complete defect detection in complex road backgrounds. In step 3, the optimized Stable Diffusion 3.5 Large model is used to expand the annotated road damage dataset, including: Use VAE-encoder to extract visual features of the input image; Dynamically adjusting generation parameters and cue word embedding vectors based on the visual features to generate synthetic images with structural diversity and style variations; Based on the synthesized image, a domain prompting strategy based on the physical morphology of road damage is introduced to generate content that is more in line with the evolution characteristics of actual cracks and potholes; Build an automated screening mechanism that uses trained convolutional neural networks to detect semantic consistency and feature fidelity of synthetic images, eliminating noisy and forged images; In step 4, the multiple backbone feature extraction architectures include: selecting two deep convolutional networks, ResNet-50 and ResNet-101, as the backbone structures, and respectively combining FPN (Pyramid Feature Network), C4 (Fourth Stage Convolution Output), and DC5 (Fifth Stage Dilated Convolution Structure) to construct feature extraction modules, and designing six differentiated feature paths; The C4 architecture extracts mid-level features from ResNet Stage 4; the DC5 architecture introduces dilated convolutions in Stage 5 to expand the receptive field and enhance global semantic expression; the FPN architecture utilizes cross-layer lateral connections and upsampling mechanisms to fuse semantic information at different scales. In step 4, the extracted multi-scale features are fused based on a scale complementary feature fusion network, including: Extract features with texture resolution that meets preset requirements from the middle and shallow layers of the backbone network; Obtaining abstract semantic representations with global perception capabilities from deep layers; At the same time, the multi-scale semantic features constructed in the FPN module are integrated to cover target information of different sizes.

2. The method according to claim 1, characterized in that In step 4, based on the fused multi-scale features, an optimal detection model based on Mask R-CNN is constructed, including: ResNet50 / 101 is used as the backbone network, and eight feature combinations are constructed with FPN, C4, DC5, and SCFN respectively. These are then substituted into the Mask R-CNN framework to form a detection model based on Mask R-CNN. Adjust the detection model hyperparameters to obtain the optimal detection model based on Mask R-CNN.

3. A pavement defect detection system based on Mask R-CNN and transfer learning, the system being used to implement the method according to any one of claims 1-2, characterized in that: The system includes: an acquisition module, a labeling module, an expansion module, a construction module and a detection module; The acquisition module is used to collect road surface disease images to form a data set; The labeling module is used to label the data set with diseases using the Labelme labeling tool; The expansion module is used to expand the annotated road damage dataset using the optimized Stable Diffusion 3.5 Large model; The construction module is used to extract multi-scale features from the expanded road damage dataset using multiple backbone feature extraction architectures, fuse the extracted multi-scale features based on a scale complementary feature fusion network (SCFN), and construct an optimal detection model based on Mask R-CNN based on the fused multi-scale features; The detection module is used to quantify the disease images to be detected using the optimal detection model based on Mask R-CNN to complete disease detection in complex road backgrounds.

4. The system according to claim 3, characterized in that The expansion module includes: an extraction unit, an adjustment unit, an enhancement unit and a screening unit; The extraction unit is used to extract visual features of the input image using the VAE-encoder; The adjustment unit is used to dynamically adjust the generation parameters and the prompt word embedding vector according to the visual features to generate a synthetic image with structural diversity and style changes; The enhancement unit is configured to introduce a domain prompting strategy based on the physical morphology of road damage according to the synthesized image, so as to generate content that is more in line with the evolution characteristics of actual cracks and potholes; The screening unit is used to build an automated screening mechanism, use a trained convolutional neural network to detect semantic consistency and feature fidelity of the synthesized image, and eliminate noisy images and forged images.

5. The system according to claim 3, wherein: In the building module, multiple backbone feature extraction architectures include: selecting two deep convolutional networks, ResNet-50 and ResNet-101, as the backbone structure, and combining FPN (Pyramid Feature Network), C4 (fourth stage convolution output), and DC5 (fifth stage dilated convolution structure) to construct feature extraction modules, designing six differentiated feature paths; Among them, the C4 structure extracts mid-level features from ResNet Stage 4; the DC5 architecture introduces dilated convolution in Stage 5 to expand the receptive field to enhance global semantic expression; the FPN structure uses cross-layer lateral connections and upsampling mechanisms to fuse semantic information of different scales.

6. The system according to claim 5, characterized in that The building block includes: a resolution feature unit, a semantic representation unit and a fusion unit; The resolution feature unit is used to extract features with texture resolution that meets preset requirements from the middle and shallow layers of the backbone network; The semantic representation unit is used to obtain an abstract semantic representation with global perception capability from a deep layer; The fusion unit is used to simultaneously fuse the multi-scale semantic features constructed in the FPN module to cover target information of different sizes.

7. The system according to claim 6, characterized in that In the building module, the optimal detection model based on Mask R-CNN is constructed based on the fused multi-scale features, including: ResNet50 / 101 is used as the backbone network, and eight feature combinations are constructed with FPN, C4, DC5, and SCFN respectively. These are then substituted into the Mask R-CNN framework to form a detection model based on Mask R-CNN. Adjust the detection model hyperparameters to obtain the optimal detection model based on Mask R-CNN.

Images