A method and system for intelligent detection of road diseases based on instance segmentation

Abstract

This invention discloses an intelligent pavement distress detection method and system based on instance segmentation. It utilizes a linear scan camera to acquire image data containing pavement distress, and preprocesses the acquired image data using adaptive histogram equalization. A multi-level feature extraction network is established to extract global features from the image data, outputting multi-scale feature maps. A spatial-channel attention mechanism is used to process the multi-scale feature maps, enhancing their feature information and fully activating the target region. A path aggregation pyramid network with transposed convolutional upsampling is used to fuse the features of the multi-scale feature maps, and the output features of the pyramid network are further enhanced through an adaptive enhancement mechanism. The enhanced multi-scale feature maps are then used for model training. The optimized instance segmentation model is used to predict the pavement distress from multiple acquired images. This invention overcomes the limitations of existing pavement distress detection methods, improves the accuracy of pavement distress detection, and solves difficulties such as cross-image distress identification.

Description

Technical Field

[0001] This invention relates to an intelligent method and system for detecting road surface defects based on instance segmentation, which belongs to the application of artificial intelligence in road defect detection. Background Technology

[0002] Regularly inspecting the quality of asphalt roads and promptly identifying pavement defects is crucial for maintaining road traffic safety. This work was initially primarily done through manual inspections, which were inefficient, costly, and susceptible to subjective interference.

[0003] With the advancement of image processing technology, many algorithms based on traditional digital image processing techniques can effectively identify some road surface defects such as cracks. However, in real-world scenarios, road surface defects vary in type and form, and the robustness and detection performance of traditional algorithms in practical applications are always unsatisfactory.

[0004] The rise of deep learning technology has spurred the rapid development of computer vision, leading to the emergence of various excellent deep learning-based visual detection algorithms. Numerous examples of applying these technologies to traffic scenarios have been published in recent years. Deep learning-based computer vision algorithms typically use backbone networks such as ResNet to extract features, then fuse these features and output the expected detection results through different detection heads. For example, Chinese patent application number CN202210481982.5 discloses a high-precision extraction and recognition method for road surface defects under complex working conditions based on artificial intelligence. Different algorithms perform differently in different application scenarios. In asphalt pavement defect detection tasks, high-precision identification and segmentation of road surface defects are usually required. However, asphalt pavement defects are typically small in area and elongated in shape. Most algorithms have low accuracy when identifying such targets. One major reason is that the feature extraction layer based on convolutional neural networks focuses too much on local image features and lacks the ability to extract image feature information that is elongated or spatially discontinuous. Furthermore, when the detection algorithm fuses the features output by the backbone network, the loss of high-level semantic information during the propagation of high-level feature maps to shallower layers affects the final model's detection performance for small and medium-sized targets.

[0005] In existing methods, most algorithms still rely on convolutional neural networks to extract features. These algorithms are not very effective in detecting asphalt pavement defects, and the quality of the feature maps is not high: there is a lot of background noise and the foreground features are not obvious enough. In addition, various target detection algorithms can only identify defects in a single image, and they are difficult to handle defects such as long cracks and repairs that span multiple images. Summary of the Invention

[0006] The technical problem solved by this invention is to address the limitations of existing deep learning-based technologies in identifying pavement defect features by providing an intelligent pavement defect detection method and system based on instance segmentation, thereby overcoming the difficulty of cross-map defect identification and improving the accuracy of pavement defect detection.

[0007] This invention is achieved using the following technical solution:

[0008] This invention first discloses an intelligent detection method for pavement defects based on instance segmentation, which specifically includes the following steps:

[0009] S100. Use a line scan camera to collect image data containing road surface defects and manually label the ground truth values ​​of road surface defects in the image data. Preprocess the collected image data through adaptive histogram equalization.

[0010] S200. Establish a multi-level feature extraction network to extract global features from image data and output multi-scale feature maps;

[0011] S300 employs a spatial-channel attention mechanism to process multi-scale feature maps, enhancing their feature information and fully activating the target areas corresponding to pavement distress features in the multi-scale feature maps.

[0012] S400 employs a path aggregation pyramid network with transposed convolution upsampling to fuse features across multi-scale feature maps and further enhances the output features of the pyramid network through an adaptive enhancement mechanism.

[0013] S500: Input the enhanced multi-scale feature map into the instance segmentation model for model training. Predict the type, spatial location and segmentation mask of road surface defects through image classification, object detection and instance segmentation of the instance segmentation model, respectively. Calculate the classification loss, bounding box regression loss and mask loss with the manually labeled ground truth values ​​in S100, and optimize the parameters of the instance segmentation model.

[0014] S600. The trained instance segmentation model is used to predict the multiple road surface images. After obtaining the prediction results of each image, the segments are stitched together to obtain the segmentation mask of the entire road surface. Adaptive morphological operations are used to process the mask matrix of each type of defect to finally obtain the defect detection results of the entire road surface.

[0015] In the instance-segmentation-based intelligent pavement distress detection method of the present invention, step S100 further includes the following sub-steps:

[0016] S101. Use the labelme annotation tool to annotate image data containing road surface defects and construct a fully supervised dataset;

[0017] S102. Obtain the image brightness in the fully supervised dataset. If the image brightness is lower than the set brightness threshold, adjust the pixel values ​​of the corresponding image using the following formula.

[0018]

[0019] In the formula p i,j Represents the pixel values ​​in the original image. The adjusted pixel values ​​are represented as follows: β is the offset factor that increases the overall base brightness of the image, and α is the scaling factor that scales the overall pixel values ​​of the original image.

[0020]

[0021] Bright represents the image brightness, which is a value between 0 and 255, and B' is the set brightness threshold.

[0022] S103. Convert the images in the fully supervised dataset from RGB format to HSV format. Histogram equalization is performed on the V channel to improve the overall contrast of the image. When processing the histogram, contrast limiting and bilinear interpolation are used to ensure the continuity of the image. Finally, the HSV format image is converted back to RGB format.

[0023] In the intelligent pavement defect detection method based on instance segmentation of the present invention, further, in step S200, the multi-level feature extraction network is constructed based on the Swin Transformer module. Its basic unit includes two connected Swin Transformer blocks. The basic unit is stacked to construct several sub-modules, and then all the sub-modules are spliced ​​together to obtain the multi-level feature extraction network.

[0024] In the intelligent pavement distress detection method based on instance segmentation of the present invention, further, in step S300, the spatial-channel attention mechanism includes a first branch and a second branch, wherein the first branch performs spatial attention enhancement on the multi-scale feature map using the following formula to obtain the first feature map F. ks :

[0025]

[0026] in This represents the initial feature map of the k-th layer output by a multi-level feature extraction network, where σ is the sigmoid function, Cat indicates concatenation of the first dimension, and MLP is a multilayer perceptron composed of linear layers. and respectively for F k The results of performing spatial max pooling and spatial average pooling;

[0027] The second branch performs channel attention enhancement on the multi-scale feature map using the following formula to obtain the second feature map F. kc :

[0028]

[0029] Where Conv5 represents a 5×5 convolution operation. and respectively for F k The results of performing channel-dimensional max pooling and channel-dimensional average pooling;

[0030] The first feature map F is obtained by the following formula. ks Second feature map F kc The fusion yields the multi-scale feature map F output in this step. kout :

[0031] F kout =Relu(GN(Conv3(F) kc +F ks ))),

[0032] In the formula, ReLU represents a nonlinear activation function, GN represents group regularization, and Conv3 represents a 3×3 convolution operation.

[0033] In the intelligent pavement distress detection method based on instance segmentation of the present invention, step S400 further includes the following sub-steps:

[0034] S401. By performing a 1×1 convolution operation, the channels of the multi-scale feature map output by S300 are adjusted to be unified to 256, resulting in a first-level feature map Fi.

[0035] S402. Utilize a path aggregation pyramid network with transposed convolution upsampling to process the first-level feature map F. i Feature fusion is performed to obtain a secondary feature map F. i ';

[0036] S403, convert the second-level feature maps of each layer of the multi-scale feature map F i 'First-level feature map F before feature fusion with its features' i Two secondary feature maps F in adjacent layers i+1 '、F i-1 The enhanced feature map F is obtained by adaptive weighting using the following formula. i ,

[0037] F i "=Conv3(ωi i F i +ω i1 F i-1 ′+ω i2 F′i +ω i3 F i+1 ′),

[0038] Where Conv3 represents a 3×3 convolution, ω i ω i1 ω i2 ω i3 These are weighted parameters, all of which are learnable parameters defined for building the path aggregation pyramid network.

[0039] In the intelligent detection method for road surface defects based on instance segmentation of the present invention, further, in step S403, the transposed convolution of the pyramid network aggregation performs double upsampling on the multi-scale feature map.

[0040] In the intelligent pavement distress detection method based on instance segmentation of the present invention, further, in step S500, the classification, detection, and segmentation detection head output of the instance segmentation model includes category prediction, target bounding box prediction, and target mask prediction, which correspond to the prediction of the type, spatial location, and segmentation mask of the pavement distress, respectively. The classification loss, bounding box regression loss, and mask loss of the instance segmentation model output prediction are weighted and calculated with the ground truth using the following formula to obtain the total loss:

[0041] L=αL cls +βL box +γL mask ,

[0042] Where L is the total loss, L cls For classification loss, L box For bounding box loss, L mask Let α be the masking loss, and β be the weights of the classification loss, bounding box regression loss, and masking loss, respectively.

[0043] Classification loss L cls and mask loss L mask Cross-entropy loss is used, and bounding box loss L is applied. box SmoothL1 loss is used;

[0044] After calculating the total loss of the instance segmentation model prediction results, the Adam optimizer is used to iteratively optimize the model parameters.

[0045] In the intelligent pavement defect detection method based on instance segmentation of the present invention, further, in step S600, the trained instance segmentation model is used to predict the pavement image. Each image yields C binary matrices of segmentation masks, where C is the number of pavement defect types in the image. The C binary matrices of segmentation masks are concatenated according to the image concatenation order to obtain the segmentation mask of the entire pavement. Adaptive morphological operation closing is used to process the mask matrix of each defect type, connecting regions belonging to the same defect to form a connected region. The depth-first search algorithm is used to traverse the binary matrix of the segmentation mask of the entire pavement to obtain the number of connected components in each matrix, that is, to obtain the quantity, area, and location information of each defect in the entire pavement, which is finally recorded in the detection result.

[0046] This invention also discloses a pavement distress intelligent detection system based on instance segmentation that employs the above-mentioned intelligent pavement distress detection method, comprising the following modules:

[0047] The image acquisition module collects image data containing road surface defects and preprocesses the collected image data through adaptive histogram equalization.

[0048] The multi-level feature extraction module internally establishes a multi-level feature extraction network to extract global features from the image data preprocessed by the image acquisition module and output multi-scale feature maps.

[0049] The feature information enhancement module uses a spatial-channel attention mechanism to process the multi-scale feature maps output by the multi-level feature extraction module, thereby enhancing their feature information and fully activating the target region.

[0050] The feature information fusion module uses a path aggregation pyramid network with transposed convolution upsampling to fuse features from the multi-scale feature maps output by the feature information enhancement module, and further enhances the output features of the pyramid network through an adaptive enhancement mechanism.

[0051] The road surface distress prediction module incorporates an instance segmentation model. Enhanced multi-scale feature maps are input into the instance segmentation model for training. The model predicts the type, spatial location, and segmentation mask of road surface distress through image classification, object detection, and instance segmentation, respectively. Classification loss, bounding box regression loss, and mask loss are calculated with ground truth values ​​to optimize the instance segmentation model parameters. The optimized instance segmentation model is then used to detect and predict multiple collected road surface images. The prediction results for each image are then concatenated to obtain the segmentation mask for the entire road surface. Adaptive morphological operations are used to process the mask matrix for each distress type, ultimately yielding the distress detection results for the entire road surface.

[0052] In addition, the present invention also discloses a computer-readable storage medium storing a computer program that is programmed or configured to implement the above-described intelligent pavement distress detection method based on instance segmentation.

[0053] The present invention has the following beneficial effects:

[0054] (1) The present invention uses adaptive histogram equalization to preprocess the collected image data to avoid overly bright or dark areas in the processed image, and obtains an image with sufficient brightness and higher contrast for processing road surface defects in the image, thus ensuring the reliability and accuracy of image data processing.

[0055] (2) This invention employs a spatial-channel attention feature enhancement mechanism to extract spatial and channel attention information from the feature map and effectively fuses them. Simultaneously considering both spatial and channel global information of the features, it effectively suppresses background noise in the feature map and enhances the feature representation of the target instance region. Compared to convolution, the attention-based architecture can better extract global features from the image, enabling global semantic information modeling. Features extracted by the attention-based backbone network are more robust and suitable for downstream visual tasks such as target detection and instance segmentation. Specifically, the self-attention mechanism allows the model to weighted aggregate information from other positions in the input sequence when processing each position, enabling the model to capture long-distance dependencies between positions in the input sequence without being limited by local receptive fields.

[0056] (3) This invention constructs a path aggregation pyramid network based on transposed convolution upsampling for image feature fusion. Considering that the features of different layers of multi-scale feature maps may contain different feature information of the same target, feature adaptive weighted fusion enhancement is used to further process the multi-scale feature maps and further enhance the output features of the pyramid network. This ensures that each layer of feature map fully contains the discriminative features required to identify the target, and can more efficiently and accurately identify road surface defects in image data.

[0057] (4) The present invention takes into account the actual road inspection environment, which usually uses line scan cameras. For some strip-shaped defects, the same defect may span multiple images. The defect segmentation results of the model are post-processed. The mask is stitched together in sequence and then the mask matrix is ​​processed by adaptive morphological operation. This can avoid the problem of identifying a defect that spans multiple images as multiple defects, and effectively improve the accuracy of defect inspection of the entire road section.

[0058] In summary, the present invention provides an intelligent pavement defect detection method based on instance segmentation, which overcomes the limitations of existing pavement defect detection methods, improves the accuracy of pavement defect detection, solves technical difficulties such as cross-map defect identification, and provides a complete intelligent pavement defect detection system for detecting the number, location, type, and severity of defects along the entire road.

[0059] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the steps of the intelligent pavement distress detection method based on instance segmentation of the present invention.

[0061] Figure 2 This is a schematic diagram of the model optimization process for intelligent detection of road surface defects based on instance segmentation in the embodiment.

[0062] Figure 3 This is a schematic diagram of the process for detecting road surface defects based on the instance segmentation model in the embodiment.

[0063] Figure 4 This is a flowchart illustrating the process of enhancing feature information in multi-scale feature maps using a spatial-channel attention mechanism in step S300 of the embodiment.

[0064] Figure 5 This is a flowchart illustrating the process of feature fusion enhancement of multi-scale feature maps in step S400 of the embodiment.

[0065] Figure 6 This is a schematic diagram of the post-processing flow for outputting the prediction results in step S600 of the embodiment.

[0066] Figure 7 The image shown is a diagram illustrating the input and output of the road surface defect detection image at each step in the embodiment. Detailed Implementation

[0067] Example

[0068] Combination Figure 1 , Figure 2 and Figure 3 As shown, the intelligent pavement distress detection method of the present invention includes the following steps:

[0069] S100: Uses a line scan camera to collect image data containing road surface defects, and preprocesses the collected image data through adaptive histogram equalization, including improving the overall brightness and contrast of the image.

[0070] Taking asphalt pavement as an example, this embodiment uses a line scan camera to acquire images of asphalt pavement. To suppress the image quality degradation caused by differences in imaging conditions, the brightness and contrast of the image are appropriately adjusted. The acquired image data is preprocessed through the following sub-steps:

[0071] S101. Use the labelme annotation tool to annotate the collected asphalt pavement image data to construct a fully supervised dataset. In the fully supervised dataset, the defect features in the image data are manually annotated as ground values ​​for subsequent instance segmentation model training.

[0072] S102. Obtain the image brightness in the dataset using the following formula.

[0073]

[0074] In the formula, (x,y) represents the pixel position in the image. The input image is sliced ​​into three channels, where W and H represent the height and width of the input image, respectively, and Bright represents the image brightness, a value between 0 and 255. If the image brightness is lower than a set brightness threshold (80 in this embodiment), the pixel values ​​of the corresponding image are adjusted using the following formula.

[0075]

[0076] In the formula p i,j Represents the pixel values ​​in the original image. The adjusted pixel values ​​are represented by β, which is an offset factor used to enhance the overall base brightness of the image. It is usually taken as a small constant, such as 20. α is a scaling factor that scales the overall pixel values ​​of the original image. The value of α is taken as follows:

[0077]

[0078] S103. After performing the above preliminary processing on the original image data, an adaptive histogram equalization method is used to further process the image to improve contrast. First, the images in the dataset are converted from RGB format to HSV format, where histogram equalization is performed on the V channel to improve the overall image contrast. To prevent the introduction of excessive noise, a contrast limit is added when processing the histogram, and bilinear interpolation is introduced to ensure the continuity of the image and avoid overly bright or dark areas in the processed image. Finally, the image is converted back to RGB format, resulting in an image with sufficient brightness and higher contrast.

[0079] S200. Establish a multi-level feature extraction network to extract global features from image data and output multi-scale feature maps.

[0080] This embodiment establishes a multi-level feature extraction network based on the Swin Transformer module to extract global features from asphalt pavement images and output multi-scale feature maps. The basic unit of the feature extraction network consists of two connected Swin Transformer blocks. Several sub-modules are then constructed by superimposing the above basic units. Finally, all sub-modules are spliced ​​together to obtain a multi-scale global semantic feature extraction network.

[0081] Specifically, the feature extraction network in this embodiment consists of four sub-modules, with the output multi-scale feature maps of 200×200×96, 100×100×192, 50×50×384, and 25×25×768 respectively. The shallow module outputs feature maps with higher resolution and fewer channels, preserving more image detail; the deep module outputs feature maps with higher channel dimensions, possessing strong semantic information. In subsequent parts of the model, the feature maps of the four scales will be fused to adapt to the road surface defect target detection and segmentation task at different scales.

[0082] S300 employs a spatial-channel attention mechanism to process multi-scale feature maps, enhancing their feature information and fully activating the target areas corresponding to pavement defects in the multi-scale feature maps.

[0083] This embodiment uses a feature information enhancement module based on a design space-channel attention mechanism to process the multi-scale feature map output in step S200, specifically as follows: Figure 4 As shown.

[0084] The spatial-channel attention mechanism includes a first branch performing spatial max pooling and spatial average pooling on the multi-scale feature maps, and a second branch performing channel max pooling and channel average pooling on the multi-scale feature maps. Let the initial feature map of the k-th layer output by the backbone network be... The first branch performs spatial attention enhancement on the initial feature map using the following formula, resulting in the first feature map F. ks :

[0085]

[0086] Where σ is the Sigmoid function, Cat represents concatenation of the first dimension, and MLP is a multilayer perceptron composed of linear layers. and These are the results of spatial max pooling and spatial average pooling applied to the original feature map, respectively.

[0087] in,

[0088] The second branch performs channel attention enhancement on the multi-scale feature map using the following formula to obtain the second feature map F. kc:

[0089]

[0090] Where Conv5 represents a 5×5 convolution operation. and These are the results of channel-dimensional max pooling and channel-dimensional average pooling, respectively, applied to the original feature map.

[0091] in,

[0092] Finally, the first feature map F ks Second feature map F kc The fusion yields the multi-scale feature map F output in this step. kout :

[0093] F kout =Relu(GN(Conv3(F) kc +F ks ))),

[0094] In the formula, ReLU represents a nonlinear activation function, GN represents group regularization, and Conv3 represents a 3×3 convolution operation.

[0095] S400: The multi-scale feature map output in step S300 is fused using a path aggregation pyramid network with transposed convolutional upsampling. The output features of the path aggregation pyramid network are further enhanced through an adaptive enhancement mechanism, specifically including the following sub-steps:

[0096] S401. By performing a 1×1 convolution operation, the channels of the multi-scale feature map output by S300 are adjusted to be uniformly 256, resulting in the first-level feature map F. i .

[0097] S402. Utilize a path aggregation pyramid network with transposed convolution upsampling to process the first-level feature map F. i Feature fusion is performed to obtain a secondary feature map F. i '.

[0098] S403, convert the second-level feature maps of each layer of the multi-scale feature map F i 'First-level feature map F before feature fusion with its features' i Secondary feature maps F of adjacent layers i+1 '、F i-1 The enhanced feature map F is obtained by adaptive weighting using the following formula. i ',

[0099] F i "=Conv3(ω i F i +ω i1 Fi-1 ′+ω i2 F i ′+ω i3 F i+1 ′),

[0100] Where Conv3 represents a 3×3 convolution, ω i ω i1 ω i2 ω i3 These are weighted parameters, all of which are learnable parameters.

[0101] This embodiment introduces transposed convolution to achieve parameterized upsampling. The optimal upsampling method is obtained through network learning, minimizing the loss of semantic information during feature propagation. The formula for calculating the output size of the transposed convolution is:

[0102] H out =(H in -1)×S-2×P+K+OP,

[0103] In the formula H in and H out Here, S represents the multi-scale feature map sizes of the input and output in the transposed convolution, P represents the stride of the transposed convolution, K represents the edge padding parameter, and K represents the kernel size. The OP parameter is used to address the issue of non-unique output sizes. In this embodiment, to ensure that the transposed convolution module performs a double upsampling, the stride S is set to 2, the edge padding parameter P is set to 1, the kernel size K is set to 3, and the OP parameter is set to 1.

[0104] like Figure 5 As shown, in this embodiment, the channels of the multi-scale feature map are first uniformly adjusted to 256 through a 1×1 convolution operation to obtain the first feature map F. i Then, the path aggregation pyramid network with added transposed convolution is used to process the first feature map F. i Feature fusion is performed to obtain a secondary feature map F. i Considering that features from different layers may contain different feature information of the same target, this embodiment designs an adaptive weighted fusion enhancement for the secondary feature map F. i Further processing ensures that each feature map layer fully contains the discriminative features required for target identification. Specifically, each layer's second-level feature map F... i 'The first-level feature map F before fusion' i Secondary feature maps F of adjacent layers i+1 '、F i-1 Adaptive weighting is performed to obtain the enhanced feature map. Figure 5 The feature map expressions for the four scales are as follows:

[0105]

[0106] F'1', F'2', F'3', and F'4' are the enhanced feature maps output from the four scale feature maps. Conv3 represents a 3×3 convolution. All weighting parameters ω are learnable parameters, defined during the construction of the path aggregation pyramid network model. These parameters are trained and optimized together during the model training iterations. The final output feature maps in this step all have 256 channels.

[0107] S500. Input the enhanced feature map output from step S400 into the instance segmentation model for model training, such as Mask R-CNN. This type of instance segmentation model has three detection heads, which perform three tasks: image classification, object detection, and instance segmentation. The classification head, detection head, and segmentation head of the instance segmentation model are used to predict the type, spatial location, and segmentation mask of road defects, respectively. The classification loss, bounding box regression loss, and mask loss are calculated with the ground truth values ​​to optimize the parameters of the instance segmentation model.

[0108] In step S500 of this embodiment, the classification, detection, and segmentation detection head output of the instance segmentation model include category prediction, target bounding box prediction, and target mask prediction, which correspond to the type, spatial location, and segmentation mask prediction of road surface defects, respectively. The classification loss, bounding box regression loss, and mask loss of the instance segmentation model output prediction are weighted and calculated using the following formula to obtain the total loss:

[0109] L=αL cls +βL box +γL mask ,

[0110] Where L is the total loss, L cls For classification loss, L box For bounding box loss, L mask Let α be the mask loss, and β be the weights of the classification loss, bounding box regression loss, and mask loss, respectively.

[0111] Classification loss L cls and mask loss L mask Cross-entropy loss is used, and the specific calculation formula is as follows:

[0112]

[0113] Where CE is L cls and L mask The loss value, y i This represents the type of pavement distress or the true value of the segmentation mask. This represents the model's prediction results for the types or segmentation masks of road surface defects.

[0114] Bounding box loss L boxThe SmoothL1 loss is used, and the specific calculation formula is as follows:

[0115]

[0116] After calculating the total loss of the instance segmentation model prediction results, the Adam optimizer is used to iteratively optimize the model parameters to reduce model loss and improve accuracy. If the number of iterations has not been reached, steps S200 to S500 are repeated. When the number of iterations reaches the set number of iterations, the iteration is stopped, and the optimized model is obtained as the asphalt pavement distress detection model.

[0117] S600. The optimized instance segmentation model is used to predict the multiple road surface images collected. The predicted road surface images are processed according to steps S200 to S400 and then input into the trained instance segmentation model. The prediction results of each image are then stitched together to obtain the segmentation mask of the entire road surface. Adaptive morphological operations are used to process the mask matrix of each type of defect to finally obtain the defect detection results of the entire road surface.

[0118] After training the model and obtaining its predictions for a single image, post-processing is performed on the results. Experiments have shown that simply identifying individual images and then stitching the results together can lead to "disconnections" in some cross-image disease patterns, meaning the same disease is identified as multiple diseases. To avoid this, this embodiment employs adaptive morphological operations to process the mask matrix for each disease type to accurately identify the disease, and finally outputs information such as the type of disease, the quantity of each type of disease, its location, and its area.

[0119] Specifically, such as Figure 6 As shown, the trained model is used to predict the images, resulting in C segmentation mask binary matrices for each image (C being the number of road surface defect types). These C matrices are then concatenated according to the image stitching order to obtain the segmentation mask for the entire road surface. To avoid identifying defects across images as multiple targets, morphological closing operations are used to process the mask matrix for each defect type, connecting regions belonging to the same defect to form a connected region, thus avoiding duplicate detection or identifying a single defect as multiple defects.

[0120] The above operations yield C segmentation mask matrices for the entire road segment, each containing information on one type of defect. A depth-first search (DFS) algorithm is then used to traverse the mask matrices, obtaining the number of connected components in each matrix and the quantity, area, and location of each defect on each road. This information is then recorded in the detection results. The predicted result after stitching together the actual image is shown below. Figure 7 As shown.

[0121] This embodiment also discloses a system employing the above-mentioned intelligent pavement distress detection method, including an image acquisition module, a multi-level feature extraction module, a feature information enhancement module, a feature information fusion module, and a pavement distress prediction module.

[0122] The image acquisition module collects image data containing road surface defects, uses the LabelMe annotation tool to label the image data containing road surface defects, constructs a fully supervised dataset, and preprocesses the acquired image data through adaptive histogram equalization. The image brightness in the dataset is obtained using the following formula.

[0123]

[0124] In the formula, (x,y) represents the pixel position in the image. The input image is sliced ​​into three channels, where W and H represent the height and width of the input image, respectively, and Bright represents the image brightness, a value between 0 and 255. If the image brightness is lower than a set brightness threshold (80 in this embodiment), the pixel values ​​of the corresponding image are adjusted using the following formula:

[0125]

[0126] In the formula p i,j Represents the pixel values ​​in the original image. The adjusted pixel value is β, which is the offset factor, and α takes the value as follows:

[0127]

[0128] After performing the aforementioned preliminary processing on the raw image data, the image acquisition module further processes the images using an adaptive histogram equalization method to enhance contrast. First, the images in the dataset are converted from RGB to HSV format, with histogram equalization applied to the V channel to improve overall image contrast. To prevent the introduction of excessive noise, a contrast limit is added during histogram processing; simultaneously, bilinear interpolation is used to ensure image continuity and avoid overly bright or dark areas in the processed image. Finally, the images are converted back to RGB format, resulting in images with sufficient brightness and higher contrast.

[0129] The multi-level feature extraction module internally establishes a multi-level feature extraction network to extract global features from the image data preprocessed by the image acquisition module and output multi-scale feature maps.

[0130] This embodiment establishes a multi-level feature extraction network based on the Swin Transformer module, extracting global features from asphalt pavement images and outputting multi-scale feature maps. The basic unit of the feature extraction network consists of two connected Swin Transformer blocks, which are then stacked to construct four sub-modules. The multi-scale feature maps output by the four modules have sizes of 200×200×96, 100×100×192, 50×50×384, and 25×25×768, respectively. The shallow module outputs feature maps with higher resolution and fewer channels, preserving more image detail information; the deep module outputs feature maps with higher channel dimensions and stronger semantic information. In subsequent parts of the model, the feature maps at the four scales will be fused to adapt to the pavement defect target detection and segmentation task at different scales.

[0131] The feature information enhancement module uses a spatial-channel attention mechanism to process the multi-scale feature maps output by the multi-level feature extraction module, thereby enhancing their feature information and fully activating the target region.

[0132] The spatial-channel attention mechanism includes a first branch performing spatial max pooling and spatial average pooling on the multi-scale feature maps, and a second branch performing channel max pooling and channel average pooling on the multi-scale feature maps. Let the initial feature map of the k-th layer output by the backbone network be... The first branch performs spatial attention enhancement on the initial feature map using the following formula, resulting in the first feature map F. ks :

[0133]

[0134] Where σ is the Sigmoid function, Cat represents concatenation of the first dimension, and MLP is a multilayer perceptron composed of linear layers. and These are the results of spatial max pooling and spatial average pooling applied to the original feature map, respectively.

[0135]

[0136] The second branch performs channel attention enhancement on the multi-scale feature map using the following formula to obtain the second feature map F. kc :

[0137]

[0138] Where Conv5 represents a 5×5 convolution operation. and These are the results of channel-dimensional max pooling and channel-dimensional average pooling, respectively, applied to the original feature map. This represents the real-field matrix after pooling the feature map.

[0139]

[0140] Finally, the first feature map F ks Second feature map F kc The fusion yields the multi-scale feature map F output in this step. kout :

[0141] F kout =Relu(GN(Conv3(F) kc +F ks ))),

[0142] In the formula, ReLU represents a nonlinear activation function, GN represents group regularization, and Conv3 represents a 3×3 convolution operation.

[0143] The feature information fusion module uses a path aggregation pyramid network with transposed convolution upsampling to perform feature fusion on the multi-scale feature maps output by the feature information enhancement module, and further enhances the output features of the pyramid network through an adaptive enhancement mechanism.

[0144] First, the channels of the multi-scale feature map output by S300 are adjusted to be uniformly 256 through a 1×1 convolution operation to obtain the first-level feature map Fi; then, a path aggregation pyramid network using transposed convolution upsampling is used to process the first-level feature map F. i Feature fusion is performed to obtain a secondary feature map F. i ';The multi-scale feature map is divided into two layers of secondary feature maps F. i 'First-level feature map F before feature fusion with its features' i Secondary feature maps F of adjacent layers i+1 '、F i-1 The enhanced feature map F is obtained by adaptive weighting using the following formula. i ,

[0145] F″ i =Conv3(ω i F i +ω i1 F i-1 ′+ω i2 F′ i +ω i3 F i+1 ′),

[0146] Where Conv3 represents a 3×3 convolution, ω i ω i1 ω i2 ω i3 These are weighted parameters, all of which are learnable parameters.

[0147] The road surface distress prediction module incorporates an instance segmentation model. It trains the model by inputting enhanced multi-scale feature maps (from the feature information fusion module) into the classification, detection, and segmentation heads. This predicts the type, spatial location, and segmentation mask of road surface distresses. Classification loss, bounding box regression loss, and mask loss are calculated separately against ground truth values ​​to optimize the instance segmentation model parameters. The optimized model is then used to predict distresses from multiple collected road surface images. The prediction results for each image are concatenated to obtain the segmentation mask for the entire road surface. Adaptive morphological operations are applied to process the mask matrix for each distress type, ultimately yielding the overall distress detection results for the entire road surface. The enhanced multi-scale feature maps are then input into the instance segmentation model for training. The instance segmentation model predicts the type, spatial location, and segmentation mask of road surface distresses through image classification, object detection, and instance segmentation, respectively.

[0148] During model training, the enhanced feature map output from the feature information fusion module is input into the detection heads of image classification, object detection, and instance segmentation for model training. This predicts the type, spatial location, and segmentation mask of road surface defects, respectively. The classification loss, bounding box regression loss, and mask loss are calculated against the ground truth values ​​to optimize the instance segmentation model parameters. The instance segmentation model's classification, detection, and segmentation head outputs include category prediction, target bounding box prediction, and target mask prediction, corresponding to the prediction of road surface defect type, spatial location, and segmentation mask, respectively. The total loss is obtained by weighting the instance segmentation model's output predictions against the ground truth values ​​using the following formula:

[0149] L=αL cls +βL box +γL mask ,

[0150] Where L is the total loss, L cls For classification loss, L box For bounding box loss, L mask Let α be the mask loss, and β be the weights of the classification loss, bounding box regression loss, and mask loss, respectively.

[0151] Classification loss L cls and mask loss L mask Cross-entropy loss is used, and the specific calculation formula is as follows:

[0152]

[0153] Where CE is L cls and L mask The loss value, y i This represents the type of pavement distress or the true value of the segmentation mask. This represents the model's prediction results for the types or segmentation masks of road surface defects.

[0154] Bounding box loss L box The SmoothL1 loss is used, and the specific calculation formula is as follows:

[0155]

[0156] After calculating the total loss of the instance segmentation model prediction results, the Adam optimizer is used to iteratively optimize the model parameters. When the number of iterations reaches the set number of iterations, the iteration stops, and the optimized model is used as the asphalt pavement distress detection model.

[0157] Multiple road surface images to be predicted are acquired by the image acquisition module and preprocessed. After feature extraction, enhancement, and fusion by a multi-level feature extraction module, feature information enhancement module, and feature information fusion module, the resulting images are input into the pre-trained segmentation model of the road surface defect prediction module. The prediction results for each image are then concatenated to obtain the segmentation mask for the entire road surface. For each image, C binary matrix segments are obtained (C being the number of road surface defect types). These C matrices are concatenated according to the image concatenation order to obtain the segmentation mask for the entire road surface. To avoid identifying defects across images as multiple targets, morphological closing operations are used to process the mask matrix for each defect type, connecting regions belonging to the same defect to form a connected region, thus avoiding duplicate detection or identifying a single defect as multiple defects.

[0158] The above operations yield C segmentation mask matrices for the entire road segment, each containing information on a type of defect. The depth-first search algorithm (DFS) is then used to traverse the mask matrices to obtain the number of connected components in each matrix, thus obtaining information such as the quantity, area, and location of each defect in each road. Finally, this information is recorded in the detection results and the prediction results are output through the road surface defect prediction module.

[0159] In other application scenarios, this embodiment is also manifested as a computer-readable storage medium that stores a road surface defect intelligent detection method based on instance segmentation as described above in this embodiment, which is programmed or configured to be applied to a server.

[0160] The computer-readable storage medium can be an internal storage unit of a hardware or software device, such as the hard drive or memory of the controller, or an external storage device of the controller, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the readable storage medium may include both internal storage units and external storage devices of the controller. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0161] In this document, the terms "upper," "lower," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used for the clarity of expressing the technical solution and for the convenience of description, and therefore should not be construed as limiting the present invention.

[0162] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0163] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent detection of pavement defects based on instance segmentation, characterized in that: Includes the following steps: S100. Use a line scan camera to collect image data containing road surface defects and manually label the ground truth values ​​of road surface defects in the image data. Preprocess the collected image data through adaptive histogram equalization. S200. Establish a multi-level feature extraction network to extract global features from image data and output multi-scale feature maps; S300 employs a spatial-channel attention mechanism to process multi-scale feature maps, enhancing their feature information. This ensures that the target regions corresponding to pavement distress features in the multi-scale feature map are fully activated. S400. A path aggregation pyramid network is used with transposed convolution upsampling to perform feature fusion on multi-scale feature maps. The output features of the pyramid network are further enhanced through an adaptive enhancement mechanism, including the following sub-steps: S401. By performing a 1×1 convolution operation, the channels of the multi-scale feature map output by S300 are adjusted to be uniformly 256, resulting in the first-level feature map F. i ; S402. Utilize a path aggregation pyramid network with transposed convolution upsampling to process the first-level feature map F. i Feature fusion is performed to obtain a secondary feature map F. i '; S403, convert the second-level feature maps of each layer of the multi-scale feature map F i 'First-level feature map F before feature fusion with its features' i Two secondary feature maps F in adjacent layers i+1 '、F i-1 The enhanced feature map F is obtained by adaptive weighting using the following formula. i , F i ″=Conv3(ω i F i +oh i1 F i-1 ′+ω i2 F i ′+ωi i3 F i+1 ′), Where Conv3 represents a 3×3 convolution, ω i ω i1 ω i2 ω i3 These are weighted parameters, all of which are learnable parameters defined for building the path aggregation pyramid network; S500: Input the enhanced multi-scale feature map into the instance segmentation model for model training. Predict the type, spatial location and segmentation mask of road surface defects through image classification, object detection and instance segmentation of the instance segmentation model, respectively. Calculate the classification loss, bounding box regression loss and mask loss with the manually labeled ground truth values ​​in S100, and optimize the parameters of the instance segmentation model. S600. The trained instance segmentation model is used to predict the multiple road surface images collected. After obtaining the prediction results of each image, the images are stitched together in the order of image stitching to obtain the segmentation mask of the entire road surface. Adaptive morphological operations are used to process the mask matrix of each type of defect. Finally, the regions belonging to the same defect are connected to form a connected region to obtain the defect detection results of the entire road surface.

2. The intelligent pavement distress detection method based on instance segmentation according to claim 1, characterized in that: Step S100 includes the following sub-steps: S101. Use the labelme annotation tool to annotate image data containing road surface defects and construct a fully supervised dataset; S102. Obtain the image brightness in the fully supervised dataset. If the image brightness is lower than the set brightness threshold, adjust the pixel values ​​of the corresponding image using the following formula. In the formula p i,j Represents the pixel values ​​in the original image. The adjusted pixel values ​​are represented as follows: β is the offset factor that increases the overall base brightness of the image, and α is the scaling factor that scales the overall pixel values ​​of the original image. Bright represents the image brightness, which is a value between 0 and 255, and B' is the set brightness threshold. S103. Convert the images in the fully supervised dataset from RGB format to HSV format. Histogram equalization is performed on the V channel to improve the overall contrast of the image. When processing the histogram, contrast limiting and bilinear interpolation are used to ensure the continuity of the image. Finally, the HSV format image is converted back to RGB format.

3. The intelligent pavement distress detection method based on instance segmentation according to claim 1, characterized in that: In step S200, the multi-level feature extraction network is built based on the Swin Transformer module. Its basic unit includes two connected Swin Transformer blocks. The basic unit is stacked to build several sub-modules, and then all the sub-modules are spliced ​​together to obtain the multi-level feature extraction network.

4. The intelligent pavement distress detection method based on instance segmentation according to claim 1, characterized in that: In step S300, the spatial-channel attention mechanism includes a first branch and a second branch, wherein the first branch performs spatial attention enhancement on the multi-scale feature map using the following formula to obtain the first feature map F. ks : in This represents the initial feature map of the k-th layer output by a multi-level feature extraction network, where σ is the sigmoid function, Cat indicates concatenation of the first dimension, and MLP is a multilayer perceptron composed of linear layers. and respectively for F k The results of performing spatial max pooling and spatial average pooling; The second branch performs channel attention enhancement on the multi-scale feature map using the following formula to obtain the second feature map F. kc : Where Conv5 represents a 5×5 convolution operation. and respectively for F k The results of performing channel-dimensional max pooling and channel-dimensional average pooling; The first feature map F is obtained by the following formula. ks Second feature map F kc The fusion yields the multi-scale feature map F output in this step. kout : F kout =Relu(GN(Conv3(F kc +F ks ))), In the formula, ReLU represents a nonlinear activation function, GN represents group regularization, and Conv3 represents a 3×3 convolution operation.

5. The intelligent pavement distress detection method based on instance segmentation according to claim 1, characterized in that: In step S403, the transposed convolution of the pyramid network aggregation performs a 2x upsampling on the multi-scale feature map.

6. The intelligent pavement distress detection method based on instance segmentation according to claim 1, characterized in that: In step S500, the classification, detection, and segmentation detection head output of the instance segmentation model includes category prediction, target bounding box prediction, and target mask prediction, which correspond to the prediction of the type, spatial location, and segmentation mask of the road surface defects, respectively. The classification loss, bounding box regression loss, and mask loss of the instance segmentation model output prediction are weighted and calculated using the following formula to obtain the total loss: L=αL cls +βL box +γL mask , Where L is the total loss, L cls For classification loss, L box For bounding box loss, L mask Let α be the masking loss, and β be the weights of the classification loss, bounding box regression loss, and masking loss, respectively. Classification loss L cls and mask loss L mask Cross-entropy loss is used, and bounding box loss L is applied. box SmoothL1 loss is used; After calculating the total loss of the instance segmentation model prediction results, the Adam optimizer is used to iteratively optimize the model parameters.

7. The intelligent pavement distress detection method based on instance segmentation according to claim 1, characterized in that: In step S600, the trained instance segmentation model is used to predict the road surface image. Each image yields C binary matrices of segmentation masks, where C is the number of road surface defects in the image. The C binary matrices of segmentation masks are concatenated according to the image stitching order to obtain the segmentation mask of the entire road surface. Adaptive morphological operation closing is used to process the mask matrix of each defect category, connecting regions belonging to the same defect to form a connected region. A depth-first search algorithm is used to traverse the binary matrix of the segmentation mask of the entire road surface to obtain the number of connected regions in each matrix, that is, to obtain the quantity, area, and location information of each defect in the entire road surface, which is finally recorded in the detection result.

8. A pavement distress intelligent detection system based on instance segmentation, characterized in that, Includes the following modules: The image acquisition module collects image data containing road surface defects and preprocesses the collected image data through adaptive histogram equalization. The multi-level feature extraction module internally establishes a multi-level feature extraction network to extract global features from the image data preprocessed by the image acquisition module and output multi-scale feature maps. The feature information enhancement module uses a spatial-channel attention mechanism to process the multi-scale feature maps output by the multi-level feature extraction module, thereby enhancing their feature information and fully activating the target region. The feature information fusion module uses a path aggregation pyramid network with transposed convolution upsampling to fuse the multi-scale feature maps output by the feature information enhancement module, and further enhances the output features of the pyramid network through an adaptive enhancement mechanism. The road surface distress prediction module incorporates an instance segmentation model. Enhanced multi-scale feature maps are input into the instance segmentation model for training. The model predicts the type, spatial location, and segmentation mask of road surface distress through image classification, object detection, and instance segmentation, respectively. Classification loss, bounding box regression loss, and mask loss are calculated with ground truth values ​​to optimize the instance segmentation model parameters. The optimized instance segmentation model is then used to detect and predict multiple collected road surface images. The prediction results for each image are then concatenated to obtain the segmentation mask for the entire road surface. Adaptive morphological operations are used to process the mask matrix for each distress type, ultimately yielding the distress detection results for the entire road surface.

Images