Road disease image semantic segmentation method based on region loss

Abstract

The application is a road disease image semantic segmentation method based on region loss. The method constructs an RLMT-Net model, adopts a feature pyramid network (FPN) for feature extraction, combines a region proposal network (RPN) and a RoIAlign layer, and processes instance segmentation in a multi-level, coarse-to-fine manner. Through the Transformer system, multi-level feature point information fusion and explicit modeling of different granularities are realized, effectively processing segmentation ambiguity. The RLMT-Net model can process unrelated nodes across the four-tuple tree level in parallel, realizing efficient instance segmentation. The F1 score and mIoU on the data set are 0.7957 and 0.8152 respectively, which can greatly improve the accuracy of automatic crack recognition for road disease recognition, and provide a new idea for the inspection of automatic road disease.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road disease treatment, specifically to a road disease image semantic segmentation method based on region loss. The method is suitable for automatic detection and analysis of road diseases and has wide application prospects in intelligent transportation systems, road maintenance management, urban planning and other fields. BACKGROUND

[0002] When the unmanned system collects road images and analyzes the reverse disease, directly applying semantic segmentation technology to identify the overall image is not suitable. First, the traditional semantic segmentation technology has relatively high requirements for training data. In the preparation of the data set, the time required for semantic segmentation is about fifteen times that of target detection. Second, in the road images taken by the unmanned aerial vehicle, there are a large number of factors that may cause the semantic segmentation identification to fail: background complexity, spot noise interference, low signal-to-noise ratio, low contrast of cracks and background, and discontinuity of cracks, etc. Other interference objects on the road surface during inspection: lane lines, vehicles in motion, lane line damage, pedestrians, roadside facilities, etc. Third, the average detection speed of the target detection technology using semantic segmentation alone is relatively low and the detection efficiency is low. Finally, in the artificial auditing stage, the target detection algorithm can quickly point out the disease location, while relying solely on the semantic segmentation result for disease positioning is relatively difficult.

[0003] Therefore, the method of the present application first performs global target detection to lock the disease area graph, and then performs semantic segmentation on these areas to accurately quantify the size of the disease, which is more in line with the needs of engineering practice. SUMMARY

[0004] The purpose of the present application is to provide a road disease image semantic segmentation method based on region loss, which can improve the semantic segmentation recognition rate in road detection and improve the road detection accuracy and speed.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0006] A road disease image semantic segmentation method based on region loss, the segmentation method comprising the following steps:

[0007] Obtain image data containing road diseases, use target detection to obtain disease area graph, and the disease area graph has the position information of the disease;

[0008] Construct an RLMT-Net model, the RLMT-Net model comprising a CNN backbone network, a feature pyramid network FPN, a quadtree, an incoherence detector, a sequence encoder, a pixel decoder and an incoherence query sequence module;

[0009] The feature pyramid network FPN has a four-level cascade structure, sequentially recorded as P2, P3, P4 and P5 from bottom to top;

[0010] The CNN backbone network is used to extract disease features, and the output of the CNN backbone network is connected to the bottom layer P2 of the feature pyramid network FPN, and the output of the CNN backbone network is connected to the Box Head operation, and the output of the Box Head operation is used as the input of each layer of the feature pyramid network FPN.

[0011] The input image is sequentially subjected to the CNN backbone network to extract the disease features of the image, and the FPN generates multi-level feature maps for downstream processing;

[0012] The outputs of the P2 layer, the P3 layer and the P4 layer of the feature pyramid network FPN are subjected to the region proposal network (RPN) and the ROIAlign layer to obtain the region of interest RoI, and the three regions of interest RoI of different scales construct a RoI feature pyramid.

[0013] The region of interest RoI obtained by processing the P4 layer of the feature pyramid network FPN through the region proposal network (RPN) and the ROIAlign layer is subjected to coarse mask head processing to obtain a coarse mask of the ROI region of interest with a scale of 28*28.

[0014] The coarse mask is input into the non-continuous detector, which includes two levels of up-sampling operations and outputs a non-continuous mask, and the non-continuous detector is used to refine the mask prediction of the non-continuous node of the quadtree.

[0015] Starting from the detection point in the highest-level feature map in the RoI feature pyramid as a root node, a multi-level quadtree is constructed, the non-continuous mask is given to the quadtree, and the quadtree is used to subdivide the non-coherent regions of each scale of the region of interest and output a non-continuous query sequence.

[0016] The non-continuous query sequence module includes a plurality of node encoders, and the output of the non-continuous query sequence module is connected to a sequence encoder, and after processing by the sequence encoder, an encoded sequence is obtained, and then the encoded sequence is input into a pixel decoder to obtain a sequential prediction result.

[0017] The RLMT-Net model is used for semantic segmentation of image data containing road diseases, and outputs disease images with segmentation labels.

[0018] Further, the output of the quadtree is a non-continuous query sequence, which has a plurality of non-continuous nodes, and each non-continuous node of the quadtree is encoded using a node encoder.

[0019] The node encoder is used for encoding each quadtree node, including fusion embedding, position embedding, the node of the quadtree has fine-grained, coarse-grained and background information, the three are fused by using the fusion embedding, and the position embedding is carried out, the spatial distance and the relationship between the nodes in each RoI are encoded by using the position embedding, the important local dependence and correlation are captured, and the encoding of the node is realized.

[0020] Further, the sequence encoder is used for integrating the nodes of each level of the quadtree, realizing global and cross-scale spatial reasoning, having three sequence encoders connected in series, the sequence encoder including a multi-head self-attention mechanism and a fully connected feedforward network, to process the incoherent point sequence, the input of the first sequence encoder being the output result of the P5 layer of the FPN and the output of the incoherent query sequence module, the other two sequence encoders being connected in series after the first sequence encoder, and the output result of the P5 layer of the FPN also being input into the other two sequence encoders separately.

[0021] The pixel decoder adopts a double-layer MLP (Multi-Layer Perceptron) form to realize decoding of the node output, and generate the final mask label.

[0022] Compared with the prior art, the beneficial effects of the present application are:

[0023] The present application can automatically extract road disease quantitative data on the basis of the Mask R-CNN model, and can meet the needs of efficient crack detection of actual engineering requirements. The RLMT-Net (RegionLoss Mask Transformer) model is compared with other semantic segmentation images in the aspect of road disease segmentation, which shows the superiority of the RLMT-Net model of the present application in maintaining stable recognition ability under various environments and the advantage of the model in road disease semantic segmentation.

[0024] The RLMT-NET model of the present application can learn better crack features and effectively prevent the segmentation result from being affected by interference factors in the image, greatly improving the problems of low recognition accuracy and poor robustness in the existing research, and being beneficial to practical application. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0026] Figure 1is a four-tree structure schematic diagram of a road disease image semantic segmentation method based on regional loss according to an embodiment of the application.

[0027] Figure 2 is a structure schematic diagram of an RLMT-Net model of a road disease image semantic segmentation method based on regional loss according to an embodiment of the application.

[0028] Figure 3 is a schematic diagram of different levels of loss regions of a four-tree.

[0029] Figure 4 is a graph showing the change of loss with the number of iterations in the training process of a road disease image semantic segmentation method based on regional loss according to an embodiment of the application.

[0030] Figure 5 is a comparison graph of test results on different public data sets according to the method of the application, (a) original image, (b) ground truth, (c) RLMT-Net, (d) SegNet, (e) U-Net, (f) FCN8s, and (g) DeepLabv3+.

[0031] Figure 6 is a comparison graph of generalization experiment results of a road disease image semantic segmentation method based on regional loss according to an embodiment of the application. DETAILED DESCRIPTION

[0032] To further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can explain the operating principle of the embodiments in conjunction with the related description of the specification. With reference to these contents, those skilled in the art should understand other possible embodiments and advantages of the present application. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0033] The method of the present application can accurately detect and segment cracks and diseases on the road under the condition of complex background, illumination change and occlusion. Road disease (Road Pavement Distress) refers to various defects, damages or destructions on the road surface, such as cracks, potholes, rutting, etc. These diseases not only affect the appearance of the road, but also may have a negative impact on driving safety and driving comfort, and even cause traffic accidents.

[0034] MaskR-CNN is an instance segmentation algorithm, which can be used for "target detection", "target instance segmentation" and "target key point detection". As an advanced instance segmentation technology, it has shown its excellent performance in many research fields, especially in dealing with the same defects with adhesion or overlap. Compared with the traditional semantic segmentation method, the algorithm can accurately identify and count each defect instance, rather than regarding a group of defects as a single whole, so as to realize the fine processing of individual defects.

[0035] The road disease image semantic segmentation method based on regional loss comprises the following steps: obtaining image data containing road diseases, using target detection to obtain a disease area graph, and the disease area graph has position information of the diseases;

[0036] An RLMT-Net model is constructed, and the RLMT-Net model comprises a CNN backbone network, a feature pyramid network FPN, a quad tree, an incoherent detector, a sequence encoder, a pixel decoder and an incoherent query sequence module.

[0037] The feature pyramid network FPN has a four-layer cascade structure, and is sequentially recorded as P2, P3, P4 and P5 from bottom to top.

[0038] The CNN backbone network is used for extracting disease features, and the output of the CNN backbone network is connected to the bottom layer P2 of the feature pyramid network FPN at the same time, and the output of the CNN backbone network is connected to a head box (BoxHead) operation, and the output of the head box (BoxHead) operation is used as the input of each layer of the feature pyramid network FPN.

[0039] The input image is sequentially subjected to the CNN backbone network to extract the disease features of the image, and the FPN generates multi-level feature mapping for downstream processing, and the outputs of the P2, P3, P4 and P5 layers are mainly used for downstream processing.

[0040] The outputs of the P2 layer, the P3 layer and the P4 layer of the feature pyramid network FPN are subjected to a region proposal network (RPN) and a ROIAlign layer to obtain a region of interest RoI, and the three different scale regions of interest RoI construct a RoI feature pyramid to effectively retain the spatial information in the RoI region and provide more accurate feature representation. i i-1 i-2 When i=4, the RoI features of three different levels {P i i-1 i-2} of the FPN are extracted, and the RoI feature pyramid with a size of {28, 56, 112} is output.

[0041] The P4 layer of the feature pyramid network FPN is processed by a region proposal network (RPN) and a ROI Align layer to obtain a coarse mask of a region of interest (RoI) with a scale of 28*28; and the RPN is used to extract a bounding box of a feature map and perform mapping.

[0042] The coarse mask is input into an incoherent detector, which includes two-level up-sampling operations (changing the scale from 28*28 to 112*112) and outputs an incoherent mask; and the incoherent mask is given to a quadtree for further decomposition of incoherent quadrant points.

[0043] A multi-level hierarchical quadtree is constructed using a detection point in the highest-level feature map in the RoI feature pyramid as a root node, the quadtree is used to subdivide incoherent regions (incoherent regions, i.e., loss regions) of each scale RoI detected and output an incoherent query sequence.

[0044] The incoherent query sequence module includes a plurality of node encoders, and the output of the incoherent query sequence module is connected to a sequence encoder, which is processed to obtain an encoded sequence, and then the encoded sequence is input into a pixel decoder to obtain a sequential prediction result.

[0045] The incoherent detector is used to refine the mask prediction of the incoherent nodes of the quadtree.

[0046] The output of the quadtree is an incoherent query sequence, i.e., a plurality of incoherent nodes, which can improve the recognition efficiency and represent the discontinuity in the image, thereby more effectively compressing the image without losing details.

[0047] Since the sequence encoder uses a Transformer model, which has the feature of permutation invariance, the order of the sequence is not important, and the node encoder can be used to encode each quadtree node without considering the order of the nodes.

[0048] The node encoder is used to encode each quadtree node, including fusion embedding and position embedding, the quadtree node has fine-grained, coarse-grained and background information, the three are fused by fusion embedding, and they are restored to the original feature dimension by a fully connected layer for position embedding to realize the encoding of the node. The function of the node encoder is to obtain fine-grained and coarse-grained features and background features of corresponding positions and levels using the feature pyramid network FPN, to perform relative position encoding of the spatial distance and relationship between nodes in each RoI using position embedding, and to capture important local dependencies and correlations, wherein the background feature considers the node features of the surrounding context and can capture local detail information.

[0049] The sequence encoder is used to integrate the nodes of each level of the quadtree, realize global and cross-scale spatial reasoning, and the multi-head self-attention mechanism is the feature of the transform, which can improve the modeling ability of the model to sequence information, enhance the representation ability, robustness and generalization ability of the model, and at the same time can improve the efficiency through parallel computing, better resist the ambiguity in language.

[0050] There are three sequence encoders in series, and the sequence encoder is composed of a multi-head self-attention mechanism and a fully connected feedforward network to process the incoherent query sequence. The input of the first sequence encoder is the output result of the P5 layer of the FPN and the output of the incoherent query sequence module, and the other two sequence encoders are connected in turn after the first sequence encoder, and the output result of the P5 layer of the FPN is also input into the other two sequence encoders separately. The size of the output result of the P5 layer of the FPN is 14*14.

[0051] The pixel decoder adopts a double-layer MLP (Multilayer Perceptron) form to realize the decoding of the node output and generate the final mask label.

[0052] Figure 1 The structure diagram of the quadtree is shown in the figure, Figure 1 The right figure is the structure of the quadtree Q-tree, the root node is 8*8 pixels, the child node is 4*4 pixels, the third level child node is 2*2 pixels, and the fourth level child node is 1*1 pixel. Using the quadtree Q-tree structure can identify the loss area, and each node is divided into four child nodes, and the data area is divided into four quadrants.

[0053] The quadtree is a tree data structure used for spatial indexing and multi-dimensional data. It recursively divides space into four sub-quadrants, suitable for processing points, regions and other objects in two-dimensional space. By recursively dividing the image space into four equal-sized subspaces and continuing the recursion until a certain number of layers is reached or certain conditions are met.

[0054] A quadtree is used to subdivide the detected incoherent regions, which refer to those parts of the image that may have some abnormalities, inconsistencies, or irregularities. Incoherent regions are those that are significantly different from their surroundings or may have issues. These regions can occur due to noise, object boundaries, occlusions, or other reasons. In the image segmentation task, incoherent regions can be those pixel blocks that are not correctly classified, or in object detection, they can be those parts that are misidentified or missed. At a yellow dot in a higher-level FPN feature P4 (e.g., 28x28 feature resolution), there are four quadrant dots in its adjacent lower-level FPN feature map P3 (e.g., 56x56 resolution), which have different granularities due to their different pyramid levels. Based on the detected incoherent points, a multi-level hierarchical quadtree is constructed, starting from using the detected points in the highest-level feature map as the root nodes. The highest-level feature map refers to the feature map with the lowest resolution and the most complete information generated during the image processing process. In this feature map, some key points (detected points) can be detected. These points represent the regions or objects of interest, or in simple terms, starting from a large region, the region is divided into smaller parts layer by layer until a fine enough region is obtained to process and analyze the details in the image. At the fine level, only the detected incoherent quadrant points can be further decomposed.

[0055] Then the mask prediction of refining the incoherent nodes of the quadtree using the incoherent detector is used, which is: first, make a prediction on a relatively rough image, find and correct some inaccuracies; then, pass this corrected information to the finer image part to ensure that the prediction of each layer is more and more accurate; through this step-by-step refinement method, the incoherent regions in the image can be more accurately found and marked. In the low-resolution prediction, the point labels belonging to the root layer of the quadtree are first corrected, and then these corrected point labels are propagated to the corresponding four quadrants of the adjacent finer layers through nearest neighbor interpolation.

[0056] The RLMT-Net model takes as input a road disease image containing cracks and the location of the cracks in the image, and processes instance segmentation in a multi-level, from coarse to fine manner based on hierarchical FPN. Unlike using single-level FPN features for each object, the RLMT-Net uses the sparse detected feature points in the incoherent image regions of each layer of the RoI feature pyramid as the input sequence of the Transforme system, and outputs the corresponding segmentation label for each pixel in the input image. The segmentation label can be represented by 0 and 1 respectively to represent the disease (crack) and the background.

[0057] Different scale region losses are obtained using the incoherent detector combined with the ROI pyramid.

[0058] Let the incoherent mask be denoted as M, the ROI pyramid has the finest scale at the bottom layer with scale 0, and the coarsest scale at the top layer with scale 2, and the intermediate scales are 1. The nearest neighbor down- and up-sampling are denoted as D↓ and U↑, respectively.

[0059] The region loss D0 represented by the binary mask obtained in the incoherent region at scale 0 is:

[0060]

[0061] The region loss D1 represented by the binary mask obtained in the incoherent region at scale 1 is:

[0062]

[0063] The region loss D2 represented by the binary mask obtained in the incoherent region at scale 2 is:

[0064]

[0065] where ⊕ denotes the logical "XOR" operation; O ↓ is the down-sampling by performing the logical "OR" operation within each 2x2 neighborhood, where l = 1.

[0066] If the original mask value M l-1 If at least one pixel in the up-sampled mask is different from the original mask, the pixels that are different are the incoherent pixels, and all the incoherent pixels form an incoherent region.

[0067] The following table is the experimental analysis result of an incoherent region. The percentage represents the area ratio of the incoherent region in the object bounding box. Recall Err is the ratio of all the false positive pixels for each object. Acc is the accuracy of the coarse mask prediction in the incoherent region. AP Coarse is measured by using the coarse mask prediction for the entire target region, while AP GT is measured by using the ground truth label to fill the incoherent region.

[0068] Table 1

[0069]

[0070] After detailed analysis of the incoherent regions, the results show that most of the prediction errors are concentrated in these regions, which account for about 43% of all false positive pixels, while the bounding box region accounts for only 14%. In the incoherent region, the accuracy of the coarse mask prediction is 56%, so selecting the P4 layer as the source of the coarse mask can significantly improve the final performance.

[0071] All the incoherent nodes from three levels of quad-tree have coarse and fine features that can capture more context and semantic information and more local detail features, respectively. These features are input into the Transforme system (composed of three parts: sequence encoder, pixel decoder, and incoherent query sequence module), and the incoherent sequence output by the quad-tree has a size of C x N, where N is the total number of nodes and C is the dimension of the feature channel. Since the order of the sequence is not important due to the permutation invariance of the Transformer model, the encoder used by the Transforme system is composed of two parts: node encoding and sequence encoding. The node encoder encodes each quad-tree node using the following four different information clues: 1) fine-grained feature extraction from the corresponding position and level of the FPN pyramid; 2) relative position encoding, which covers the spatial distance and relationship between nodes in each RoI, capturing important local dependencies and correlations; and 3) surrounding context node features, capturing local detail information.

[0072] For each node, as shown in Figure 2 the fine-grained features, coarse segmentation clues, and context features are first connected and fused, and the fused features are then reduced to the original feature dimension through a fully connected layer. Subsequently, the position embedding is added to the resulting feature vector.

[0073] On the quad-tree, the point labels are sequentially corrected from the root layer of the quad-tree, and then these corrected point labels are propagated to the corresponding four quadrants of the adjacent finer layer through nearest neighbor interpolation, which is effectively performed in a step-by-step manner on incoherent nodes until the finest quad-tree level is reached. The entire RLMT-Net model framework can be trained end-to-end, and the multi-task loss function of the entire RLMT-Net model is represented as follows:

[0074]

[0075] Here, represents the refinement of the loss between the segmentation labels output by the model and their true value labels. The binary cross-entropy loss is used to detect incoherent regions. represents the coarse mask prediction loss, λ {1,2,3} represents the hyperparameters {1.0, 1.0, 0.5}.

[0076] In the method of this embodiment, the parameters of each layer of the feature pyramid network FPN are as follows:

[0077] Table 2 Parameters of each layer of the feature pyramid network FPN

[0078]

[0079] The experimental environment of this embodiment is implemented based on the hardware in Table 3:

[0080] Table 3

[0081]

[0082] Dataset preparation:

[0083] The data set is selected by using a drone to shoot, and labelme is used to label the pictures of road diseases collected by the drone. The data set used for training the RLMT-Net model is different from the data set for target detection in that all cracks are classified as one class, and other pit and repair classes are retained. The data set UAV-CR for semantic segmentation, i.e., the private data set, is prepared.

[0084] In order to verify the effectiveness of the improved model, experiments are performed on other public data sets, such as: (1) Crack500, which is obtained by shooting 500 pavement cracks images with a mobile phone at Temple University; (2) CFD data set, which only contains 118 urban concrete road crack images with a resolution of about 480x320; (3) Crack675, which is obtained by shooting 600 pavement crack images with a vehicle-mounted motion camera. At the same time, the effects of the popular FCN, SegNet, Unet, DeepLabv3+ models in the field of image semantic segmentation are compared with the effect of the RLMT-Net model of the application.

[0085] Experimental process: First, qualitative analysis is performed on the private data, and the effectiveness of the improved model is compared, and then the results of other models are compared horizontally on the same data set.

[0086] During training, the stochastic gradient descent (SGD) is used with mini-batch. Data augmentation is used to avoid overfitting in model training by using flipping, shifting and rotation. To overcome overfitting, the early stopping method is used, which takes effect when the validation loss shows continuous growth for five consecutive epochs. On the private data set, the model is trained for 200 epochs, 50 epochs are used for frozen training, the learning rate is set to 0.0001, the batch size is set to 8, and the learning rate is reduced by 10 factors every 100 epochs. On the Crack500 data set, the model is trained for 200 epochs, the learning rate is set to 0.0001, the batch size is set to 8, and the learning rate is reduced by 10 factors every 50 epochs. Adam is used as the optimizer to adjust the hyperparameters, and the weight decay is 0.000001.

[0087] 80% of the data set is used for training and verification, constituting a trainvalset, and 20% is used for testset testing. In the trainvalset, the cross-validation training method is used, in which 90% of the data is used for training and 10% of the data is used for verification.

[0088] In image semantic segmentation, commonly used indicators for evaluating the performance of a model include IoU (Intersection over Union), pixel accuracy, mean accuracy, mean IoU (mIoU), F1 score, and Dice coefficient. IoU measures the degree of overlap between the model's predicted regions and the true labels; mean accuracy and mIoU are the average values of the pixel accuracy and IoU for each class, which more comprehensively evaluate the performance differences between different classes; F1 score combines precision and recall, suitable for evaluating multi-class segmentation tasks; Dice coefficient also measures the overlap between predicted regions and true regions, evaluating the accuracy and generalization ability of the model by calculating the ratio of intersection to average size.

[0089] Their calculation methods are as follows:

[0090]

[0091] Pixel accuracy measures the proportion of correctly classified pixels to the total number of pixels, denoted by P all represents the total number of pixels P true represents the number of correctly classified pixels, and pixel accuracy PixelAccuracy can be represented as:

[0092]

[0093] Mean accuracy:

[0094]

[0095] F1 score combines precision and recall and can be represented as:

[0096]

[0097] Training process:

[0098] Figure 4 The training process loss changes are shown in FIG. 1, where the y-axis represents the loss and the x-axis represents the number of iterations. The average loss of FCN, SegNet, Unet, and DeepLabv3+ models is compared, Figure 4 is the average loss on the left training set, and the right is the average loss on the validation set, where the green line represents the RLMT-NET model of the present application; the black line in the SegNet graph represents; the yellow curve in DeepLabv3+; the red line in the Unet graph; and the purple line represents FCN.

[0099] From Figure 4 , it can be seen that after 200 iterations, the UNet model and the RLMT-NET model training loss are all at a particularly low level, but the RLMT-Net model has the lowest validation loss. The training loss of the SegNet model still has a tendency to continue to decline. The DeepLabv3+ model has the highest training loss and validation loss.

[0100] The segmentation performance of the RLMT-Net model under different data sets is listed in Table 4. The RLMT-Net has the highest recognition result in Crack675, which contains some road shadow areas, but the MPA of the RLMT-Net in this data set is as high as 81.25%, indicating that the background noise area has little effect on the segmentation ability of the model. Thus, it is verified that the RLMT-Net proposed in this paper can be used in actual engineering and has strong generalization and migration ability under various road conditions. However, the recognition accuracy of complex road disease morphology needs to be improved, rather than the generalization ability of the model under complex road background. The continuity of road cracks needs to be improved, and thus the feature analysis ability of road diseases can be improved to provide a basis for accurately calculating the damage degree of road diseases.

[0101] Table 4

[0102]

[0103] In order to further study the generalization ability of the RLMT-Net model, the RLMT-Net model trained on the private data set is used to predict the CRACK500 data set, and the F1 score of 0.61 can be obtained.

[0104] Figure 5 The segmentation results of road cracks, block cracks, and images with distortion conditions are shown in Figure 5 As shown in the first six rows of

[0105] In order to more intuitively see the generalization effect and compare it with the prediction results of other models, Figure 6 The RLMT-Net model corresponding to the prediction results of several challenging pictures selected from the UAV-CR data set is shown in Figure 6As can be seen, the cracks in the first row of pictures are very thin, but the RLMT-Net model basically predicts the position and shape of the cracks except for a few missed predictions. The cracks in the second row of pictures have interference features, but the RLMT-Net model is not affected and still accurately predicts the position of the cracks in the image. Combining the foregoing analysis and research can draw that the RLMT-Net model trained on the UAV-CR dataset still has good performance on the CRACK500 dataset, which shows that the RLMT-Net model has good generalization ability.

[0106] The method calculates the accuracy, precision, recall, F1 score and mIoU of each model on the four datasets, and provides the following table for comprehensive comparison. In addition, the experimental results of other researchers using the same evaluation indicators and datasets are compared in Table 5.

[0107] Table 5

[0108]

[0109]

[0110] From the above table, it can be seen that the F1 score and mIoU of the RLMT-NET model on the UAV-CR dataset, the CRACK675 dataset, the CRACK500 dataset and the CDF are the best. On the UAV-CR dataset, the F1 score and mIoU of the RLMT-NET model reached 0.7957 and 0.8152, respectively, which was 6.56% and 2.30% higher than the second highest UNet model, respectively. On the CRACK675 dataset, the F1 score and mIoU of the RLMT-NET model reached 0.7382 and 0.7913, respectively, which was 8.27% and 2.20% higher than the second highest UNet model, respectively. On the CRACK500 dataset, the F1 score and mIoU of the RLMT-NET model reached 0.7046 and 0.7498, respectively, which was 2.03% and 1.15% higher than the second highest DeepLabv3+ model, respectively. On the UAV-PDD dataset, the F1 score and mIoU of the RLMT-NET model reached 0.6924 and 0.7601, respectively, which was 1.78% and 0.98% higher than the second highest UNet model, respectively.

[0111] To sum up, by means of the above technical scheme of the present application, a road disease image semantic segmentation method based on a region loss is provided, and according to all experimental results and comparison, it can be seen that the RLMT-NET model in the present application can learn better crack features, and can effectively prevent the segmentation result from being affected by interference factors in the image, greatly improving the problems of low recognition accuracy and poor robustness of the existing research, which is of great help to practical application

[0112] The above merely describes preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0113] The unmentioned part of the present application is applicable to the prior art.

Claims

1. A method for road disease image semantic segmentation based on region loss, characterized in that, The segmentation method comprises the following steps: Obtain image data containing road diseases, and obtain a disease area graph by target detection, wherein the disease area graph contains position information of the diseases; An RLMT-Net model is constructed, which comprises a CNN backbone network, a feature pyramid network FPN, a quadtree, an incoherent detector, a sequence encoder, a pixel decoder and an incoherent query sequence module; The feature pyramid network FPN has a four-layer cascaded structure, and is sequentially recorded as P2, P3, P4 and P5 from bottom to top; The CNN backbone network is used for extracting disease features, and the output of the CNN backbone network is connected to the bottom P2 layer of the feature pyramid network FPN, and the output of the CNN backbone network is also connected to a box head operation, and the output of the box head operation is used as the input of each layer of the feature pyramid network FPN; The input image is sequentially subjected to the CNN backbone network to extract disease features of the image, and the FPN generates multi-level feature mappings for downstream processing; The outputs of the P2 layer, the P3 layer and the P4 layer of the feature pyramid network FPN are subjected to a region proposal network (RPN) and a ROI Align layer to obtain regions of interest RoI, and the three regions of interest RoI of different scales construct a RoI feature pyramid; The regions of interest RoI obtained by processing the P4 layer of the feature pyramid network FPN through the region proposal network (RPN) and the ROI Align layer are subjected to coarse mask head processing to obtain a coarse mask of the ROI region of interest with a scale of 28*28; The coarse mask is input into the incoherent detector, which comprises two-level up-sampling operations and outputs an incoherent mask, and the incoherent detector is used to refine the mask prediction of the incoherent node of the quadtree; Starting from a detection point in the highest-level feature mapping in the RoI feature pyramid as a root node, a multi-level quadtree is constructed, the incoherent mask is given to the quadtree, and the quadtree is used to subdivide the non-coherent regions of each scale of the detected regions of interest and output an incoherent query sequence; The incoherent query sequence module comprises a plurality of node encoders, and the output of the incoherent query sequence module is connected to a sequence encoder, and after processing by the sequence encoder, an encoded sequence is obtained, which is then input into a pixel decoder to obtain a sequential prediction result; The RLMT-Net model is used for semantic segmentation of image data containing road diseases, and a disease image with a segmentation label is output.

2. The segmentation method of claim 1, wherein, The output of the quadtree is an incoherent query sequence, which has a plurality of incoherent nodes, and each incoherent node of the quadtree is encoded by using a node encoder; The node encoder is used for encoding each node of the quadtree, which comprises fusion embedding and position embedding, the node of the quadtree has fine-grained, coarse-grained and background information, the three are fused by using the fusion embedding, and the position embedding is used to encode the spatial distance and relationship between the nodes in each RoI, capture important local dependencies and correlations, and realize the encoding of the nodes.

3. The segmentation method of claim 1, wherein, The sequence encoder is used for integrating nodes of each level of the quadtree, and achieves global and cross-scale spatial reasoning, and has three sequence encoders connected in series, the sequence encoders include a multi-head self-attention mechanism and a fully connected feedforward network to process the incoherent point sequence, the input of the first sequence encoder is the output of the P5 layer of the FPN and the output of the incoherent query sequence module, the other two sequence encoders are connected in series after the first sequence encoder, and the output of the P5 layer of the FPN is also separately input into the other two sequence encoders; The pixel decoder is in the form of a double-layer MLP (Multi-Layer Perceptron) to decode the node output and generate the final mask label.

4. The segmentation method of claim 1, wherein, The parameters of each layer of the feature pyramid network FPN are as follows:

Images