An interactive semantic segmentation data annotation method for pavement damage images and its application

Abstract

An interactive semantic segmentation data annotation method for pavement damage images based on significant edge guidance: obtain pavement inspection images, denoise them, and enhance the contrast between damage and background to complete the preprocessing of pavement inspection images; construct a U-shaped semantic segmentation network with a residual SwinTransformer structure as an encoder and decoder; construct a basic Swin Transformer, image block segmentation, linear embedding layer, and image block merging module; perform preliminary semantic segmentation training on the obtained road inspection images to obtain a rough pavement damage segmentation model; preliminarily judge pavement damage based on the trained pavement damage segmentation model, and input it into the interactive semantic segmentation model to promote the update of the interactive semantic segmentation model; compare it with the manually annotated semantic labels, and stop updating if the threshold requirement is reached; output semantic labels, complete semi-automatic annotation, and improve the accuracy of road damage prediction.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence and relates to a labeling method, in particular to an interactive semantic segmentation data labeling method for road surface disease images. Background Art

[0002] Semantic segmentation is an important branch in the field of computer vision. Through pixel-level classification operations based on specific rules, pixels of the same type are classified into one category and labeled as corresponding categories, thereby achieving pixel-level understanding of the image. In recent years, with the emergence of large-scale public datasets, semantic segmentation technology has developed rapidly and has achieved outstanding results in different scenarios such as autonomous driving, medical imaging, and remote sensing imaging. By drawing on the application of semantic segmentation models in other fields, more and more researchers have devoted themselves to the development of semantic models for road surface disease detection and have achieved certain results and applications.

[0003] However, there are two difficulties in intelligent road inspection. On the one hand, road maintenance inspection data is massive. By the end of 2022, the total mileage of my country's highways will reach 5.35 million kilometers, including 177,000 kilometers of expressways. It is difficult to achieve partial manual annotation of the inspection images of expressways alone, and complete the pixel-level pavement disease recognition that relies on the semantic segmentation model. Faced with massive inspection data, the efficiency and accuracy of pavement disease recognition have a lot of room for improvement. On the other hand, the existing semantic segmentation models that rely on fully convolutional neural networks generally have the problem of rough target edges, which cannot achieve accurate perception of pavement diseases and affect the training effect of the semantic segmentation model. Therefore, the development of refined semi-automatic pavement disease annotation tools is of great significance to improving the efficiency and accuracy of pavement disease recognition.

[0004] The existing semantic segmentation data annotation methods are highly dependent on manual work, and have problems such as low efficiency and high error rate; the annotation quality cannot be controlled; the annotation tool is not accurate enough. At the same time, the existing pavement disease recognition methods based on semantic segmentation are relatively rough in the division of the disease edge, and cannot achieve refined edge division and regional segmentation. More importantly, the research on semi-automatic annotation of pavement diseases based on interactive semantic segmentation models is still relatively limited, and there is a model matching problem that the general model cannot be applied to specific scenarios. In summary, in order to ensure high-efficiency and high-quality pavement disease data annotation and realize pixel-level refined pavement disease identification, it is of great significance to develop an interactive semantic segmentation data annotation method for pavement disease images based on significant edge guidance to serve the intelligent maintenance of road infrastructure.

[0005] There are still few studies on the existing interactive image data annotation methods, which generally include drawing lines, drawing borders, clicking edge points, clicking positive and negative sample points, etc., and the image data is annotated by selecting the marked area, formulating rules for training based on the marked pixels, obtaining the pre-trained model, completing the image segmentation, etc. For example, the prior art 1 (Gao Ruipu, Li Li. Interactive image semantic segmentation method using superpixel encoding [J]. Surveying and Mapping Geographic Information, 2022, 47(S1):148-152.DOI:10.14188 / j.2095-6045.2022274) and the prior art 2 (Zuo Xiangmei, Zhao Zhen, Gou Tingting. RGB-D image semantic segmentation based on interactive conditional random fields [J]. Computer Applications and Software, 2017, 34(03):174-180.) disclose the annotation methods based on interactive learning, in which, Figure 1 and Figure 2 They represent interactive annotation methods for general medical imaging data and RGB-D images respectively. Figure 1 The represented method generates positive sample points in the segmented object after obtaining the user's interactive information, encodes the interactive information, obtains the feature map of the input network, realizes the training of the samples, and then uses the intersection over union (IoU) evaluation indicator to generate positive or negative points for the evaluation results. Figure 2 The represented method first performs an initial segmentation of the image based on a conditional random field, and dynamically adjusts the model through a small amount of manual interaction to reflect the current scene information, thereby achieving the purpose of improving the segmentation effect.

[0006] However, the existing interactive data annotation methods have not been extended to the field of pavement disease annotation, and there are certain scene matching problems:

[0007] Existing interactive data annotation methods require high user input. The position and number of annotations for the same object will affect the annotation results, and the amount of interaction is large.

[0008] It is currently not possible to semi-automatically mark road surface defects, especially road cracks. Semi-automatic marking is even more difficult when it comes to fine cracks. Summary of the invention

[0009] In order to solve the defects in the prior art, the present invention discloses an interactive semantic segmentation data annotation method for pavement damage images based on significant edge guidance, and specifically discloses the following technical solutions:

[0010] A method for interactive semantic segmentation data annotation of pavement damage images, in particular, a method for interactive semantic segmentation data annotation of pavement damage images based on significant edge guidance, which is characterized by:

[0011] Step 1, obtaining a road inspection image, reducing noise on the road inspection image based on a median filtering method, enhancing the contrast between the defect and the background, and completing preprocessing of the road inspection image;

[0012] Step 2: construct a U-shaped semantic segmentation network with the residual Swin Transformer structure as the encoder and decoder;

[0013] Step 3: construct the basic Swin Transformer module, image block segmentation module, linear embedding layer and image block merging module;

[0014] Step 4: Based on the residual idea, a residual Swin Transformer module is constructed to enhance the nonlinearity of crack feature learning;

[0015] Step 5: Perform preliminary semantic segmentation training on the acquired road inspection images to obtain a rough road surface disease segmentation model;

[0016] Step 6: Preliminarily judge the pavement damage based on the trained pavement damage segmentation model, input it into the interactive semantic segmentation model, and click on the significant edge information of damages such as cracks and potholes to promote the update of the interactive semantic segmentation model;

[0017] Step 7: Compare with the manually annotated semantic labels, and stop updating if the threshold requirement is reached;

[0018] Step 8: Output semantic labels, complete semi-automatic labeling, and improve the accuracy of road damage prediction.

[0019] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0020] The present invention also discloses an electronic device, characterized in that it comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the above method when they are executed. Beneficial Effects

[0021] 1. Ability to label road surface damage data efficiently and with high quality, and achieve pixel-level refined road surface damage identification;

[0022] 2. The ability to label different road surface defects has been enhanced, so that the identification of road surface defects is not limited to the identification of road surface cracks;

[0023] 3. It can greatly save manpower expenses and minimize the impact of subjective influence on marking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The first is an interactive annotation method for common medical imaging data and RGB-D images in the prior art;

[0025] Figure 2 The second interactive annotation method for common medical imaging data and RGB-D images in the prior art;

[0026] Figure 3 A flowchart of a method for interactive semantic segmentation data annotation of pavement damage images based on significant edge guidance provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a road surface disease image semantic segmentation model based on significant edge guidance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] A method for interactive semantic segmentation data annotation of pavement damage images, in particular, a method for interactive semantic segmentation data annotation of pavement damage images based on significant edge guidance, comprising the following steps:

[0029] Step 1: Obtain a road inspection image, perform noise reduction on the road inspection image based on a median filtering method, enhance the contrast between the defect and the background, and complete the preprocessing of the road inspection image.

[0030] The gray value of each pixel in the collected road inspection image is set to the median gray value of all pixels in the neighborhood window of the pixel. The two-dimensional median filter output can be expressed as:

[0031] (1)

[0032] in, , represent the original road inspection image and the processed road inspection image respectively. Represents the two-dimensional template dimension, Represents the pixel coordinate point, Represents the numerical value of the template's horizontal and vertical axes. After median filtering, isolated disease noise points in the inspection image can be effectively eliminated.

[0033] Step 2: Construct a U-shaped semantic segmentation network with a residual Swin Transformer structure as the encoder and decoder. The left structure of the U-shaped semantic segmentation network is the encoder, and the right structure is the decoder. The encoder and decoder structures are each divided into three levels, and each Swin Transformer unit adopts a residual connection structure to increase the nonlinear learning ability of the model. Finally, the fully convolutional neural network is used as the prediction head to complete the pixel-level segmentation and prediction of pavement diseases. In more detail, in the left encoder structure, an image segmentation module is set at the beginning stage, and three residual Swin Transformer modules are followed by a linear embedding layer and two tile merging modules respectively. At the same time, the bottom of the left structure is a tile merging module, and the above modules are assembled in a sequential connection manner. In the right decoder structure, the corresponding modules are constructed in a symmetrical manner, which includes a tile merging module from bottom to top, three residual Swin Transformer modules are followed by three tile expansion modules respectively, and a linear projection module, and the above are assembled in a sequential connection manner. At the same time, the left structure and the right structure are connected horizontally in sequence, and a residual Swin Transformer module is set at the bottom to connect the left and right structures.

[0034] Step 3: Construct the basic Swin Transformer module, image block segmentation module, linear embedding layer and image block merging module.

[0035] The road inspection image is input into the image segmentation module, and the road inspection image is segmented. The size of each image block is 4*4. After the image segmentation operation, each block is flattened. As a result, the dimension of the image block becomes [H / 4, W / 4, 16*3], where H represents the feature map height and W represents the feature map width. Specifically, except for the linear embedding layer used in the first stage, the second and third stages both use the block merging module. The image segmentation module contains five key steps, namely, the image block embedding process, the local self-attention calculation process, the global self-attention calculation process, the Transformer encoding process, and the position encoding process. Among them, the linear embedding layer linearly transforms the channels of the road disease pixels, that is, the dimension of the image changes from [H / 4, W / 4, 16*3] to [H / 4, W / 4, C]. Specifically, the linear embedding layer performs a linear transformation according to the weight matrix, and adds a bias vector to obtain the output vector. C is the number of feature map channels, which is set to 96 in this case. The block merging module also includes linear transformation operations according to the weight matrix, but adds image block reorganization based on shape transformation. Its main function is to halve the height and width of each block through downsampling, pixel splicing, and layer normalization operations, thereby doubling the number of channels of the image.

[0036] Step 4: Based on the residual idea, a residual Swin Transformer module is constructed to enhance the nonlinearity of crack feature learning.

[0037] The residual Swin Transformer module is composed of a normal window Transformer module and a moving window Transformer module connected in residual form. For the normal Swin Transformer module sequence, its calculation formula is shown in the following equations (2)-(5):

[0038] (2)

[0039] (3)

[0040] (4)

[0041] (5)

[0042] in, represents linear normalization, represents the normal window Transformer attention mechanism, represents the moving window Transformer attention mechanism, represents a multi-layer perceptron, represents the feature input value, Represents the feature output value after being processed by the ordinary window attention mechanism, Represents the feature output value after being processed by the multi-layer perceptron. Represents the feature output value after being processed by the moving window attention mechanism, represents the feature output value after multiple perceptron processing. Specifically, the calculation formula of the residual Swin Transformer module is shown in the following formula (6):

[0043] (6)

[0044] in, represents the residual operation, represents the characteristic input value, Represents the feature output value after the residual operation.

[0045] Step 5: Perform preliminary semantic segmentation training on the acquired road inspection images to obtain a rough road surface disease segmentation model.

[0046] Semantic segmentation training is performed on road inspection images. The deep learning training framework used is pytorch, and the MMSegmentation toolbox is used for basic model configuration. Specifically, the initial learning rate is 0.00001, the number of iterations is 12, the batch size is 2, the momentum parameter is set to 0.9, the optimizer is set to SGD, and the weight decay is set to 0.0005. The loss function used is the Dice loss function, and the specific calculation formula is shown in the following formula (7):

[0047] (7)

[0048] in, is the predicted disease value and the true disease value The pixel intersection between and The predicted disease value and the true disease value .

[0049] Step 6: Preliminarily judge the pavement damage based on the trained pavement damage segmentation model, input it into the interactive semantic segmentation model, and click on the significant edge information of damages such as cracks and potholes to promote the update of the interactive semantic segmentation model.

[0050] The preliminary pavement defect semantic segmentation results are input into the interactive semantic segmentation model. By clicking on the significant edges of cracks, potholes and other defects in the annotation interface, the similarity between the two-pixel features in the neighborhood is calculated. The calculation formula is shown in the following formula (8):

[0051] (8)

[0052] Among them is Similarity calculation function, and is the neighborhood pixel, is the average distance between features. By calculating the similarity results, the significant disease area is determined.

[0053] Step 7: Compare with the manually annotated semantic labels. If the threshold requirement is reached, stop updating.

[0054] The interactive pavement disease semantic segmentation results are compared with the true labels, and the intersection over union (IoU) is used as the calculation parameter to compare the prediction accuracy. If the IoU result exceeds 90%, it is a qualified prediction result, and the update of one round of training is stopped and the next round of training begins. The calculation formula of the intersection over union (IoU) is shown in the following formula (9):

[0055] (9)

[0056] in, Represents the intersection of pixels in the diseased area, Represents the union of pixels in the diseased area.

[0057] Step 8: Output semantic labels, complete semi-automatic labeling, and improve the accuracy of road damage prediction.

[0058] The obtained semantic label results are classified and output to complete the semi-automatic pavement disease semantic labeling. On the basis of expanding the original disease dataset, the pavement disease dataset is trained for secondary semantic segmentation model. During the training process, the pytorch framework and mmsegmentation toolbox are still used. The initial learning rate is set to 0.00002, the number of iterations is 24, the batch size is 4, the momentum parameter is set to 0.9, the optimizer is set to SGD, the weight decay is set to 0.0005, and the mixed loss function (Equation 10, Equation 11) is used for training to improve the prediction accuracy of pavement diseases.

[0059] (10)

[0060]

[0061] X is the predicted disease value and Y is the label disease value.

[0062] The present invention proposes an interactive semantic segmentation data annotation method for pavement disease images based on significant edge guidance. It innovatively utilizes the significant edge guidance module and the pixel-level semantic segmentation network model to realize interactive pavement disease image data interaction and semi-automatic pavement crack image annotation, which greatly reduces the manpower annotation required for road network-level engineering inspection images, saves costs while improving the efficiency and accuracy of pavement disease identification.

[0063] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An interactive semantic segmentation data annotation method for pavement damage images based on salient edge guidance, which is characterized by: Step 1: Obtain a road inspection image, perform noise reduction on the road inspection image based on the median filtering method, enhance the contrast between the defect and the background, and complete the preprocessing of the road inspection image; set the gray value of each pixel in the collected road inspection image to the median gray value of all pixels in the neighborhood window of the pixel, and the two-dimensional median filtering output can be expressed as: (1) in, , represent the original road inspection image and the processed road inspection image respectively. Represents the two-dimensional template dimension, Represents the pixel coordinate point, Represents the value of the template's horizontal and vertical axes; after median filtering, isolated noise points in the inspection image can be effectively eliminated; Step 2: construct a U-shaped semantic segmentation network with the residual Swin Transformer structure as the encoder and decoder; Step 3: construct a basic Swin Transformer module, an image block segmentation module, a linear embedding layer, and an image block merging module; the segmentation module is used for enhancement; Step 4: Based on the residual idea, a residual Swin Transformer module is constructed to enhance the nonlinearity of crack feature learning. The residual Swin Transformer module is connected by a normal window Transformer module and a moving window Transformer module in a residual form. For the normal Swin Transformer module sequence, its calculation formula is shown in the following equations (2)-(5): (2) (3) (4) (5) in, represents linear normalization, represents the normal window Transformer attention mechanism, represents the moving window Transformer attention mechanism, represents a multi-layer perceptron, represents the feature input value, Represents the feature output value after being processed by the ordinary window attention mechanism, Represents the feature output value after being processed by the multi-layer perceptron. Represents the feature output value after being processed by the moving window attention mechanism, represents the feature output value after multiple perceptron processing; specifically, the residual Swin Transformer module calculation formula is shown in the following formula (6): (6) in, represents the residual operation, represents the feature input value, Represents the feature output value after the residual operation; Step 5: Perform preliminary semantic segmentation training on the acquired road inspection images to obtain a rough road surface disease segmentation model; perform semantic segmentation training on the road inspection images, using the deep learning training framework pytorch and the MMSegmentation toolbox for basic model configuration; Step 6: Preliminarily judge the pavement damage according to the trained pavement damage segmentation model, input it into the interactive semantic segmentation model, click on the significant edge information of cracks and potholes to promote the update of the interactive semantic segmentation model; input the preliminary pavement damage semantic segmentation result into the interactive semantic segmentation model, click on the significant edge of cracks and potholes through the annotation interface, and calculate the similarity between the neighboring 2-pixel features. The calculation formula is shown in the following formula (8): (8) Among them is Similarity calculation function, and is the neighborhood pixel, is the average distance between features; by calculating the similarity results, the significant disease area is determined; Step 7: Compare with the manually annotated semantic labels. If the threshold requirement is reached, the update is stopped. The interactive pavement disease semantic segmentation results are compared with the real labels, and the intersection over union (IoU) is used as the calculation parameter to compare the prediction accuracy. If the IoU result exceeds 90%, it is a qualified prediction result, and the update of one round of training is stopped and the next round of training is started. The calculation formula of the intersection over union (IoU) is shown in the following formula (9): (9) in, Represents the intersection of pixels in the diseased area, Represents the union of pixels in the diseased area; Step 8: Output semantic labels, complete semi-automatic labeling, and improve the accuracy of road damage prediction.

2. The method for interactive semantic segmentation of pavement damage images based on significant edge guidance according to claim 1 is characterized by: Step 2 further includes the following contents: the encoder and decoder structures are divided into three levels respectively, and each Swin Transformer unit adopts a residual connection structure to increase the nonlinear learning ability of the model; finally, a fully convolutional neural network is used as the prediction head to complete the pixel-level segmentation and prediction of road surface diseases.

3. The method for interactive semantic segmentation of pavement damage images based on significant edge guidance according to claim 1 is characterized by: Step 3 further includes the following contents: inputting the road inspection image into the image segmentation module, performing a segmentation operation on the road inspection image, each image block is 4*4 in size, and flattening each block after the image segmentation operation, thereby the dimension of the image block becomes [H / 4, W / 4, 16*3], where H represents the feature map height and W represents the feature map width.

4. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein the program controls the device where the non-volatile storage medium is located to execute the method according to any one of claims 1 to 3 when the program is executed.

5. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method described in any one of claims 1 to 3 when executed.

Images