Method for automatically marking crack semantic segmentation and improving segmentation effect

Abstract

The invention discloses a method for automatically marking crack semantic segmentation and improving a segmentation effect. According to the method, the minimum value is continuously searched for the crack pixel in the image neighborhood after preprocessing, the preliminary semantic segmentation crack picture can be optimized, and the precision and efficiency of crack pixel segmentation are improved. In addition, the invention further provides an automatic labeling method, each pixel of the crack does not need to be manually labeled, and the efficiency of crack pixel segmentation is remarkably improved. And after the crack pixel list and the corresponding crack binary image are obtained, a crack segmentation scheme of axis extraction and watershed algorithm filling is adopted, so that the problems of strong subjectivity and insufficient refinement of crack edge pixel judgment when a deep learning algorithm is directly adopted for segmentation or manual segmentation is also avoided, and the crack pixel segmentation quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation, and in particular relates to a method for automatically labeling crack semantic segmentation and improving segmentation effect. Background Art

[0002] In the field of house, road, bridge, and tunnel inspection, cracks on the surface of structures are crucial to the assessment of the health status of structures. Since there are many cracks, their distribution is random, and some locations (such as the bottom of bridges and the top of tunnels) are difficult for personnel to reach, machine vision methods are often used to extract dimensional information such as length and width after taking pictures of the cracks on site. In order to extract the length and width information of the cracks in a refined manner, it is usually necessary to segment the cracks at the pixel level, and then obtain the size information of the cracks by counting the crack pixels.

[0003] Crack segmentation algorithms are mainly divided into feature recognition-based algorithms (such as threshold segmentation method, edge detection method, etc.) and deep learning-based algorithms based on their principles. Among them, deep learning-based algorithms have been widely used in recent years because of their advantages in eliminating interference factors, segmentation accuracy, segmentation efficiency, etc.

[0004] The crack segmentation algorithm based on deep learning usually requires that the cracks to be processed have similar features to the cracks trained when building the model. However, in the actual crack image segmentation needs, due to differences in crack image acquisition scenes, environments, locations, lighting, occlusion, morphology, etc., there may be large differences between the training samples and the actual samples. Limited by the generalization ability of the model, when the features of the collected crack images are significantly different, such as using the crack segmentation model trained on house cracks to process bridge and tunnel cracks, the crack segmentation effect is usually poor, and only a small part of the crack area can be segmented, which cannot meet the needs of crack parameter extraction. In addition, when training a crack segmentation model based on deep learning, a large number of labeled crack segmentation samples are required. Pure manual labeling is usually inefficient, and the labeling accuracy at the crack edge is poor, which affects the training of the crack segmentation model.

[0005] Therefore, the present invention proposes a method for improving the crack segmentation effect. In order to solve the problem that there is a large difference between training samples and actual samples, resulting in poor segmentation effect, the segmentation effect can be optimized. In order to solve the problem of low annotation efficiency of crack segmentation algorithm, an open source existing similar crack segmentation model can be used for preliminary segmentation, and then the segmentation effect can be optimized using this method to quickly and automatically construct a crack segmentation training sample set. Summary of the invention

[0006] The purpose of the present invention is to provide a method for automatic annotation of deep learning crack semantic segmentation and improvement of segmentation effect, which can optimize images with poor crack segmentation effect and improve the segmentation effect. In addition, it can also be used to automatically construct a crack segmentation sample set based on deep learning.

[0007] The present invention aims at the crack segmentation problem in the field of machine vision, and at the problems of poor generalization ability of deep learning image segmentation and large workload of segmented image annotation. An automated segmentation method is proposed, which utilizes part of the existing segmented area and extends the crack area to the actual boundary of the image through multiple factors such as grayscale and direction, so as to improve the accuracy and efficiency of crack segmentation.

[0008] The specific implementation method is as follows:

[0009] Step 1: Perform image preprocessing on the crack image and extract the crack axis of the preliminary segmentation.

[0010] Step 2: Perform connected domain analysis on the initially segmented fracture axis, extract the area of ​​each connected domain and the coordinates of the internal points, and obtain the filtered connected domain according to the screening criteria.

[0011] Step 3: Calculate the average gray value of each connected domain coordinate after screening, and compare it with the average gray value of the original crack image to determine the input image.

[0012] Step 4: Extract the starting point and end point of each connected domain after screening and add them to the endpoint list; add the adjacent points of the starting point and end point in the connected domain to the direction list, find the minimum grayscale value in the neighborhood and extract its coordinates.

[0013] Step 5: Determine in sequence whether the grayscale difference, direction difference, whether to traverse the neighborhood, whether to traverse the endpoint list, and whether the line connecting the endpoint and the grayscale minimum value coincides with the point inside the crack pixel, until entering the judgment of whether to traverse the endpoint list to obtain the crack binary map.

[0014] Step 6: Extract the axis of the crack binary image to obtain the optimized crack axis binary image, obtain the boundary of the crack in the input image, obtain the optimized crack grayscale segmentation image, mark the optimization result after crack segmentation in the crack grayscale image, and obtain the optimized crack segmentation image.

[0015] The beneficial effects brought by the present invention: The present invention can optimize the preliminary semantic segmentation crack image by continuously searching for the minimum value to find crack pixels in the image neighborhood after preprocessing, thereby improving the accuracy and efficiency of crack pixel segmentation. In addition, the present invention provides an automatic labeling method, which does not require manual labeling of each crack pixel, significantly improving the efficiency of crack pixel segmentation. After obtaining the crack pixel list, the crack segmentation scheme using axis extraction and watershed algorithm filling also avoids the problem of strong subjectivity and lack of refinement in crack edge pixel judgment when directly using deep learning algorithm segmentation or manual segmentation, thereby improving the quality of crack pixel segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1This is a specific flow chart of Example S1 of the present application.

[0017] Figure 2 This is a specific flow chart of Example S2 of the present application.

[0018] Figure 3 This is a specific flow chart of Example S3 of this application.

[0019] Figure 4 This is a specific flow chart of Example S4-S5 of this application.

[0020] Figure 5 This is a specific flow chart of Example S6 of the present application. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] The present application provides a method for automatic annotation of crack semantic segmentation and improving segmentation effect, comprising the following steps:

[0023] Step 1: Preprocess the crack image to extract the crack axis of the preliminary segmentation;

[0024] Step 2: Perform connected domain analysis on the initially segmented fracture axis, extract the area of ​​each connected domain and the coordinates of the internal points, and obtain the screened connected domain according to the screening criteria;

[0025] Step 3: Calculate the average gray value of each connected domain coordinate after screening, and compare it with the average gray value of the original crack image to determine the input image;

[0026] Step 4: Extract the starting point and end point of each connected domain after screening, and add them to the endpoint list; add the adjacent points of the starting point and end point in the connected domain to the direction list, find the minimum gray value in the neighborhood and extract its coordinates;

[0027] Step 5: sequentially judge whether the grayscale difference, direction difference, whether to traverse the neighborhood, whether to traverse the endpoint list, whether the connection line between the endpoint and the grayscale minimum value coincides with the point inside the crack pixel, and whether the conditions are met until entering the judgment of whether to traverse the endpoint list to obtain the crack binary map;

[0028] Step 6: Extract the axis of the crack binary image to obtain the optimized crack axis binary image, obtain the boundary of the crack in the input image, obtain the optimized crack grayscale segmentation image, mark the optimization result after crack segmentation in the crack grayscale image, and obtain the optimized crack segmentation image.

[0029] Further, the step 1 specifically includes:

[0030] Step 1.1: converting the crack image to be automatically labeled or the crack image to be improved in segmentation effect into a grayscale image to obtain the crack grayscale image to be automatically labeled or the crack grayscale image to be improved in segmentation effect;

[0031] Step 1.2: Perform preliminary semantic segmentation on the crack grayscale image to be automatically annotated, and obtain a binary image after preliminary semantic segmentation; perform semantic segmentation on the crack grayscale image for which the segmentation effect is to be improved, and obtain a binary image for semantic segmentation to be improved;

[0032] Step 1.3: Use image morphological dilation operation on the crack area of ​​the binary image after preliminary semantic segmentation or the binary image of semantic segmentation to be improved, connect the narrower crack discontinuities, smooth the crack contours, and extract the crack axis of the preliminary segmentation.

[0033] Furthermore, the screening criteria in step 2 are specifically:

[0034] There are fracture axis diagrams with connected domain area > X, and only connected domains with area > X are retained;

[0035] If there is no fracture axis diagram with a connected domain area greater than X, all of them will be retained.

[0036] Furthermore, in step 3, the average grayscale value of the original crack image is compared to determine that the input image is:

[0037] If the average gray value in each connected domain is greater than the average gray value of the original crack image, the crack gray image to be processed is inverted as the input image; otherwise, the crack gray image to be processed is directly used as the input image.

[0038] Further, the step 4 specifically includes:

[0039] Step 4.1: Extract the starting point and end point of each connected domain after screening and add them to the endpoint list;

[0040] Step 4.2: Use the adjacent points of the starting point in the connected domain and the adjacent points of the end point in the connected domain to add them to the direction list;

[0041] Step 4.3: Put all the points in the direction list into the crack pixel list, which is used to judge the end condition and determine the crack pixels after the segmentation is completed;

[0042] Step 4.4: Extract an endpoint in the endpoint list and the corresponding point in the direction list, perform mean filtering and adaptive histogram equalization preprocessing on the input image, and then find the minimum grayscale value in the neighborhood with the endpoint coordinate as the center point and extract its coordinates.

[0043] Further, the step 5 specifically includes:

[0044] Determine whether the difference between the grayscale minimum value in the neighborhood and the grayscale value at the endpoint coordinates is less than a (a is the set threshold) and the average direction difference between the direction from the endpoint to the grayscale minimum value and the corresponding N points in the direction list is less than b degrees (b is the set direction difference).

[0045] When the condition is not satisfied, it is determined whether the neighborhood has been traversed;

[0046] When the conditions are met, it is determined whether the line connecting the endpoint and the grayscale minimum value coincides with the point inside the crack of the crack pixel.

[0047] Furthermore, when the condition is not satisfied, it is determined whether the neighborhood has been traversed. If yes, it is determined whether to traverse the endpoint list directly. Otherwise, it is determined whether to search for a new grayscale minimum value and coordinates again.

[0048] Furthermore, when the condition is met, it is determined whether the line connecting the endpoint and the grayscale minimum value coincides with the point inside the crack of the crack pixel; if so, the coordinates of the point inside the line connecting the endpoint to the coincident position are taken and put into the crack pixel list, and the judgment of whether to traverse the endpoint list is directly entered; otherwise, the coordinates inside the line connecting the endpoint to the grayscale minimum value position are taken and put into the crack pixel list, the endpoint is updated to the grayscale minimum point, and the N adjacent points are taken as the starting point direction, and the operation is repeated until the judgment of whether to traverse the endpoint list is entered.

[0049] Furthermore, when judging whether the endpoint list has been traversed, if not all points in the endpoint list have been traversed, the next endpoint in the endpoint list and its corresponding starting point direction are selected, and step 4 is repeated; if the endpoint list has been traversed, all points in the crack pixel list are extracted, annotated with a special grayscale value, and a crack binary map is obtained.

[0050] Further, the step 6 specifically includes:

[0051] Step 6.1: Extract the axis of the crack binary image to obtain an optimized crack axis binary image;

[0052] Step 6.2: In the input image, the crack axis pixels in the crack binary image are used as annotations, and the non-crack pixels that are inverted after the crack pixels are dilated are used as background annotations. The watershed algorithm is used to fill the crack boundary, thereby completing the optimization of the crack segmentation and obtaining the optimized crack grayscale segmentation map;

[0053] Step 6.3: Mark the optimization result after crack segmentation in the crack image to be processed to obtain the optimized crack segmentation map.

[0054] The implementation steps of the present invention are further described below with reference to the accompanying drawings and specific examples:

[0055] like Figure 1-Figure 5 As shown, this embodiment provides a method for deep learning crack semantic segmentation automatic annotation and improving segmentation effect, which specifically includes the following steps:

[0056] S1: Perform grayscale conversion on the crack image to be automatically labeled or the crack image to be improved in segmentation effect, and obtain the crack grayscale image to be automatically labeled or the crack grayscale image with improved segmentation effect. Use the existing relevant open source crack segmentation model, similar crack segmentation model, etc. to perform semantic segmentation on the crack grayscale image to be automatically labeled, and obtain a binary image after preliminary semantic segmentation; use the crack segmentation model of the segmentation effect to be improved to perform semantic segmentation on the crack grayscale image with improved segmentation effect, and obtain the binary image to be improved in semantic segmentation. Use image morphological dilation operation on the crack area of ​​the binary image after preliminary semantic segmentation or the binary image of semantic segmentation to be improved, connect the narrower crack discontinuities, smooth the crack contour, and then use axis extraction algorithms such as K3M and Zhang-Suen to extract the crack axis of the preliminary segmented crack. The specific process is as follows: Figure 1 shown.

[0057] S2: Perform connected domain analysis on the initially segmented crack axis, and extract the area and internal point coordinates of each connected domain. For the crack axis map with a connected domain area > X (X is the set connected domain area threshold; to ensure the optimization effect, X is usually determined based on the actual number of crack pixels in each connected domain in the image set to be processed), only the connected domain with an area > X is retained; if there is no crack axis map with a connected domain area > X, all are retained, thereby obtaining the screened connected domain. The specific process is as follows: Figure 2 shown.

[0058] S3: Calculate the average grayscale value of each connected domain coordinate after screening on the grayscale image of the crack to be processed, and compare it with the average grayscale value of the grayscale image of the crack to be processed.

[0059] If the average gray value in each connected domain is greater than the average gray value of the original crack image, then the corresponding gray area is subtracted from 255 for each pixel of the crack gray image to be processed, and the inverse of the crack gray image to be processed is obtained as the input image; otherwise, the crack gray image to be processed is directly used as the input image. The specific process is as follows: Figure 3 shown.

[0060] Usually, the grayscale value of crack pixels is much smaller than the grayscale value of the background area, and subsequent segmentation is also performed under this premise. However, due to background differences, the grayscale value of crack pixels may be much larger than the grayscale value of the background area, so the grayscale value needs to be negated.

[0061] Figure 4 The specific process of S4-S5 is provided, which is as follows:

[0062] S4: Extract the starting point and end point of each connected domain after screening and add them to the endpoint list. Use the N points (usually determined according to the number of points in the connected domain of the image set to be processed to ensure the optimization effect) in the connected domain backward from the starting point and the N points in the connected domain forward from the end point to add them to the direction list.

[0063] All points in the direction list are put into the crack pixel list to judge the end condition and determine the crack pixels after the segmentation is completed. When there is a large area of ​​connected domain, remove the small connected domain to avoid the small connected domain being the result of wrong segmentation and affecting further segmentation.

[0064] Extract an endpoint in the endpoint list and the corresponding N points in the direction list, perform mean filtering and adaptive histogram equalization preprocessing on the input image, and then use the endpoint coordinates as the center point to find the minimum grayscale value in M ​​neighborhoods (such as 3×3 neighborhoods, 5×5 neighborhoods, 10×10 neighborhoods, etc., which need to be determined based on the image set to be processed) and extract its coordinates.

[0065] S5: Determine whether the conditions that the difference between the grayscale minimum value in the neighborhood and the grayscale value at the endpoint coordinates is < a (needs to be determined based on the image set to be processed) and the difference between the direction from the endpoint to the grayscale minimum value and the average direction of the corresponding N points in the direction list is < b degrees (usually 90 degrees, but can also be adjusted according to the degree of tortuosity of the crack direction in the actual image) are met.

[0066] When the condition is not satisfied, it is determined whether the M neighborhood has been traversed. If the M neighborhood has not been traversed, a new grayscale minimum value and coordinates are searched again. If the M neighborhood has been traversed, it is directly determined whether the endpoint list has been traversed.

[0067] When the above conditions are met, it is determined whether the line connecting the endpoint and the grayscale minimum value coincides with the point inside the crack pixel. If they coincide, the coordinates of the point inside the line connecting the endpoint to the coincident position are taken and put into the crack pixel list, and the judgment of whether to traverse the endpoint list is directly entered; if they do not coincide, the coordinates inside the line connecting the endpoint to the grayscale minimum value position are taken and put into the crack pixel list, the endpoint is updated to the grayscale minimum value point, and the N points adjacent to it are taken as the starting point direction, and the operation is repeated until the judgment of whether to traverse the endpoint list is entered.

[0068] In this embodiment, when determining whether the endpoint list has been traversed, if not all points in the endpoint list have been traversed, the next endpoint in the endpoint list and its corresponding starting point direction are selected, and the step of searching for the minimum value in the neighborhood is repeated.

[0069] If the endpoint list has been traversed, all points in the crack pixel list are extracted and annotated with a special grayscale value, such as 255, to obtain a crack binary map, thus completing the crack semantic segmentation automatic annotation / segmentation effect improvement.

[0070] S6: Extract the axis of the crack binary image to obtain the optimized crack axis binary image. In the input image, the crack pixels in the crack axis binary image are marked as 1, and the crack pixels are image-expanded Q times (need to be determined based on the crack width distribution in the image set) and then the inverted non-crack pixels are marked as background 2. The watershed algorithm is used to fill and obtain the crack boundary, thereby completing the optimization of crack segmentation and obtaining the optimized crack grayscale segmentation map. Subsequently, the optimization result after crack segmentation is marked in the crack image to be processed to obtain the optimized crack segmentation map. The specific process is as follows: Figure 5 shown.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for automatic annotation of crack semantic segmentation and improving segmentation effect, characterized in that: The following steps are involved: Step 1: Preprocess the crack image to extract the crack axis of the preliminary segmentation; Step 2: Perform connected domain analysis on the initially segmented fracture axis, extract the area of ​​each connected domain and the coordinates of the internal points, and obtain the screened connected domain according to the screening criteria; Step 3: Calculate the average gray value of each connected domain coordinate after screening, and compare it with the average gray value of the original crack image to determine the input image; Step 4: Extract the starting point and end point of each connected domain after screening, and add them to the endpoint list; add the adjacent points of the starting point and end point in the connected domain to the direction list, find the minimum gray value in the neighborhood and extract its coordinates; Step 5: sequentially judge whether the grayscale difference, direction difference, whether to traverse the neighborhood, whether to traverse the endpoint list, whether the connection line between the endpoint and the grayscale minimum value coincides with the point inside the crack pixel, and whether the conditions are met until entering the judgment of whether to traverse the endpoint list to obtain the crack binary map; Step 6: Extract the axis of the crack binary image to obtain the optimized crack axis binary image, obtain the boundary of the crack in the input image, obtain the optimized crack grayscale segmentation image, mark the optimization result after crack segmentation in the crack grayscale image, and obtain the optimized crack segmentation image.

2. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1: converting the crack image to be automatically labeled or the crack image to be improved in segmentation effect into a grayscale image to obtain the crack grayscale image to be automatically labeled or the crack grayscale image to be improved in segmentation effect; Step 1.2: Perform preliminary semantic segmentation on the crack grayscale image to be automatically annotated, and obtain a binary image after preliminary semantic segmentation; perform semantic segmentation on the crack grayscale image for which the segmentation effect is to be improved, and obtain a binary image for semantic segmentation to be improved; Step 1.3: Use image morphological dilation operation on the crack area of ​​the binary image after preliminary semantic segmentation or the binary image of semantic segmentation to be improved, connect the narrower crack discontinuities, smooth the crack contours, and extract the crack axis of the preliminary segmentation.

3. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 1, characterized in that: The screening criteria in step 2 are specifically: There are fracture axis diagrams with connected domain area > X, and only connected domains with area > X are retained; If there is no fracture axis diagram with a connected domain area greater than X, all of them will be retained; Among them, X is the set connected domain area threshold.

4. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 1, characterized in that: In step 3, the average grayscale value of the original crack image is compared to determine that the input image is: If the average gray value in each connected domain is greater than the average gray value of the original crack image, the gray image of the crack to be processed is inverted as the input image; Otherwise, the crack grayscale image to be processed is directly used as the input image.

5. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 1, characterized in that: The step 4 specifically includes: Step 4.1: Extract the starting point and end point of each connected domain after screening and add them to the endpoint list; Step 4.2: Use the adjacent points of the starting point in the connected domain and the adjacent points of the end point in the connected domain to add them to the direction list; Step 4.3: Put all the points in the direction list into the crack pixel list, which is used to judge the end condition and determine the crack pixels after the segmentation is completed; Step 4.4: Extract an endpoint in the endpoint list and the corresponding point in the direction list, perform mean filtering and adaptive histogram equalization preprocessing on the input image, and then find the minimum grayscale value in the neighborhood with the endpoint coordinate as the center point and extract its coordinates.

6. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 1, characterized in that: The step 5 specifically includes: Determine whether the difference between the grayscale minimum value in the neighborhood and the grayscale value at the endpoint coordinates is less than the set threshold and whether the average direction difference between the direction from the endpoint to the grayscale minimum value and the corresponding N points in the direction list is less than the set direction difference. When the condition is not satisfied, it is determined whether the neighborhood has been traversed; When the conditions are met, it is determined whether the line connecting the endpoint and the grayscale minimum value coincides with the point inside the crack of the crack pixel.

7. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 6, characterized in that: When the condition is not satisfied, it is determined whether the neighborhood has been traversed. If yes, it is determined whether to traverse the endpoint list directly. Otherwise, it is searched for a new grayscale minimum value and coordinates again.

8. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 6, characterized in that: When the above conditions are met, it is determined whether the line connecting the endpoint and the grayscale minimum value coincides with the point inside the crack of the crack pixel; If yes, take the coordinates of the points inside the line from the endpoint to the coincident position and put them into the crack pixel list, and directly proceed to the judgment of whether to traverse the endpoint list; Otherwise, take the coordinates of the line from the endpoint to the grayscale minimum position and put them into the crack pixel list, update the endpoint to the grayscale minimum point, and take the N adjacent points as the starting point direction, and repeat the operation until it is determined whether to traverse the endpoint list.

9. The method for automatic crack semantic segmentation and segmentation effect improvement according to claim 1 or 6, characterized in that: When determining whether the endpoint list has been traversed, if all points in the endpoint list have not been traversed, then select the next endpoint in the endpoint list and its corresponding starting point direction, and repeat step 4; If the endpoint list has been traversed, all points in the crack pixel list are extracted and annotated with a special gray value to obtain a crack binary map.

10. The method for automatic annotation of crack semantic segmentation and improving segmentation effect according to claim 1, characterized in that: The step 6 specifically includes: Step 6.1: Extract the axis of the crack binary image to obtain an optimized crack axis binary image; Step 6.2: In the input image, the crack axis pixels in the crack binary image are used as annotations, and the non-crack pixels that are inverted after the crack pixels are dilated are used as background annotations. The watershed algorithm is used to fill the crack boundary, thereby completing the optimization of the crack segmentation and obtaining the optimized crack grayscale segmentation map; Step 6.3: Mark the optimization result after crack segmentation in the crack image to be processed to obtain the optimized crack segmentation map.

Images