A Method for Automatic Annotation of Crack Semantic Segmentation and Improvement of Segmentation Effect

Abstract

The present invention discloses a method for automatic annotation of crack semantic segmentation and improvement of segmentation effect. Through preprocessing, the present invention continuously searches for the minimum value in the image neighborhood to find crack pixels, which can optimize the initially semantically segmented crack images and improve the accuracy and efficiency of crack pixel segmentation. In addition, the present invention also provides an automatic annotation method, which does not require manual annotation of each pixel of the crack, significantly improving the efficiency of crack pixel segmentation. After obtaining the crack pixel list and the corresponding crack binary map, the crack segmentation scheme of axis extraction and watershed algorithm filling is also adopted, which also avoids the problem of strong subjectivity and insufficient refinement in the judgment of crack edge pixels when directly using deep learning algorithms for segmentation or manual segmentation, and improves the quality of crack pixel segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation, and particularly relates to a method for automatic annotation of crack semantic segmentation and improvement of segmentation effect. Background Art

[0002] In the fields of building, road surface, bridge, and tunnel inspection, cracks on the surface of structures are crucial for the assessment of structural health status. Due to the large number of cracks, random distribution positions, and the fact that some positions (such as the bottom of bridges and the top of tunnels) are difficult for personnel to reach, machine vision methods are often used to capture cracks on-site and extract size information such as their length and width. To accurately extract the length and width information of cracks, pixel-level segmentation of cracks is usually required, and then the size information of cracks is obtained by counting the crack pixels.

[0003] Crack segmentation algorithms are mainly divided into feature recognition-based algorithms (such as threshold segmentation method, edge detection method, etc.) and deep learning-based algorithms according to their principles. Among them, deep learning-based algorithms have been widely used in recent years due to their advantages in interference factor elimination, segmentation accuracy, and segmentation efficiency.

[0004] For deep learning-based crack segmentation algorithms, the cracks to be processed usually need to have similar characteristics to the training cracks when constructing the model. However, in actual crack image segmentation requirements, due to differences in crack image acquisition scenarios, environments, positions, lighting, occlusion, morphology, etc., there may be significant differences between training samples and actual samples. Limited by the generalization ability of the model, when the characteristics of the collected crack images are significantly different, such as using a crack segmentation model trained with building cracks to process bridge and tunnel cracks, the crack segmentation effect is usually poor, and only a small part of the crack area can be segmented, unable to meet the requirements of crack parameter extraction. In addition, when training a deep learning-based crack segmentation model, a large number of labeled crack segmentation samples are required. Pure manual annotation usually has low efficiency and poor annotation accuracy at the crack edges, affecting the training of the crack segmentation model.

[0005] Therefore, the present invention proposes a method for improving the crack segmentation effect. Aiming at the problem of poor segmentation effect caused by the significant differences between training samples and actual samples, it can optimize the segmentation effect; aiming at the problem of low annotation efficiency of crack segmentation algorithms, the open-source existing similar crack segmentation models can be used for preliminary segmentation and then the segmentation effect can be optimized by this method to quickly and automatically construct a crack segmentation training sample set. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for automatic annotation of deep learning crack semantic segmentation and improvement of segmentation effect, which can optimize images with poor crack segmentation effect, improve the segmentation effect, and can also be used to automatically construct a crack segmentation sample set based on deep learning.

[0007] In view of the problem of crack segmentation in the field of machine vision, aiming at the problems of poor generalization ability of deep learning image segmentation and large workload of segmented image annotation, the present invention proposes an automatic segmentation method that uses some existing segmented regions to extend the crack region to the actual boundary of the image through various factors such as gray scale and direction, so as to improve the accuracy and efficiency of crack segmentation.

[0008] The specific implementation method is as follows:

[0009] Step 1: Perform image preprocessing on the crack image to extract the initially segmented crack axis.

[0010] Step 2: Perform connected component analysis on the initially segmented crack axis, extract the area and internal point coordinates of each connected component, and obtain the filtered connected components according to the screening criteria.

[0011] Step 3: Calculate the average gray value of each filtered connected component's coordinates and compare it with the average gray value of the original crack image to determine the input image.

[0012] Step 4: Extract the starting point and ending point of each filtered connected component and add them to the endpoint list; add the adjacent points of the starting point and ending point within the connected component to the direction list, find the minimum gray value in the neighborhood and extract its coordinates.

[0013] Step 5: Sequentially judge whether the conditions of gray difference, direction difference, whether to traverse the neighborhood, whether to traverse the endpoint list, and whether the connection line between the endpoint and the minimum gray value coincides with the crack pixels inside the crack are satisfied until the judgment of whether to traverse the endpoint list is entered to obtain the crack binary image.

[0014] Step 6: Perform axis extraction on the crack binary image to obtain the optimized crack axis binary image, obtain the boundary of the crack in the input image, obtain the optimized crack gray scale segmentation image, and then mark the optimized result after crack segmentation in the crack gray scale image to obtain the optimized crack segmentation image.

[0015] The beneficial effects brought by the present invention: By continuously searching for the minimum value in the image neighborhood after preprocessing to find crack pixels, the present invention can optimize the initially semantically segmented crack image, improve the accuracy and efficiency of crack pixel segmentation. In addition, the present invention provides an automatic annotation method, which does not require manual annotation of each pixel of the crack, significantly improving the efficiency of crack pixel segmentation. After obtaining the crack pixel list, the crack segmentation scheme of axis extraction and watershed algorithm filling is also adopted, which avoids the problem of strong subjectivity and insufficient refinement in the judgment of crack edge pixels when directly using deep learning algorithms for segmentation or manual segmentation, and improves the quality of crack pixel segmentation. Description of the Drawings

[0016] Figure 1This is the specific flowchart of Embodiment S1 of the present application.

[0017] Figure 2 This is the specific flowchart of Embodiment S2 of the present application.

[0018] Figure 3 This is the specific flowchart of Embodiment S3 of the present application.

[0019] Figure 4 This is the specific flowchart of Embodiments S4 - S5 of the present application.

[0020] Figure 5 This is the specific flowchart of Embodiment S6 of the present application. Detailed implementation manners

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0022] The present application provides a method for automatic annotation of crack semantic segmentation and improvement of segmentation effect, including the following steps:

[0023] Step 1: Perform image pre - processing on the crack image to extract the preliminarily segmented crack axis;

[0024] Step 2: Perform connected - component analysis on the preliminarily segmented crack axis, extract the area of each connected - component and the internal point coordinates, and obtain the filtered connected - components according to the screening criteria;

[0025] Step 3: Calculate the average gray value of each filtered connected - component coordinate, and compare it with the average gray value of the original crack image to determine the input image;

[0026] Step 4: Extract the starting point and ending point of each filtered connected - component and add them to the endpoint list; add the adjacent points of the starting point and ending point within the connected - component to the direction list, find the minimum gray value in the neighborhood and extract its coordinates;

[0027] Step 5: Sequentially judge whether the conditions of gray - level difference, direction difference, whether the neighborhood is traversed, whether the endpoint list is traversed, and whether the connection line between the endpoint and the minimum gray value coincides with the crack pixels inside the crack are satisfied, until the judgment of whether the endpoint list is traversed is entered to obtain the crack binary image;

[0028] Step 6: Perform axis extraction on the crack binary image to obtain the optimized crack axis binary image, obtain the boundary of the crack in the input image, obtain the optimized crack gray - level segmentation image, and then mark the optimized result after crack segmentation in the crack gray - level image to obtain the optimized crack segmentation image.

[0029] Further, step 1 specifically includes:

[0030] Step 1.1: Convert the crack image to be automatically labeled or with segmentation effect to be improved into a grayscale image, obtaining the grayscale crack image to be automatically labeled or the grayscale crack image with segmentation effect to be improved.

[0031] Step 1.2: Perform preliminary semantic segmentation on the grayscale crack image to be automatically labeled to obtain a binary image after preliminary semantic segmentation; perform semantic segmentation on the grayscale crack image with segmentation effect to be improved to obtain a binary image with semantic segmentation to be improved.

[0032] Step 1.3: Perform an image morphological dilation operation on the crack region of the binary image after preliminary semantic segmentation or the binary image with semantic segmentation to be improved, connect the narrow crack discontinuities, smooth the crack contour, and extract the axis of the preliminarily segmented crack.

[0033] Further, the screening criteria in step 2 are specifically:

[0034] For the crack axis diagram with a connected domain area > X, only retain the connected domain with an area > X.

[0035] For the crack axis diagram without a connected domain area > X, retain all.

[0036] Further, in step 3, compare with the average grayscale value of the original crack image to determine the specific input image as:

[0037] If the average grayscale value within each connected domain is greater than the average grayscale value of the original crack image, then take the inverse of the crack grayscale image to be processed as the input image; otherwise, directly use the crack grayscale image to be processed as the input image.

[0038] Further, step 4 specifically includes:

[0039] Step 4.1: Extract the starting point and ending point of each connected domain after screening and add them to the endpoint list.

[0040] Step 4.2: Use the adjacent point backward from the starting point within the connected domain and the adjacent point forward from the ending point within the connected domain and add them to the direction list.

[0041] Step 4.3: Put all the points in the direction list into the crack pixel list, which is used to judge the end condition and determine the crack pixels after segmentation is completed.

[0042] Step 4.4: Extract an endpoint from the endpoint list and the corresponding point from the direction list. After preprocessing the input image with mean filtering and adaptive histogram equalization, find the minimum grayscale value within the neighborhood with the endpoint coordinates as the center point and extract its coordinates.

[0043] Further, step 5 specifically includes:

[0044] Determine whether the condition that the difference between the minimum gray value in the neighborhood and the gray value at the endpoint coordinate is < a (a is a set threshold) and the difference between the direction from the endpoint to the minimum gray value and the average direction of the corresponding N points in the direction list is < b degrees (b is a set direction difference) is satisfied;

[0045] When the condition is not satisfied, determine whether the neighborhood has been traversed;

[0046] When the condition is satisfied, determine whether there is an overlap between the line connecting the endpoint and the minimum gray value and the points inside the crack pixels of the crack.

[0047] Furthermore, when the condition is not satisfied, determine whether the neighborhood has been traversed. If so, directly enter the judgment of whether the endpoint list has been traversed. Otherwise, re - find the new minimum gray value and its coordinates.

[0048] Furthermore, when the condition is satisfied, determine whether there is an overlap between the line connecting the endpoint and the minimum gray value and the points inside the crack pixels of the crack. If so, take the coordinates of the points inside the line connecting the endpoint to the overlapping position and put them into the crack pixel list, and directly enter the judgment of whether the endpoint list has been traversed. Otherwise, take the coordinates inside the line connecting the endpoint to the minimum gray value position and put them into the crack pixel list, update the endpoint to the minimum gray value point, and take its adjacent N points as the starting direction, and repeat the operation until entering the judgment of whether the endpoint list has been traversed.

[0049] Further, when judging whether the endpoint list has been traversed, if not all points in the endpoint list have been traversed, select the next endpoint in the endpoint list and its corresponding starting direction, and repeat step 4. If the endpoint list has been traversed, extract all points in the crack pixel list and mark them with a special gray value to obtain a binary crack image.

[0050] Further, step 6 specifically includes:

[0051] Step 6.1: Extract the axis of the binary crack image to obtain an optimized binary crack axis image;

[0052] Step 6.2: In the input image, using the crack axis pixels in the binary crack image as the annotation, and the non - crack pixels obtained by taking the inverse after image dilation of the crack pixels as the background annotation, perform watershed algorithm filling to obtain the boundary of the crack, thereby completing the optimization of crack segmentation and obtaining an optimized gray - scale segmentation image of the crack;

[0053] Step 6.3: Mark the optimized result after crack segmentation in the crack image to be processed to obtain an optimized crack segmentation image.

[0054] The following further illustrates the implementation steps of the present invention in conjunction with the accompanying drawings and specific examples:

[0055] As Figures 1 - 5 shown, this embodiment provides a method for automatic annotation of deep learning crack semantic segmentation and improving the segmentation effect, which specifically includes the following steps:

[0056] S1: Convert the crack image to be automatically annotated or whose segmentation effect is to be improved into grayscale to obtain the grayscale image of the crack to be automatically annotated or the grayscale image of the crack whose segmentation effect is to be improved. Use existing relevant open-source crack segmentation models, similar crack segmentation models, etc. to perform semantic segmentation on the grayscale image of the crack to be automatically annotated to obtain a binary image after preliminary semantic segmentation; use the crack segmentation model whose segmentation effect is to be improved to perform semantic segmentation on the grayscale image of the crack whose segmentation effect is to be improved to obtain a binary image of the semantic segmentation to be improved. Perform an image morphological dilation operation on the crack region of the binary image after preliminary semantic segmentation or the binary image of the semantic segmentation to be improved to connect the narrow crack discontinuities and smooth the crack contour, and then use axis extraction algorithms such as K3M and Zhang-Suen to extract the axis of the preliminarily segmented crack. The specific process is as Figure 1 shown.

[0057] S2: Perform connected component analysis on the axis of the preliminarily segmented crack to extract the area and internal point coordinates of each connected component. For the crack axis diagram with a connected component area > X (X is the set connected component area threshold; to ensure the optimization effect, X is usually determined according to the actual number of crack pixels in each connected component in the image set to be processed), only retain the connected components with an area > X; if there is no crack axis diagram with a connected component area > X, retain all of them to obtain the filtered connected components. The specific process is as Figure 2 shown.

[0058] S3: Calculate the average grayscale value at the coordinates of each filtered connected component on the grayscale image of the crack to be processed and compare it with the average grayscale value of the grayscale image of the crack to be processed.

[0059] If the average grayscale value within each connected component is greater than the average grayscale value of the original crack image, subtract the corresponding grayscale value from 255 for each pixel point of the grayscale image of the crack to be processed to obtain the inverted grayscale image of the crack to be processed as the input image; otherwise, directly use the grayscale image of the crack to be processed as the input image. The specific process is as Figure 3 shown.

[0060] Usually, the grayscale value of crack pixels is much smaller than that of the background area, and subsequent segmentation is carried out on this premise. However, due to background differences, there may also be cases where the grayscale value of crack pixels is much greater than that of the background area. Therefore, it is necessary to invert the grayscale value.

[0061] Figure 4 The specific processes of S4 - S5 are provided, specifically as follows:

[0062] S4: Extract the starting and ending points of each connected component after screening, and add them to the endpoint list. Add the N points (usually determined according to the number of points in the connected components of the image set to be processed to ensure the optimization effect) behind the starting point in the connected component and the N points in front of the ending point in the connected component to the direction list.

[0063] Put all the points in the direction list into the crack pixel list, which is used to judge the end condition and determine the crack pixels after segmentation. When there are large connected components, remove the small connected components to avoid the small connected components being the result of incorrect segmentation and affecting further segmentation.

[0064] Extract one endpoint from the endpoint list and the corresponding N points in the direction list. After preprocessing the input image by mean filtering and adaptive histogram equalization, find the minimum gray value within the M neighborhood (such as 3×3 neighborhood, 5×5 neighborhood, 10×10 neighborhood, etc., which needs to be determined according to the images in the image set to be processed) with the endpoint coordinates as the center point and extract its coordinates.

[0065] S5: Judge whether the condition that the difference between the minimum gray value in the neighborhood and the gray value at the endpoint coordinates < a (which needs to be determined according to the images in the image set to be processed) and the difference between the direction from the endpoint to the minimum gray value and the average direction of the corresponding N points in the direction list < b degrees (usually 90 degrees, but can also be adjusted according to the tortuosity of the crack direction in the actual image) is satisfied.

[0066] When the above condition is not satisfied, then judge whether the M neighborhood has been traversed. If the M neighborhood has not been traversed, find a new minimum gray value and its coordinates again. If the M neighborhood has been traversed, directly enter the judgment of whether the endpoint list has been traversed.

[0067] When the above condition is satisfied, then judge whether there is an overlap between the line connecting the endpoint and the minimum gray value and the points inside the crack of the crack pixels. If there is an overlap, take the coordinates of the points within the line connecting the endpoint to the overlapping position and put them into the crack pixel list, and directly enter the judgment of whether the endpoint list has been traversed; if there is no overlap, take the coordinates within the line connecting the endpoint to the minimum gray value position and put them into the crack pixel list, update the endpoint to the minimum gray value point, and take its adjacent N points as the starting direction, and repeat the operation until entering the judgment of whether the endpoint list has been traversed.

[0068] In this embodiment, when judging whether the endpoint list has been traversed, if not all the points in the endpoint list have been traversed, select the next endpoint in the endpoint list and its corresponding starting direction, and repeat the step of finding the minimum value within the neighborhood.

[0069] If the endpoint list has been traversed, extract all the points in the crack pixel list, and use a special gray value, such as 255, for annotation to obtain the crack binary image, that is, complete the automatic annotation / improvement of the crack semantic segmentation effect.

[0070] S6: Extract the axis of the crack binary image to obtain the optimized crack axis binary image. In the input image, the crack pixels in the crack axis binary image are marked as 1, and the non-crack pixels obtained by taking the inverse after the crack pixels are dilated Q (determined according to the crack width distribution in the image set) times are used as the background and marked as 2. Then, the watershed algorithm is used for filling to obtain the boundary of the crack, thereby completing the optimization of crack segmentation and obtaining the optimized crack grayscale segmentation image. Subsequently, the optimized result after crack segmentation is marked in the crack image to be processed to obtain the optimized crack segmentation image. The specific process is as Figure 5 shown.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for automatic annotation of crack semantic segmentation and improving the segmentation effect, characterized in that, It includes the following steps: Step 1: Perform image preprocessing on the crack image to extract the preliminarily segmented crack axis; Step 2: Conduct connected component analysis on the preliminarily segmented crack axis, extract the area of each connected component and the internal point coordinates, and obtain the filtered connected components according to the screening criteria; Step 3: Calculate the average gray value under the coordinates of each filtered connected component and compare it with the average gray value of the original crack image to determine the input image; specifically: If the average gray value within each connected component is greater than the average gray value of the original crack image, then take the inverse of the crack gray image to be processed as the input image; Otherwise, directly use the crack gray image to be processed as the input image; Step 4: Extract the starting point and ending point of each filtered connected component and add them to the endpoint list; add the adjacent points of the starting point and ending point within the connected component to the direction list, find the minimum gray value within the neighborhood of the endpoints in the endpoint list and extract its coordinates; Step 5: Sequentially judge whether the conditions of gray difference, direction difference, whether the neighborhood has been traversed, whether the endpoint list has been traversed, and whether there is a coincidence between the connection line of the endpoint and the minimum gray value and the positions of the crack pixels within the crack are satisfied until the judgment of whether the endpoint list has been traversed is entered, and obtain the crack binary image; Specifically: Judge whether the condition that the difference between the minimum gray value within the neighborhood and the gray value at the endpoint coordinates < the set threshold and the average direction difference between the direction from the endpoint to the minimum gray value and the corresponding N points in the direction list < the set direction difference is satisfied; When the above condition is not satisfied, then judge whether the neighborhood has been traversed; If yes, directly enter the judgment of whether the endpoint list has been traversed, otherwise find a new minimum gray value and coordinates again; When the above condition is satisfied, then judge whether there is a coincidence between the connection line of the endpoint and the minimum gray value and the positions of the crack pixels within the crack; If yes, take the coordinates of the points within the connection line from the endpoint to the coincidence position and put them into the crack pixel list, and directly enter the judgment of whether the endpoint list has been traversed; Otherwise, take the coordinates within the connection line from the endpoint to the minimum gray value position and put them into the crack pixel list, update the endpoint to the minimum gray value point, and take its adjacent N points as the starting direction, and repeat the operation until the judgment of whether the endpoint list has been traversed is entered; When judging whether the endpoint list has been traversed, if not all points in the endpoint list have been traversed, then select the next endpoint in the endpoint list and its corresponding starting direction, and repeat the operation of finding the minimum gray value within the neighborhood in Step 4; If the endpoint list has been traversed, then extract all points in the crack pixel list and mark them with a special gray value to obtain the crack binary image; Step 6: Extract the axis of the crack binary image to obtain the optimized crack axis binary image, obtain the boundary of the crack in the input image, obtain the optimized crack gray segmentation image, and then mark the optimized result after crack segmentation in the crack gray image to obtain the optimized crack segmentation image; specifically including: Step 6.1: Extract the axis of the crack binary image to obtain the optimized crack axis binary image; Step 6.2: In the input image, using the pixels of the crack axis in the binary crack image as the annotation, and the non-crack pixels obtained by taking the inverse of the dilated crack pixels as the background annotation, perform watershed algorithm filling to obtain the boundary of the crack, thereby completing the optimization of crack segmentation and obtaining the optimized grayscale segmentation map of the crack; Step 6.3: Mark the optimized result after crack segmentation in the crack image to be processed to obtain the optimized crack segmentation map.

2. A method for automatic annotation of crack semantic segmentation and improvement of segmentation effect according to claim 1, characterized in that, The specific steps of Step 1 include: Step 1.1: Convert the crack image to be automatically annotated or whose segmentation effect is to be improved into a grayscale image to obtain the grayscale crack image to be automatically annotated or the grayscale crack image whose segmentation effect is to be improved; Step 1.2: Perform preliminary semantic segmentation on the grayscale crack image to be automatically annotated to obtain the binary image after preliminary semantic segmentation; perform semantic segmentation on the grayscale crack image whose segmentation effect is to be improved to obtain the binary image of the semantic segmentation to be improved; Step 1.3: Perform image morphological dilation operation on the crack region of the binary image after preliminary semantic segmentation or the binary image of the semantic segmentation to be improved, connect the narrow crack discontinuities, smooth the crack contour, and extract the axis of the preliminarily segmented crack.

3. A method for automatic annotation of crack semantic segmentation and improvement of segmentation effect according to claim 1, characterized in that, The specific screening criteria in Step 2 are: For the crack axis map with a connected domain area > X, only retain the connected domains with an area > X; If there is no crack axis map with a connected domain area > X, retain all; where X is the set threshold of the connected domain area.

4. A method for automatic annotation of crack semantic segmentation and improvement of segmentation effect according to claim 1, characterized in that, The specific steps of Step 4 include: Step 4.1: Extract the starting point and ending point of each connected domain after screening and add them to the endpoint list; Step 4.2: Add the adjacent points behind the starting point in the connected domain and the adjacent points in front of the ending point in the connected domain to the direction list; Step 4.3: Put all the points in the direction list into the crack pixel list, which is used to judge the end condition and determine the crack pixels after segmentation is completed; Step 4.4: Extract an endpoint from the endpoint list and the corresponding point from the direction list. After preprocessing the input image with mean filtering and adaptive histogram equalization, find the minimum grayscale value in the neighborhood with the endpoint coordinates as the center point and extract its coordinates.

Images