A crack segmentation method, device, equipment and computer readable storage medium

Abstract

The application discloses a crack segmentation method, device and equipment and a computer readable storage medium. The method comprises the following steps: preprocessing an obtained engineering structure surface image to obtain an input image block; inputting the input image block into a semantic segmentation network; using a direction perception strip hole block in the semantic segmentation network to perform direction differentiation enhancement on extracted crack features to obtain enhanced features; outputting a crack probability map through a mask branch in the semantic segmentation network; outputting a skeleton probability map through a skeleton branch in the semantic segmentation network; wherein the semantic segmentation network is obtained through joint supervision training of a crack mask label and a skeleton label, the skeleton label is generated from the crack mask label and is used for indicating a crack center line position; obtaining a crack mask result according to the crack probability map and obtaining a skeleton result according to the skeleton probability map. The application reduces the probability of crack in the segmentation of fine cracks and takes into account the integrity of the crack area and the continuity of the center line.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a crack segmentation method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the advancement of deep learning technology and the growing demand for automated inspection of engineering structures, crack recognition methods based on semantic segmentation networks have gradually become an important means of detecting surface defects in engineering structures. However, fine cracks on the surface of engineering structures account for a small percentage of pixels in images. When traditional semantic segmentation networks identify such fine cracks, the prediction results are prone to problems such as crack breaks, missing ends, and local discontinuities. In addition, conventional convolutional feature extraction methods mainly focus on pixel relationships within local neighborhoods, and are insufficient in capturing the directional continuity features of fine cracks.

[0003] In summary, effectively addressing issues such as crack fractures, missing ends, and local discontinuities in prediction results, as well as insufficient capture of the directional continuity characteristics of fine cracks, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a crack segmentation method that reduces the fracture of fine cracks while taking into account the integrity of the crack area and the continuity of the centerline; another purpose of this application is to provide a crack segmentation device, equipment and computer-readable storage medium.

[0005] To solve the above-mentioned technical problems, this application provides the following technical solution: A crack segmentation method, comprising: Acquire an image of the surface of the engineering structure and preprocess the image to obtain an input image block; The input image patch is input into a semantic segmentation network. The extracted crack features are enhanced using direction-aware stripe hole blocks within the semantic segmentation network, resulting in enhanced features. This enhancement process includes: enhancing the crack features laterally using a horizontal stripe hole branch; enhancing the crack features vertically using a vertical stripe hole branch; extracting local detail information from the crack features using a local detail branch; and extracting long-range context information from the crack features using a long-range context branch. After extracting the long-range context information from the crack features using the long-range context branch, the process further includes: generating directional weights for each branch based on the crack features; and weighting and fusing the outputs of the horizontal stripe hole branch, the vertical stripe hole branch, the local detail branch, and the long-range context branch based on these directional weights. The enhanced features are output as a crack probability map through the mask branch in the semantic segmentation network, and the enhanced features are output as a skeleton probability map through the skeleton branch in the semantic segmentation network; wherein, the semantic segmentation network is obtained through joint supervised training of crack mask labels and skeleton labels, and the skeleton labels are generated by the crack mask labels and are used to indicate the position of the crack centerline; The crack masking result is obtained based on the crack probability map, and the skeleton result is obtained based on the skeleton probability map.

[0006] In one specific embodiment of this application, the extracted crack features are enhanced with directional differentiation using the direction-aware strip hole blocks in the semantic segmentation network, including: At the high-level feature layer of the encoder, the bottleneck layer between the encoder and the decoder, or the skip connection fusion position between the encoder and the decoder in the semantic segmentation network, the direction-aware strip hole block is used to enhance the direction of the crack feature.

[0007] In one specific embodiment of this application, the process of generating the skeleton label is further included, the process of generating the skeleton label includes: The crack mask label is subjected to noise removal and hole filling; The filled mask is then subjected to skeletonization processing to extract the center line of the crack; The extracted centerline is expanded according to a preset radius to obtain the skeleton label.

[0008] In one specific embodiment of this application, after obtaining the skeleton result based on the skeleton probability map, the method further includes: Perform connected component analysis on the skeleton results and extract the skeleton lines of each connected component; Calculate the spacing between the endpoints of the skeleton lines of adjacent connected components; When the spacing is less than a preset threshold, adjacent connected regions are marked as suspected broken segments, and the number of broken segments and their endpoint positions are output.

[0009] In one specific embodiment of this application, it further includes: A sliding window method is used to divide a large-scale engineering structure surface image into blocks, resulting in multiple input image blocks; Each of the input image blocks is input into the semantic segmentation network to obtain the crack probability map and skeleton probability map corresponding to each image block; For overlapping areas of adjacent image blocks, the crack probability map and skeleton probability map of each image block are fused using the average probability or maximum probability method.

[0010] A crack segmentation device, comprising: An input image block acquisition module is used to acquire surface images of engineering structures and preprocess the surface images of engineering structures to obtain input image blocks; The enhanced feature acquisition module is used to input the input image patch into a semantic segmentation network, and use the direction-aware strip hole block in the semantic segmentation network to perform directional differential enhancement on the extracted crack features to obtain enhanced features. The enhanced feature acquisition module includes: a horizontal direction feature enhancement submodule, used to enhance crack features in the horizontal direction using the horizontal strip hole branch; a vertical direction feature enhancement submodule, used to enhance crack features in the vertical direction using the vertical strip hole branch; a local detail information extraction submodule, used to extract local detail information of crack features using the local detail branch; and a long-distance context information extraction submodule, used to extract long-distance context information of crack features using the long-distance context branch. It also includes: a direction weight generation module, used to generate direction weights corresponding to each branch according to the crack features after extracting long-distance context information of crack features using the long-distance context branch; and a weighted fusion module, used to perform weighted fusion of the outputs of the horizontal strip hole branch, the vertical strip hole branch, the local detail branch, and the long-distance context branch based on the direction weights. The probability map output module is used to output a crack probability map by passing the enhanced features through the mask branch in the semantic segmentation network, and to output a skeleton probability map by passing the enhanced features through the skeleton branch in the semantic segmentation network; wherein, the semantic segmentation network is obtained through joint supervised training of crack mask labels and skeleton labels, and the skeleton labels are generated by the crack mask labels and are used to indicate the position of the crack centerline. The result acquisition module is used to obtain the crack mask result based on the crack probability map and the skeleton result based on the skeleton probability map.

[0011] A crack segmentation device, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the crack segmentation method as described above when executing the computer program.

[0012] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the crack segmentation method as described above.

[0013] The crack segmentation method provided in this application includes: acquiring an image of the surface of an engineering structure and preprocessing the image to obtain an input image block; inputting the input image block into a semantic segmentation network, and using the direction-aware strip hole block in the semantic segmentation network to perform directional differential enhancement on the extracted crack features to obtain enhanced features; wherein, using the direction-aware strip hole block in the semantic segmentation network to perform directional differential enhancement on the extracted crack features includes: using a horizontal strip hole branch to perform horizontal feature enhancement on the crack features; using a vertical strip hole branch to perform vertical feature enhancement on the crack features; using a local detail branch to extract local detail information of the crack features; and using a long-range context branch to extract long-range context information of the crack features. The process involves extracting long-range contextual information from crack features using long-range contextual branches, followed by: generating directional weights for each branch based on the crack features; weighting and fusing the outputs of the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range contextual branch based on the directional weights; outputting a crack probability map through the mask branch in the semantic segmentation network and outputting a skeleton probability map through the skeleton branch in the semantic segmentation network; wherein the semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels, and the skeleton labels are generated by the crack mask labels to indicate the position of the crack centerline; obtaining the crack mask result based on the crack probability map and the skeleton result based on the skeleton probability map.

[0014] As can be seen from the above technical solution, by using direction-aware striped void blocks in the semantic segmentation network to enhance the directional differences of crack features in different directions, the network can simultaneously capture the continuity features of cracks in the horizontal, vertical, and long-distance directions. This overcomes the shortcomings of ordinary convolution, which only focuses on local neighborhoods and is difficult to model the complete structure of slender cracks. Simultaneously, the enhanced features are provided to both the mask branch and the skeleton branch, outputting crack probability maps and skeleton probability maps respectively. Furthermore, the semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels. The skeleton labels are generated by the crack mask labels and indicate the position of the crack centerline. This ensures that the network is simultaneously constrained by both the integrity of the crack region and the continuity of the centerline during the training phase. Therefore, it can solve the technical problems of segmentation breakage, missing ends, and local discontinuities in fine cracks in traditional semantic segmentation networks. It achieves multi-directional and multi-scale feature enhancement of fine cracks, enabling the semantic segmentation network to more completely segment continuous fine cracks, improving its ability to perceive the directional continuity of fine cracks, reducing the segmentation breakage of fine cracks, and balancing the integrity of the crack region and the continuity of the centerline.

[0015] Accordingly, this application also provides a crack segmentation device, equipment, and computer-readable storage medium corresponding to the above-mentioned crack segmentation method, which have the above-mentioned technical effects, and will not be repeated here. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating one implementation of a crack segmentation method in this application. Figure 2 This is a schematic diagram of the structure of a direction-sensing strip cavity block in an embodiment of this application; Figure 3 This is a structural block diagram of a pixel-level fine crack recognition system according to an embodiment of this application; Figure 4 This is a schematic diagram of a mask-skeleton dual-branch supervision and consistency constraint in an embodiment of this application; Figure 5 This is a structural block diagram of a crack segmentation device according to an embodiment of this application; Figure 6 This is a structural block diagram of a crack segmentation device according to an embodiment of this application; Figure 7This is a schematic diagram of the specific structure of a crack segmentation device provided in an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] See Figure 1 , Figure 1 This is a flowchart illustrating an implementation of a crack segmentation method according to an embodiment of this application. The method may include the following steps: S101: Acquire the surface image of the engineering structure and preprocess the surface image of the engineering structure to obtain the input image block.

[0021] Images of the engineering structure's surface are acquired. These images can be obtained from drone aerial photography, close-up images from inspection vehicles, handheld cameras, robot cameras, or fixed monitoring cameras. The image objects can include at least one of the following: bridge beam bottoms, bridge piers, tunnel linings, road surfaces, underground utility tunnels, or the surface of concrete components. For clarity, the following explanation uses the segmentation of fine cracks on a bridge surface as an example.

[0022] The acquired engineering structure surface image undergoes preprocessing, which may include at least one of the following: image cropping, size unification, format unification, normalization, brightness equalization, blur removal, and data augmentation. For example, images may be uniformly adjusted to a preset size, or grayscale and RGB format images may be uniformly converted to the same format. Furthermore, brightness equalization may be applied to reduce the impact of uneven illumination on subsequent segmentation. Images with excessive blur or indistinguishable crack details can be directly discarded. Through preprocessing of the engineering structure surface image, input image blocks are obtained.

[0023] S102: Input the input image patch into the semantic segmentation network, and use the direction-aware strip hole block in the semantic segmentation network to perform directional differential enhancement on the extracted crack features to obtain the enhanced features.

[0024] The method involves using direction-aware strip hole blocks in a semantic segmentation network to enhance the extracted crack features through directional differentiation. This includes: enhancing crack features laterally using a horizontal strip hole branch; enhancing crack features vertically using a vertical strip hole branch; extracting local detail information from crack features using a local detail branch; extracting long-range context information from crack features using a long-range context branch; and after extracting long-range context information from crack features using a long-range context branch, the method further includes: generating directional weights for each branch based on crack features; and weighting and fusing the outputs of the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch based on the directional weights.

[0025] The preprocessed input image patch is fed into the semantic segmentation network. The semantic segmentation network classifies each pixel in the input image patch to distinguish between crack pixels and background pixels. The semantic segmentation network can include an encoder and a decoder. The encoder extracts multi-scale features layer by layer from shallow texture to deep semantics, and the decoder upsamples layer by layer to restore spatial resolution so that the size of the output segmentation result is consistent with the size of the input image. The semantic segmentation network can be any of the following: U-Net, nested U-Net (UNet++), Attention U-Net, Residual U-Net, DeepLabv3+, Pyramid Scene Parsing Network (PSPNet), Semantic Segmentation Transformer (SegFormer), or Sliding Window Transformer U-Net (Swin-Unet). For ease of explanation, the following description will primarily use the U-Net as an example.

[0026] After the input image patch enters the semantic segmentation network, the encoder in the semantic segmentation network extracts features from the input image patch, obtaining crack features at different levels. The semantic segmentation network embeds direction-aware striped hole blocks, which are used to perform directional differential enhancement on the extracted crack features. Directional differential enhancement refers to targeted feature enhancement based on different directional attributes of the crack features, enabling the network to better capture the continuous features of the crack in different directions. Unlike conventional square convolutional kernels, direction-aware striped hole blocks can process features of the crack in different directions, such as horizontal and vertical, thus adapting to the morphological characteristics of fine cracks extending in different directions. The features enhanced by the direction-aware striped hole blocks are the enhanced features.

[0027] It should be noted that there are multiple ways to implement the direction-aware striped perforated block. The direction-aware striped perforated block can perform directional decomposition on the extracted crack features, for example, by performing striped perforated convolution operations on the crack features along the horizontal and vertical directions respectively, thereby extracting the feature responses of the crack in different directions. The direction-aware striped perforated block can combine multi-scale perforated convolution, expanding the receptive field by setting different perforation rates to adapt to cracks of different widths and curvatures. The direction-aware striped perforated block can also include adaptive direction weights, automatically adjusting the enhancement intensity in different directions based on the input features. The above implementation methods are not mutually exclusive and can be combined as needed.

[0028] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a direction-aware strip hole block in an embodiment of this application. After the input feature X enters the direction-aware strip hole block, it can first undergo 1×1 convolution for channel compression or channel transformation to obtain intermediate feature X'. Subsequently, the intermediate feature X' enters four parallel branches: the horizontal strip hole branch, the vertical strip hole branch, the local detail branch, and the long-range context branch.

[0029] Lateral strip hollowing is used to extract features of fine cracks extending horizontally. Lateral strip hollowing can employ a 1×k strip convolution or its hollowed-out form, where k represents the size of the convolution kernel along the horizontal direction. k can be set according to the actual crack width characteristics; for example, k can be 5, 7, 9, or other odd numbers. When the crack to be detected mainly extends horizontally and is narrow, a larger k value can be set, such as k=9, to capture longer horizontal continuity features; when the crack direction is relatively random, a moderate k value can be set, such as k=5 or k=7. Lateral strip hollowing can also be implemented using hollowed-out convolution, introducing a hollowing rate d into the strip convolution to further expand the receptive field without increasing the number of parameters. The hollowing rate d can be set as needed, for example, to 1, 2, 4, etc. When the crack extends significantly horizontally and the spacing between adjacent cracks is large, a larger hollowing rate can be set, such as d=4; when the cracks are densely distributed, a smaller hollowing rate can be set, such as d=1 or d=2. Transverse strip void branches are suitable for enhancing the characteristic response of transverse cracks and near-transverse crack segments.

[0030] Longitudinal strip void branches are used to extract features of fine cracks extending vertically. Longitudinal strip void branches can employ k×1 strip convolution or its voided form. Similar to transverse strip void branches, the kernel size k and void ratio d of the longitudinal strip void branch can be set according to the longitudinal extension characteristics of the crack. For example, when the cracks at the bottom of a bridge beam are mainly vertical cracks, a larger longitudinal k value can be set, such as k=9, while a moderate void ratio, such as d=2, can be set to enhance the feature capture capability of vertical cracks. Longitudinal strip void branches are suitable for enhancing longitudinal cracks, vertical crack branches, and crack structures in longitudinal textures.

[0031] Local detail branches are used to extract detailed information such as crack edges, endpoints, and local textures. Local detail branches can employ a standard 3×3 convolution operation. Since strip convolution primarily focuses on the extension direction of linear structures, it may overlook local details at crack edges and endpoints. Therefore, setting up local detail branches can compensate for the shortcomings of strip convolution in extracting local details. A 3×3 convolution can model the pixel relationships within a local 3×3 neighborhood, effectively capturing local texture variations, edge transitions, and subtle branches and endpoints of the crack. In practical applications, the kernel size for local detail branches is not limited to 3×3; other conventionally sized square convolution kernels, such as 5×5, can also be used.

[0032] Long-range context branching is used to extract long-range contextual information of curved cracks. Long-range context branching can employ dilated convolution, deformable convolution, or large-kernel convolution to expand the receptive field. Dilated convolution, by setting a larger dilation rate, achieves a larger receptive field, thereby capturing the contextual relationships of the same crack over a wider area. For example, when the crack exhibits a significant curved shape, a larger dilation rate, such as d=4 or d=6, can be set to enable the network to perceive the continuity of the crack along the entire curved path. Deformable convolution can be used in long-range context branching, adaptively adjusting the sampling position of the convolution kernel to adapt to the irregular curved shape of the crack. Long-range context branching can reduce the problem of the same crack being segmented into multiple discontinuous small segments.

[0033] The four branches can run in parallel, sequentially in a preset order, or a combination of partially parallel and partially sequential operations. The outputs of each branch can be merged according to a preset fusion method. The outputs of the four branches can be concatenated through channels and then fused using a 1×1 convolution, or the outputs of the four branches can be added element-wise.

[0034] By employing horizontal and vertical stripe hole branches to capture the continuity features of cracks in different directions, utilizing local detail branches to ensure that detailed information such as crack edges and endpoints is not lost, and utilizing long-distance context branches to capture the contextual relationships of cracks over a larger range, multi-directional and multi-scale feature enhancement of fine cracks is achieved, enabling the semantic segmentation network to more completely segment continuous fine cracks.

[0035] After obtaining the outputs of the four branches—horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch—directional weight fusion is performed. This adaptively generates directional weights for each branch based on the crack features, and then weights the outputs of each branch based on these directional weights. Directional weight fusion is not simply a fixed addition or concatenation of the branch outputs; rather, it automatically determines which branch should dominate the response for crack features of different directions and shapes based on the characteristics of the input features.

[0036] The intermediate feature X' or the original feature X input to the orientation-aware strip hole block is subjected to global average pooling, followed by channel compression via 1×1 convolution. Then, it is activated by a sigmoid activation function or a normalized exponential (softmax) activation function to generate orientation weights for each branch. For example, orientation weights αH for the horizontal strip hole branch, αV for the vertical strip hole branch, αL for the local detail branch, and αC for the long-range context branch can be generated. The outputs of the four branches can then be weighted and fused using the following formula: ; Where Y represents the output feature after orientation enhancement; H(X'), V(X'), L(X'), and C(X') represent the outputs obtained after processing the intermediate feature X' by the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch, respectively; αH, αV, αL, and αC represent the orientation weights corresponding to the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch, respectively; concat represents the channel concatenation operation; Conv1×1 represents the 1×1 convolution operation; X represents the original input feature input to the orientation-aware strip hole block, which is added to the fused feature through a residual connection (i.e., +X) to alleviate the gradient vanishing problem in deep networks and preserve the original information. The orientation weights can be generated by the Sigmoid function, with each weight independent and ranging from 0 to 1, or they can be generated by the Softmax function, with the sum of all weights being 1. For example, when using the Sigmoid function to generate weights, the weights of each branch are calculated independently and are not affected by the weights of other branches; when using the Softmax function to generate weights, the sum of the weights of the four branches is normalized to 1, and there is a competitive relationship between the weights.

[0037] By setting directional weights, the model increases the response weights of the lateral branches when the cracks mainly extend horizontally, increases the response weights of the longitudinal branches when the cracks mainly extend vertically, and enhances the response weights of the local detail branches and long-distance context branches when the cracks are curved, discontinuous, or have many endpoints. Through adaptive adjustment of directional weights, differentiated feature enhancement for cracks of different shapes and directions is achieved, thereby improving the semantic segmentation network's ability to perceive the continuity of fine crack directions.

[0038] S103: The enhanced features are output as a crack probability map through the mask branch in the semantic segmentation network, and the enhanced features are output as a skeleton probability map through the skeleton branch in the semantic segmentation network.

[0039] The semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels. The skeleton labels are generated from the crack mask labels and are used to indicate the location of the crack centerline.

[0040] After obtaining enhanced features from the orientation-aware strip hole block, the enhanced features are gradually upsampled by the decoder to restore a spatial resolution matching the input image block. At the end of the decoder, there are two output branches: a mask branch and a skeleton branch. The mask branch outputs a crack probability map based on the enhanced features, where the value of each pixel in the crack probability map represents the probability that the pixel belongs to a crack region. The skeleton branch outputs a skeleton probability map based on the enhanced features, where the value of each pixel in the skeleton probability map represents the probability that the pixel belongs to the crack centerline.

[0041] Semantic segmentation networks require training before use. During training, the semantic segmentation network is jointly supervised by crack mask labels and skeleton labels. Crack mask labels are pixel-by-pixel binary annotations of crack regions in an image, marking crack pixels as 1 (or using other preset first label values) and background pixels as 0 (or using other preset second label values). Skeleton labels are generated from crack mask labels through skeletonization and dilation, and are used to indicate the location of the crack's centerline. The centerline of the crack region can be extracted by skeletonizing the crack mask labels, thus obtaining the skeleton labels. As an additional supervisory signal, the skeleton labels, together with the crack mask labels, guide the training of the semantic segmentation network, enabling the network to learn not only the boundaries of the crack regions but also the continuity features of the crack centerline.

[0042] The mask branch and the skeleton branch share the enhanced features of the orientation-aware strip hole blocks, but each has independent output layer parameters. During the network's forward propagation, the enhanced features flow into both the mask branch and the skeleton branch simultaneously. Each branch performs its own convolution operation, ultimately outputting a crack probability map and a skeleton probability map, respectively. This approach allows the semantic segmentation network to simultaneously obtain crack region information (crack probability map) and crack centerline information (skeleton probability map) in a single inference, eliminating the need to run two separate network models.

[0043] S104: Obtain the crack masking result based on the crack probability map, and obtain the skeleton result based on the skeleton probability map.

[0044] After obtaining the crack probability map and the skeleton probability map, the crack masking result and the skeleton result are obtained based on the crack probability map and the skeleton probability map, respectively.

[0045] For the crack probability map, thresholding can be performed based on a preset threshold or an adaptive threshold to obtain a crack masking result. The crack masking result is a binary image, where pixels with values ​​greater than or equal to the threshold are identified as crack pixels, and pixels with values ​​less than the threshold are identified as background pixels. The threshold can be set according to the actual situation. For example, in scenarios where the integrity of fine cracks is highly important, such as bridge structural safety assessment, a lower probability threshold, such as 0.3 to 0.4, can be set to retain more crack candidate pixels; in scenarios where the false detection rate is high, such as preliminary screening, a higher threshold, such as 0.6 to 0.7, can be set to reduce false positives in non-crack areas. The adaptive threshold can be dynamically determined based on the overall probability distribution of the crack probability map, for example, the segmentation threshold can be automatically selected based on the Otsu method.

[0046] For the skeleton probability map, a preset threshold or an adaptive threshold can also be used to binarize the skeleton probability map to obtain the skeleton result. The skeleton result can be a binary skeleton image or extracted skeleton line coordinate data. The threshold of the skeleton result can be set independently of the threshold of the crack mask result. For example, in scenarios requiring detailed analysis of crack connectivity, a lower skeleton threshold, such as 0.2 to 0.3, can be set; in scenarios where only a general determination of crack direction is needed, a higher skeleton threshold, such as 0.5 to 0.6, can be set.

[0047] As can be seen from the above technical solution, by using direction-aware striped hole blocks in the semantic segmentation network to enhance the directional differences of crack features in different directions, the network can simultaneously capture the continuity features of cracks in the horizontal, vertical, and long-distance directions. This overcomes the shortcomings of ordinary convolution, which only focuses on local neighborhoods and is difficult to fully model the structure of slender cracks. Simultaneously, the enhanced features are provided to both the mask branch and the skeleton branch, outputting crack probability maps and skeleton probability maps respectively. Furthermore, the semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels. The skeleton labels are generated by the crack mask labels and indicate the position of the crack centerline. This ensures that the network is simultaneously constrained by both the integrity of the crack region and the continuity of the centerline during the training phase. Therefore, it can solve the technical problems of segmentation breakage, missing ends, and local discontinuities in fine cracks in traditional semantic segmentation networks. It achieves multi-directional and multi-scale feature enhancement of fine cracks, enabling the semantic segmentation network to more completely segment continuous fine cracks, improving its ability to perceive the directional continuity of fine cracks, alleviating the problem of segmentation breakage, and achieving a technical effect that balances the integrity of the crack region and the continuity of the centerline.

[0048] It should be noted that, based on the above embodiments, this application also provides corresponding improvement solutions. In subsequent embodiments, steps that are the same as or corresponding to those in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other. These improvements will not be elaborated upon in the following improved embodiments.

[0049] In one specific embodiment of this application, the extraction of crack features is enhanced by directional differentiation using direction-aware strip hole blocks in a semantic segmentation network, which may include the following steps: In the high-level feature layer of the encoder, the bottleneck layer between the encoder and the decoder, or the skip connection fusion position between the encoder and the decoder in the semantic segmentation network, the direction-aware strip hole block is used to enhance the direction of the crack feature.

[0050] The embedding position of the orientation-aware strip hole block in the semantic segmentation network can be flexibly set according to the feature distribution of the crack and the specific architecture of the semantic segmentation network.

[0051] Direction-aware striped aperture blocks can be embedded in the higher-level feature layers of the encoder. The features output from these higher-level feature layers have a high degree of semantic abstraction and a low spatial resolution. Embedding direction-aware striped aperture blocks at this location allows for directional enhancement of the overall crack orientation using high-level semantic features. Furthermore, since the spatial resolution of the features has decreased at this stage, the computational cost of striped aperture convolution is relatively low. For example, when using U-Net as the semantic segmentation network, direction-aware striped aperture blocks can be embedded after the last or penultimate convolutional layer of the encoder.

[0052] Orientation-aware stripe hole blocks can also be embedded in the bottleneck layer between the encoder and decoder. The bottleneck layer is the transition layer between the encoder and decoder, where the features have the highest level of semantic abstraction. Embedding orientation-aware stripe hole blocks in the bottleneck layer can globally enhance the orientation of the crack features, providing an orientation-enhanced feature representation for subsequent upsampling recovery by the decoder.

[0053] Orientation-aware stripe holes can also be embedded at skip connection fusion locations between the encoder and decoder. At these fusion locations, shallow high-resolution features from the encoder are fused with deep features upsampled by the decoder. Embedding orientation-aware stripe holes at these locations allows for orientation enhancement of the fused features by utilizing both shallow detail information and deep semantic information. For example, at a skip connection layer in U-Net, the features from the encoder and decoder are first concatenated or added together, and then orientation-aware stripe holes are applied to the fused features for enhancement.

[0054] It should be noted that the three embedding locations mentioned above are not mutually exclusive. In practical applications, orientation-aware stripe hole blocks can be embedded simultaneously at multiple locations within a semantic segmentation network as needed. For example, one orientation-aware stripe hole block can be embedded simultaneously in the last layer of the encoder and in the bottleneck layer. The choice of embedding location depends on the specific characteristics of the cracks and the design of the network architecture. In scenarios where crack orientation changes are complex, with both obvious horizontal and vertical cracks, orientation-aware stripe hole blocks can be embedded at multiple locations to achieve more comprehensive orientation enhancement. In scenarios with high computational efficiency requirements, only one orientation-aware stripe hole block can be embedded in the bottleneck layer to achieve orientation enhancement while maintaining computational efficiency.

[0055] By embedding direction-aware strip hole blocks into different locations in the semantic segmentation network, it is possible to flexibly adapt to crack features at different levels, achieve multi-level direction differentiation enhancement, and further improve the network's ability to perceive and segment the direction continuity of fine cracks.

[0056] In one specific embodiment of this application, the method may further include a skeleton tag generation process, which may include the following steps: Step 1: Noise removal and hole filling for cracked masking labels; Step 2: Perform skeletonization processing on the filled mask and extract the center line of the crack; Step 3: Expand the extracted centerline according to a preset radius to obtain the skeleton label.

[0057] For ease of description, the three steps above can be combined for explanation.

[0058] During the training phase of a semantic segmentation network, crack mask labels and skeleton labels are needed as supervision signals. Crack mask labels can be obtained through manual or semi-automatic annotation, while skeleton labels are automatically generated from crack mask labels.

[0059] First, noise removal and hole filling are performed on the crack mask labels. Noise removal refers to removing small, isolated noise points in the crack mask labels, which may originate from mislabeling during manual annotation or artifacts introduced during image preprocessing. Noise removal can be achieved through morphological opening operations, i.e., first performing an erosion operation on the crack mask to remove small noise points, and then performing a dilation operation to restore the main crack area. Hole filling refers to filling the small holes inside the crack area, which may be caused by incomplete annotation or the presence of filler material inside the crack. Hole filling can be achieved through morphological closing operations, i.e., first dilation followed by erosion, or by identifying and filling internal hole regions through connected component analysis.

[0060] After noise removal and hole filling, skeletonization is performed on the filled mask to extract the centerline of the crack. Skeletonization is the process of reducing the crack region to a centerline with a width of one or a few pixels, transforming cracks of different widths into centerlines of uniform width. Skeletonization can employ any of the following algorithms: morphological thinning (such as the Zhang-Suen thinning algorithm), distance transform centerline extraction algorithm, or topology-preserving thinning algorithm. Morphological thinning algorithms iteratively remove pixels from the crack region's edges while maintaining the crack's topological connectivity and endpoints, gradually thinning the crack region to a single-pixel-wide centerline. Distance transform centerline extraction algorithms first calculate the distance from each pixel within the crack region to the nearest background pixel, then extract the centerline along the ridge of the distance value. Different skeletonization algorithms may produce slightly different extraction results; the appropriate algorithm should be selected based on the actual morphological characteristics of the crack. For example, for relatively straight cracks with uniform width, the Zhang-Suen thinning algorithm can be used to obtain regular skeleton lines; for curved cracks with large width variations, the distance transformation method can be used to obtain skeleton lines that better match the actual direction of the crack.

[0061] After extracting the centerline of the crack, the extracted centerline is dilated by a preset radius to obtain the skeleton label. The purpose of the dilation is to avoid the adverse effects of differences in crack boundary thickness in manual annotation on training, while maintaining the constraint of the continuity of the crack centerline. Through dilation, the centerline, originally one pixel wide, is expanded into a strip-shaped supervision region with a preset radius, giving the network a certain margin of error when learning skeleton features and improving training stability. The preset radius can be set according to image resolution, crack width, and annotation quality. For example, when the image resolution is high (e.g., 1024×1024 pixels) and the crack is wide (e.g., crack width approximately 5 to 10 pixels), a larger preset radius (e.g., 3 pixels) can be set to ensure that the skeleton supervision region can better cover the crack center area; when the image resolution is low (e.g., 256×256 pixels) and the crack is thin (e.g., crack width approximately 1 to 3 pixels), a smaller preset radius (e.g., 1 pixel) can be set to avoid the skeleton supervision region exceeding the actual range of the crack. Generally, the preset radius can be set to 1 to 3 pixels. The result of the expansion is the skeleton label, which is used to supervise the skeleton branches during subsequent training.

[0062] It should be noted that the skeleton label is not used to replace the crack mask label, but rather as an additional supervisory signal that works together with the crack mask label in the network training. The crack mask label constrains the model to identify crack regions (i.e., region integrity), while the skeleton label constrains the model to identify the crack centerline and connectivity (i.e., centerline continuity). The joint training of the two enables the model to obtain complete crack regions while reducing crack breaks and missed end detections.

[0063] In one specific embodiment of this application, after obtaining the skeleton result based on the skeleton probability map, the method may further include the following steps: Step 1: Perform connected component analysis on the skeleton results and extract the skeleton lines of each connected component; Step 2: Calculate the spacing between the endpoints of the skeleton lines of adjacent connected components; Step 3: When the spacing is less than the preset threshold, mark the adjacent connected components as suspected broken segments and output the number of broken segments and their endpoint positions.

[0064] For ease of description, the three steps above can be combined for explanation.

[0065] After obtaining the skeleton results, further post-processing analysis can be performed on the skeleton results to extract the fracture information and endpoint location of the cracks, providing reference data for engineering assessment and maintenance decisions.

[0066] First, connected component analysis is performed on the skeleton result to extract each connected component, obtaining the skeleton line corresponding to each connected component. Connected component analysis can employ connected component labeling algorithms, such as the Two-Pass algorithm or the Seed-Filling algorithm. By scanning each foreground pixel in the skeleton result and assigning a unique identifier to each connected component, interconnected skeleton pixels are grouped into the same connected component. The skeleton line of each connected component is the set of all skeleton pixels within that connected component.

[0067] Next, the spacing between the endpoints of the skeleton lines of adjacent connected components is calculated. For each skeleton line of a connected component, the endpoints of the skeleton line are first identified. An endpoint is a pixel in the skeleton line that has only one adjacent foreground pixel (or has one adjacent foreground pixel). This can be done by traversing each pixel in the skeleton line and counting the number of foreground pixels in its 8-neighborhood. After identifying the endpoints of all connected components, the Euclidean distance between the endpoints of each pair of adjacent connected components is calculated. The determination of adjacent connected components can be defined according to actual needs. For example, the K spatially closest pairs of connected components can be considered adjacent connected components, or pairs of connected components with endpoint spacing within a certain range (e.g., spacing not exceeding 50 pixels) can be considered adjacent connected components.

[0068] Next, it is determined whether the distance between the endpoints of the skeleton lines of adjacent connected components is less than a preset threshold. The preset threshold is used to determine whether two adjacent connected components belong to two segments caused by the breakage of the same crack. The preset threshold can be set according to the image resolution and the actual distribution of cracks. For example, for high-resolution engineering images (e.g., a single pixel corresponds to an actual size of 0.1 mm), if the crack breakage distance is within 1 mm, it is considered the same crack, and the preset threshold can be set to 10 pixels; for low-resolution engineering images (e.g., a single pixel corresponds to an actual size of 0.5 mm), if the crack breakage distance is within 1 mm, it is considered the same crack, and the preset threshold can be set to 2 pixels. Generally, the preset threshold can be set to 3 to 15 pixels according to the typical width of the crack and the engineering acceptance standards.

[0069] When the distance between the endpoints of the skeleton lines of adjacent connected components is less than a preset threshold, the adjacent connected components are marked as suspected breaks. A suspected break indicates that the adjacent connected components are highly likely to belong to two segments of the same crack, and the break occurred during the segmentation process due to local discontinuities predicted by the network. In this case, the number of broken segments and all endpoint positions can be output. The endpoint positions can be output in pixel coordinates or actual engineering coordinates for subsequent manual verification or further crack continuity analysis.

[0070] When the distance between the endpoints of the skeleton lines of adjacent connected regions is greater than or equal to a preset threshold, the adjacent connected regions can be considered to belong to different cracks and are not marked as suspected fracture segments. In addition, after identifying suspected fracture segments, it is possible to further determine whether the fracture segments need to be connected and repaired based on the endpoint positions of adjacent fracture segments and the direction of the crack. For example, connecting lines at the fracture points can be added to the visualization results to provide engineers with a more intuitive overall view of the crack.

[0071] Based on the skeleton results, connected component analysis and fracture detection can automatically identify and mark possible predicted fractures in the semantic segmentation network, providing engineers with quantitative information such as the number of fracture segments and endpoint locations of cracks, and assisting in post-processing connection of cracks and engineering safety assessment.

[0072] In one specific embodiment of this application, the method may further include the following steps: Step 1: Divide the large-scale engineering structure surface image into blocks using a sliding window method to obtain multiple input image blocks; Step 2: Input each input image block into the semantic segmentation network to obtain the crack probability map and skeleton probability map corresponding to each image block; Step 3: For the overlapping areas of adjacent image blocks, fuse the crack probability map and skeleton probability map of each image block using the average probability or maximum probability method.

[0073] For ease of description, the three steps above can be combined for explanation.

[0074] For large-scale engineering surface images, such as panoramic images with a resolution of 4000×6000 pixels or higher, directly inputting them into a semantic segmentation network may exceed the network's input size limit or lead to insufficient video memory. Therefore, a sliding window approach can be used to segment large images into blocks. Sliding window segmentation involves using a fixed-size window to slide across the image, extracting one window region at a time as the input image block. The window size can be set according to the preset input size of the semantic segmentation network. For example, if the network's preset input size is 512×512 pixels, the window size can be set to 512×512 pixels. The step size of the sliding window can be set to the window's side length minus the size of the overlapping region. The overlapping region is used to ensure that cracks are not truncated when located at the boundary of an image block, and to maintain a certain pixel overlap between adjacent image blocks. The size of the overlapping region can be set according to the characteristics of the actual image. For example, when the cracks in the image to be processed are relatively dense, a larger overlap area can be set (e.g., the overlap area is 20% to 25% of the window side length) to reduce the impact of crack truncation at the boundary; when the cracks in the image to be processed are relatively sparse, a smaller overlap area can be set (e.g., the overlap area is 10% to 15% of the window side length) to improve processing efficiency. A large image is divided into several input image blocks with overlapping areas through sliding window scanning.

[0075] Each input image block is fed into the semantic segmentation network to obtain a crack probability map and a skeleton probability map for each input image block. The processing of each input image block can be sequential, i.e., inputting them into the network one by one, or parallel, i.e., batch processing multiple image blocks when hardware resources permit, without any restriction.

[0076] For overlapping regions of adjacent image patches, the crack probability map and skeleton probability map of each image patch are fused using either average probability or maximum probability. Since there are overlapping regions between adjacent input image patches, pixels in the overlapping regions are predicted separately by the adjacent image patches, resulting in multiple prediction results. To merge the prediction results of each image patch into a unified result for the entire image, it is necessary to fuse the prediction probabilities of the overlapping regions.

[0077] An average probability fusion method can be used. For each pixel in the overlapping region, the average of the predicted probabilities of each image patch for that pixel is taken as the final predicted probability. The average probability fusion method can combine the prediction results of multiple image patches, reduce the random fluctuation of individual prediction results, and is suitable for scenarios where crack features are relatively uniform and the prediction quality of adjacent image patches is similar.

[0078] The method also employs maximum probability fusion, where for each pixel in the overlapping region, the maximum predicted probability of that pixel across all image blocks is taken as the final predicted probability. Maximum probability fusion preserves the strong response of crack pixels and is suitable for scenarios where cracks are fine and there is a risk of local breakage. For example, when a crack shows a slight decrease in predicted probability near the boundary of an image block, maximum probability fusion can select a higher predicted probability value from adjacent image blocks, thereby reducing the risk of crack breakage at the block boundary.

[0079] After fusion, the crack mask and skeleton results for the entire large image are obtained, facilitating engineers to perform overall analysis and evaluation of cracks in the entire image. By using sliding window segmentation and overlapping region fusion, the problem of large images not being able to be directly input into semantic segmentation networks is solved, while ensuring the continuity of cracks at image block boundaries.

[0080] It should be noted that, in this embodiment of the application, the semantic segmentation network can be trained using the following joint loss function: ; Where L_total represents the total loss; M is the crack mask label; S is the skeleton label; P_m is the crack probability map output by the mask branch of the semantic segmentation network; P_s is the skeleton probability map output by the skeleton branch of the semantic segmentation network; Skel(P_m) represents the predicted skeleton obtained by differentiable approximation skeletonization of the crack probability map P_m (e.g., using max pooling morphological approximation); L_focal is the focal loss, used to solve the class imbalance problem caused by fewer crack pixels and more background pixels; L_dice is the Dice loss, used to improve the overlap between the predicted region and the real crack region; L_skel is the skeleton loss, used to constrain the continuity of the crack centerline, and can be any of the binary cross-entropy loss, Dice loss, Tversky loss, or centerline distance loss; L_cons is the mask-skeleton consistency loss. Focal Loss is used to constrain the predicted skeleton Skel(P_m) derived from the mask branch to maintain consistency with the skeleton probability map P_s output by the skeleton branch, thus avoiding inconsistencies between skeleton predictions and crack region predictions. λ1, λ2, λ3, and λ4 are preset weighting coefficients used to control the contribution ratio of each loss to the total loss. The weighting coefficients λ1, λ2, λ3, and λ4 can be set based on the training data and validation set results. For example, when the proportion of crack pixels is extremely low (e.g., less than 1%), the value of λ1 can be increased to enhance the mitigation effect of Focal Loss on class imbalance; when crack connectivity is crucial for engineering evaluation, the values ​​of λ3 and λ4 can be increased to strengthen skeleton and consistency constraints.

[0081] During training, the parameters of the semantic segmentation network can be updated based on the joint loss function using the Adaptive Moment Estimation (Adam) optimizer, the AdamW optimizer, or the Stochastic Gradient Descent (SGD) optimizer. A maximum number of training epochs can be preset during training. Training is stopped and the model parameters are saved when the loss value on the validation set no longer decreases within a certain number of consecutive epochs or reaches the maximum number of training epochs. On the validation set, multiple metrics can be comprehensively evaluated, including Intersection over Union (IoU), Dice coefficient, precision, recall, F1 score, and skeleton continuity. The model weights with the optimal comprehensive metrics are selected as the final semantic segmentation network parameters.

[0082] Furthermore, the semantic segmentation network in this embodiment can further reduce missed detections at crack ends by introducing an optional endpoint loss. The endpoint loss can be implemented by identifying the endpoint positions of the crack skeleton lines and imposing additional supervisory constraints. The total loss function can be extended as follows: ; Where L_end is the endpoint loss, and λ5 is the preset endpoint loss weight coefficient. This endpoint loss is an optional enhancement and can be enabled or disabled depending on the actual application scenario.

[0083] In this embodiment, the skeletonization of the skeleton label can be achieved, but is not limited to, using any one of the following algorithms: morphological thinning algorithm, distance transformation centerline extraction algorithm, or topology-preserving thinning algorithm. The strip convolutions in the transverse and longitudinal strip hole branches of the direction-aware strip hole block can be replaced with any one of the following: spatially separable convolution, large kernel convolution, deformable convolution, directional convolution, rotational convolution, strip attention, or axial attention. The kernel size k of the transverse and longitudinal strip hole branches can be 3, 5, 7, 9, 11, or other values ​​besides those described above. The hole ratio d can be 1, 2, 3, 4, 6, or other values ​​besides those described above. The branches can use the same k and d values, or different k and d values. The number of parallel branches can also be expanded from four to include multiple branches with more directions, such as 0°, 45°, 90°, 135°, etc.

[0084] See Figure 3 , Figure 3This is a structural block diagram of a pixel-level fine crack recognition system according to an embodiment of this application. System module structure (in order of data flow): Data acquisition and preprocessing: The image acquisition module is responsible for acquiring images of the engineering structure surface, with image sources including cameras, drones, inspection vehicles, and other equipment; the data preprocessing module performs slicing, enhancement, and skeleton label generation on the acquired images. Model training phase: The model training module uses the joint loss function to train the model and optimize the parameters; Core network structure (connected to training output): The dual-output module receives the trained network from the model training module and outputs a mask map and a skeleton map respectively; the orientation-aware strip hole feature enhancement module is embedded in the network and uses multi-directional convolution with strip holes to differentially enhance the crack features in different directions; the improved U-Net module provides encoder, decoder and skip connections, and is the backbone structure for feature extraction and recovery. Output and Evaluation: The results evaluation module receives the raw output from the improved U-Net module, calculates evaluation indicators such as crossover ratio, data coefficients, F1 score, and continuity, and verifies the model performance; the application output module receives the final mask map and skeleton map from the dual output module and outputs engineering application information such as crack location, area, length, and fracture segment.

[0085] See Figure 4 , Figure 4 This diagram illustrates a mask-skeleton dual-branch supervision and consistency constraint in an embodiment of this application. The input layer takes image I, the surface image of the engineering structure to be identified, as input to the entire network model. An improved U-Net with a direction-aware stripe hole module extracts features from image I, and direction-aware stripe hole blocks enhance the directionality of crack features. Then, a dual-branch output is performed. The backbone network splits into two parallel output branches at the decoder end, each handling different supervision tasks. The mask branch outputs a crack probability map (each pixel represents the probability of belonging to a crack region), which is compared with the mask label M to supervise the integrity of the crack region. The skeleton branch outputs a skeleton probability map (each pixel represents the probability of belonging to the crack centerline), which is compared with the skeleton label S to supervise the continuity of the crack centerline. Consistency constraints are applied to the crack probability map and the skeleton probability map, aligning the prediction results from the mask branch and the skeleton branch to ensure that the crack region prediction obtained by the mask branch and the crack centerline prediction obtained by the skeleton branch are spatially consistent, avoiding contradictions such as the mask predicting the presence of a crack but the skeleton predicting no centerline, or vice versa.

[0086] The method provided in this application embodiment can be deployed on local servers, edge computing devices, inspection vehicles, drones, robots, mobile terminals or cloud platforms, and is applicable to the segmentation and identification of surface cracks in bridges, tunnels, road surfaces, pipe corridors and concrete components.

[0087] Corresponding to the above method embodiments, this application also provides a crack segmentation device, and the crack segmentation device described below can be referred to in correspondence with the crack segmentation method described above.

[0088] See Figure 5 , Figure 5 This is a structural block diagram of a crack segmentation device according to an embodiment of this application. The device may include: The input image block acquisition module 51 is used to acquire the surface image of the engineering structure and preprocess the surface image of the engineering structure to obtain the input image block; The enhanced feature acquisition module 52 is used to input the input image patch into the semantic segmentation network, and use the direction-aware strip hole block in the semantic segmentation network to perform directional differential enhancement on the extracted crack features to obtain enhanced features. The enhanced feature acquisition module includes: a horizontal direction feature enhancement submodule, used to enhance the crack features in the horizontal direction using the horizontal strip hole branch; a vertical direction feature enhancement submodule, used to enhance the crack features in the vertical direction using the vertical strip hole branch; a local detail information extraction submodule, used to extract the local detail information of the crack features using the local detail branch; and a long-distance context information extraction submodule, used to extract the long-distance context information of the crack features using the long-distance context branch. It also includes: a direction weight generation module 53, used to generate the direction weights corresponding to each branch according to the crack features after extracting the long-distance context information of the crack features using the long-distance context branch; and a weighted fusion module 54, used to perform weighted fusion of the outputs of the horizontal strip hole branch, the vertical strip hole branch, the local detail branch, and the long-distance context branch based on the direction weights. The probability map output module 55 is used to output a crack probability map through the mask branch in the semantic segmentation network and output a skeleton probability map through the skeleton branch in the semantic segmentation network. The semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels. The skeleton labels are generated by crack mask labels and are used to indicate the position of the crack center line. Module 56 is used to obtain crack masking results based on crack probability maps and skeleton results based on skeleton probability maps.

[0089] As can be seen from the above technical solution, by using direction-aware striped void blocks in the semantic segmentation network to enhance the directional differences of crack features in different directions, the network can simultaneously capture the continuity features of cracks in the horizontal, vertical, and long-distance directions. This overcomes the shortcomings of ordinary convolution, which only focuses on local neighborhoods and is difficult to model the complete structure of slender cracks. Simultaneously, the enhanced features are provided to both the mask branch and the skeleton branch, outputting crack probability maps and skeleton probability maps respectively. Furthermore, the semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels. The skeleton labels are generated by the crack mask labels and indicate the position of the crack centerline. This ensures that the network is simultaneously constrained by both the integrity of the crack region and the continuity of the centerline during the training phase. Therefore, it can solve the technical problems of segmentation breakage, missing ends, and local discontinuities in fine cracks in traditional semantic segmentation networks. It achieves multi-directional and multi-scale feature enhancement of fine cracks, enabling the semantic segmentation network to more completely segment continuous fine cracks, improving its ability to perceive the directional continuity of fine cracks, reducing the segmentation breakage of fine cracks, and balancing the integrity of the crack region and the continuity of the centerline.

[0090] In one specific embodiment of this application, the enhanced feature acquisition module 52 is specifically a module that uses direction-aware strip hole blocks to perform directional differential enhancement of crack features at the high-level feature layer of the encoder, the bottleneck layer between the encoder and the decoder, or the skip connection fusion position between the encoder and the decoder in the semantic segmentation network.

[0091] In one specific embodiment of this application, the device may further include a skeleton label generation module, which may include: The noise removal and hole filling submodule is used for noise removal and hole filling of cracked mask labels; The centerline extraction submodule is used to perform skeletonization processing on the filled mask and extract the centerline of the crack; The skeleton label acquisition submodule is used to expand the extracted centerline according to a preset radius to obtain the skeleton label.

[0092] In one specific embodiment of this application, the device may further include: The skeleton line extraction module is used to perform connected component analysis on the skeleton results obtained from the skeleton probability map and extract the skeleton lines of each connected component. The spacing calculation module is used to calculate the spacing between the endpoints of the skeleton lines of adjacent connected domains; The module for outputting the number of broken segments and endpoint locations is used to mark adjacent connected regions as suspected broken segments when the spacing is less than a preset threshold, and output the number of broken segments and endpoint locations.

[0093] In one specific embodiment of this application, the device may further include: The input image block acquisition module is used to divide a large-scale engineering structure surface image into blocks using a sliding window method to obtain multiple input image blocks. The probability map acquisition module is used to input each input image block into the semantic segmentation network to obtain the crack probability map and skeleton probability map corresponding to each image block; The probability map fusion module is used to fuse the crack probability map and skeleton probability map of each image block in the overlapping area of ​​adjacent image blocks using the average probability or maximum probability method.

[0094] For the method embodiments described above, see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the crack segmentation device provided in this application, which may include: Memory 332 is used to store computer programs; The processor 322 is used to implement the steps of the crack segmentation method in the above method embodiment when executing a computer program.

[0095] For details, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the specific structure of a crack segmentation device provided in this embodiment. The crack segmentation device can vary significantly depending on its configuration or performance. It may include a processor (central processing unit, CPU) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the crack segmentation device 301.

[0096] The crack segmentation device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0097] The steps in the crack segmentation method described above can be implemented by the structure of the crack segmentation device.

[0098] Corresponding to the above method embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps: The process involves acquiring surface images of the engineering structure and preprocessing them to obtain input image patches. These patches are then input into a semantic segmentation network. The extracted crack features are enhanced using direction-aware stripe hole blocks within the semantic segmentation network, resulting in enhanced features. This enhancement process includes: using a horizontal stripe hole branch to enhance crack features in the horizontal direction; using a vertical stripe hole branch to enhance crack features in the vertical direction; using a local detail branch to extract local detail information from crack features; and using a long-range context branch to extract long-range context information from crack features. After extracting long-range contextual information of crack features using long-range context branches, the process further includes: generating directional weights for each branch based on the crack features; weighted fusion of the outputs of the horizontal strip hole branch, the vertical strip hole branch, the local detail branch, and the long-range context branch based on the directional weights; outputting a crack probability map through the mask branch in the semantic segmentation network and outputting a skeleton probability map through the skeleton branch in the semantic segmentation network; wherein, the semantic segmentation network is trained under joint supervision by crack mask labels and skeleton labels, and the skeleton labels are generated by the crack mask labels and are used to indicate the position of the crack centerline; the crack mask result is obtained from the crack probability map, and the skeleton result is obtained from the skeleton probability map.

[0099] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0102] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for crack segmentation, characterized in that, include: Acquire an image of the surface of the engineering structure and preprocess the image to obtain an input image block; The input image patch is input into a semantic segmentation network. The extracted crack features are enhanced using direction-aware stripe hole blocks within the semantic segmentation network, resulting in enhanced features. This direction-aware stripe hole enhancement includes: enhancing the crack features laterally using a horizontal stripe hole branch; enhancing the crack features vertically using a vertical stripe hole branch; extracting local detail information from the crack features using a local detail branch; extracting long-range context information from the crack features using a long-range context branch; and further extracting the crack features using a long-range context branch. Following the long-range context information of the crack features, the method further includes: generating directional weights corresponding to each branch based on the crack features; and weighting and fusing the outputs of the horizontal stripe hole branch, the vertical stripe hole branch, the local detail branch, and the long-range context branch based on the directional weights. Specifically, the intermediate features or the original features input to the direction-aware stripe hole block are subjected to global average pooling, channel compression via 1×1 convolution, and then directional weights corresponding to each branch are generated using a Sigmoid activation function or a normalized exponential activation function. The weighted fusion of the outputs of the horizontal stripe hole branch, the vertical stripe hole branch, the local detail branch, and the long-range context branch includes: ； Where Y represents the output feature after orientation enhancement; H(X'), V(X'), L(X'), and C(X') represent the outputs obtained after processing the intermediate feature X' by the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch, respectively; αH, αV, αL, and αC represent the orientation weights corresponding to the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch, respectively; concat represents the channel concatenation operation; Conv1×1 represents the 1×1 convolution operation; X represents the original input feature input to the orientation-aware strip hole block, which is added to the fused feature through residual connection to alleviate the gradient vanishing problem in deep networks and preserve the original information; The enhanced features are output as a crack probability map through the mask branch in the semantic segmentation network, and the enhanced features are output as a skeleton probability map through the skeleton branch in the semantic segmentation network; wherein, the semantic segmentation network is obtained through joint supervised training of crack mask labels and skeleton labels, and the skeleton labels are generated by the crack mask labels and are used to indicate the position of the crack centerline; The crack masking result is obtained based on the crack probability map, and the skeleton result is obtained based on the skeleton probability map.

2. The crack segmentation method according to claim 1, characterized in that, The extracted crack features are enhanced with directional differentiation using the direction-aware strip hole blocks in the semantic segmentation network, including: At the high-level feature layer of the encoder, the bottleneck layer between the encoder and the decoder, or the skip connection fusion position between the encoder and the decoder in the semantic segmentation network, the direction-aware strip hole block is used to enhance the direction of the crack feature.

3. The crack segmentation method according to claim 1, characterized in that, It also includes the process of generating the skeleton tag, which includes: The crack mask label is subjected to noise removal and hole filling; The filled crack mask label is skeletonized to extract the center line of the crack. The extracted centerline is expanded according to a preset radius to obtain the skeleton label.

4. The crack segmentation method according to claim 1, characterized in that, After obtaining the skeleton result based on the skeleton probability map, the process also includes: Perform connected component analysis on the skeleton results and extract the skeleton lines of each connected component; Calculate the spacing between the endpoints of the skeleton lines of adjacent connected components; When the spacing is less than a preset threshold, adjacent connected regions are marked as suspected broken segments, and the number of broken segments and their endpoint positions are output.

5. The crack segmentation method according to any one of claims 1 to 4, characterized in that, Also includes: A sliding window method is used to divide a large-scale engineering structure surface image into blocks, resulting in multiple input image blocks; Each of the input image blocks is input into the semantic segmentation network to obtain the crack probability map and skeleton probability map corresponding to each image block; For overlapping areas of adjacent image blocks, the crack probability map and skeleton probability map of each image block are fused using the average probability or maximum probability method.

6. A crack dividing device, characterized in that, include: An input image block acquisition module is used to acquire surface images of engineering structures and preprocess the surface images of engineering structures to obtain input image blocks; The enhanced feature acquisition module is used to input the input image patch into the semantic segmentation network, and use the direction-aware strip hole block in the semantic segmentation network to perform directional differential enhancement on the extracted crack features to obtain the enhanced features; The enhanced feature acquisition module includes: a lateral feature enhancement submodule, used to enhance crack features in the lateral direction using lateral strip hole branches; a longitudinal feature enhancement submodule, used to enhance crack features in the longitudinal direction using longitudinal strip hole branches; a local detail information extraction submodule, used to extract local detail information of crack features using local detail branches; a long-range context information extraction submodule, used to extract long-range context information of crack features using long-range context branches; and a direction weight generation module, used to generate direction weights based on the crack features after extracting long-range context information using long-range context branches. The seam feature generates directional weights for each branch; the weighted fusion module is used to weight and fuse the outputs of the horizontal strip hole branch, the vertical strip hole branch, the local detail branch, and the long-range context branch based on the directional weights; wherein, the intermediate features or the original features input to the direction-aware strip hole block are subjected to global average pooling, channel compression is performed through 1×1 convolution, and then the directional weights for each branch are generated by the Sigmoid activation function or the normalized exponential activation function; the weighted fusion of the outputs of the horizontal strip hole branch, the vertical strip hole branch, the local detail branch, and the long-range context branch includes: ； Where Y represents the output feature after orientation enhancement; H(X'), V(X'), L(X'), and C(X') represent the outputs obtained after processing the intermediate feature X' by the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch, respectively; αH, αV, αL, and αC represent the orientation weights corresponding to the horizontal strip hole branch, vertical strip hole branch, local detail branch, and long-range context branch, respectively; concat represents the channel concatenation operation; Conv1×1 represents the 1×1 convolution operation; X represents the original input feature input to the orientation-aware strip hole block, which is added to the fused feature through residual connection to alleviate the gradient vanishing problem in deep networks and preserve the original information; The probability map output module is used to output a crack probability map by passing the enhanced features through the mask branch in the semantic segmentation network, and to output a skeleton probability map by passing the enhanced features through the skeleton branch in the semantic segmentation network; wherein, the semantic segmentation network is obtained through joint supervised training of crack mask labels and skeleton labels, and the skeleton labels are generated by the crack mask labels and are used to indicate the position of the crack centerline. The result acquisition module is used to obtain the crack mask result based on the crack probability map and the skeleton result based on the skeleton probability map.

7. A crack segmentation device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the crack segmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the crack segmentation method as described in any one of claims 1 to 5.

Images