Road and bridge crack segmentation network detection method based on boundary enhancement and warning

Abstract

The invention discloses a road and bridge crack segmentation network detection method based on boundary enhancement and warning. The method comprises the following steps: according to a road and bridge crack image sample data set; designing a boundary warning module and a boundary enhancement module based on snakelike convolution, introducing an attention mechanism, designing a wavelet down-sampling attention module based on discrete wavelet transform, and constructing a road and bridge crack detection model of a road and bridge crack segmentation network based on boundary enhancement and warning; training and verifying a road and bridge crack detection model by using the road and bridge crack image sample data set to obtain a trained road and bridge crack detection model; and taking the to-be-processed road and bridge crack image as the input of the trained model, performing crack boundary identification on the to-be-processed road and bridge crack image, and outputting a crack detection result of the to-be-processed road and bridge crack image. According to the method provided by the invention, the road and bridge crack segmentation precision is effectively improved, and the problem of low detection result precision caused by the polymorphism and noise complexity of the road and bridge cracks is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a road and bridge crack segmentation network detection method based on boundary enhancement and warning. Background Art

[0002] With the continuous development of transportation infrastructure, roads and bridges, as key transportation hubs, have a direct impact on traffic flow and public safety due to their safety and reliability. Cracks are a common form of damage in road and bridge structures. If not detected and repaired in a timely manner, they may lead to further structural deterioration, increase maintenance costs, and even cause serious safety hazards. However, existing crack detection methods face significant challenges in terms of accuracy and efficiency.

[0003] At present, traditional crack detection methods mostly rely on manual inspection or use simple image processing technology. These methods often have difficulty dealing with the polymorphism and complex noise of road and bridge cracks, resulting in low accuracy and stability of detection results. In addition, traditional methods also have shortcomings in computational complexity and processing efficiency, making it difficult to meet the needs of large-scale applications and real-time monitoring.

[0004] In order to improve the accuracy and efficiency of crack detection, automated detection methods based on deep learning have emerged in recent years. However, these methods still face the following problems: loss of high-frequency information, noise interference, unclear boundary information, etc.

[0005] Therefore, faced with the diversity of road and bridge cracks and the complexity of noise, how to effectively improve the segmentation accuracy of road and bridge cracks and ensure the accuracy of crack detection has become a problem that needs further research and resolution. Summary of the invention

[0006] In view of the above-mentioned deficiencies of the prior art, the present invention provides a road and bridge crack segmentation network detection method based on boundary enhancement and warning, so as to improve the segmentation accuracy of road and bridge cracks and solve the problem that the polymorphism and complex noise of road and bridge cracks lead to low accuracy of detection results.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A road and bridge crack segmentation network detection method based on boundary enhancement and warning comprises the following steps:

[0009] Obtain a sample dataset of road and bridge crack images;

[0010] Design boundary warning module and boundary enhancement module based on snake convolution, introduce attention mechanism, design wavelet downsampling attention module based on discrete wavelet transform, and build a road and bridge crack detection model based on boundary enhancement and warning road and bridge crack segmentation network;

[0011] Using the road bridge crack image sample data set to train and verify the road bridge crack detection model, to obtain a trained road bridge crack detection model;

[0012] Obtain a road bridge crack image to be processed, use the road bridge crack image to be processed as the input of a trained road bridge crack detection model, perform crack boundary recognition on the road bridge crack image to be processed, and output a crack detection result of the road bridge crack image to be processed.

[0013] As a preferred solution, the road and bridge crack detection model includes a cascaded multi-scale convolution module, a boundary warning module, three multi-scale attention boundary enhancement units, a multi-scale attention upsampling unit, three multi-scale dense upsampling units and a head network; wherein the multi-scale attention boundary enhancement unit includes a wavelet downsampling attention module, a multi-scale convolution module and a boundary enhancement module connected in sequence, the multi-scale attention upsampling unit includes a wavelet downsampling attention module, a multi-scale convolution module and a dense upsampling convolution module connected in sequence, the multi-scale dense upsampling unit includes a multi-scale convolution module and a dense upsampling convolution module connected in sequence, and the head network is a 1×1 convolution layer;

[0014] The input of the road and bridge crack detection model is used as the input of the multi-scale convolution module, and the output of the multi-scale convolution module is used as the input of the boundary warning module and the first multi-scale attention boundary enhancement unit respectively; the output of each multi-scale attention boundary enhancement unit is used as the input of the next one in the road and bridge crack segmentation network; the output of the third multi-scale attention boundary enhancement unit is used as the input of the multi-scale attention upsampling unit; the output of the multi-scale attention upsampling unit is spliced ​​with the output of the third multi-scale attention boundary enhancement unit as the input of the third multi-scale dense upsampling unit; the input of each multi-scale dense upsampling unit is a splicing graph of the output of the corresponding multi-scale attention boundary enhancement unit in the road and bridge crack segmentation network and the previous output of the multi-scale dense upsampling unit; the output of the boundary warning module is spliced ​​with the output of the multi-scale convolution module and then with the output of the first multi-scale dense upsampling unit as the input of the head network, and the output of the head network is used as the overall output of the road and bridge crack segmentation model.

[0015] As a preferred solution, the wavelet downsampling attention module includes a cascaded Haar discrete wavelet transform unit, a SoftMax function and a local average function; wherein, the input feature map is decomposed by the Haar discrete wavelet transform unit, and the high-frequency component and the low-frequency component are output, which are represented as HH, HL, LH, and LL respectively, and the HL, LH, and LL components are introduced into the attention mechanism, and the HL and LH containing high-frequency features are added and normalized by the SoftMax function; then, a 5×5 convolution kernel is used to perform a local average operation, and the normalized feature map is denoised; thereafter, the low-frequency component LL is multiplied by the output of the local average function, and the attention map is output; finally, the attention map is weighted and added with the low-frequency component LL to obtain the output feature of the wavelet downsampling attention module.

[0016] As a preferred solution, the processing process of the wavelet downsampling attention module is shown by the following formula:

[0017]

[0018] Where Mean is the local average function, size is the local window size, and α is the weight factor.

[0019] As a preferred solution, the boundary warning module includes a cascaded serpentine convolution connection unit with a convolution kernel of 3, a serpentine convolution connection unit with a convolution kernel of 5, and a normalized convolution activation unit; the serpentine convolution connection unit with a convolution kernel of 3 includes parallel 1×3, 3×1 serpentine convolution layers, 3×3 convolution layers, a first connection layer and a first 1×1 convolution layer, the serpentine convolution connection unit with a convolution kernel of 5 includes parallel 1×5, 5×1 serpentine convolution layers, 5×5 convolution layers, a second connection layer and a second 1×1 convolution layer, and the normalized convolution activation unit includes a cascaded third 1×1 convolution layer, a BatchNorm normalization layer and a ReLU activation function layer; wherein the parallel result of the 1×3, 3×1 serpentine convolution layer and the 3×3 convolution layer is cascaded with the first connection layer, and the parallel result of the 1×5, 5×1 serpentine convolution layer and the 5×5 convolution layer is cascaded with the second connection layer;

[0020] Among them, the input feature map of the boundary warning module is used as the input of the serpentine convolution connection unit with a convolution kernel of 3, the output of the serpentine convolution connection unit with a convolution kernel of 3 is used as the input of the serpentine convolution connection unit with a convolution kernel of 5, the output of the serpentine convolution connection unit with a convolution kernel of 5 is added to the input feature map and used as the input of the normalized convolution activation unit, and the output of the normalized convolution activation unit is used as the output of the boundary warning module.

[0021] As a preferred solution, the processing process of the boundary warning module is shown by the following formula:

[0022]

[0023] In the formula, SF(X In ,k) is the output feature of snake convolution, F(X In ,k) is the output feature of the snake convolution connection unit with convolution kernel k, X residual are the residual output features of two sets of snake convolution and ordinary convolution with convolution kernels of 3 and 5 along the x and y directions, k is the convolution kernel size, Cat is the channel splicing, DSConv 1×k and DSConv k×1 They are snake convolutions along the x and y directions, Conv k×k The convolution kernel is k standard convolution, Conv 1×1 is a 1×1 convolution, X In is the input feature of the boundary warning module, BN is batch normalization, ReLU is the activation function, X Out It is the boundary mask image output by the boundary warning module.

[0024] As a preferred solution, the boundary warning module also introduces a boundary loss function, which is used to optimize the model's learning of crack boundary features based on the boundary features extracted by the boundary warning module and the boundary features of the real mask;

[0025] The extraction process of the boundary features of the real mask is shown in the following formula:

[0026]

[0027] In the formula, mask is the real mask image, D is a 5x5 operator, and mask d is the image after the original image is expanded, mask boundary is the boundary image of the true mask.

[0028] As a preferred solution, the boundary enhancement module includes a long dependency branch, a boundary enhancement branch and a fusion unit;

[0029] The long dependency branch includes a local feature extraction unit and a global feature extraction unit connected in sequence; the local feature extraction unit includes a cascaded 3×3CBS module and a 1×1CBS module, wherein the CBS module includes a cascaded convolutional layer, a BatchNorm normalization layer and a SiLU activation function layer, and the global feature extraction unit includes a cascaded Unfold function, a Transformer module and a Fold function, wherein the Transformer module includes a cascaded self-attention unit and a feedforward network, the self-attention unit includes a cascaded layer normalization layer and a self-attention layer, and the feedforward network includes a layer normalization layer, a fully connected layer, a SiLU activation function regularization layer and a fully connected regularization layer;

[0030] The boundary enhancement branch includes a Gaussian blur function and a Laplace transform function connected in sequence; the fusion unit includes a 1×1CBS module, a connection layer and a 3×3CBS module connected in sequence;

[0031] In the boundary enhancement module, the feature map input to the boundary enhancement module is respectively used as the input of the long dependency branch and the boundary enhancement branch, wherein the input of the long dependency branch is used as the input of the local feature extraction unit, the output of the local feature extraction unit is used as the input of the global feature extraction unit, the output of the global feature extraction unit is used as the output of the long dependency branch, the input of the other boundary enhancement branch is used as the input of the Gaussian blur function, the output of the Gaussian blur function is subtracted from the input of the boundary enhancement branch, and the output is a differential Gaussian feature, the output of the Gaussian blur function is used as the input of the Laplace transform function, the output of the Laplace transform function is multiplied with the differential Gaussian feature, and the output is a Gaussian Laplace feature, the Gaussian Laplace feature is added to the input of the boundary enhancement branch to obtain the output of the boundary enhancement branch; the output of the long dependency branch is used as the input of the 1×1CBS module in the fusion unit, the output of the 1×1CBS module in the fusion unit is spliced ​​with the output of the boundary enhancement branch, and then passed through the 3×3CBS module to obtain the output feature as the boundary enhancement module.

[0032] As a preferred solution, the processing process of the long dependency branch includes the following steps:

[0033] The input feature map is extracted through the 3×3CBS module and the 1×1CBS module to obtain local features. The local features are extracted using the Unfold operation to extract the extraction sliding window in the input feature map according to the window size (ph, pw). The sliding window is input into the Transformer module based on the self-attention mechanism and rearranged through the Fold operation to obtain the global features.

[0034] The calculation formula of the self-attention mechanism is as follows:

[0035]

[0036] In the formula, Q, K, and V represent different linearly transformed vectors of the sliding window, d head is the number of self-attention heads.

[0037] As a preferred solution, the processing process of the boundary enhancement branch includes the following steps:

[0038] The extraction process of the differential Gaussian feature is shown in the following formula:

[0039] DoG = I*(G(x,y,σ)-1);

[0040]

[0041] The Gaussian Laplace feature extraction process is shown in the following formula:

[0042]

[0043] The output characteristics of the boundary enhancement module are shown in the following formula:

[0044] I output =DOG·LOG+I;

[0045] Where DoG is the result of the input feature after differential Gaussian, G(x, y, σ) is the Gaussian blur function, σ is the blur degree, x and y are pixel coordinates, LOG is the result of the input feature after Gaussian Karplas, I is the input feature of the boundary enhancement module, and I output is the output feature after boundary enhancement.

[0046] Compared with the prior art, the present invention has the following technical effects:

[0047] (1) The present invention decomposes the signal into components of different scales by designing a downsampling attention module based on wavelet transform. The downsampling attention mechanism retains important high-frequency information and ensures the accurate retention of crack details. At the same time, in the road and bridge crack images, the high-frequency components often contain noise. The present invention separates the noise from the edge information of the cracks through wavelet transform, thereby reducing the interference of noise while retaining the edge features.

[0048] (2) The boundary warning module in the present invention combines morphological operations and serpentine convolution to remove small noise points and fill holes in the cracks while ensuring the shape of the cracks. By applying serpentine convolution, the crack direction and shape can be better captured, and the feature extraction capability of slender and complex structures is improved. This method effectively reduces the noise interference in the image, enhances the boundary features of the cracks through the refined processing of the crack boundaries, avoids the negative impact of noise on the detection results, and improves the stability and reliability of the detection.

[0049] (3) The boundary enhancement module designed by the present invention combines convolutional neural networks and Transformer to perform dual extraction of local and global features, thereby better understanding the contextual information of the crack image and enhancing the ability to recognize complex crack morphologies; at the same time, the boundary enhancement module also introduces differential Gaussian and Gaussian Laplace transforms to further enhance boundary information, wherein Gaussian blur is used to smooth the crack image to reduce image noise, and Laplace transform can be used to highlight and enhance the image edge, which helps to identify the boundaries between different areas, thereby assisting segmentation. These methods improve the accuracy of crack segmentation by refining edges and improving boundary detection, making the model more sensitive and accurate in responding to boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0051] Figure 1 The present invention discloses a flow chart of a road and bridge crack segmentation network detection method based on boundary enhancement and warning;

[0052] Figure 2 It is a structural schematic diagram of the road and bridge crack segmentation model of the present invention;

[0053] Figure 3 A schematic diagram of a down-sampling mechanism based on wavelet transform according to an embodiment of the present invention;

[0054] Figure 4 Schematic diagram of a boundary warning module BWM according to an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of snake convolution position learning according to an embodiment of the present invention;

[0056] Figure 6 Schematic diagram of a boundary enhancement module BEM according to an embodiment of the present invention;

[0057] Figure 7 This is a comparison chart of the prediction results of each model on the Deepcrack test set in the verification experiment of the embodiment of the present invention;

[0058] Figure 8 This is a comparison chart of the prediction results of each model on the Crack500 test set in the verification experiment of the embodiment of the present invention;

[0059] Fig. 9 This is a comparison chart of the prediction results of each model in the CamCrack789 test set and the self-built data set Crack149 in the verification experiment of the embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0061] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0062] The existing traditional crack detection methods rely on manual inspection or use simple image processing technology. These methods often have difficulty dealing with the polymorphism and complex noise of road and bridge cracks, resulting in low accuracy and stability of the detection results. In order to improve the accuracy and efficiency of crack detection, automated detection methods based on deep learning have emerged in recent years. However, these methods still face the following problems: loss of high-frequency information, noise interference, unclear boundary information, etc.

[0063] In order to solve the above problems, the present invention proposes a road and bridge crack segmentation network detection method based on boundary enhancement and warning. The flow chart of the method is as follows: Figure 1 As shown, including:

[0064] Obtain a sample dataset of road and bridge crack images;

[0065] Design boundary warning module and boundary enhancement module based on snake convolution, introduce attention mechanism, design wavelet downsampling attention module based on discrete wavelet transform, and build a road and bridge crack detection model based on boundary enhancement and warning road and bridge crack segmentation network;

[0066] Using the road bridge crack image sample data set to train and verify the road bridge crack detection model, to obtain a trained road bridge crack detection model;

[0067] Obtain a road bridge crack image to be processed, use the road bridge crack image to be processed as the input of a trained road bridge crack detection model, perform crack boundary recognition on the road bridge crack image to be processed, and output a crack detection result of the road bridge crack image to be processed.

[0068] The road and bridge crack segmentation network described in the present invention effectively improves the accuracy of road and bridge crack segmentation and reduces the computational complexity by designing advanced image processing modules, including a wavelet downsampling attention module based on discrete wavelet transform, a boundary warning module and a boundary enhancement module, thereby providing a new solution for road and bridge crack detection.

[0069] In order to better introduce the technical solution of the present invention, a more detailed description is given in the following parts.

[0070] 1. Image preprocessing

[0071] In the specific implementation of the application, the sample data set of road and bridge crack images for training and verifying the road and bridge crack detection model can be selected from public data sets, such as DeepCrack, Crack500 and CamCrack789. Secondly, the self-built road and bridge crack image sample data set Crack149 is used for generalization experiments. Before model training, the data set is pre-processed by scaling, cropping, rotating, normalizing and enhancing to ensure that the data is suitable for model training. After the model is trained and predicted, the output results of the model need to be post-processed. Among them, in the example verification, the DeepCrack data set contains 537 different types of crack images, and some images contain more noise, such as puddles, spots, stains, occlusions, and shadows. These images are mainly divided into two groups: a training set containing 300 images and a test set containing 237 images. The resolution of all images is 544×384; the Crack500 dataset contains 500 road pavement images with a size of 2000×1500, which are collected using smartphones. The dataset contains backgrounds such as dark light, bright light, water stains, and shadows. The original author cropped Crack500 into 3368 images, of which the training set contains 1896 images, the test set contains 1124 images, and the validation set contains 348 images; the CamCrack789 dataset contains a dataset of 789 crack images with a resolution of 640×480. The dataset is divided by the original author into 546 training sets and 243 training sets. This dataset contains many complex backgrounds, such as dark light, bright light, water stains, shadows, fallen leaves, etc.; the Crack149 self-built dataset includes 149 self-built crack images with a resolution of 1000×800, most of which are dark or bright concrete crack images, including some shrinkage crack images.

[0072] 2. Road and bridge crack detection model

[0073] like Figure 2 As shown, the road and bridge crack detection model includes a cascaded multi-scale convolution module, a boundary warning module, three multi-scale attention boundary enhancement units, a multi-scale attention upsampling unit, three multi-scale dense upsampling units and a head network; wherein the multi-scale attention boundary enhancement unit includes a wavelet downsampling attention module, a multi-scale convolution module and a boundary enhancement module connected in sequence, the multi-scale attention upsampling unit includes a wavelet downsampling attention module, a multi-scale convolution module and a dense upsampling convolution module connected in sequence, the multi-scale dense upsampling unit includes a multi-scale convolution module and a dense upsampling convolution module connected in sequence, and the head network is a 1×1 convolution layer;

[0074] The input of the road and bridge crack detection model is used as the input of the multi-scale convolution module, and the output of the multi-scale convolution module is used as the input of the boundary warning module and the first multi-scale attention boundary enhancement unit respectively; the output of each multi-scale attention boundary enhancement unit is used as the input of the next one in the road and bridge crack segmentation network; the output of the third multi-scale attention boundary enhancement unit is used as the input of the multi-scale attention upsampling unit; the output of the multi-scale attention upsampling unit is spliced ​​with the output of the third multi-scale attention boundary enhancement unit as the input of the third multi-scale dense upsampling unit; the input of each multi-scale dense upsampling unit is a splicing graph of the output of the corresponding multi-scale attention boundary enhancement unit in the road and bridge crack segmentation network and the previous output of the multi-scale dense upsampling unit; the output of the boundary warning module is spliced ​​with the output of the multi-scale convolution module and then with the output of the first multi-scale dense upsampling unit as the input of the head network, and the output of the head network is used as the overall output of the road and bridge crack segmentation model.

[0075] Next, each module in the road and bridge crack detection model is explained in detail.

[0076] 2.1 Wavelet Downsampling Attention Module

[0077] like Figure 3As shown in the figure, the wavelet downsampling attention module includes a cascaded Haar discrete wavelet transform unit, a SoftMax function and a local average function; wherein, the input feature map is decomposed by the Haar discrete wavelet transform unit, and the high-frequency component and the low-frequency component are output, which are represented as HH, HL, LH, and LL respectively. The HL, LH, and LL components are introduced into the attention mechanism, and the HL and LH containing high-frequency features are added and normalized by the SoftMax function; then, a 5×5 convolution kernel is used to perform a local average operation to reduce the noise of the normalized feature map; then, the low-frequency component LL is multiplied by the output of the local average function to obtain the attention map; finally, the attention map is weighted and added with the low-frequency component LL to obtain the output feature of the wavelet downsampling attention module.

[0078] Specifically, the present invention decomposes the input signal into low-frequency and high-frequency components in a discrete wavelet transform (DWT) of a two-dimensional Haar wavelet transform, wherein the expressions of the low-frequency filter h and the high-frequency filter g of the Haar wavelet are: For a two-dimensional discrete wavelet transform (DWT), you need to apply low-frequency filters and high-frequency filters to the rows and columns of the input signal. Assume there is an input signal X with dimensions (H, W). First apply the row transform, then apply the column transform. The specific steps are as follows:

[0079] For each row, calculate the low-frequency and high-frequency components:

[0080]

[0081] For each column, calculate the low-frequency and high-frequency components:

[0082]

[0083] Where i and j represent the row and column indices respectively.

[0084] The processing process of the wavelet downsampling attention module is shown in the following formula:

[0085]

[0086] Where Mean is the local average function, size is the local window size, and α is the weight factor.

[0087] 2.2. Boundary warning module

[0088] like Figure 4As shown, the boundary warning module includes a cascaded snake convolution connection unit with a convolution kernel of 3, a snake convolution connection unit with a convolution kernel of 5, and a normalized convolution activation unit; the snake convolution connection unit with a convolution kernel of 3 includes parallel 1×3, 3×1 snake convolution layers, 3×3 convolution layers, a first connection layer and a first 1×1 convolution layer, the snake convolution connection unit with a convolution kernel of 5 includes parallel 1×5, 5×1 snake convolution layers, 5×5 convolution layers, a second connection layer and a second 1×1 convolution layer, and the normalized convolution activation unit includes a cascaded third 1×1 convolution layer, a BatchNorm normalization layer and a ReLU activation function layer; wherein the parallel result of the 1×3, 3×1 snake convolution layers and the 3×3 convolution layers is cascaded with the first connection layer, and the parallel result of the 1×5, 5×1 snake convolution layers and the 5×5 convolution layers is cascaded with the second connection layer;

[0089] Among them, the input feature map of the boundary warning module is used as the input of the serpentine convolution connection unit with a convolution kernel of 3, the output of the serpentine convolution connection unit with a convolution kernel of 3 is used as the input of the serpentine convolution connection unit with a convolution kernel of 5, the output of the serpentine convolution connection unit with a convolution kernel of 5 is added to the input feature map and used as the input of the normalized convolution activation unit, and the output of the normalized convolution activation unit is used as the output of the boundary warning module.

[0090] In the specific implementation, taking advantage of the dynamic snake convolution, a boundary warning module is designed. The learned boundary information is combined with the boundary information of the real mask to calculate the boundary loss to warn the boundary and avoid introducing noise information. The following first introduces the offset position learning of the dynamic snake convolution in detail, and then describes the output process of the boundary warning module.

[0091] Dynamic snake convolution (DSConv) is a convolution operation designed for tubular structures, aiming to improve the feature extraction capability of slender and complex structures. It introduces a learnable snake path so that the convolution kernel can adaptively select the path according to the image content, thereby better adapting to the feature distribution of different image regions and ensuring the continuity of attention.

[0092] For a standard (2C+1)×(2C+1) convolution kernel, the dynamic snake convolution takes the x-axis direction as an example, and the specific position of each grid in K is expressed as: K i±c =(x i±c ,y i±c ), c = {0, 1, 2, ..., C} represents the horizontal distance to the center grid. The selection of the basis ± for each grid position in the convolution kernel K is an accumulation process. i Initially, the position of the grid away from the center depends on the position of the previous grid: i In comparison, K i+1An offset δ=[σ|σ∈]-1,1] is added. Therefore, the offset needs to be accumulated so that the convolution kernel conforms to the linear morphology structure.

[0093] Figure 5 Schematic diagram of snake convolution position learning, Figure 5 It can be seen that the position in the x-axis direction becomes:

[0094]

[0095] The position in the y-axis direction becomes:

[0096]

[0097] Since the offset δ is usually fractional, bilinear interpolation is implemented as:

[0098] B(K',K)=b(K x ,K' x )·b(K y ,K' y );

[0099] like Figure 5 This is a snake convolution with a convolution kernel of 3. Due to the position offset in the two directions of the x-axis and the y-axis, it covers a 9×9 range during the deformation process.

[0100] The boundary warning module is mainly composed of two residual modules consisting of snake convolutions and ordinary convolutions in different directions and sizes to better capture the complex morphological information of linear cracks. The 1×k DSConv and k×1 DSConv in the boundary warning module represent two different directions of snake convolution, respectively along the x-axis and y-axis, where k is the size of the convolution kernel. The output of the boundary warning module is a boundary mask image, and its processing process is shown in the following formula:

[0101]

[0102] In the formula, SF(X In ,k) is the output feature of snake convolution, F(X In ,k) is the output feature of the snake convolution connection unit with convolution kernel k, X residual are the residual output features of two sets of snake convolution and ordinary convolution with convolution kernels of 3 and 5 along the x and y directions, k is the convolution kernel size, Cat is the channel splicing, DSConv 1×k and DSConv k×1 They are snake convolutions along the x and y directions, Conv k×k The convolution kernel is k standard convolution, Conv 1×1 is a 1×1 convolution, X Inis the input feature of the boundary warning module, BN is batch normalization, ReLU is the activation function, X Out It is the boundary mask image output by the boundary warning module.

[0103] At the same time, the boundary information learned by the boundary warning module needs to be calculated with the boundary of the real mask. The method used to extract the boundary features of the real mask is a 5x5 operator, which is as follows:

[0104]

[0105] After using this operator to dilate the image, the difference between the original real mask image and the original mask image is calculated to obtain the boundary features of the real mask, that is, the image after dilation minus the original image. The extraction process is shown in the following formula:

[0106]

[0107] In the formula, mask is the real mask image, D is a 5x5 operator, and mask d is the image after the original image is expanded, mask boundary is the boundary image of the true mask.

[0108] 2.3 Boundary Enhancement Module

[0109] In order to effectively encode local and global information, a boundary enhancement module (BEM) is proposed, which combines convolutional neural network (CNN) and Transformer for feature extraction and enhances boundary information using difference-of-Gaussian and Laplacian of Gaussian.

[0110] like Figure 6 As shown, the boundary enhancement module includes a long dependency branch, a boundary enhancement branch and a fusion unit;

[0111] The long dependency branch includes a local feature extraction unit and a global feature extraction unit connected in sequence; the local feature extraction unit includes a cascaded 3×3CBS module and a 1×1CBS module, wherein the CBS module includes a cascaded convolutional layer, a BatchNorm normalization layer and a SiLU activation function layer, and the global feature extraction unit includes a cascaded Unfold function, a Transformer module and a Fold function, wherein the Transformer module includes a cascaded self-attention unit and a feedforward network, the self-attention unit includes a cascaded layer normalization layer and a self-attention layer, and the feedforward network includes a layer normalization layer, a fully connected layer, a SiLU activation function regularization layer and a fully connected regularization layer;

[0112] In the specific implementation, when establishing long-term dependencies, the input image is first divided into non-overlapping image blocks, and each image block is regarded as a sequence; then, a set of lightweight convolutional neural network layers are applied to each image block to extract local features. These local features capture the details and local structures within the image block, which helps to identify the local features and textures of the crack image; next, the local features are used to capture global features using Transformer to establish long-term dependencies. The local feature sequence of each image block is input into the Transformer model, in which the self-attention mechanism can effectively capture the relationship between image blocks. This branch enables the boundary enhancement module to understand the local and global features of the image, thereby better understanding the contextual information of the crack image.

[0113] The calculation formula of the self-attention mechanism is as follows:

[0114]

[0115] In the formula, Q, K, and V represent different linearly transformed vectors of the sliding window, d head is the number of self-attention heads.

[0116] The boundary enhancement branch includes a Gaussian blur function and a Laplace transform function connected in sequence; the fusion unit includes a 1×1CBS module, a connection layer and a 3×3CBS module connected in sequence.

[0117] In the specific implementation, when the boundary enhancement branch is established, it mainly includes two calculations: Gaussian blur and Laplace transform. The boundary features are enhanced by multiplying the result pixels of Gaussian Laplace with the difference Gaussian of the original features. Among them, the binary continuous Gaussian blur function is as follows:

[0118]

[0119] G(x,y,σ) is the Gaussian blur function, σ is the blur degree, and x and y are pixel coordinates.

[0120] Laplace is the divergence of the gradient. To find the Laplace operator for a binary continuous Gaussian function, two partial derivatives need to be found. The calculation process of Gaussian Laplace is as follows:

[0121]

[0122] Since x and y are symmetric in the function, we can get:

[0123]

[0124] From this, the Laplace operator formula of the Gaussian function is expressed as follows:

[0125]

[0126] Differential high-precision edge feature extraction is to subtract the input feature from the input feature after Gaussian blurring. The formula is as follows:

[0127] DoG = I*(G(x,y,σ)-1);

[0128] Gaussian Laplace edge feature extraction is to perform Laplace calculation after Gaussian blurring the input features. The formula is as follows:

[0129]

[0130] The output characteristics of the boundary enhancement module are shown in the following formula:

[0131] I output =DOG·LOG+I;

[0132] Where DoG is the result of the input feature after differential Gaussian, LOG is the result of the input feature after Gaussian Karplas, I is the input feature of the boundary enhancement module, and I output is the output feature after boundary enhancement.

[0133] In the boundary enhancement module, the feature map input to the boundary enhancement module is respectively used as the input of the long dependency branch and the boundary enhancement branch, wherein the input of the long dependency branch is used as the input of the local feature extraction unit, the output of the local feature extraction unit is used as the input of the global feature extraction unit, the output of the global feature extraction unit is used as the output of the long dependency branch, the input of the other boundary enhancement branch is used as the input of the Gaussian blur function, the output of the Gaussian blur function is subtracted from the input of the boundary enhancement branch, and the output is a differential Gaussian feature, the output of the Gaussian blur function is used as the input of the Laplace transform function, the output of the Laplace transform function is multiplied with the differential Gaussian feature, and the output is a Gaussian Laplace feature, the Gaussian Laplace feature is added to the input of the boundary enhancement branch to obtain the output of the boundary enhancement branch; the output of the long dependency branch is used as the input of the 1×1CBS module in the fusion unit, the output of the 1×1CBS module in the fusion unit is spliced ​​with the output of the boundary enhancement branch, and then passed through the 3×3CBS module to obtain the output feature as the boundary enhancement module.

[0134] 3. Training of road and bridge crack detection model

[0135] In the specific implementation, the road bridge crack detection model is trained in the following way: after the road bridge crack detection model is built through the pytorch tool, the training set obtained from the public data set and the self-built data set are used as the input of the road bridge crack detection model, and the DiceBceLoss loss function is used to minimize the DiceBceLoss loss function to update the parameters of the road bridge crack detection model to optimize the road bridge crack detection model. In the specific implementation application, the training set batchsize is 4, the test set batchsize is 1, the input image size is 256×256, and each image is predicted separately and then integrated into one image.

[0136] 4. Experimental description

[0137] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.

[0138] This example selects training sets, test sets, and generalization experiments from four datasets: DeepCrack, Crack500, CamCrack789, and Crack149, and uses nine public algorithm models: SegNet, HED, LMNet, RCF, DeeplabV3++, TransUnet, DeepcrackZou, DeepcrackLiu, and CrackFormer-II for experiments. All network comparison methods are performed under the same experimental conditions, using the DiceBceLoss loss function. The batchsize of the training set is 4, the batchsize of the test set is 1, and the input image size is 256×256.

[0139] 4.1 Evaluation Indicators

[0140] Precision (P) and recall (R) are performance metrics calculated for semantic segmentation tasks. In the experiment, the precision and recall can be calculated by the predicted cracks and the true masks. In addition, the F1 metric (2·P·R / (P+R)) can be used as an overall performance evaluation indicator. In the experiment, the evaluation indicators used precision (P), recall (R), mean intersection over union (mIoU), pixel accuracy (PA), mean average precision (mPA), and two indicators based on the F1 metric: the best F1 (ODS) under a fixed threshold and the dataset aggregate F1 (OIS) under the best threshold for each image.

[0141] 4.2 Experimental Results

[0142] The performance of the road and bridge crack detection model (BEWCrack) of this embodiment and nine comparison models are evaluated on four datasets. DeepCrack and Crack500 are used for training and testing, and CamCrack789 and Crack149 are used for generalization experiments.

[0143] DeepCrack dataset: Table 1 shows the comparison results of BEWCrack and 9 comparison methods on the DeepCrack dataset. The BEWCrack model proposed in this paper achieves a recall rate of 88.02%, a precision rate of 91.15%, an mIoU of 90.08%, a PA of 99.11%, and an mPA of 93.82%, which is better than the comparison methods. This method obtains the highest ODS (89.56%) and the highest OIS (88.73%). Figure 7 Visualization of the segmentation results of BEWCrack and other comparison methods on DeepCrack.

[0144] Table 1 Comparison results of BEWCrack and 9 comparison methods on DeepCrack dataset

[0145]

[0146]

[0147] Crack500 Dataset:

[0148] The BEWCrack model is compared with 9 contrasting methods on the Crack500 dataset. The results are shown in Table 2. The BEWCrack model achieves 79.1% recall, 70.17% precision, 78% mean intersection over union (mIoU), 96.95% pixel accuracy (PA), and 88.55% mean pixel accuracy (mPA), which are all better than the contrasting methods. In addition, the proposed method also achieves the best performance in terms of ODS (74.37%) and OIS (74.17%). Figure 8 Visualization of the segmentation results of BEWCrack and other comparison methods on Crack500.

[0149] Table 2 Comparison results of BEWCrack model and 9 comparison methods on Crack500 dataset

[0150]

[0151] CamCrack789 and Crack149 datasets:

[0152] Using the weights trained on the DeepCrack dataset, generalization experiments were conducted on the CamCrack789 and Crack149 datasets. On the CamCrack789 dataset, except for the recall rate and average pixel accuracy which are slightly lower than TransUnet, other indicators are the best. On the Crack149 self-built dataset, the precision is slightly lower than LMNet, but the recall rate and other comprehensive indicators are the best. The generalization results of the CamCrack789 dataset are shown in Table 3, and the generalization results of the Crack149 dataset are shown in Table 4.

[0153] Some of the prediction results are as follows Fig. 9 as shown.

[0154] Table 3 Generalization results of BEWCrack model and 9 comparison methods on CamCrack789 dataset

[0155]

[0156]

[0157] Table 4 Generalization results of BEWCrack model and 9 comparison methods on Crack149 dataset

[0158]

[0159] It can be seen from the above that the road and bridge crack segmentation model (BEWCrack) proposed in the present invention has the best comprehensive index compared with the 9 compared models.

[0160] 5. Overview

[0161] In summary, the method of the present invention has the following technical advantages:

[0162] (1) The present invention decomposes the signal into components of different scales by designing a downsampling attention module based on wavelet transform. The downsampling attention mechanism retains important high-frequency information and ensures the accurate retention of crack details. At the same time, in the road and bridge crack images, the high-frequency components often contain noise. The present invention separates the noise from the edge information of the cracks through wavelet transform, thereby reducing the interference of noise while retaining the edge features.

[0163] (2) The boundary warning module in the present invention combines morphological operations and serpentine convolution to remove small noise points and fill holes in the cracks while ensuring the shape of the cracks. By applying serpentine convolution, the crack direction and shape can be better captured, and the feature extraction capability of slender and complex structures is improved. This method effectively reduces the noise interference in the image, enhances the boundary features of the cracks through the refined processing of the crack boundaries, avoids the negative impact of noise on the detection results, and improves the stability and reliability of the detection.

[0164] (3) The boundary enhancement module designed by the present invention combines convolutional neural networks and Transformer to perform dual extraction of local and global features, thereby better understanding the contextual information of the crack image and enhancing the ability to recognize complex crack morphologies; at the same time, the boundary enhancement module also introduces differential Gaussian and Gaussian Laplace transforms to further enhance boundary information, wherein Gaussian blur is used to smooth the crack image to reduce image noise, and Laplace transform can be used to highlight and enhance the image edge, which helps to identify the boundaries between different areas, thereby assisting segmentation. These methods improve the accuracy of crack segmentation by refining edges and improving boundary detection, making the model more sensitive and accurate in responding to boundaries.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A road and bridge crack segmentation network detection method based on boundary enhancement and warning, characterized in that: The steps include: Obtain a sample dataset of road and bridge crack images; Design boundary warning module and boundary enhancement module based on snake convolution, introduce attention mechanism, design wavelet downsampling attention module based on discrete wavelet transform, and build a road and bridge crack detection model based on road and bridge crack segmentation network with boundary enhancement and warning. Using the road bridge crack image sample data set to train and verify the road bridge crack detection model, to obtain a trained road bridge crack detection model; Obtain a road bridge crack image to be processed, use the road bridge crack image to be processed as the input of a trained road bridge crack detection model, perform crack boundary recognition on the road bridge crack image to be processed, and output a crack detection result of the road bridge crack image to be processed.

2. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 1 is characterized in that: The road and bridge crack detection model includes a cascaded multi-scale convolution module, a boundary warning module, three multi-scale attention boundary enhancement units, a multi-scale attention upsampling unit, three multi-scale dense upsampling units and a head network; wherein the multi-scale attention boundary enhancement unit includes a wavelet downsampling attention module, a multi-scale convolution module and a boundary enhancement module connected in sequence, the multi-scale attention upsampling unit includes a wavelet downsampling attention module, a multi-scale convolution module and a dense upsampling convolution module connected in sequence, the multi-scale dense upsampling unit includes a multi-scale convolution module and a dense upsampling convolution module connected in sequence, and the head network is a 1×1 convolution layer; The input of the road and bridge crack detection model is used as the input of the multi-scale convolution module, and the output of the multi-scale convolution module is used as the input of the boundary warning module and the first multi-scale attention boundary enhancement unit respectively; the output of each multi-scale attention boundary enhancement unit is used as the input of the next one in the road and bridge crack segmentation network; the output of the third multi-scale attention boundary enhancement unit is used as the input of the multi-scale attention upsampling unit; the output of the multi-scale attention upsampling unit is spliced ​​with the output of the third multi-scale attention boundary enhancement unit as the input of the third multi-scale dense upsampling unit; the input of each multi-scale dense upsampling unit is a splicing graph of the output of the corresponding multi-scale attention boundary enhancement unit in the road and bridge crack segmentation network and the previous output of the multi-scale dense upsampling unit; the output of the boundary warning module is spliced ​​with the output of the multi-scale convolution module and then with the output of the first multi-scale dense upsampling unit as the input of the head network, and the output of the head network is used as the overall output of the road and bridge crack segmentation model.

3. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 2 is characterized in that: The wavelet downsampling attention module includes a cascaded Haar discrete wavelet transform unit, a SoftMax function and a local average function; wherein, the input feature map is decomposed by the Haar discrete wavelet transform unit, and the high-frequency component and the low-frequency component are output, which are represented as HH, HL, LH, and LL respectively, and the HL, LH, and LL components are introduced into the attention mechanism, and the HL and LH containing high-frequency features are added and normalized by the SoftMax function; then, a 5×5 convolution kernel is used to perform a local average operation, and the normalized feature map is denoised; thereafter, the low-frequency component LL is multiplied by the output of the local average function, and the attention map is output; finally, the attention map is weighted and added with the low-frequency component LL to obtain the output feature of the wavelet downsampling attention module.

4. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 3 is characterized in that: The processing process of the wavelet downsampling attention module is shown by the following formula: Where Mean is the local average function, size is the local window size, and α is the weight factor.

5. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 2 is characterized in that: The boundary warning module includes a cascaded snake convolution connection unit with a convolution kernel of 3, a snake convolution connection unit with a convolution kernel of 5, and a normalized convolution activation unit; the snake convolution connection unit with a convolution kernel of 3 includes parallel 1×3, 3×1 snake convolution layers, 3×3 convolution layers, a first connection layer and a first 1×1 convolution layer, the snake convolution connection unit with a convolution kernel of 5 includes parallel 1×5, 5×1 snake convolution layers, 5×5 convolution layers, a second connection layer and a second 1×1 convolution layer, and the normalized convolution activation unit includes a cascaded third 1×1 convolution layer, a BatchNorm normalization layer and a ReLU activation function layer; wherein the parallel result of the 1×3, 3×1 snake convolution layers and the 3×3 convolution layers is cascaded with the first connection layer, and the parallel result of the 1×5, 5×1 snake convolution layers and the 5×5 convolution layers is cascaded with the second connection layer; Among them, the input feature map of the boundary warning module is used as the input of the serpentine convolution connection unit with a convolution kernel of 3, the output of the serpentine convolution connection unit with a convolution kernel of 3 is used as the input of the serpentine convolution connection unit with a convolution kernel of 5, the output of the serpentine convolution connection unit with a convolution kernel of 5 is added to the input feature map and used as the input of the normalized convolution activation unit, and the output of the normalized convolution activation unit is used as the output of the boundary warning module.

6. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 5 is characterized in that: The processing process of the boundary warning module is shown by the following formula: In the formula, SF(X In ,k) is the output feature of snake convolution, F(X In ,k) is the output feature of the snake convolution connection unit with convolution kernel k, X residual are the residual output features of two sets of snake convolution and ordinary convolution with convolution kernels of 3 and 5 along the x and y directions, k is the convolution kernel size, Cat is the channel splicing, DSConv 1×k and DSConv k×1 They are snake convolutions along the x and y directions, Conv k×k The convolution kernel is k standard convolution, Conv 1×1 is a 1×1 convolution, X In is the input feature of the boundary warning module, BN is batch normalization, ReLU is the activation function, X Out It is the boundary mask image output by the boundary warning module.

7. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 5 is characterized in that: The boundary warning module also introduces a boundary loss function, which is used to optimize the model's learning of crack boundary features based on the boundary features extracted by the boundary warning module and the boundary features of the real mask; The extraction process of the boundary features of the real mask is shown in the following formula: In the formula, mask is the real mask image, D is a 5x5 operator, and mask d is the image after the original image is expanded, mask boundary is the boundary image of the true mask.

8. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 2 is characterized in that: The boundary enhancement module includes a long dependency branch, a boundary enhancement branch and a fusion unit; The long dependency branch includes a local feature extraction unit and a global feature extraction unit connected in sequence; the local feature extraction unit includes a cascaded 3×3CBS module and a 1×1CBS module, wherein the CBS module includes a cascaded convolutional layer, a BatchNorm normalization layer and a SiLU activation function layer, and the global feature extraction unit includes a cascaded Unfold function, a Transformer module and a Fold function, wherein the Transformer module includes a cascaded self-attention unit and a feedforward network, the self-attention unit includes a cascaded layer normalization layer and a self-attention layer, and the feedforward network includes a layer normalization layer, a fully connected layer, a SiLU activation function regularization layer and a fully connected regularization layer; The boundary enhancement branch includes a Gaussian blur function and a Laplace transform function connected in sequence; the fusion unit includes a 1×1CBS module, a connection layer and a 3×3CBS module connected in sequence; In the boundary enhancement module, the feature map input to the boundary enhancement module is respectively used as the input of the long dependency branch and the boundary enhancement branch, wherein the input of the long dependency branch is used as the input of the local feature extraction unit, the output of the local feature extraction unit is used as the input of the global feature extraction unit, the output of the global feature extraction unit is used as the output of the long dependency branch, the input of the other boundary enhancement branch is used as the input of the Gaussian blur function, the output of the Gaussian blur function is subtracted from the input of the boundary enhancement branch, and the output is a differential Gaussian feature, the output of the Gaussian blur function is used as the input of the Laplace transform function, the output of the Laplace transform function is multiplied with the differential Gaussian feature, and the output is a Gaussian Laplace feature, the Gaussian Laplace feature is added to the input of the boundary enhancement branch to obtain the output of the boundary enhancement branch; the output of the long dependency branch is used as the input of the 1×1CBS module in the fusion unit, the output of the 1×1CBS module in the fusion unit is spliced ​​with the output of the boundary enhancement branch, and then passed through the 3×3CBS module to obtain the output feature as the boundary enhancement module.

9. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 8 is characterized in that: The processing process of the long dependency branch includes the following steps: The input feature map is extracted through the 3×3CBS module and the 1×1CBS module to obtain local features. The local features are extracted using the Unfold operation to extract the extraction sliding window in the input feature map according to the window size (ph, pw). The sliding window is input into the Transformer module based on the self-attention mechanism and rearranged through the Fold operation to obtain the global features. The calculation formula of the self-attention mechanism is as follows: In the formula, Q, K, and V represent different linearly transformed vectors of the sliding window, d head is the number of self-attention heads.

10. The road and bridge crack segmentation network detection method based on boundary enhancement and warning according to claim 8 is characterized in that: The processing process of the boundary enhancement branch includes the following steps: The extraction process of the differential Gaussian feature is shown in the following formula: DoG = I*(G(x,y,σ)-1); The Gaussian Laplace feature extraction process is shown in the following formula: The output characteristics of the boundary enhancement module are shown in the following formula: I output =DOG·LOG+I; Where DoG is the result of the input feature after differential Gaussian, G(x, y, σ) is the Gaussian blur function, σ is the blur degree, x and y are pixel coordinates, LOG is the result of the input feature after Gaussian Karplas, I is the input feature of the boundary enhancement module, and I output is the output feature after boundary enhancement.

Images