A pavement crack segmentation method based on a multi-frequency network

Abstract

The application provides a pavement crack segmentation method based on a multi-frequency network, comprising the following steps: S1), constructing a frequency division network; S2), preprocessing a pavement crack image X and a pavement crack label image Y; S3), inputting a training set into the frequency division network to perform model training; S4), inputting a test set image into the trained training model, processing by an encoder, and extracting high-frequency and low-frequency features of the crack test set image; S5), sending the high-frequency and low-frequency features into a decoder, fully utilizing the extracted high-frequency and low-frequency features of the crack through a frequency division maximum upper pooling operation, gradually recovering image details, and finally obtaining an accurate crack segmentation map. The application combines a frequency division convolution residual block and a frequency division maximum upper pooling operation, prevents the loss of extracted high-frequency and low-frequency features, improves the performance of the model, simultaneously utilizes the high-frequency and low-frequency features of the crack image, improves the feature extraction capability of the network for the crack image, and thus improves the crack segmentation effect.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-frequency network crack segmentation method based on frequency-division convolutional residual blocks and frequency-division maximum pooling. Background Technology

[0002] Road surface cracks are a typical surface defect in engineering structures. Road surfaces are frequently subjected to fatigue stress and cyclic loads, leading to structural defects. Cracks reduce local stiffness, causing material discontinuities that pose a significant threat to road surface service life and traffic safety. Proper road maintenance and upgrades can extend road surface service life, reduce fuel consumption, and improve safety.

[0003] Traditional manual methods for inspecting road surface defects are labor-intensive, time-consuming, and disruptive to traffic. To address these technical issues, existing technologies primarily employ automated road surface crack detection methods based on computer vision. This approach offers advantages such as low cost and high accuracy, effectively replacing manual labor and becoming the mainstream defect detection method.

[0004] However, in real-world scenarios, the complexity of crack topology, various crack widths, lighting conditions caused by weather, anti-skid textures that closely resemble cracks, and road surface features such as leaves, road markings, shadows, and manhole covers pose significant challenges to crack segmentation. Existing machine learning methods can only extract shallow crack features and are sensitive to noise in images, thus failing to extract crack features effectively.

[0005] Secondly, existing technologies also utilize convolutional neural networks to automatically extract deep crack features, achieving significant breakthroughs in crack detection and segmentation. However, in crack segmentation, the severe imbalance in the number of categories in crack images causes the network to prioritize background information. Most existing methods mitigate this problem by altering the weights of different categories in the loss function. While this improves model accuracy, it still results in the loss of detailed information and the omission of fine cracks. To retain more detailed information, most existing crack segmentation methods employ multi-scale feature fusion. Although multi-scale feature fusion can significantly improve model accuracy, these methods stack multiple levels of features together in some way, thus offering limited accuracy improvement and generating more spatial redundancy, increasing computational resource consumption, and extending inference time. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a road surface crack segmentation method based on a multi-frequency network. This invention has high operating efficiency and can simultaneously extract both the main information and detailed information of the crack.

[0007] The technical solution of this invention is: a method for pavement crack segmentation based on multi-frequency networks, comprising the following steps:

[0008] S1) Construct a frequency division network, which includes an encoder and a decoder;

[0009] S2) Preprocess the road surface crack image X and the road surface crack label image Y, and divide the images into training set and test set;

[0010] S3) The preprocessed training set from step S2) is put into the frequency division network described in step S1) for model training, high and low frequency features of road surface crack images are extracted, and crack segmentation is performed based on the extracted high and low frequency features to finally obtain the training model.

[0011] S4) Input the test set images into the trained model trained in step S3), and after processing by the encoder, extract the high and low frequency features of the crack test set images.

[0012] S5) The high and low frequency features from step S4) are sent to the decoder. The extracted high and low frequency features of the crack are fully utilized through the frequency division maximum pooling operation to gradually restore the image details and finally obtain an accurate crack segmentation map.

[0013] Preferably, in step S1), the encoder consists of five progressively increasing frequency-division convolutional residual blocks and max pooling operations, and the decoder consists of five frequency-division max pooling operations and frequency-division convolutions for feature extraction.

[0014] Preferably, in step S1), the frequency-division convolution residual block uses frequency-division convolution to divide the feature map into high-frequency groups and low-frequency groups, and two or three of the high-frequency groups and low-frequency groups form a frequency-division convolution residual block.

[0015] Preferably, in step S1), residual connections are used in each frequency division convolution residual block.

[0016] Preferably, in step S1), after the frequency-division convolutional residual block divides the feature map into high-frequency group and low-frequency group, BN and ReLU are used to process the feature map of each group separately.

[0017] Preferably, in step S1), if the frequency-division convolution residual block has two layers, then the frequency-division convolution residual block is represented as follows:

[0018]

[0019] Among them, Y H Y L These are the high- and low-frequency feature maps of the output; X H X LF represents the high-frequency and low-frequency feature maps of the input, respectively. H (X H ,{W i H}), F L (X L ,{W i L}) represent the residual mappings of the high-frequency group and the low-frequency group, respectively, and σ and θ represent BN and ReLU, respectively; W i H W i L Let X represent the residual mappings of the high-frequency group and the low-frequency group, respectively. H and X L Weights when passing through a neuron; X H and F H (X H ,{W i H}), X L and F L (X L ,{W i L The number of channels in the feature map must be the same. If they are not the same, linear projection is performed when performing a shortcut connection. W is the transformation matrix, which is used when performing linear projection to change the number of channels in the feature map.

[0020] The expression for the residual mapping is as follows:

[0021] F=θ(W2×σ(θ(W1×X)))

[0022] In the formula, W2 and W1 represent the weights of X when it passes through the neuron in the residual mapping, and X represents the road surface crack image;

[0023] Preferably, in step S1), the frequency-division convolution divides the feature map into different groups. The frequency-division convolution divides the feature map according to channels and extracts the high- and low-frequency features of the image respectively. The expression of the frequency-division convolution is as follows:

[0024] Y H =Y H→H +Y L→H =f H→H +f L→H ;

[0025] Y L =Y L→L +Y H→L =f L→L +f H→L ;

[0026] In the formula, f H→H f L→LThese represent ordinary convolutions that update information from high-frequency features to high-frequency features or from low-frequency features to low-frequency features, respectively.

[0027] f L→H f H→L These represent a series of operations that exchange information from high-frequency features to low-frequency features or from low-frequency features to high-frequency features, including upsampling, downsampling, and regular convolution.

[0028] Preferably, in step S1), the max pooling operation can fully utilize the high and low frequency features extracted by the encoder. For the frequency division max pooling operation, only the output at position (p, q) is considered, and its expression is as follows:

[0029]

[0030]

[0031] In the formula, The training parameters for the convolution operation are represented; (p, q) represent the position coordinates.

[0032] Indicates adjacent positions, k represents the kernel size of the convolution operation, and Ψ X The upper pooling operation is represented by the following expression:

[0033]

[0034] In the formula, Γ={i|0≤i≤h×w-1} represents the max pooling index, and h and w represent the height and width of the feature map, respectively.

[0035] Preferably, in step S2), the preprocessing of the road surface crack image X and the road surface crack label image Y includes random rotation, horizontal flipping, vertical flipping, random brightness adjustment, random gamma value adjustment, random contrast adjustment, random affine transformation, and random saturation adjustment; each operation has a 50% probability of being triggered.

[0036] Preferably, in step S3), the total loss L during model training is:

[0037] L=αL WBCE +βL Dice ;

[0038] In the formula, α and β represent the cross-entropy loss L, respectively. WBCE And Dice lost L Dice The weight.

[0039] Preferably, in step S3), the cross-entropy loss L WBCE Represented as:

[0040]

[0041] In the formula, Y + Y - Let represent cracked pixels and non-cracked pixels in the labeled image, respectively; P represents the sigmoid function that converts the feature map into a crack prediction map, and w p It was used to solve the problem of class imbalance in crack images.

[0042] Preferably, in step S3), the Dice loss L Dice The expression is:

[0043]

[0044] In the formula, N is the number of pixels in the image, p n y represents the neural network's prediction at the nth pixel. n This represents the crack label of the nth pixel, ε = 1 × 10 -5 .

[0045] The beneficial effects of this invention are as follows:

[0046] 1. This invention utilizes frequency-division convolutional residual blocks to construct a frequency-division encoder, which can extract high and low frequency information from crack images separately, reducing information loss;

[0047] 2. This invention utilizes frequency division maximum pooling to ensure that the decoder can decode the high and low frequency information of the crack, making full use of the high and low frequency information of the crack to accurately recover the detailed information of the crack; it improves the network's ability to extract crack image features, thereby improving the crack segmentation effect;

[0048] 3. Compared with existing methods, this invention uses multi-frequency feature representation to achieve multi-scale feature representation, rather than stacking multi-scale feature maps. This not only further improves the model accuracy, but also reduces spatial redundancy and model inference time.

[0049] 4. Compared with existing crack segmentation methods, this invention can predict accurate crack segmentation results on CFD datasets. In the quantitative comparison of crack segmentation effects, the three comparison indicators, precision (PR), recall (RE) and F1 score, all surpass existing methods. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram of the frequency division network of the present invention;

[0052] Figure 3This is a schematic diagram of the structure of the frequency division convolution residual block of the present invention.

[0053] Figure 4 This is a schematic diagram of the frequency division maximum pooling structure of the present invention.

[0054] Figure 5 This is a comparison chart of the crack segmentation performance of the method of this invention and existing algorithms on a CFD dataset. Detailed Implementation

[0055] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0056] like Figure 1 As shown, this embodiment provides a method for pavement crack segmentation based on a multi-frequency network, including the following steps:

[0057] S1) Construct a frequency division network, the structure of which is as follows: Figure 2 As shown, the frequency division network includes an encoder and a decoder; in this embodiment, the encoder consists of five progressively layered frequency division convolutional residual blocks and max pooling operations, and the decoder consists of five frequency division max pooling operations and frequency division convolutions for feature extraction.

[0058] S2) Preprocess the road surface crack image X and the road surface crack label image Y, and divide the images into training set and test set;

[0059] S3) The preprocessed training set from step S2) is put into the frequency division network described in step S1) for model training, high and low frequency features of road surface crack images are extracted, and crack segmentation is performed based on the extracted high and low frequency features to finally obtain the training model.

[0060] S4) Input the test set images into the trained model trained in step S3), and after processing by the encoder, extract the high and low frequency features of the crack test set images.

[0061] S5) The high and low frequency features from step S4) are sent to the decoder. The extracted high and low frequency features of the crack are fully utilized through the frequency division maximum pooling operation to gradually restore the image details and finally obtain an accurate crack segmentation map.

[0062] Preferably, in step S1), the frequency-division convolution residual block uses frequency-division convolution to divide the feature map into high-frequency groups and low-frequency groups, and two or three of the high-frequency groups and low-frequency groups form a frequency-division convolution residual block.

[0063] Preferably, in step S1), residual connections are used in each frequency division convolution residual block.

[0064] Preferably, in step S1), after the frequency-division convolutional residual block divides the feature map into high-frequency group and low-frequency group, BN and ReLU are used to process the feature map of each group separately.

[0065] Preferably, in step S1), if the frequency-division convolution residual block has two layers, then the frequency-division convolution residual block is represented as follows:

[0066]

[0067] Among them, Y H Y L These are the high-frequency and low-frequency feature maps of the output, respectively; X H X L F represents the high-frequency and low-frequency feature maps of the input, respectively. H (X H ,{W i H}), F L (X L ,{W i L}) represent the residual mappings of the high-frequency group and the low-frequency group, respectively, and σ and θ represent BN and ReLU, respectively; W i H W i L Let X represent the residual mappings of the high-frequency group and the low-frequency group, respectively. H and X L Weights when passing through a neuron; X H and F H (X H ,{W i H}), X L and F L (X L ,{W i L The number of channels in the feature map must be the same. If they are not the same, linear projection is performed when performing a shortcut connection. W is the transformation matrix, which is used when performing linear projection to change the number of channels in the feature map.

[0068] The expression for the residual mapping is as follows:

[0069] F=θ(W2×σ(θ(W1×X)))

[0070] In the formula, W2 and W1 represent the weights of X when it passes through the neuron in the residual mapping, and X represents the road surface crack image;

[0071] Preferably, in step S1), the frequency-division convolution divides the feature map into different groups. The frequency-division convolution divides the feature map according to channels and extracts the high- and low-frequency features of the image respectively. The expression of the frequency-division convolution is as follows:

[0072] Y H =Y H→H +Y L→H =f H→H +f L→H ;

[0073] Y L =Y L→L +Y H→L =f L→L +f H→L ;

[0074] In the formula, f H→H f L→L These represent ordinary convolutions that update information from high-frequency features to high-frequency features or from low-frequency features to low-frequency features, respectively.

[0075] f L→H f H→L These represent a series of operations that exchange information from high-frequency features to low-frequency features or from low-frequency features to high-frequency features. These operations include upsampling, downsampling, and regular convolution.

[0076] Preferably, in step S1), the max pooling operation can fully utilize the high and low frequency features extracted by the encoder. For the frequency division max pooling operation, only the output at position (p, q) is considered, and its expression is as follows:

[0077]

[0078]

[0079] In the formula, The training parameters for the convolution operation are represented; (p, q) represent the position coordinates.

[0080] Indicates adjacent positions, k represents the kernel size of the convolution operation, and Ψ X The upper pooling operation is represented by the following expression:

[0081]

[0082] In the formula, Γ={i|0≤i≤h×w-1} represents the max pooling index, and h and w represent the height and width of the feature map, respectively.

[0083] Preferably, in step S2), this embodiment obtains a CFD pavement crack dataset, consisting of 118 pavement crack images and their corresponding pavement crack label images. The dataset is divided into a training set containing 94 images and a test set containing 24 images in an 8:2 ratio. The preprocessing of the pavement crack image X and the pavement crack label image Y includes random rotation, horizontal flipping, vertical flipping, random brightness adjustment, random gamma value adjustment, random contrast adjustment, random affine transformation, and random saturation adjustment. Each operation has a 50% probability of being triggered.

[0084] Preferably, in step S3), the total loss L during model training is:

[0085] L=αL WBCE +βL Dice ;

[0086] In the formula, α and β represent the cross-entropy loss L, respectively. WBCE And Dice lost L Dice The weight.

[0087] Preferably, in step S3), the cross-entropy loss L WBCE Represented as:

[0088]

[0089] In the formula, Y + Y - Let represent cracked pixels and non-cracked pixels in the labeled image, respectively; P represents the sigmoid function that converts the feature map into a crack prediction map, and w p It was used to solve the problem of class imbalance in crack images.

[0090] Preferably, in step S3), the Dice loss L Dice The expression is:

[0091]

[0092] In the formula, N is the number of pixels in the image, p n y represents the neural network's prediction at the nth pixel. n This represents the crack label of the nth pixel, ε = 1 × 10 -5 .

[0093] Preferably, in step S4), the encoder includes a total of five frequency-division convolutional residual blocks and downsampling operations, such as... Figure 3 As shown, specifically:

[0094] The first to fifth layers of the encoder are represented as follows:

[0095]

[0096]

[0097]

[0098]

[0099]

[0100] In the formula, These represent the high-frequency and low-frequency feature maps output by the first layer encoder, respectively. These represent the high-frequency and low-frequency feature maps output from the second layer encoder to the second layer encoder, respectively.

[0101] downsample(OCRB(·)) represents the downsampling operation, and OCRB(·) represents the frequency division convolution residual block.

[0102] Preferably, in step S5), the structure of the decoder can be found in [reference needed]. Figure 4 As shown, the first to fifth layers of the decoder are represented as follows:

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] In the formula, These represent the high- and low-frequency feature maps of the outputs of the first, second, third, fourth, and fifth layers of the decoder, respectively.

[0109] OctConvBR(·) represents a combination of frequency division convolution, BN, and ReLU operations; OMU(·) represents frequency division max pooling operation.

[0110] Then, a 1×1 frequency-division convolution is used to reduce the number of channels in the output feature map to 1, as shown in the following expression:

[0111]

[0112] In the formula, Y outThis represents the output feature after a 1×1 frequency-division convolution; OctConv(·) is the frequency-division convolution. Finally, a sigmoid activation function is used to obtain the predicted probability map Y. t Its expression is as follows:

[0113] Y t =Sigmoid(Y out );

[0114] Sigmoid(·) represents the sigmoid activation function.

[0115] To illustrate the advantages of the network proposed in this invention, Table 1 presents a quantitative scoring table for crack segmentation using PR, RE, and F1 values ​​of the proposed method and some existing algorithms on a CFD dataset, which is widely used in the field of pavement crack segmentation. As can be seen from Table 1, the proposed method (Ours) achieves the best PR, RE, and F1 values, indicating that Ours achieves the best pavement crack segmentation performance.

[0116] Table 1. Quantitative analysis of the method of the present invention and existing algorithms on CFD datasets.

[0117] method PR RE F1 FCN 0.7923 0.8194 0.8056 HED 0.8674 0.8574 0.8624 U-Net 0.9118 0.9157 0.9137 DeepCrack_Zou 0.8565 0.8733 0.8624 DeepCrack_Liu 0.9179 0.8806 0.8989 FPHBN 0.8869 0.8797 0.8833 ECDFFNet 0.9162 0.9292 0.9227 DMA-Net 0.8926 0.9042 0.8984 Ours 0.9296 0.9276 0.9273

[0118] In addition, such as Figure 5 As shown in the figure, a comparison of the crack segmentation performance of the proposed method and existing algorithms on a CFD dataset is presented. The figure demonstrates that the proposed method can segment cracks more comprehensively, and it also performs better on fine cracks.

[0119] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A method for pavement crack segmentation based on multi-frequency networks, characterized in that, Includes the following steps: S1) Construct a frequency division network containing an encoder and a decoder; S2) Preprocess the road surface crack image X and the road surface crack label image Y, and divide the images into training set and test set; S3) The preprocessed training set in step S2) is put into the frequency division network constructed in step S1) for model training. Then, high and low frequency features of road surface crack images are extracted, and crack segmentation is performed based on the extracted high and low frequency features to finally obtain the training model. S4) Input the test set images into the trained model trained in step S3), and after processing by the encoder, extract the high and low frequency features of the crack test set images. S5) The high and low frequency features from step S4) are sent to the decoder. The extracted high and low frequency features of the crack are fully utilized through the frequency division maximum pooling operation to gradually restore the image details and finally obtain an accurate crack segmentation map. In step S1), the encoder consists of five progressively increasing frequency-division convolutional residual blocks and max pooling operations, with residual connections used in each frequency-division convolutional residual block; the decoder consists of five frequency-division max pooling operations and frequency-division convolutions for feature extraction. The frequency-division convolutional residual block uses frequency-division convolution to divide the feature map into high-frequency groups and low-frequency groups, and two or three of the high-frequency groups and low-frequency groups form a frequency-division convolutional residual block; The frequency-division convolutional residual block divides the feature map into high-frequency and low-frequency groups, and then uses BN and ReLU to process the feature map of each group separately. In step S1), if each frequency-division convolutional residual block has two layers, then the frequency-division convolutional residual block is represented as follows: ； ； in, These are the high- and low-frequency feature maps of the output, respectively. These represent the high-frequency and low-frequency feature maps of the input, respectively. , These represent the residual mappings for the high-frequency group and the low-frequency group, respectively. , These represent BN and ReLU operations, respectively. , Representing the residual mappings of the high-frequency group and the low-frequency group respectively and Weights when passing through neurons; and , and The number of channels must be the same; if they are not the same, linear projection is performed when performing a shortcut connection. W is the transformation matrix, which is used when performing linear projection to change the number of channels in the feature map. The expression for the residual mapping is as follows: ； In the formula, These represent the weights of X as it passes through the neuron in the residual mapping, where X represents the road surface crack image. In step S1), the frequency-division convolution divides the feature map into different groups. The frequency-division convolution divides the feature map according to channels and extracts the high- and low-frequency features of the image respectively. The expression of the frequency-division convolution is as follows: ； ； In the formula, These represent ordinary convolutions that update information from high-frequency features to high-frequency features and from low-frequency features to low-frequency features, respectively. These represent a series of operations for exchanging information from low-frequency features to high-frequency features and from high-frequency features to low-frequency features, respectively. These operations include upsampling, downsampling, and regular convolution. In step S1), the max pooling operation can fully utilize the high and low frequency features extracted by the encoder. For the frequency division max pooling operation, only the output at position (p, q) is considered, and its expression is as follows: ； ； In the formula, Represents the trainable parameters of the convolution operation; Indicates position coordinates; Indicates adjacent positions. This indicates the kernel size of the convolution operation. The upper pooling operation is represented by the following expression: In the formula, This represents the max-pooling index, and h and w represent the height and width of the feature map, respectively.

2. The method for pavement crack segmentation based on a multi-frequency network according to claim 1, characterized in that: In step S2), the preprocessing of the road surface crack image X and the road surface crack label image Y includes random rotation, horizontal flipping, vertical flipping, random brightness adjustment, random gamma value adjustment, random contrast adjustment, random affine transformation, and random saturation adjustment; each operation has a 50% probability of being triggered.

3. The method for pavement crack segmentation based on a multi-frequency network according to claim 1, characterized in that: In step S3), during the model training process, the total loss L is: ； In the formula, These represent the cross-entropy loss, respectively. and loss The weights; The aforementioned cross-entropy loss Represented as: ； In the formula, Let represent cracked pixels and non-cracked pixels in the labeled image, respectively; P represents the sigmoid function that transforms the feature map into a crack prediction map. It was used to address the class imbalance problem in crack images; The aforementioned Dice loss The expression is: ； In the formula, N is the number of pixels in the image. This represents the neural network's prediction at the nth pixel. This represents the crack label at the nth pixel. .

4. The method for pavement crack segmentation based on a multi-frequency network according to claim 1, characterized in that: The first to fifth layers of the encoder are represented as follows: ； ； ； ； ； In the formula, These represent the high-frequency and low-frequency feature maps output by the first layer encoder, respectively. These represent the high-frequency and low-frequency feature maps output by the second to fifth layers of the encoder, respectively; downsample(OCRB(·)) represents the downsampling operation described in the above steps, and OCRB(·) represents the frequency-division convolution residual block.

5. The method for pavement crack segmentation based on a multi-frequency network according to claim 1, characterized in that: The first to fifth layers of the decoder are represented as follows: ； ； ； ； ； In the formula, These represent the high- and low-frequency feature maps of the outputs of the first, second, third, fourth, and fifth layers of the decoder, respectively. OctConvBR(·) represents a combination of frequency division convolution, BN, and ReLU operations; OMU(·) represents frequency division max pooling operation; Then, a 1×1 frequency-division convolution is used to reduce the number of channels in the output feature map to 1, as shown in the following expression: ； In the formula, This represents the output feature after a 1×1 frequency-division convolution; OctConv(·) is the frequency-division convolution; Finally, a sigmoid activation function is used to obtain the predicted probability map Yt, the expression of which is as follows: ； Sigmoid(·) represents the sigmoid activation function.

Images