Pavement crack segmentation method and device based on multi-scale fusion network, and storage medium

Abstract

The invention relates to a pavement crack segmentation method and device based on a multi-scale fusion network, and a storage medium. The method comprises the steps of obtaining a pavement crack image data set, and dividing the pavement crack image data set into a training set and a test set; predicting the pavement crack image in the test set by using the trained pavement crack segmentation model based on the multi-scale fusion network to obtain a pavement crack segmentation result; the construction and training process of the pavement crack segmentation model based on the multi-scale fusion network specifically comprises the following steps: preprocessing a pavement crack image in a training set; constructing a multi-scale fusion network; training a multi-scale fusion network by using the preprocessed training set; and when the loss function of the multi-scale fusion network converges, obtaining a final pavement crack segmentation model. Compared with the prior art, the method has the advantages of accurately capturing the pavement crack boundary and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of crack image segmentation, and in particular to a pavement crack segmentation method, device and storage medium based on a multi-scale fusion network. Background Art

[0002] With the expansion of cities and the development of transportation networks, the total area of ​​pavement continues to increase, which brings greater challenges to pavement maintenance. The traditional method of pavement crack detection relying on manpower is not only inefficient, but also limited in accuracy, and also poses safety risks to workers. Therefore, it is particularly important to develop a system that can automatically identify pavement cracks. The goal of such a system is to be able to receive pavement images or videos as input and output crack detection results. Although traditional automated detection technologies already exist, they often perform poorly in terms of efficiency and accuracy. In recent years, researchers have tended to use deep learning technology to improve pavement crack detection algorithms. By using a large amount of training data, deep learning models can learn to accurately identify and segment the area where cracks are located. However, cracks themselves are diverse, including width, length, and texture complexity, which requires the detection system to be flexible and robust enough to deal with various types of cracks.

[0003] In pavement maintenance work, it is very important to use automated detection equipment based on deep learning, which can not only significantly improve detection efficiency, but also improve detection accuracy. However, in order for deep learning models to have strong detection capabilities, a large number of data sets are required for training. Although a rich pavement crack data set is provided on the open source project hosting platform, there is still relatively little research on accurate segmentation of crack edges. Summary of the invention

[0004] The purpose of the present invention is to provide a pavement crack segmentation method, device and storage medium based on a multi-scale fusion network in order to overcome the defects of the above-mentioned prior art that it is difficult to accurately capture the crack boundary and the edges are blurred or unclear.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, a pavement crack segmentation method based on a multi-scale fusion network is provided, comprising the following steps: S1, obtaining a pavement crack image dataset and dividing it into a training set and a test set; S2, using a trained pavement crack segmentation model based on a multi-scale fusion network to predict the pavement crack images in the test set to obtain a pavement crack segmentation result; wherein the construction and training process of the pavement crack segmentation model based on the multi-scale fusion network specifically comprises: preprocessing the pavement crack images in the training set; constructing a multi-scale fusion network; training the multi-scale fusion network using the preprocessed training set; and obtaining a final pavement crack segmentation model when the loss function of the multi-scale fusion network converges.

[0007] As a preferred technical solution, the pavement crack image dataset includes DeepCrack500, Crack200 and SDNET2018.

[0008] As a preferred technical solution, the pavement crack images in the training set are preprocessed, specifically including: dividing the pavement crack images into transverse crack images, vertical crack images and mesh crack images; and denoising the divided pavement crack images using median filtering.

[0009] As a preferred technical solution, the multi-scale fusion network includes a symmetrical encoder-decoder structure and an attention mechanism module and a pyramid pooling module respectively connected between the encoder and the decoder.

[0010] As a preferred technical solution, the attention mechanism module adopts the SENet attention network.

[0011] As a preferred technical solution, the network structure of the encoder includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer which are connected in sequence.

[0012] As a preferred technical solution, the multi-scale fusion network specifically includes: adding a corresponding attention mechanism module after the last convolution block of each convolution layer, and connecting the output end of each attention mechanism module to the upsampling output end of the corresponding convolution layer of the decoder to obtain a first fusion output end; adding corresponding pyramid pooling modules to the fourth convolution layer and the fifth convolution layer respectively, and connecting the output ends of each pyramid pooling module to each other to obtain a second fusion output end, and the second fusion output end is connected to the decoder.

[0013] As a preferred technical solution, the loss function is a binary cross entropy loss function, which is expressed as:

[0014]

[0015] Where N represents the number of pixels in the pavement crack image, y represents the true label value, the positive class value is 1, and the negative class value is 0. Represents the predicted probability value.

[0016] According to a second aspect of the present invention, there is provided a pavement crack segmentation device based on a multi-scale fusion network, comprising a memory, a processor, and a program stored in the memory, wherein the method described is implemented when the processor executes the program.

[0017] According to a third aspect of the present invention, there is provided a storage medium having a program stored thereon, wherein the program implements the method described above when executed.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. The pavement crack segmentation method provided by the present invention designs a pavement crack segmentation model based on a multi-scale fusion network. The model is an improvement on the traditional encoding-decoding segmentation model. By using the model to segment the pavement crack image, the crack edge information can be captured more accurately and clearly, and it is more suitable for the pavement crack background;

[0020] 2. In the pavement crack segmentation model provided by the present invention, the multi-scale fusion network includes a symmetrical encoder-decoder structure and an attention mechanism module and a pyramid pooling module respectively connected between the encoder and the decoder. Specifically, the SegNet network is used as the basic framework, and the attention mechanism module and the pyramid pooling module are added to improve the SegNet network, so that the improved SegNet network structure can capture small and important features such as crack edges and integrate multi-scale feature information of cracks of different lengths, widths and irregular shapes, thereby accurately capturing the pavement crack boundaries and realizing accurate segmentation of the crack edges. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the implementation process of the pavement crack segmentation method provided in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the structure of a multi-scale fusion network provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the SENet attention network structure provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a PPM network structure provided in an embodiment of the present invention;

[0025] Figure 5 This is a comparison diagram of segmentation results provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0027] Example

[0028] This embodiment provides a pavement crack segmentation method based on a multi-scale fusion network, including:

[0029] Step S1, obtaining a pavement crack image dataset and dividing it into a training set and a test set.

[0030] Step S2, using the trained pavement crack segmentation model based on the multi-scale fusion network to predict the pavement crack images in the test set to obtain the pavement crack segmentation results; wherein, the construction and training process of the pavement crack segmentation model based on the multi-scale fusion network specifically includes: first, preprocessing the pavement crack images in the training set; second, constructing a multi-scale fusion network; then, using the preprocessed training set to train the multi-scale fusion network; finally, when the loss function of the multi-scale fusion network converges, the final pavement crack segmentation model is obtained.

[0031] like Figure 1 As shown, the specific implementation process of the above method in this embodiment is as follows:

[0032] Step 1: Obtain a pavement crack image dataset and divide the dataset into a training set and a test set in a ratio of 8:2.

[0033] Specifically, in this embodiment, open source databases are searched, and the pavement crack image datasets finally obtained include DeepCrack500, Crack200 and SDNET2018.

[0034] Step 2: preprocess the pavement crack images in the training set.

[0035] Specifically, firstly, according to the distribution law of cracks in the pavement crack image dataset, the pavement crack images are divided into three categories: transverse crack images, vertical crack images and network crack images.

[0036] Next, median filtering is used to denoise the divided pavement crack image, eliminate isolated noise points, and make the pixel value closer to the true value. The calculation formula is as follows:

[0037] g(x,y)=med{f(xm,yn)} (1)

[0038] Where f(x, y) represents the original image, g(x, y) represents the denoised image, med represents the middle grayscale value, and m and n represent the offset of the filter window in the horizontal and vertical directions relative to the center pixel position, respectively.

[0039] Step 3: Build a multi-scale fusion network.

[0040] like Figure 2 As shown in the figure, the multi-scale fusion network uses SegNet as the basic framework, specifically: the basic framework is a symmetrical encoder-decoder structure and an attention mechanism module and a pyramid pooling module (PPM) connected between the encoder and the decoder respectively.

[0041] The encoder and decoder have the same structure and are symmetrically connected. The encoder downsamples through convolution and pooling to reduce the resolution of the feature image, and the decoder upsamples through deconvolution and depooling to accurately restore the resolution of the original image. The network structure of the encoder includes five convolutional layers connected in sequence: the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer. The size of the first convolutional layer is 64*3*3, with a step size of 2; the size of the second convolutional layer is 128*3*3, with a step size of 2; the size of the third convolutional layer is 256*3*3, with a step size of 2; the size of the fourth convolutional layer is 512*3*3, with a step size of 2; the size of the fifth convolutional layer is 512*3*3, with a step size of 2; the padding mode of the five convolutional layers is Valid. Correspondingly, the structure of the decoder is symmetrical and the same as the above structure.

[0042] The multi-scale fusion network designed in this embodiment is based on the above basic framework, and uses the added attention mechanism module and pyramid pooling module to perform multi-scale fusion of image features. The specific connection relationship of the network includes:

[0043] (1) After the last convolution block of each convolutional layer of the encoder, a corresponding attention mechanism module is added, and the output of each attention mechanism module is connected to the upsampled output of the corresponding convolutional layer of the decoder to obtain the first fusion output;

[0044] (2) Corresponding pyramid pooling modules are added to the fourth convolutional layer and the fifth convolutional layer respectively, and the output ends of the pyramid pooling modules are connected to each other to obtain a second fusion output end, and the second fusion output end is connected to the decoder.

[0045] Based on the above network structure improvement, the multi-scale feature fusion strategy adopted in this embodiment specifically includes:

[0046] (1) In this embodiment, the attention mechanism module adopts the SENet attention network. Each attention mechanism module connects the feature map output by it with the corresponding upsampled output of the decoder through multi-scale fusion, and outputs the first fused feature map through the first fusion output terminal.

[0047] Among them, through multi-scale fusion, the feature map output contains the regional feature information of the main cracks in the pavement crack image, that is, the information of the entire crack area where the crack is located; the corresponding up-sampling output of the decoder enhances the image features after the previous encoder reduces the resolution and restores the original crack features before the resolution is reduced. The first fused feature map retains the image feature information before the resolution is reduced, and emphasizes the key areas through the attention mechanism module. The decoder up-samples and restores the spatial resolution of the feature and maintains the high-level feature information, thereby enriching the image information of the crack.

[0048] (2) Then, corresponding pyramid pooling modules are added to the fourth convolutional layer and the fifth convolutional layer of the encoder respectively, and image features are fused between different pyramid pooling modules. A second fused feature map is output through the second fusion output terminal, and then the second fused feature map is passed to the decoder, so that it can obtain more image features, which is convenient for the subsequent more accurate segmentation of the edge features of the cracks.

[0049] The pyramid pooling module is mainly used to capture information at different scales. Since the number of channels of the fourth and fifth convolutional layers of the encoder has not changed, but the size of the feature map has been reduced, the pyramid pooling module is added to perform feature fusion in order to extract information under different feature maps. These feature information are fused together to obtain the second fused feature map including the crack edge features. Finally, the second fused feature map is used to generate the final pixel-level crack segmentation image through the Softmax layer binary classification. In this way, the specific location and boundary of the generated crack can be better segmented, making the crack segmentation accuracy higher. In addition, in addition to the crack edge information, the second fused feature map also includes crack texture features (such as fracture lines), crack shape features (such as horizontal, vertical, and mesh), crack connectivity features (generally cracks are continuous, but there are also interruptions), etc.

[0050] like Figure 3As shown, the SENet attention network introduced in this embodiment is a lightweight and computationally efficient channel attention mechanism, which aims to enhance the network's ability to capture key features by dynamically adjusting the weights of each channel in the feature map. The core components of the SENet attention network include: Squeeze, Excitation, and Scale. In the squeeze stage, the M×N×C feature map is compressed into a one-dimensional feature vector of 1×1×C through global average pooling, which retains all the information of the original feature map. In the excitation stage, this one-dimensional vector is processed through two fully connected layers: first, the dimension is reduced to 1×1×C / r, where r is the reduction ratio of the channel; then the ReLU activation function is used to increase the nonlinear characteristics, and then the dimension is restored to 1×1×C through another fully connected layer, and the Sigmoid function is used to generate the weights of each channel. In the scaling stage, the channel weights output by the excitation stage are multiplied by the original feature map channel by channel to adjust the contribution value of each channel, effectively improving the performance of the crack image segmentation task.

[0051] like Figure 4 As shown in the figure, the pyramid pooling module first extracts high-level feature representations from a series of convolutional layers. Subsequently, these feature maps are divided into several parts, each of which is globally average pooled using pooling windows of different sizes (such as 1×1, 2×2, 3×3, and 6×6). This operation enables each pooling layer to focus on objects of different sizes, thereby capturing richer details; then, each pooled feature map is restored to the size of the original feature map by upsampling, usually using a bilinear interpolation upsampling method; finally, all upsampled feature maps are concatenated with the original feature map to form a new feature map. This new feature map contains multi-scale information, which helps the model better understand and identify targets in the image and improves the performance of the model in semantic segmentation tasks.

[0052] Step 4: Use the preprocessed training set to train the multi-scale fusion network, and when the loss function of the multi-scale fusion network converges, obtain the final pavement crack segmentation model and save it.

[0053] Specifically, the loss function is the binary cross entropy loss function, expressed as:

[0054]

[0055] Where N represents the number of pixels in the pavement crack image, y represents the true label value (positive value is 1, negative value is 0), Represents the predicted probability value.

[0056] Step 5: Use the pavement crack segmentation model obtained in step 4 to predict the pavement crack images in the test set to obtain the pavement crack segmentation results.

[0057] In this embodiment, for the constructed multi-scale fusion network, in order to verify the performance of this method, it is compared with the original segmentation network and Unet network. The experimental results are as follows: Figure 5 As shown in Table 1:

[0058] Table 1 Comparison of performance indicators of this method and existing methods

[0059]

[0060]

[0061] Through the above experiments, it can be seen that the method proposed in this embodiment has improved precision, recall and F1-score, indicating the applicability of multi-scale fusion in the multi-pavement crack segmentation method. This method can effectively solve the problem of difficulty in accurately capturing crack boundaries and blurred or unclear edges, thereby improving the segmentation accuracy.

[0062] Furthermore, the present embodiment also provides a pavement crack segmentation device based on a multi-scale fusion network, including a memory, a processor, and a program stored in the memory, and the processor implements the aforementioned method when executing the program. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit to a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the steps in the aforementioned embodiments. For example, in some embodiments, the steps in the aforementioned embodiments may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the aforementioned method in any other appropriate manner (e.g., by means of firmware). The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, the hardware logic components of exemplary types that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0063] Further, the present embodiment also provides a storage medium on which a program is stored, and the aforementioned method is implemented when the program is executed. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine or completely on a remote machine or server. In the context of the present invention, a computer-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk-read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0064] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A pavement crack segmentation method based on a multi-scale fusion network, characterized in that: The following steps are involved: S1, obtain a pavement crack image dataset and divide it into a training set and a test set; S2, using the trained pavement crack segmentation model based on the multi-scale fusion network to predict the pavement crack images in the test set to obtain a pavement crack segmentation result; The construction and training process of the pavement crack segmentation model based on the multi-scale fusion network specifically includes: Preprocessing the pavement crack images in the training set; Construct a multi-scale fusion network; Using the preprocessed training set to train the multi-scale fusion network; When the loss function of the multi-scale fusion network converges, the final pavement crack segmentation model is obtained.

2. The pavement crack segmentation method based on multi-scale fusion network according to claim 1 is characterized in that: The pavement crack image datasets include DeepCrack500, Crack200 and SDNET2018.

3. The pavement crack segmentation method based on multi-scale fusion network according to claim 1 is characterized in that: Preprocessing the pavement crack images in the training set specifically includes: The pavement crack image is divided into a transverse crack image, a vertical crack image and a mesh crack image; The divided pavement crack image is denoised using median filtering.

4. The pavement crack segmentation method based on multi-scale fusion network according to claim 1 is characterized in that: The multi-scale fusion network includes a symmetrical encoder-decoder structure and an attention mechanism module and a pyramid pooling module respectively connected between the encoder and the decoder.

5. The pavement crack segmentation method based on multi-scale fusion network according to claim 4 is characterized in that: The attention mechanism module adopts the SENet attention network.

6. The pavement crack segmentation method based on multi-scale fusion network according to claim 4 is characterized in that: The network structure of the encoder includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer which are connected in sequence.

7. The pavement crack segmentation method based on multi-scale fusion network according to claim 6 is characterized in that: The multi-scale fusion network specifically includes: After the last convolution block of each convolution layer, a corresponding attention mechanism module is added, and the output end of each attention mechanism module is connected to the up-sampled output end of the corresponding convolution layer of the decoder to obtain a first fusion output end; Corresponding pyramid pooling modules are added to the fourth convolutional layer and the fifth convolutional layer respectively, and output ends of the pyramid pooling modules are connected to each other to obtain a second fusion output end, and the second fusion output end is connected to the decoder.

8. The pavement crack segmentation method based on multi-scale fusion network according to claim 1 is characterized in that: The loss function is a binary cross entropy loss function, expressed as: Where N represents the number of pixels in the pavement crack image, y represents the true label value, the positive class value is 1, and the negative class value is 0. Represents the predicted probability value.

9. A road crack segmentation device based on a multi-scale fusion network, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images