A road damage detection method based on RSUNet

Abstract

This application relates to a road defect detection method based on RSUNet. A road defect detection model based on RSUNet is constructed. The model includes a CNN encoder, a CNN decoder, and a Swin transformer module. The model can automatically detect cracks, reduce the use of manpower and material resources, and accelerate the process of road crack detection. On the other hand, the model can simultaneously target multiple road scenarios and accurately identify most defects. After saving the trained model weights, batch detection can be directly performed on the images to be detected.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular, to a road damage detection method based on RSUNet. Background Art

[0002] In the field of highway maintenance, the rapid and efficient identification of pavement defects is crucial. Whether on concrete or asphalt roads, cracks are a common type of pavement defect. If not promptly repaired and treated, repeated traffic and natural erosion can lead to further deterioration of the pavement's condition. Cracks can gradually evolve into potholes, potentially causing traffic accidents and posing a significant threat to road traffic.

[0003] Traditional pavement crack detection relies on manual visual inspection, requiring on-site investigations and traffic blockades. This is not only time-consuming and labor-intensive, but also inefficient and subject to human influence. For example, if personnel are constantly changing during maintenance, different people may make different judgments about the same road image. Furthermore, high-precision detection equipment is complex and expensive. Therefore, both the personnel and equipment requirements are insufficient to meet the needs of road maintenance.

[0004] In recent years, the application of deep learning technology in computer vision has become increasingly widespread, and researchers both domestically and internationally have conducted extensive research on automated pavement crack detection. If automated detection methods could replace dangerous and complex manual labor, it would not only save time and effort, reduce subsequent maintenance costs, but also mitigate traffic hazards. Existing technologies for automated pavement crack detection include image segmentation based on convolutional neural networks, ultrasonic image processing, laser-based image processing, and automatic crack detection based on morphology and cross-curvature. However, these technologies are primarily targeted at single pavement types or pavements with a limited number of defect types, and suffer from low road defect detection accuracy. Summary of the Invention

[0005] In order to overcome at least one deficiency in the prior art, the present application provides a road damage detection method based on RSUNet.

[0006] First, a road damage detection model based on RSUNet is provided, including: a CNN encoder, a CNN decoder, and a swin transformer module;

[0007] The CNN encoder includes multiple encoding units and multiple residual networks, and the multiple encoding units are connected through multiple residual networks (ResNet). The CNN decoder includes multiple decoding units, and the Swin transformer module includes multiple Swin transformer units. Each Swin transformer unit is connected to an encoding unit and a decoding unit. The Swin transformer unit is used to input the output of the encoding unit to the decoding unit.

[0008] The input of the CNN encoder is the road image to be detected, and the output of the CNN encoder is the feature map; the input of the CNN decoder is the feature map, and the output is the road disease segmentation image.

[0009] In one embodiment, each of the plurality of encoding units includes two convolutional layers, and the plurality of encoding units include a first encoding unit, a second encoding unit, a third encoding unit, a fourth encoding unit, and a fifth encoding unit;

[0010] The input of the first encoding unit is the road image to be detected. The first encoding unit performs convolution on the road image to be detected to obtain a first convolution result E1, that is, the output F1 of the first encoding unit;

[0011] The input of the second encoding unit is the output F1 of the first encoding unit. The second encoding unit convolves the output F1 of the first encoding unit to obtain a second convolution result E2. The output F1 of the first encoding unit passes through the residual network to obtain a first residual result G1. The second convolution result E2 and the first residual result G1 are spliced ​​together to obtain a first splicing result, i.e., the output F2 of the second encoding unit.

[0012] The input of the third encoding unit is the output F2 of the second encoding unit. The third encoding unit convolves the output F2 of the second encoding unit to obtain a third convolution result E3; the second convolution result E2 passes through the residual network to obtain a second residual result G2; the output F2 of the second encoding unit passes through the residual network to obtain a third residual result G3; the first residual result G1 and the third residual result G3 are residually added to obtain a first residual addition result S1; the result of the first residual addition result S1 passing through the residual network, the second residual result G2, and the third convolution result E3 are spliced ​​to obtain a second splicing result, i.e., the output F3 of the third encoding unit;

[0013] The input of the fourth encoding unit is the output F3 of the third encoding unit. The fourth encoding unit convolves the output F3 of the third encoding unit to obtain a fourth convolution result E4; the third convolution result E3 passes through the residual network to obtain a fourth residual result G4; the first residual addition result S1 passes through the residual network to obtain a fifth residual result G5; the first residual result G1 and the fifth residual result G5 are residually added to obtain a second residual addition result S2; the second residual addition result S2 is spliced ​​with the result of the residual network, the fourth residual result G4 and the fourth convolution result E4 to obtain a third splicing result, that is, the output F4 of the fourth encoding unit;

[0014] The input of the fifth coding unit is the output F4 of the fourth coding unit. The fifth coding unit performs a convolution on the output F4 of the fourth coding unit to obtain a fifth convolution result E5; the fourth convolution result E4 passes through the residual network to obtain a sixth residual result G6; the second residual addition result S2 passes through the residual network to obtain a seventh residual result G7; the first residual result G1 and the seventh residual result G7 are residually added to obtain a third residual addition result S3; the sixth residual result G6 and the third residual addition result S3 are spliced ​​with the result of the residual network and the fifth convolution result E5 to obtain a fourth splicing result; the fourth splicing result is convolved once to obtain the output F5 of the fifth coding unit.

[0015] In one embodiment, each decoding unit in the plurality of decoding units includes a concatenation layer and two convolutional layers; the plurality of decoding units include a first decoding unit, a second decoding unit, a third decoding unit, and a fourth decoding unit; the plurality of swin transformer units include a first swin transformer unit, a second swin transformer unit, a third swin transformer unit, and a fourth swin transformer unit;

[0016] The input of the first decoding unit is the up-sampled output of the CNN encoder and the output of the first swintransformer unit;

[0017] The input of the second decoding unit is the output of the second swin transformer unit and the output of the first decoding unit after upsampling;

[0018] The input of the third decoding unit is the output of the third swin transformer unit and the up-sampled output of the second decoding unit;

[0019] The input of the fourth decoding unit is the output of the fourth swin transformer unit and the up-sampled output of the third decoding unit, and the output is a road damage segmentation image.

[0020] Secondly, a road damage detection method based on RSUNet is provided, comprising:

[0021] Acquire the road image to be detected;

[0022] Input the road image to be detected into the road damage detection model based on RSUNet to obtain the road damage segmentation image;

[0023] The RSUNet-based road damage detection model is the RSUNet-based road damage detection model according to any one of claims 1-3.

[0024] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned road defect detection method based on RSUNet is implemented.

[0025] In a fourth aspect, a computer program product includes a computer program / instruction, which, when executed by a processor, implements the above-mentioned road damage detection method based on RSUNet.

[0026] Compared with the existing technology, the present application has the following beneficial effects: the RSUNet-based road damage detection method of the present application constructs a road damage detection model based on RSUNet, which can automatically detect cracks, reduce the use of manpower and material resources, and speed up the process of road crack detection; on the other hand, the model can simultaneously target multiple road scenes and accurately identify most diseases. After saving the trained model weights, batch detection can be directly performed on the images to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:

[0028] Figure 1 The following is a structural block diagram of a road damage detection model based on RSUNet according to an embodiment of the present application;

[0029] Figure 2 A comparison chart of road hazard detection results between the model of this application and the existing model is shown. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0031] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0032] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0033] This application embodiment provides a road damage detection model based on RSUNet. Figure 1 The structural block diagram of the road damage detection model based on RSUNet provided in accordance with the embodiment of the present application is shown in FIG. Figure 1 , the model includes: CNN encoder, CNN decoder and swin transformer modules.

[0034] The CNN encoder includes multiple encoding units and multiple residual networks, and the multiple encoding units are connected through multiple residual networks (ResNets). The CNN decoder includes multiple decoding units, and the Swin transformer module includes multiple Swin transformer units. Each Swin transformer unit is connected to an encoding unit and a decoding unit. The Swin transformer unit is used to input the output of the encoding unit to the decoding unit. The input of the CNN encoder is the road image to be detected, and the output of the CNN encoder is a feature map. The input of the CNN decoder is the feature map, and the output is a road disease segmentation image.

[0035] In the above embodiment, the road damage segmentation image includes the location information of the road damage, so the area of ​​the road damage can be determined based on the segmented image. The road damage detection model of this embodiment is based on RSUNet, which is applied to road damage recognition. The SwinTransformer structure is used for horizontal splicing. A local attention mechanism is added to each layer to reduce the computational workload, and a cross-layer window mechanism is used to enhance global perception capabilities, which can better capture the global and local features of the feature map.

[0036] In one embodiment, each of the plurality of encoding units includes two convolutional layers, and the plurality of encoding units include a first encoding unit, a second encoding unit, a third encoding unit, a fourth encoding unit, and a fifth encoding unit;

[0037] The input of the first encoding unit is the road image to be detected. The first encoding unit performs convolution on the road image to be detected to obtain a first convolution result E1, that is, the output F1 of the first encoding unit, with 64 channels.

[0038] The input of the second encoding unit is the output F1 of the first encoding unit. The second encoding unit convolves the output F1 of the first encoding unit to obtain a second convolution result E2; the output F1 of the first encoding unit passes through the residual network to obtain a first residual result G1; the second convolution result E2 and the first residual result G1 are spliced ​​to obtain a first splicing result, that is, the output F2 of the second encoding unit; the number of channels is 128.

[0039] The input of the third encoding unit is the output F2 of the second encoding unit. The third encoding unit convolves the output F2 of the second encoding unit to obtain a third convolution result E3; the second convolution result E2 passes through the residual network to obtain a second residual result G2; the output F2 of the second encoding unit passes through the residual network to obtain a third residual result G3; the first residual result G1 and the third residual result G3 are residually added to obtain a first residual addition result S1; the first residual addition result S1 is spliced ​​with the result of the residual network, the second residual result G2 and the third convolution result E3 to obtain a second splicing result, that is, the output F3 of the third encoding unit; the number of channels is 256.

[0040] The input of the fourth encoding unit is the output F3 of the third encoding unit. The fourth encoding unit convolves the output F3 of the third encoding unit to obtain a fourth convolution result E4; the third convolution result E3 passes through the residual network to obtain a fourth residual result G4; the first residual addition result S1 passes through the residual network to obtain a fifth residual result G5; the first residual result G1 and the fifth residual result G5 are residually added to obtain a second residual addition result S2; the second residual addition result S2 passes through the result of the residual network, the fourth residual result G4 and the fourth convolution result E4, and the third splicing result is obtained, that is, the output F4 of the fourth encoding unit; the number of channels is 512.

[0041] The input of the fifth encoding unit is the output F4 of the fourth encoding unit. The fifth encoding unit performs a convolution on the output F4 of the fourth encoding unit to obtain a fifth convolution result E5; the fourth convolution result E4 passes through the residual network to obtain a sixth residual result G6; the second residual addition result S2 passes through the residual network to obtain a seventh residual result G7; the first residual result G1 and the seventh residual result G7 are residually added to obtain a third residual addition result S3; the sixth residual result G6 and the third residual addition result S3 are spliced ​​with the result of the residual network and the fifth convolution result E5 to obtain a fourth splicing result; the fourth splicing result is convolved once to obtain the output F5 of the fifth encoding unit, and the number of channels is 1024.

[0042] In one embodiment, each decoding unit in the plurality of decoding units includes a concatenation layer and two convolutional layers; the plurality of decoding units include a first decoding unit, a second decoding unit, a third decoding unit, and a fourth decoding unit; the plurality of swin transformer units include a first swin transformer unit, a second swin transformer unit, a third swin transformer unit, and a fourth swin transformer unit;

[0043] The input of the first decoding unit is the up-sampled output of the CNN encoder and the output of the first swin transformer unit. Here, the number of channels of both is 512. The up-sampled output of the CNN encoder and the output of the first swin transformer unit are concatenated, and the number of channels of the concatenated result is 1024.

[0044] The input of the second decoding unit is the output of the second swin transformer unit and the up-sampled output of the first decoding unit; here, the number of channels of both is 256, and the number of channels of the spliced ​​result is 512.

[0045] The input of the third decoding unit is the output of the third swin transformer unit and the up-sampled output of the second decoding unit; here, the number of channels of both is 64, and the number of channels of the spliced ​​result obtained by splicing is 128.

[0046] The input of the fourth decoding unit is the up-sampled output of the fourth swin transformer unit and the output of the third decoding unit. Here, the number of channels of the up-sampled output of the fourth swin transformer unit and the output of the third decoding unit is 32. The two are spliced ​​together to obtain a spliced ​​result with 64 channels. The output of the fourth decoding unit is a road disease segmentation image with 1 channel.

[0047] Here, the input of the first swin transformer unit is the output of the fourth encoding unit, the input of the second swin transformer unit is the output of the third encoding unit, the input of the third swin transformer unit is the output of the second encoding unit, and the input of the fourth swin transformer unit is the output of the first encoding unit.

[0048] Furthermore, the RSUNet-based road defect detection model of the present application needs to be trained before being used for road defect detection; during the training process, training is performed on the NVIDIA RTX 3090 GPU, using Adam as the optimizer for model training, with a momentum value of 0.9, a weight decay of 0.0001, and a learning rate of 0.01.

[0049] A linear array camera is used to capture road images and preprocess the road images to be inspected. First, the multiple captured images are adjusted according to the size of the model input, and a dataset is constructed. The dataset is enhanced by performing operations such as rotation, flipping, and cropping to increase the diversity of the dataset and improve the robustness of the model. The model input image is an RGB image, which contains rich color and spatial information about the road. In order to better perform gradient updates and training, the images in the dataset need to be normalized. The dataset is divided into a training set and a test set. The preprocessed training set is fed into the constructed network model. A set of weights is obtained through training, which is then verified using the test set. The trained weights are saved, and finally a trained RSUNet-based road disease detection model is obtained.

[0050] The present application also provides a road damage detection method based on RSUNet, including:

[0051] Obtain a road image to be detected; input the road image to be detected into the RSUNet-based road damage detection model to obtain a road damage segmentation image; the RSUNet-based road damage detection model is based on the model in the previous article.

[0052] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned RSUNet-based road damage detection method.

[0053] An embodiment of the present application provides a computer program product, including a computer program / instruction, which implements the above-mentioned RSUNet-based road damage detection method when executed by a processor.

[0054] In order to further verify the effectiveness of the model and method of this application, the network performance of the model of this application is compared with the existing model through three indicators: Precision, Recall and F1 score. Table 1 shows the network performance evaluation results.

[0055] Table 1 Network performance evaluation results after improvement

[0056]

[0057] In Table 1, ST stands for SwinTransformer. As shown in Table 1, the model proposed in this application, which combines ResNet and SwinTransformer, achieves optimal results in road crack detection, improving performance. Precision increased from 71.90% to 74.85%, a 2.95% improvement; Recall increased from 64.98% to 67.63%, a 2.65% improvement; and F1-score increased from 68.26% to 71.05%, a 2.79% improvement.

[0058] Figure 2 The following figure shows the comparison of the road disease detection results of the model of this application and the existing model, where the first column is the original road image, the second column is the real road crack map, the third column is the road crack map detected by the TransUnet model, the fourth column is the road crack map detected by the TransUnet+ST model, and the fifth column is the road crack map detected by the model of this application. Figure 2 As can be seen, the road crack segmentation results using the basic TransUnet network model are relatively rough, the crack features are not refined enough, and are easily affected by factors such as noise. However, the model proposed in this application can restore the features of small edges in the image, while reducing noise interference and making the edge features smooth, which is beneficial for road crack image segmentation.

[0059] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A road damage detection model based on RSUNet, characterized by: include: CNN encoder, CNN decoder and swin transformer modules; The CNN encoder includes multiple encoding units and multiple residual networks, and the multiple encoding units are connected through the multiple residual networks Resnet; the CNN decoder includes multiple decoding units, and the swin transformer module includes multiple swin transformer units, each of which is connected to one encoding unit and one decoding unit; the swin transformer unit is used to input the output of the encoding unit to the decoding unit; The input of the CNN encoder is the road image to be detected, and the output of the CNN encoder is a feature map; the input of the CNN decoder is the feature map, and the output is a road damage segmentation image; Each of the plurality of encoding units includes two convolutional layers, and the plurality of encoding units include a first encoding unit, a second encoding unit, a third encoding unit, a fourth encoding unit, and a fifth encoding unit; The input of the first encoding unit is a road image to be detected, and the first encoding unit performs convolution on the road image to be detected to obtain a first convolution result E1, that is, an output F1 of the first encoding unit; The input of the second encoding unit is the output F1 of the first encoding unit. The second encoding unit convolves the output F1 of the first encoding unit to obtain a second convolution result E2. The output F1 of the first encoding unit passes through the residual network to obtain a first residual result G1. The second convolution result E2 and the first residual result G1 are spliced ​​to obtain a first splicing result, i.e., the output F2 of the second encoding unit. The input of the third encoding unit is the output F2 of the second encoding unit. The third encoding unit convolves the output F2 of the second encoding unit to obtain a third convolution result E3; the second convolution result E2 passes through the residual network to obtain a second residual result G2; the output F2 of the second encoding unit passes through the residual network to obtain a third residual result G3; The first residual result G1 and the third residual result G3 are residually added to obtain a first residual addition result S1; the first residual addition result S1 is spliced ​​with the result of the residual network, the second residual result G2 and the third convolution result E3 to obtain a second splicing result, i.e., the output F3 of the third encoding unit; The input of the fourth encoding unit is the output F3 of the third encoding unit. The fourth encoding unit convolves the output F3 of the third encoding unit to obtain a fourth convolution result E4; the third convolution result E3 passes through the residual network to obtain a fourth residual result G4; the first residual addition result S1 passes through the residual network to obtain a fifth residual result G5; Perform residual addition on the first residual result G1 and the fifth residual result G5 to obtain a second residual addition result S2; The second residual addition result S2 is spliced ​​with the result of the residual network, the fourth residual result G4 and the fourth convolution result E4 to obtain a third splicing result, that is, the output F4 of the fourth encoding unit; The input of the fifth encoding unit is the output F4 of the fourth encoding unit. The fifth encoding unit performs a convolution on the output F4 of the fourth encoding unit to obtain a fifth convolution result E5; the fourth convolution result E4 is passed through the residual network to obtain a sixth residual result G6; the second residual addition result S2 is passed through the residual network to obtain a seventh residual result G7; the first residual result G1 and the seventh residual result G7 are residually added to obtain a third residual addition result S3; The sixth residual result G6, the result of the third residual addition result S3 after the residual network and the fifth convolution result E5 are spliced ​​to obtain a fourth splicing result; the fourth splicing result is convolved once to obtain the output F5 of the fifth encoding unit.

2. The model according to claim 1, characterized in that Each decoding unit in the plurality of decoding units includes a splicing layer and two convolutional layers; the plurality of decoding units include a first decoding unit, a second decoding unit, a third decoding unit and a fourth decoding unit; the plurality of swin transformer units include a first swin transformer unit, a second swin transformer unit, a third swin transformer unit and a fourth swin transformer unit; The input of the first decoding unit is the up-sampled output of the CNN encoder and the output of the first swintransformer unit; The input of the second decoding unit is the output of the second swin transformer unit and the output of the first decoding unit after upsampling; The input of the third decoding unit is the output of the third swin transformer unit and the output of the second decoding unit after upsampling; The input of the fourth decoding unit is the result of upsampling the output of the fourth swin transformer unit and the output of the third decoding unit, and the output is a road damage segmentation image.

3. A road damage detection method based on RSUNet, characterized in that: include: Acquire the road image to be detected; Inputting the road image to be detected into a road damage detection model based on RSUNet to obtain a road damage segmentation image; The RSUNet-based road damage detection model is the RSUNet-based road damage detection model according to any one of claims 1-2.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the road damage detection method based on RSUNet according to claim 3.

5. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the road damage detection method based on RSUNet according to claim 3.

Images