A method for super-resolution reconstruction of road defect images based on improved SwinIR

Abstract

The application discloses a kind of based on improved SwinIR's road surface defect image super-resolution reconstruction method, specifically comprising the following steps: step 1: obtaining road surface defect image, constructs first data set;Step 2: generate second data set based on first data set;Step 3: construct the road surface defect image super-resolution reconstruction model based on improved SwinIR;Step 4: design the improved L1 loss function combined with wasserstein distance;Step 5: the training set and validation set in second data set are used to train model;Step 6: the performance of the model is evaluated using the test set of second data set;Step 7: the model weight obtained by step 5 is input into the improved SwinIR model together with the first data set, and a high-resolution image is output.Compared with the prior art, the road surface image super-resolution reconstruction method based on improved SwinIR proposed in the application can effectively improve the resolution of road surface defect image and enhance the road surface defect detail recovery capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a road defect image super-resolution reconstruction method based on improved SwinIR. BACKGROUND

[0002] With the growth of the service life of highways and the influence of environmental factors (such as temperature changes, traffic loads, rain erosion, etc.), road defect problems gradually appear, such as cracks, potholes, etc. If these defects are not discovered and repaired in time, not only will the service life of the highway be greatly shortened, but also traffic accidents may be caused. Therefore, regular road inspection is crucial to ensuring the safety of highways and extending their service life.

[0003] However, the current road defect detection method still faces many challenges. The traditional road defect image acquisition method is affected by the quality of the imaging device and external environment (such as changes in lighting, weather conditions, etc.), resulting in low-quality images with blurred details. Especially when the defects are very fine, the imaging device has difficulty in clearly capturing these features. In addition, road defect images are often disturbed by noise, further reducing the clarity of the images and affecting the accuracy of subsequent defect identification and road performance analysis. If low-quality images are directly used for defect detection, the results will often have large errors. Therefore, super-resolution reconstruction on low-resolution, blurred road image data has become a key to improving detection accuracy.

[0004] Traditional image processing methods mainly use interpolation algorithms such as bilinear interpolation and cubic interpolation. These methods are simple to calculate and fast to run, and can meet real-time processing needs. However, since interpolation methods only rely on existing pixel information in low-resolution images, they fail to fully exploit potential image features, and when dealing with complex textures and fine structures, the reconstructed images will have problems such as missing details and blurred edges, which will affect the accuracy of crack identification. In contrast, deep learning-based super-resolution reconstruction methods can learn the mapping relationship between low-resolution and high-resolution images through an end-to-end approach, significantly improving image detail restoration, providing clearer and more accurate input images for road defect detection, and having important application value in highway maintenance and safety management. However, as the number of layers of deep learning models increases, the model size continues to expand, and deployment on resource-constrained vehicle platforms or edge devices still faces significant challenges. SUMMARY

[0005] In view of the problems in the prior art, the present application provides a road defect image super-resolution reconstruction method based on an improved SwinIR. Through the improved SwinIR model, the quality of the road defect image affected by device performance and environmental interference is effectively enhanced, and the subtle defects are more easily identified, thereby improving the accuracy of subsequent road defect detection. At the same time, the method reduces the computational overhead while maintaining the reconstruction effect, and can be applied to actual scenarios with limited resources.

[0006] The technical solution provided by the present application comprises the following steps:

[0007] Step 1: Obtain a road defect image and construct a first data set; the road images in the first data set can be captured by a digital camera or obtained from a drive recorder or a monitoring video.

[0008] Step 2: Generate a second data set based on the first data set; the second data set generates a low-resolution image pair corresponding to a high-resolution image by downsampling the first data set by two times, and divides it into a training set, a validation set and a test set.

[0009] Step 3: Construct a road defect image super-resolution reconstruction model based on an improved SwinIR; further, the step 3 comprises steps 3.1 to 3.3:

[0010] Step 3.1: input the low-quality image training set and validation set of the second data set into a shallow feature extraction module; the shallow feature extraction module uses a 3x3 convolution layer to extract initial features and generate a preliminary feature map;

[0011] Step 3.2: input the preliminary feature map into a deep feature extraction module; the deep feature extraction module is composed of 5 RSTB modules, 5 SimAM attention mechanisms and a 3x3 convolution; each RSTB module includes 6 STL layers and a deep separable convolution module;

[0012] Specifically, a SimAM attention mechanism is added after each RSTB module of the SwinIR model to optimize the feature extraction process; and a deep separable convolution is introduced inside the RSTB module to reduce the number of model parameters;

[0013] Step 3.3: send the output of the deep feature extraction module to a high-quality repair module for image reconstruction; the high-quality repair module is composed of an upsampling module and a 3x3 convolution;

[0014] Specifically, the upsampling module uses the PixelShuffle method to upsample the feature map to improve the image resolution; the upsampled feature map is further refined by a 3x3 convolution layer to effectively repair the image details, and finally outputs a high-quality reconstructed image.

[0015] Step 4: design an improved L1 loss function combined with Wasserstein distance;

[0016] Specifically, the improved L1 loss function has a calculation formula as follows:

[0017] L Total = loss_weight x L L1 + wass_weight x L Wass (1)

[0018] wherein, L L1 represents L1Loss, used to measure the absolute difference between the predicted value and the true value; L Wass represents the Wasserstein distance loss, used to measure the distribution difference between the predicted value and the true value; loss_weight and wass_weight are the weights of L1Loss and Wasserstein distance respectively, which are taken as 1.0 and 0.1 respectively here;

[0019] The calculation formula of L1 Loss is as follows:

[0020]

[0021] The calculation formula of Wasserstein distance is as follows:

[0022]

[0023] wherein, f(x i ) and y i represent the predicted value and the corresponding true value of the i-th sample respectively, n is the number of samples, and Hist(·) represents the histogram operation.

[0024] Step 5: training the improved SwinIR-based road defect image super-resolution reconstruction model in step 3 by using the training set and the validation set of the second data set, and saving the trained model and the weight;

[0025] Further, the step 5 specifically includes steps 5.1 to 5.4:

[0026] Step 5.1: setting the training parameters of the improved SwinIR-based road defect image super-resolution reconstruction model, including: reconstruction ratio, learning rate, optimizer, iteration number, and batch size;

[0027] Step 5.2: input the training set and validation set image pairs of the second data set into the improved SwinIR pavement defect image super-resolution reconstruction model, calculate the gradient of the loss function on the model parameters using the back propagation algorithm, update the parameters of the model according to the gradient, and gradually reduce the loss function;

[0028] Step 5.3: monitor the loss function value and performance indicators during training, stop training when the loss function of the training set no longer decreases, and the evaluation indicators PSNR and SSIM no longer improve, to avoid model overfitting;

[0029] Step 5.4: after training is completed, select the optimal model from it and save the training weights.

[0030] Step 6: test the optimal model selected in step 5.4 using the test set of the second data set, and obtain the final pavement defect super-resolution reconstruction model;

[0031] Further, step 6 specifically includes steps 6.1 to 6.3:

[0032] Step 6.1: input the test set image pairs into the optimal model selected in step 5 for testing;

[0033] Step 6.2: calculate the model performance indicator PSNR, and the specific calculation formula of PSNR is as follows:

[0034]

[0035] Where x(i,j) and y(i,j) represent the values of image x and image y at the i,j pixel position, respectively; m and n represent the width and height of the image, and MAX x is the maximum pixel value in image x;

[0036] Calculate the model performance indicator SSIM, and the specific calculation formula of SSIM is as follows:

[0037]

[0038] Where μ x and μ y respectively represent the local mean of two images x and y, which is used to estimate the brightness; c1 and c2 are constant terms to prevent the divisor from being zero; and represent the local variance of images x and y, which is used to measure the contrast of the image; σ xy represents the covariance of images x and y, which is used to measure the structural similarity between images;

[0039] Step 6.3: Evaluate the performance indicators of the test set. If the PSNR and SSIM indicators of the test set are similar to those of the training set, it indicates that the model meets the requirements of generalization, and the final road defect image super-resolution reconstruction model based on the improved SwinIR is obtained.

[0040] Step 7: Input the first data set and the training weight saved in step 5 into the improved SwinIR road defect image super-resolution reconstruction model for reconstruction, i.e. to obtain the final super-resolution reconstruction image.

[0041] Compared with the prior art, the beneficial effects of the present application are:

[0042] (1) The present application discloses a road defect image super-resolution reconstruction method based on improved SwinIR. The method introduces a SimAM attention mechanism in the deep feature extraction module of SwinIR to enhance the model's feature perception and extraction ability for road defect areas, effectively improving the recognition of road defect areas.

[0043] (2) The method introduces a depth separable convolution inside the RSTB module to reduce the computational complexity while maintaining the reconstruction ability of road defect image details, achieving model lightweight while ensuring that the reconstruction effect is not significantly affected.

[0044] (3) The method uses an L1 loss function improved based on Wasserstein distance to improve the optimization effect of the model in the super-resolution reconstruction task, making the reconstructed road defect image clearer and reducing the interference of artifacts, thereby improving the accuracy of subsequent road defect detection. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the road defect image super-resolution reconstruction method based on the improved SwinIR of the present application;

[0046] Figure 2 The architecture diagram of the road defect image super-resolution reconstruction model based on the improved SwinIR of the present application;

[0047] Figure 3 The schematic diagram of the SimAM attention mechanism;

[0048] Figure 4 The internal schematic diagram of the RSTB module;

[0049] Figure 5 The depth separable convolution processing flow; DETAILED DESCRIPTION

[0050] In order to make the technical solutions, structural features, achieved purposes and advantages of the present application clearer and more comprehensible, the present application will be further described in detail below in conjunction with specific embodiments and the accompanying drawings; it should be noted that the specific embodiments described herein are only used to more clearly explain the present application and do not limit the present application;

[0051] Figure 1 is a flowchart of the road defect image super-resolution reconstruction method based on the improved SwinIR disclosed in the present application, and the implementation process is as follows:

[0052] Step 1: Obtain the road defect image and construct the first data set; in the first data set, the road defect image can be captured by a digital camera, obtained from a drive recorder or a monitoring video.

[0053] In this embodiment, in order to better evaluate the reconstruction effect of the road defect image super-resolution reconstruction method based on the improved SwinIR disclosed in the present application, the Japan data in the published RDD2022 data set is used as the first data set.

[0054] Step 2: Generate a second data set based on the first data set, the second data set is generated by two times down-sampling of the first data set, to generate a low-resolution image pair corresponding to a high-resolution image, and divide it into a training set, a validation set and a test set.

[0055] Preferably, the divided data set includes: 6320 groups of high-resolution-low-resolution image pairs as the training set, 790 groups as the test set, and 10 groups as the validation set. In this embodiment, the size of the low-resolution image is 300x300, and the size of the corresponding high-resolution image is 600x600.

[0056] Step 3: Construct a road defect image super-resolution reconstruction model based on the improved SwinIR, the improved SwinIR model structure is as shown in Figure 2 The model construction process specifically includes steps 3.1 to 3.3:

[0057] Step 3.1: input the low-quality image training set and validation set of the second data set into the shallow feature extraction module, the shallow feature extraction module uses a 3x3 convolution layer to extract initial features to generate a preliminary feature map.

[0058] Step 3.2: input the preliminary feature map into the deep feature extraction module, the deep feature extraction module is composed of 5 RSTB modules, 5 SimAM attention mechanisms and 1 3x3 convolution, wherein each RSTB module includes 6 STL layers and 1 deep separable convolution module.

[0059] Specifically, a SimAM attention mechanism is added after each RSTB module of the SwinIR model to optimize the feature extraction process; and depthwise separable convolution is introduced inside the RSTB module to reduce the number of model parameters.

[0060] Furthermore, the SimAM attention mechanism structure diagram is as follows: Figure 3 As shown: It is a 3-D parameterless attention module ( Figure 3 c), with 1-D attention ( Figure 3 a) 2-D attention compared to ( Figure 3 b) SimAM focuses on the importance of both channel and spatial features, and can derive three-dimensional weights without adding extra parameters to the network. SimAM does not require adding extra training layers and parameters to the original network, can evaluate the importance of each neuron, and has high computational efficiency. The energy function formula for each neuron is defined as shown in Equation (1):

[0061]

[0062] In the formula: e t This represents the attention weights obtained; M is the number of samples; w t and b t These represent the bias and weight, respectively; y represents the sample label; x i Let w represent the feature vector of the i-th sample; λ is the coefficient of the regularization term in the formula; t represents the mean of the current feature channel, where the parameter w t and b t There is a fast, closed-form solution that can be solved without iteration or other optimization algorithms, as shown in Equations (2) and (3):

[0063]

[0064] Where: μ t and σ t 2 denoted as the mean and variance of all neurons except t, where λ is the coefficient of the regularization term;

[0065] Furthermore, a lightweight RSTB module is designed for the deep feature extraction stage, such as... Figure 4 As shown, by introducing depthwise separable convolution, the number of model parameters is significantly reduced.

[0066] Depthwise separable convolution mainly includes two steps: depthwise convolution and pointwise convolution. Its processing flow is as follows: Figure 5As shown. First, the deep convolution separately performs KxK convolution on each channel of the input feature, only calculates in the spatial dimension, without changing the number of channels, so as to extract local features, second, the point-by-point convolution uses 1x1 convolution, fuses the information of all channels, and adjusts the number of channels to enhance the feature expression ability.

[0067] Step 3.3: The output of the deep feature extraction module is sent to the high-quality repair module for image reconstruction, and the high-quality repair module is composed of an up-sampling module and a 3x3 convolution;

[0068] Specifically, the up-sampling module uses the PixelShuffle method to up-sample the feature map to improve the image resolution; the up-sampled feature map is further refined through the 3x3 convolution layer, effectively repairing the image details, and finally outputting the high-quality reconstructed image.

[0069] Step 4: Design an improved L1 loss function combined with Wasserstein distance, and its calculation formula is:

[0070] L Total =loss_weight×L L1 +wass_weight×L Wass (4)

[0071] Wherein, L L1 1Loss, used to measure the absolute difference between the predicted value and the true value; L Wass Wasserstein distance loss, used to measure the distribution difference between the predicted value and the true value; loss_weight and wass_weigiht are the weights of L1Loss and Wasserstein distance respectively, which are taken as 1.0 and 0.1 respectively here;

[0072] The calculation formula of L1 Loss is:

[0073]

[0074] The calculation formula of Wasserstein distance is:

[0075]

[0076] Wherein, f(x i ) and y i represent the predicted value and the corresponding true value of the i-th sample respectively, n is the number of samples, and Hist(·) represents the histogram operation;

[0077] Step 5: Train the improved SwinIR-based road defect image super-resolution reconstruction model described in step 3 using the training set and validation set of the second data set, and save the trained model and weights. The step 5 further comprises steps 5.1 to 5.4:

[0078] Step 5.1: Set the training parameters of the improved SwinIR-based road defect image super-resolution reconstruction model, including reconstruction ratio, learning rate, optimizer, iteration rounds, and batch size.

[0079] In this embodiment, the optimizer reconstruction ratio is 2, the initial learning rate lr is 2e-4, the optimizer is Adam, the iteration rounds total_iter is 500000, and the batch size is 1.

[0080] Step 5.2: Input the image pairs of the training set and validation set of the second data set into the improved SwinIR road defect image super-resolution reconstruction model, calculate the gradient of the loss function with respect to the model parameters using the backpropagation algorithm, and update the model parameters according to the gradient to gradually reduce the loss function.

[0081] Specifically, in each training iteration, first, the model calculates the feature representation of the low-resolution image through forward propagation, and gradually restores the high-resolution details through hierarchical window multi-head self-attention (W-MSA) and RSTB modules, finally generating a super-resolution reconstructed image.

[0082] Secondly, the error between the reconstructed image and the real high-resolution image is calculated, and the loss function (such as the improved L1LOSS) is calculated based on the error;

[0083] Thirdly, the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm, and the model parameters are updated through the optimization algorithm.

[0084] Repeat the above process, and gradually optimize the model parameters to make the generated high-resolution image closer to the real image, thereby improving the super-resolution reconstruction effect.

[0085] Step 5.3: Monitor the loss function value and performance indicators during training. When the loss function of the training set no longer decreases, and the evaluation indicators PSNR and SSIM no longer improve, stop training to avoid model overfitting.

[0086] Step 5.4: After training is completed, select the optimal model from it and save the training weights.

[0087] Step 6: Test the optimal model selected in step 5.4 using the test set of the second data set, and obtain the final road defect super-resolution reconstruction model. The step 6 further comprises steps 6.1 to 6.3:

[0088] Step 6.1: input the test set image pair into the optimal model selected in step 5 for testing.

[0089] Step 6.2: calculate the model performance indicator PSNR, which is mainly used to evaluate the quality of images after compression or reconstruction processing. The higher the value, the smaller the image distortion, and the better the reconstruction effect. The specific calculation formula of PSNR is as follows:

[0090]

[0091] where x(i,j) and y(i,j) represent the values of the i,j pixel position of images x and y respectively; m and n represent the width and height of the image, MAX x is the maximum pixel value in image x.

[0092] Calculate the model performance indicator SSIM, which is a combination of the distortion of image contrast, image structure and image brightness. The output result is between [0, 1], and the closer the value of SSIM to 1, the smaller the image distortion, that is, the better the image reconstruction effect. The specific calculation formula of SSIM is as follows:

[0093]

[0094] where μ x and μ y respectively represent the local mean of two images x and y, which is used to estimate the brightness; c1 and c2 are constant terms to prevent the divisor from being zero; and represent the local variance of images x and y, which is used to measure the contrast of the image; σ xy represents the covariance of images x and y, which is used to measure the structural similarity between images.

[0095] Model parameter quantity is used to measure the complexity of neural network model, that is, the total number of weights that the model needs to learn and store in the training and inference process. The smaller the parameter quantity, the lighter the model, the lower the computational resource consumption, and it is suitable for actual deployment. Therefore, under the premise of ensuring image reconstruction effect, the smaller the parameter quantity is, the better.

[0096] Step 6.3: evaluate the performance indicators of the test set. If the PSNR and SSIM indicators of the test set are similar to those of the training set, it means that the model meets the requirements of generalization, and the final pavement defect image super-resolution reconstruction model based on improved SwinIR is obtained.

[0097] Step 7: input the first data set and the training weight saved in step 5 into the improved SwinIR pavement defect image super-resolution reconstruction model for reconstruction, that is, obtain the final super-resolution reconstruction image.

[0098] In this embodiment, in order to verify the effect of the pavement defect image super-resolution reconstruction model based on the improved SwinIR disclosed by the present application, the EDSR model, the HAN model, the SwinIR model, the HAT model and the model proposed by the present application are used to test on the second data set, and the evaluation results are shown in Table 1, wherein the evaluation indexes PSNR, SSIM and model parameters of the pavement defect image super-resolution reconstruction data of the model proposed by the present application are better than those of other comparative models.

[0099] Table 1 comparison experiment results

[0100]

[0101] In this embodiment, in order to further verify the effectiveness of the improved SwinIR model, the present application inputs the reconstruction images before and after the improvement into the YOLOv7 model for pavement defect detection, and compares and analyzes the detection results in detail, as shown in Table 2, the improved model has improved in the indexes of map@0.5, Precision, Recall and the like.

[0102] Table 2 pavement defect detection results before and after the improved SwinIR algorithm

[0103]

[0104] In this embodiment, in order to verify the effectiveness of the improved SwinIR model disclosed by the present application, an ablation experiment is carried out based on the self-made data set, and the evaluation results are shown in Table 3, wherein the evaluation indexes PSNR, SSIM and model parameters of the pavement defect image super-resolution reconstruction data of the model proposed by the present application are better than those of other models.

[0105] Table 3 ablation experiment results

[0106]

[0107] The above only describes one embodiment of the present application, and does not limit the patent scope of the present application. For those skilled in the art, the present application can have various changes and variations, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for pavement defect image super-resolution reconstruction based on improved SwinIR, characterized in that, Specifically comprising the following steps: Step 1: Obtain the road defect image and construct the first data set; in the first data set, the road defect image can be captured by a digital camera, obtained from a drive recorder or a monitoring video; Step 2: Generate a second data set based on the first data set, which is generated by downsampling the first data set by two times, generating a low-resolution image pair corresponding to a high-resolution image, and dividing it into a training set, a validation set and a test set; Step 3: Construct a road defect image super-resolution reconstruction model based on the improved SwinIR, which includes a shallow feature extraction module, a deep feature extraction module and a high-quality repair module, and the construction of the model further includes steps 3.1 to 3.3: Step 3.1: Input the low-quality image training set and validation set of the second data set into the shallow feature extraction module, which uses a 3x3 convolution layer to extract initial features and generate a preliminary feature map; Step 3.2: Input the preliminary feature map into the deep feature extraction module, which is composed of 5 RSTB modules, 5 SimAM attention mechanisms and 1 3x3 convolution, wherein each RSTB module contains 6 STL layers and 1 deep separable convolution module; Step 3.3: Send the output of the deep feature extraction module to the high-quality repair module for image reconstruction, which is composed of an upsampling module and a 3x3 convolution; the upsampling module uses the PixelShuffle method to upsample the feature map to improve the image resolution; the upsampled feature map is further refined by a 3x3 convolution layer to effectively repair the image details, and finally outputs a high-quality reconstructed image; Step 4: Design an improved L1 loss function combined with Wasserstein distance, whose calculation formula is: L Total = loss_weight x L L1 + wass_weight x L Wass (1) wherein, L L1 represents L1Loss, used to measure the absolute difference between the predicted value and the true value; L Wass represents the Wasserstein distance loss, used to measure the distribution difference between the predicted value and the true value; loss_weight and wass_weight are the weights of L1Loss and the Wasserstein distance, respectively; The calculation formula of L1 loss is: The calculation formula of Wasserstein distance is: where f(x i ) and y i represent the predicted value and the corresponding true value of the i-th sample, respectively, n is the number of samples, and Hist(·) represents the histogram operation. Step 5: Train the road defect image super-resolution reconstruction model based on the improved SwinIR in step 3 using the training set and validation set of the second data set, and save the trained model and weights, wherein step 5 further includes steps 5.1 to 5.4: Step 5.1: Set the training parameters of the road defect image super-resolution reconstruction model based on the improved SwinIR; Step 5.2: Input the training set and validation set images of the second data set into the improved SwinIR road defect image super-resolution reconstruction model, calculate the gradient of the loss function on the model parameters using the back propagation algorithm, and update the parameters of the model according to the gradient to gradually reduce the loss function; Step 5.3: Monitor the loss function value and performance indicators during training, stop training when the loss function of the training set no longer decreases and the evaluation indicators PSNR and SSIM no longer improve to avoid model overfitting; Step 5.4: After training is completed, select the optimal model and save the training weights; Step 6: Test the optimal model selected in step 5.4 using the test set of the second data set, and evaluate the test results. If the PSNR and SSIM of the model meet the generalization requirements, a final pavement defect image super-resolution reconstruction model based on the improved SwinIR is obtained. Step 6 further includes steps 6.1 to 6.3: Step 6.1: Input the test set image pair into the optimal model selected in step 5 for testing; Step 6.2: Calculate the model performance indicator PSNR, and the specific calculation formula of PSNR is as follows: where x(i,j) and y(i,j) represent the value of image x and image y at the i,j pixel position, respectively; m and n represent the width and height of the image, MAX x is the maximum pixel value in image x; Calculate the model performance indicator SSIM, and the specific calculation formula of SSIM is as follows: where μ x and μ y denote the local mean of the two images x and y, respectively, to estimate the intensity; c1 and c2 are constant terms to prevent the denominator from being zero; and denote the local variance of the images x and y, respectively, to measure the contrast of the images; σ xy denotes the covariance of the images x and y, respectively, to measure the structural similarity between the images; Step 6.3: Evaluate the performance indicators of the test set. If the PSNR and SSIM indicators of the test set are similar to those of the training set, it indicates that the model meets the generalization requirements, and a final pavement defect image super-resolution reconstruction model based on the improved SwinIR is obtained; Step 7: Input the first data set and the training weight saved in step 5 into the improved SwinIR pavement defect image super-resolution reconstruction model for reconstruction, and obtain the final super-resolution reconstruction image.

Images