Highway asphalt pavement disease perception method based on fusion convolutional neural network

Abstract

The application discloses a kind of based on fusion convolutional neural network's highway asphalt pavement disease perception method, comprising: using optimized Mobilenet V3 network replaces Deeplabv3+ backbone network, constructs DL-M-PDS network;Improvement network parameter, constructs DL-M2-PDS optimization network for highway asphalt pavement disease segmentation;Through the optimization network, obtain road surface pixel-level disease information, and with original image superposition;Faster-RCNN, Yolov5s, SSD model is improved adaptively, constructs FR-PDD, Yolov5s-PDD, SSD-PDD network model for highway asphalt pavement disease detection;Input superposition disease image, constructs FCNN-PDP-FR, FCNN-PDP-Yolov5s and FCNN-PDP-SSD network for highway asphalt pavement disease perception;Training network and optimizing parameter, preferred model carries out highway asphalt pavement disease perception.The application fuses highway asphalt pavement disease segmentation model and highway asphalt pavement disease detection classification model to realize the segmentation disease while improving the detection classification precision of model, can provide technical support for the intelligent development of highway maintenance.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent highway and road maintenance, and particularly relates to a highway asphalt pavement disease perception method based on a fusion convolutional neural network. BACKGROUND

[0002] In recent years, with the completion of the planning and construction target of highways in China and the continuous improvement and development of the highway traffic network, the maintenance of highways has become increasingly prominent, and how to improve the informatization, specialization and intelligent level of highway maintenance through technical means is one of the problems to be solved in the field of highway maintenance. The detection, classification and pixel-level accurate positioning of highway asphalt pavement diseases can more intuitively present the overall situation of the pavement diseases, and are of great significance for the maintenance of highways. They are an important basis for realizing intelligent road maintenance.

[0003] The target detection network can only realize the detection and classification of highway asphalt pavement diseases, and can detect and classify the detected pavement diseases by using a rectangular frame or a square frame, and cannot intuitively segment the disease area. The semantic segmentation network can segment the detected pavement diseases at the pixel level, but its detection positioning and classification ability is poor. Therefore, the application provides a highway asphalt pavement disease perception method based on a fusion convolutional neural network. SUMMARY

[0004] The application aims to overcome the deficiencies in the prior art and provide a highway asphalt pavement disease perception method based on a fusion convolutional neural network. The Deeplabv3+ network, the Mobilenet V3 network and the Faster-RCNN, Yolov5s and SSD network are optimized and fused to construct the fusion convolutional neural network FCNN-PDP-FR, FCNN-PDP-Yolov5s and FCNN-PDP-SSD for asphalt pavement disease perception. The precise positioning, detection and classification of highway asphalt pavement diseases can provide technical support for the intelligent development of highway maintenance.

[0005] The application provides a highway asphalt pavement disease perception method based on a fusion convolutional neural network, which comprises the following steps:

[0006] S1: replacing the Deeplabv3+ backbone network with an optimized Mobilenet V3 network to construct a DL-M-PDS network for highway asphalt pavement disease segmentation;

[0007] S2: improving the network parameters to construct a DL-M2-PDS optimized network for highway asphalt pavement disease segmentation;

[0008] S3: Obtain the pixel-level segmentation disease information of the pavement through the DL-M2-PDS pavement disease segmentation model;

[0009] S4: Superimpose the pixel-level information on the original image;

[0010] S5: Adaptively improve the Faster-RCNN, Yolov5s, and SSD models to construct the FR-PDD, Yolov5s-PDD, and SSD-PDD fusion convolutional neural network models for highway asphalt pavement disease detection;

[0011] S6: Superimpose the highway asphalt pavement disease image and input it into the FR-PDD, Yolov5s-PDD, and SSD-PDD to construct the FCNN-PDP-FR, FCNN-PDP-Yolov5s, and FCNN-PDP-SSD three networks for highway asphalt pavement disease perception;

[0012] S7: Train the model and optimize the parameters, and preferably select the FCNN-PDP-FR, FCNN-PDP-Yolov5s, and FCNN-PDP-SSD network models for highway asphalt pavement disease perception.

[0013] Further, the specific steps for optimizing the Mobilenet V3 in step S1 are as follows:

[0014] S1-1: Add a hollow convolution (Atrous conv) as needed in the bneck basic structure;

[0015] S1-2: Add additional batch normalization and NL activation functions after each bneck depth convolution.

[0016] Further, the main content of improving the network parameters in step S2 is: optimizing the output stride to be 16; and optimizing the hollow convolution step length rate to be [3, 5, 8, 11].

[0017] Further, the specific content of step S3 is:

[0018] S3-1: Input the original image of highway asphalt pavement disease, and normalize the network image size to 513x513x3;

[0019] S3-2: Perform feature extraction, and set the scale ratio of the input image to the high-dimensional feature map to be 16;

[0020] S3-3: Use the spatial pyramid pooling module to extract multi-scale information, concatenate the feature maps of different scales with the image-level features, and fuse to obtain a new feature image;

[0021] S3-4: Interpolating the high-dimensional feature map to the original image size, outputting the binary road disease segmentation image.

[0022] Further, the specific content of the step S4 is: setting the transparency of the binary image to 0.3, setting the transparency of the original image to 0.7, and superimposing the binary image and the original image by using the transparency blending function.

[0023] Further, the specific content of the step S5 is:

[0024] S5-1: According to the characteristics of the road disease image, the Faster-RCNN network is adaptively improved, the area of the anchor frame is kept unchanged (128*128, 256*256, 512*512), the anchor frame aspect ratio is adjusted (1:1, 1:4, 4:1), and the Faster-RCNN basic network structure is kept unchanged, and the FR-PDD road asphalt pavement disease detection network is constructed;

[0025] S5-2: According to the characteristics of the road disease image, the Yolov5s network is adaptively improved, the GT coordinates of the labeled data set are clustered and regressed, 9 most likely anchor frame sizes are generated, and the Yolov5s network data enhancement part is removed, and the Yolov5s-PDD road asphalt pavement disease detection network is constructed;

[0026] S5-3: According to the characteristics of the road disease image, the input image parameters of the SSD network are adjusted, VGG16 is used as the backbone network, the aspect ratio of the prior frame is set to (1, 2, 3, 1 / 2, 1 / 3), and the SSD-PDD road asphalt pavement disease detection network is constructed.

[0027] Further, the construction method of the fusion convolutional neural network FCNN-PDP-FR for asphalt pavement disease perception in the step S6 is:

[0028] ① Extraction: The network automatically normalizes the image, the backbone network VGG16 uses shared convolutional layers to extract full image features, and the feature map is obtained;

[0029] ② RPN: 20000 anchor frames are generated by series of anchor frame sliding convolution operations, whether the anchor frame contains a road disease target is distinguished, and the anchor frame containing the target is regressed;

[0030] ③ Classification: Map the target candidate region to the feature map and adjust the size, determine the class, and then regress again to correct the precise position of the road disease;

[0031] ④ Output: Draw the detection frame result, and obtain the final road disease perception result.

[0032] Further, the construction method of the fusion convolutional neural network FCNN-PDP-Yolov5s for asphalt pavement disease perception in step S6 is as follows:

[0033] ① Extraction: feature extraction is performed through the core components of Yolov5s to obtain a feature map, and the main steps include downsampling, slicing, blocking, splicing, etc.;

[0034] ② Combination: pyramid combination is completed through FPN+PAN to obtain prediction results of different dimensions, and the extracted pavement disease features are mixed and combined by Neck;

[0035] ③ Prediction: prediction boxes of different sizes are generated for the three-dimensional feature maps, and the prediction boxes are subjected to bounding box regression operation and target box screening operation to obtain the final detection and classification results;

[0036] ④ Classification: the network draws the prediction box information on the input pavement image according to the coordinates of the prediction box, and labels the pavement disease category corresponding to the prediction box.

[0037] Further, the construction method of the fusion convolutional neural network FCNN-PDP-Yolov5s for asphalt pavement disease perception in step S6 is as follows:

[0038] ① Extraction: normalize the image, extract features through the backbone network VGG16, and finally obtain six layers of feature maps;

[0039] ② Generation: generate prior boxes of different sizes and same proportions at each point on each layer of feature maps;

[0040] ③ Classification: classification and regression are completed through convolution, and after step-by-step classification and regression for each prior box, a non-maximum suppression operation is performed to obtain the target detection result;

[0041] ④ Output: draw the detection result to the input highway asphalt pavement disease image.

[0042] Further, the specific steps of step S7 are as follows:

[0043] S7-1: provide feedback of validation set data through loss function curve, and adjust the hyperparameters such as iteration number, learning rate, and decay weight of the model in sequence;

[0044] S7-2: use AP and mAP to evaluate the performance of the three networks FCNN-PDP-FR, FCNN-PDP-Yolov5s, and FCNN-PDP-SSD, and select the optimal highway asphalt pavement disease perception method through comprehensive comparison.

[0045] Beneficial effects: compared with the prior art, the present application fuses the highway asphalt pavement disease segmentation model and the highway asphalt pavement disease detection classification model to realize the segmentation of diseases and improve the detection classification accuracy of the model, and can provide technical support for the intelligent development of highway maintenance. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Schematic diagram of a dilated convolution structure.

[0047] Figure 2 Schematic diagram of superimposed binary image, wherein (a) is a transverse cracking pavement image, (b) is a pothole pavement image, (c) is a net cracking pavement image, and (d) is an other disease pavement image, and each image from left to right is the original image, the binary image, and the superimposed image.

[0048] Figure 3 Schematic diagram of FR-PDD highway asphalt pavement disease detection network anchor frame

[0049] Figure 4 Structure diagram of a fusion convolutional neural network FCNN-PDP-FR.

[0050] Figure 5 Structure diagram of a fusion convolutional neural network FCNN-PDP-Yolov5s.

[0051] Figure 6 Partial component diagram of a fusion convolutional neural network FCNN-PDP-Yolov5s

[0052] Figure 7 Structure diagram of a fusion convolutional neural network FCNN-PDP-SSD. DETAILED DESCRIPTION

[0053] The present application will be further illustrated below in combination with the drawings and specific embodiments.

[0054] The present application provides a highway asphalt pavement disease perception method based on a fusion convolutional neural network, specifically including the following steps:

[0055] S1: replace the Deeplabv3+ backbone network with an optimized Mobilenet V3 network to construct a DL-M-PDS network for highway asphalt pavement disease segmentation, which specifically comprises:

[0056] S1-1: add a dilated convolution (Atrous conv) as needed in the bneck basic structure, and the dilated convolution structure is as shown in Figure 1 ;

[0057] S1-2: add an additional batch normalization and NL activation function after each bneck depth convolution, which specifically comprises:

[0058] The network structure of MobileNetV3 can be divided into three parts: the starting part is one convolutional layer, which extracts features through 3x3 convolution; the middle part is multiple convolutional layers; and the last part replaces the full connection through two 1x1 convolutional layers to output the category. The network structures of MobileNetV3 before and after improvement are shown in Tables 1 and 2. Among them, bneck is the basic structure of the network, ExpSize is the number of channels added in the BottleNeck module, #out is the number of output channels, SE is whether to use channel attention mechanism, NL is the type of activation function, including HS (h-swish), RE (ReLU); NBN is no BN (normalization) operation, s is the stride (step), the network uses convolution operation for downsampling, and no pooling operation is used.

[0059] Table 1 MobileNetV3 network structure

[0060]

[0061]

[0062] Table 2 Improved MobileNetV3 network structure

[0063] Input Operator Exp size #out SE Atrous conv NL s 2242x3 conv2d,3x3 - 16 - - HS 2 1122x16 bneck,3x3 16 16 √ √ RE 2 562x16 bneck,3x3 72 24 - √ RE 2 282x24 bneck,3x3 88 24 - - RE 1 282x24 bneck,5x5 96 40 √ √ HS 2 142x40 bneck,5x5 240 40 √ - HS 1 142x40 bneck,5x5 240 40 √ - HS 1 142x40 bneck,5x5 120 48 √ - HS 1 142x48 bneck,5x5 144 48 √ - HS 1 142x48 bneck,5x5 288 96 √ √ HS 2 72x96 bneck,5x5 576 96 √ - HS 1 72x96 bneck,5x5 576 96 √ - HS 1 72x96 conv2d,1x1 - 576 √ - HS 1 72x576 pool,7x7 - - - - - 1 12x576 conv2d 1x1,NBN - 1280 - - HS 1 12x1280 conv2d 1x1,NBN - k - - - 1

[0064] S2: Improve network parameters and construct DL-M2-PDS optimization network for highway asphalt pavement disease segmentation: optimize the output stride to be 16; and optimize the dilated convolution step length rate to be [3, 5, 8, 11].

[0065] S3: Obtain pixel-level segmentation disease information of the pavement through the DL-M2-PDS pavement disease segmentation model, which specifically comprises:

[0066] S3-1: Input the highway asphalt pavement disease original image, and normalize the segmentation network image size to 513x513x3;

[0067] S3-2: Enter the segmentation network coding part, use MobilenetV3 as the backbone network to extract features, and the ratio of the size of the input image to the high-dimensional feature map is 16, that is, the size of the feature map is (H / 16)x(w / 16);

[0068] S3-3: Use the spatial pyramid pooling module ASPP to extract multi-scale information, the ASPP dilated convolution step length rate is [3, 5, 8, 11], and different scale feature maps and image-level features are concatenated together, and a new feature map is obtained through 1x1 convolution fusion;

[0069] S3-4: Interpolate the high-dimensional feature map to the original image size by decoding part and output the predicted image, that is, get the binarized pavement disease segmentation image.

[0070] S4: Superimpose the pixel-level information on the original image: set the transparency α of the binarized image obtained by segmentation to 0.3, set the transparency α of the original image to 0.7, both images are in RGB format, and use the transparency blending function to superimpose the binarized image and the original image to obtain a superimposed image with more semantic information, which is used as the input of FCNN-PDP-FR, and the partial sample is shown as Figure 2 , from left to right, the original image, the binarized image, and the superimposed image. The superimposition principle is:

[0071] Im gC=Im gA·(1-α)+Im gB·α

[0072] S5: Adaptive improvement is made on Faster-RCNN, Yolov5s, and SSD models to build FR-PDD, Yolov5s-PDD, and SSD-PDD fusion convolutional neural network models for highway asphalt pavement disease detection:

[0073] S5-1: According to the characteristics of the pavement disease image, the Faster-RCNN network is adaptively improved, the anchor box area is kept unchanged according to the size characteristics of the highway pavement disease itself (128×128, 256×256, 512×512), and the anchor box aspect ratio is adjusted to (1:1, 1:4, 4:1), as shown in Figure 3 , and the Faster-RCNN basic network structure is kept unchanged to build the FR-PDD highway asphalt pavement disease detection network;

[0074] S5-2: According to the characteristics of the pavement disease image, the Yolov5s network is adaptively improved, and the GT coordinates of the labeled data set are clustered and regressed to generate nine most likely anchor box sizes to improve the generation efficiency and accuracy of the adaptive anchor box. The nine anchor boxes generated have lengths and widths of [33.75, 524.8], [42.5, 263.68], [45.0, 564.48], [73.75, 78.08], [110.0, 117.76], [197.5, 96.0], [313.75, 174.08], [629.38, 153.6], and [631.25, 92.16]. Since the data has been enhanced, the data enhancement part of the YOLO-V5s network is removed to avoid repeated operations, and the Yolov5s-PDD highway asphalt pavement disease detection network is built.

[0075] S5-3: According to the characteristics of the pavement disease image, the input image parameters of the SSD network are adjusted to 300x300x3, VGG16 is used as the backbone network, the aspect ratio of the prior box is set to (1, 2, 3, 1 / 2, 1 / 3), and the SSD-PDD highway asphalt pavement disease detection network is constructed.

[0076] S6: Superimpose the highway asphalt pavement disease image into FR-PDD, Yolov5s-PDD and SSD-PDD, respectively, to construct three networks for highway asphalt pavement disease perception: FCNN-PDP-FR, FCNN-PDP-Yolov5s and FCNN-PDP-SSD:

[0077] The structure of the fusion convolutional neural network FCNN-PDP-FR for asphalt pavement disease perception is shown in Figure 4 The third part is the highway asphalt pavement disease target detection and classification network part, which specifically includes the following steps:

[0078] ① Extraction: input the superimposed pavement disease image into the FCNN-PDP-FR network, and the network automatically normalizes the pavement disease image; the backbone network VGG16 extracts the features of the pavement disease image, i.e., uses shared convolutional layers to extract the features of the whole image to obtain a feature map;

[0079] ② RPN: the pavement disease feature map is input into the RPN layer, and the RPN network uses a series of fixed-size anchor boxes to perform sliding convolution operation on the feature map to generate 20000 anchor boxes; the Softmax layer distinguishes whether the generated anchor boxes contain pavement disease targets, and if they do, they are retained as foreground, and if they do not, they are discarded as background; at the same time, the anchor boxes containing targets are regressed and corrected using bbox reg (bounding box regression) to obtain accurate pavement disease target prediction regions.

[0080] ③ Classification: the pooling layer maps the different size pavement disease target candidate regions generated by RPN to the feature map of the entire pavement image, retains its image features and adjusts the feature map to a fixed size, and through the full connection layer, the Softmax layer of the classification link calculates the range of the Pooling result on the feature map to determine the class, and again through bbox reg (bounding box regression) to regress and correct the final pavement disease detection frame to obtain the accurate position of the detection frame.

[0081] ④ Output: draw the detection frame result on the input image according to the generated coordinates, and output the final FCNN-PDP-FR pavement disease perception result.

[0082] The fusion convolutional neural network FCNN-PDP-Yolov5s structure for asphalt pavement disease perception is shown in Figure 5 The detailed component diagram is shown in Figure 6 The third part is the highway asphalt pavement disease target detection and classification network part, which specifically includes the following steps:

[0083] ① Extraction: input the obtained superimposed highway asphalt pavement disease image into the Yolov5s-PDD backbone network, extract the features of the input pavement disease image, and the main steps include downsampling, slicing, blocking, splicing, etc. to obtain the feature map;

[0084] ② Combination: the Neck mixes and combines the extracted pavement disease features, and passes these features to the prediction layer. The core design structure of the Neck part is FPN+PAN to complete pyramid combination to obtain prediction results of different dimensions. The obtained feature map dimensions are 76x76x255, 38x38x255, and 19x19x255.

[0085] ③ Prediction: generate prediction boxes of different sizes for the three-dimensional feature maps, and calculate the classification loss, positioning loss, and confidence loss for each prediction box. Then select the prediction box with the smallest loss, i.e. perform bounding box regression and nms (non-maximum suppression) operation on the prediction box to obtain the final detection and classification result.

[0086] ④ Classification: the network draws the prediction box information on the input pavement image according to the coordinates of the prediction box, and labels the pavement disease category corresponding to the prediction box.

[0087] The fusion convolutional neural network FCNN-PDP-SSD structure for asphalt pavement disease perception is shown in Figure 7 The third part is the highway asphalt pavement disease target detection and classification network part, which specifically includes the following steps:

[0088] ① Extraction: input the obtained highway asphalt pavement disease superimposed image into the SSD-PDD, the network automatically normalizes the pavement image to 300x300x3, and extracts the features of the input pavement disease image through the backbone network VGG16. After 4-step convolution (conv4), the corresponding feature map is obtained, and the convolution operation is continued, finally obtaining six layers of feature maps;

[0089] ② Generation: generate prior boxes of different sizes and same proportions at each point on each layer of feature maps. Each layer of feature maps is divided into a×a grids, and each grid center point is the center point of the prior box. Each center point has at least two square prior boxes and several rectangular prior boxes. The aspect ratio of the prior box is set to (1, 2, 3, 1 / 2, 1 / 3).

[0090] ③Classification: classification and regression are performed on the prior box, as shown in the structural diagram, Conv4, Conv7, Conv8, Conv9, Conv10 and Conv11 all point to 8732 classifications and the last non-maximum suppression. Taking Conv4 as an example, the feature map obtained after Conv4 has a size of 38x38x512, and 38x38x4=5776 prior boxes will be generated on the feature map. Classification and regression are completed by 3x3x(4x(Classes+4)) convolution, and similarly, 6 prior boxes are generated for each pixel in the Conv7, Conv8 and Conv9 feature maps, and 4 prior boxes are generated for each pixel in the Conv10 and Conv11 feature maps, totaling 8732 prior boxes. After classification and regression of each prior box at each level, the non-maximum suppression operation is performed, and the target detection result is obtained;

[0091] ④Output: draw the detection result into the input highway asphalt pavement disease image.

[0092] S7: Model training and parameter optimization, preferably FCNN-PDP-FR, FCNN-PDP-Yolov5s and FCNN-PDP-SSD network models are used for highway asphalt pavement disease perception:

[0093] The data enhancement methods such as cropping, flipping, scaling, translation, adding Gaussian noise and adjusting brightness are used to obtain 3741 images of road asphalt pavement disease detection, classification and segmentation, of which 1381 are transverse cracks, 1056 are longitudinal cracks, 570 are others, 462 are potholes and 272 are net cracks. The 3741 image data sets are manually annotated and labeled as five types of transverse cracks, longitudinal cracks, net cracks, potholes and others. The perception accuracy (AP) and average perception accuracy (mAP) of each type of disease perceived by the three fusion perception models FCNN-PDP-FR, FCNN-PDP-Yolov5s and FCNN-PDP-SSD are counted, as shown in Table 4, and compared with the three road asphalt pavement disease detection models (Table 3) without fusion processing FR-PDD, Yolov5s-PDD and SSD-PDD. According to the data analysis of Table 3 and Table 4, the detection and classification accuracy of the fusion network model for transverse cracks and longitudinal cracks has been greatly improved, and the average detection accuracy mAP has also been greatly improved without affecting the real-time performance. This shows that the pixel-level segmentation of the asphalt pavement disease image by the segmentation model and the superposition can effectively improve the information contained in the pavement image, and further improve the detection and classification accuracy of the network model. Compared with the three different structures of the fusion model, the comprehensive performance ranking is FCNN-PDP-Yolov5s>FCNN-PDP-FR>FCNN-PDP-SSD, which shows that the FCNN-PDP-Yolov5s network for road asphalt pavement disease perception constructed in the patent has obvious superiority, higher average accuracy and better real-time performance, which is beneficial to practical engineering application.

[0094] Table 3 AP and mAP statistical table of non-fusion model

[0095]

[0096] Table 4 AP and mAP statistical table of fusion model

[0097]

[0098] The technical means disclosed in the present application scheme are not limited to the technical means disclosed in the above-mentioned embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that for ordinary skilled persons in the art, without departing from the principles of the present application, some improvements and refinements can be made, which are also considered within the protection scope of the present application.

Claims

1. A method for perceiving asphalt pavement distress on highways based on fused convolutional neural networks, characterized in that: Includes the following steps: S1: Replace the Deeplabv3+ backbone network with an optimized Mobilenet V3 network to construct a DL-M-PDS network for highway asphalt pavement distress segmentation; S2: Improve network parameters and construct a DL-M2-PDS optimized network for segmentation of asphalt pavement distresses; S3: Obtain pixel-level segmentation information of pavement defects through the DL-M2-PDS pavement defect segmentation model; S4: Overlay pixel-level information onto the original image; S5: Adaptive improvements were made to Faster-RCNN, Yolov5s, and SSD models to construct a FR-PDD, Yolov5s-PDD, and SSD-PDD fusion convolutional neural network model for detecting asphalt pavement defects on highways. S6: Superimpose images of highway asphalt pavement defects into FR-PDD, Yolov5s-PDD, and SSD-PDD respectively to construct three networks for highway asphalt pavement defect perception: FCNN-PDP-FR, FCNN-PDP-Yolov5s, and FCNN-PDP-SSD. S7: Train the model and optimize the parameters, and use FCNN-PDP-FR, FCNN-PDP-Yolov5s and FCNN-PDP-SSD network models to perceive asphalt pavement defects on highways; The specific content of step S5 is as follows: S5-1: Based on the characteristics of pavement distress images, the Faster-RCNN network is adaptively improved. The area of ​​the anchor frame remains unchanged at 128×128, 256×256, and 512×512, while the aspect ratio of the anchor frame is adjusted to 1:1, 1:4, and 4:

1. The basic network structure of Faster-RCNN remains unchanged, and the FR-PDD highway asphalt pavement distress detection network is constructed. S5-2: Based on the characteristics of pavement distress images, the YOLOv5s network is adaptively improved. The GT coordinates of the labeled dataset are clustered and regressed to generate 9 most likely anchor box sizes. At the same time, the data augmentation part of the YOLOv5s network is removed, and the YOLOv5s-PDD highway asphalt pavement distress detection network is constructed. S5-3: Based on the characteristics of pavement distress images, adjust the input image parameters of the SSD network, use VGG16 as the backbone network, and set the aspect ratio of the prior bounding boxes to 1, 2, 3, 1 / 2, and 1 / 3 to construct the SSD-PDD highway asphalt pavement distress detection network. The method for constructing the FCNN-PDP-FR network for asphalt pavement distress perception in step S6 is as follows: ① Extraction: The network automatically normalizes the image, and the backbone network VGG16 uses shared convolutional layers to extract features from the entire image to obtain a feature map; ②RPN: 20,000 anchor frames are generated through a series of anchor frame sliding convolution operations. The anchor frames are distinguished as to whether they contain pavement distress targets, and regression is performed on the anchor frames that contain targets. ③Classification: Map the target candidate region onto the feature map and adjust its size. After determining the category, regress and correct again to obtain the precise location of road surface defects; ④ Output: Draw the detection box results and obtain the final pavement defect perception results; The method for constructing the FCNN-PDP-Yolov5s network for asphalt pavement distress perception in step S6 is as follows: ① Extraction: Feature maps are obtained by extracting features from the core components of Yolov5s. The main steps include downsampling, slicing, block division, and stitching. ② Combination: Pyramid combination is completed through FPN+PAN to obtain prediction results of different dimensions. Neck mixes and combines the extracted pavement distress features. ③Prediction: Generate prediction boxes of different sizes from the feature maps of the three dimensions, and perform bounding box regression and target box filtering operations on the prediction boxes to obtain the final detection and classification results; ④ Classification: The network draws the prediction box information on the input pavement image based on the coordinates of the prediction box and labels the pavement distress category corresponding to the prediction box; The method for constructing the FCNN-PDP-SSD network for asphalt pavement distress perception in step S6 is as follows: ① Extraction: The normalized image is used to extract features through the backbone network VGG16, resulting in a six-layer feature map; ② Generation: Generate prior boxes of different sizes but the same proportions at each point on each layer of the feature map; ③ Classification: Classification and regression are completed through convolution. After classifying and regressing each prior box step by step, non-maximum suppression is performed to obtain the target detection result. ④ Output: Plot the detection results onto the input image of highway asphalt pavement defects.

2. The method for perceiving highway asphalt pavement defects based on fused convolutional neural networks according to claim 1, characterized in that: The specific steps for optimizing Mobilenet V3 in step S1 are as follows: S1-1: Add dilated convolutions to the basic bneck structure as needed; S1-2: Add additional batch normalization and NL activation function after each bneck depth convolution.

3. The method for perceiving highway asphalt pavement defects based on fused convolutional neural networks according to claim 1, characterized in that: The main contents of improving network parameters in step S2 are: optimizing the output stride to 16; optimizing the dilated convolution stride rate to [3,5,8,11].

4. The method for perceiving highway asphalt pavement defects based on fused convolutional neural networks according to claim 1, characterized in that: The specific content of step S3 is as follows: S3-1: Input the original image of highway asphalt pavement distress, with the normalized segmentation network image size being 513×513×3; S3-2: Perform feature extraction, setting the scale ratio of the input image to the high-dimensional feature map to 16; S3-3: The spatial pyramid pooling module is used to extract multi-scale information, and the feature maps of different scales are concatenated with pixel-level information features to obtain a new feature image; S3-4: Interpolate the high-dimensional feature map to the original image size and output a binarized road surface defect segmentation image.

5. The method for perceiving highway asphalt pavement defects based on fused convolutional neural networks according to claim 1, characterized in that: The specific content of step S4 is as follows: set the transparency of the binarized image to 0.3, set the transparency of the original image to 0.7, and use a transparency blending function to overlay the binarized image and the original image.

6. The method for perceiving highway asphalt pavement defects based on fused convolutional neural networks according to claim 1, characterized in that: The specific steps of step S7 are as follows: S7-1: Provide validation set data feedback through the loss function curve, and adjust the number of model iterations, learning rate, and decay weight hyperparameters in turn; S7-2: Performance evaluation of three networks, FCNN-PDP-FR, FCNN-PDP-Yolov5s, and FCNN-PDP-SSD, was conducted using AP and mAP. The optimal method for detecting asphalt pavement defects on highways was selected through comprehensive comparison.

Images