A road damage warning method based on deep learning

Abstract

The present invention provides a road damage warning method based on deep learning, comprising the following steps: S101, constructing a road damage dataset, or using an existing public road damage dataset: the road damage dataset is collected by installing a camera on a car during driving on the road, the collected image data is manually annotated, and the damaged position is framed and typed using an annotation tool; S102, all images in the road damage dataset are preprocessed, and a bilateral filtering operation is performed to obtain a processed road damage dataset; the present invention can quickly extract features of road damage images by partially convolving the convolution layers in the FCB module and the FCP module, and improving the pooling layer, thereby enhancing the ability of the road damage warning model to process complex road scenes and reducing feature redundancy of road damage images. In the CSAttention module, a channel attention mechanism is introduced, the model is simple, and the running speed is fast.

Description

Technical Field

[0001] The present invention relates to the technical field of road damage warning information processing, and in particular to a road damage warning method based on deep learning. Background Art

[0002] With the rapid construction and operation of the road traffic network, there are certain deficiencies in road management and maintenance capabilities. If not resolved in a timely manner, it may pose a threat to the safety of pedestrians, vehicles, etc. At present, road pavement detection is mainly divided into two methods: manual detection and automated detection. Manual detection has the disadvantages of large workload, low efficiency, poor safety, and large differences in results, and cannot meet the requirements of rapid and accurate detection of road damage. However, the existing automated detection equipment is expensive, and some still require manual assistance to complete the entire detection work, so there are still certain limitations.

[0003] Based on this, the use of image sensors as road surface data collection equipment and road damage detection based on deep learning is still in the development stage, and the detection effect is still far from the actual demand;

[0004] The hardware costs of some existing technologies are too expensive, and some methods use equipment such as radar and laser, which are too costly to be applied on a large scale in practice.

[0005] The existing part uses deep learning technology to detect road damage, which uses a large number of convolutional layers and causes feature redundancy.

[0006] Some existing methods do not consider comprehensive feature information and are prone to missing information, resulting in poor road damage warning model effects. Therefore, a road damage warning method based on deep learning is proposed. Summary of the invention

[0007] In view of this, an embodiment of the present invention hopes to provide a road damage warning method based on deep learning to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.

[0008] The technical solution of the embodiment of the present invention is implemented as follows: A road damage warning method based on deep learning comprises the following steps:

[0009] S101. Construct a road damage dataset or use an existing public road damage dataset:

[0010] The road damage dataset is collected by installing a camera on a car while driving on the road. The collected image data is manually annotated, and the damage location is selected and the type is annotated using an annotation tool.

[0011] S102, pre-process all images in the road damage dataset and perform bilateral filtering to obtain a processed road damage dataset:

[0012] After the road damage image is processed by bilateral filtering, the road damage features become more obvious and the image as a whole becomes smoother;

[0013] S103, constructing a road damage warning model, and inputting the preprocessed road damage image into the road damage warning model:

[0014] The road damage warning model is used to detect the damage location and damage type on the road;

[0015] S104, dividing the road damage data set, and then training and verifying the road damage warning model;

[0016] S105. After the road damage warning model is trained, the road damage warning model is used to detect road damage, and timely warning is issued for areas with road damage.

[0017] In some embodiments, in S103, a road damage warning model is constructed, and the specific steps of inputting the pre-processed road damage image into the road damage warning model are as follows:

[0018] S1031. Design an FCP module and an FCB module. The FCP module is connected to an HPooling layer after the FCB module to quickly extract features of the road damage image, enhance the ability of the road damage warning model to handle complex road scenes, and reduce feature redundancy of the road damage image.

[0019] The FCP module is composed of a Conv layer, a ReLU layer, a LConv layer, a ReLU layer, a Conv layer, a ReLU layer, and a HPooling layer connected in series in sequence. The road damage feature map F1 is input into the Conv layer, and the result is input into the ReLU layer to obtain the road damage feature map F2. The road damage feature map F2 is then input into the LConv layer, and the result is input into the ReLU layer to obtain the road damage feature map F3. The road damage feature map F3 is then input into the Conv layer, and the result is input into the ReLU layer to obtain the road damage feature map F4. The road damage feature map F3 and the road damage feature map F4 are concat to obtain the road damage feature map F5, which is the output result of the FCB module. The road damage feature map F5 is then input into the HPooling layer to obtain the road damage feature map F6, which is the output result of the FCP module.

[0020] S1032. Designed the CSAttention module to achieve balanced perception of the visual field, which can perceive the characteristics of the entire space, taking into account global and local information, while also reducing interference with irrelevant areas in the road damage image;

[0021] S1033, constructing a feature extraction part in the road damage warning model, inputting a road damage image, generating a final output road damage feature map, taking into account global and local information, while also reducing interference with irrelevant areas in the road damage image, and making feature extraction of small targets at damaged locations in the road damage more effective;

[0022] S1034. Input the road damage feature map output by S1033 into the classifier and regressor respectively, output the predicted bounding box and target category, and finally remove the overlapping bounding boxes through non-maximum suppression, output the predicted bounding box and category, and finally realize the detection of the road damage warning model.

[0023] In some embodiments, in S1031, the specific operation steps are as follows:

[0024] S1031-1, input the road damage feature map F1 to a Conv layer with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1, to obtain an output result, where the output channel is half of the input channel, and then input the result to a ReLU layer to obtain a road damage feature map F2;

[0025] S1031-2, input F2 into the LConv layer, the LConv layer evenly takes 1 / 4 of the input channels of F2 for feature extraction (that is, each channel in this 1 / 4 of the channels has the same number of channels as the adjacent channels belonging to this 1 / 4), uses a convolution with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1 for feature extraction, and the remaining 3 / 4 of the input channels remain unchanged and are directly copied to the output result to obtain the output result, and then the result is input into the ReLU layer to obtain the road damage feature map F3;

[0026] S1031-3, input the road damage feature map F3 to a Conv layer with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1, to obtain an output result, and then input the result to a ReLU layer to obtain a road damage feature map F4;

[0027] S1031-4, concat the road damage feature maps F3 and F4 to obtain a road damage feature map F5, which is the final output result of the FCB module. Then, the road damage feature map F5 is input into the HPooling layer with a pooling window size of 4×4, a step size of 2, and a zero padding of 1 to obtain a road damage feature map F6, which is the final output result of the FCP module.

[0028] HPooling layer workflow:

[0029] The maximum value is obtained for each row in each pooling window, and then the average of the results is obtained. In addition, the maximum value is obtained for each column in the pooling window, and then the average of the results is obtained. The two average results are weighted to obtain the value of the pixel point on the corresponding output feature map.

[0030] In some embodiments, in S1031, the calculation formula is as follows:

[0031]

[0032] Where k is the size of the HPooling layer, a m,n is the pixel point in the mth row and nth column of the corresponding k×k pooling window in the input feature map (m=1,…,k,n=1,…,k), is the pixel point on the corresponding output feature map, and p is learned through model training, with the initial value set to 0.5.

[0033] In some embodiments, in S1032, the specific steps are as follows:

[0034] S1032-1. Input the road damage feature map X1 into three Conv layers respectively, where the Conv layer is composed of 2C convolution kernels with a convolution kernel size of 1×1. Reduce the dimension of the output of the first Conv layer to obtain the road damage feature map X2. The Conv layer with a convolution kernel size of 1×1 changes the number of channels of the input feature map, which depends on the number of convolution kernels of the Conv layer. The number of convolution kernels of the Conv layer is equal to the number of channels of the output feature map. Reduce the dimension of the output of the second Conv layer and input it into the Softmax layer, then transpose the dimension of the result to obtain the road damage feature map X3. Reduce the dimension of the output of the third Conv layer and input it into the Softmax layer to obtain the road damage feature map X5.

[0035] S1032-2, performing matrix multiplication on the road damage feature map X2 and the road damage feature map X3 to obtain a road damage feature map X4;

[0036] S1032-3, performing matrix multiplication on the road damage feature map X4 and the road damage feature map X5, and then restoring the flattened shape to the shape of a two-dimensional feature map to obtain a road damage feature map X6;

[0037] S1032-4. Input the road damage feature map X6 into the Conv layer to obtain the road damage feature map X7.

[0038] In some embodiments, the dimensionality reduction is to flatten the remaining dimensions, that is, to merge the height and width dimensions into one dimension.

[0039] In some embodiments, in S1033, the specific steps are as follows:

[0040] S1033-1. Build a Conv layer and a ReLU layer with a convolution kernel size of 1×1 to form a Conv1×1+ReLU module, input the road damage image into the Conv1×1+ReLU module, and pass the output result into the FCP module to obtain the road damage feature map Y1;

[0041] S1033-2, stack four FCP modules in sequence, and the output results of the first three FCP modules are input into the next FCP module after passing through a Conv layer with a convolution kernel size of 1×1. The road damage feature maps output by the four FCP modules are recorded as Y1, Y2, Y3, and Y4 in sequence;

[0042] S1033-3, input the road damage feature maps Y1, Y2, Y3, and Y4 into four CSAttention modules respectively to obtain road damage feature maps Y'1, Y'2, Y'3, and Y'4;

[0043] S1033-4, concat the road damage feature maps Y1, Y2, Y3, and Y4 with the road damage feature maps Y'1, Y'2, Y'3, and Y'4, respectively, to obtain road damage feature maps Y"1, Y"2, Y"3, and Y"4;

[0044] S1033-5, input the road damage feature map Y"4 into the FCB module to obtain the road damage feature map Y"'4, then pass the road damage feature map Y"'4 through Conv1×1+Upsampling (a Conv layer with a convolution kernel size of 1×1 and 2 times upsampling, where the upsampling method uses the nearest neighbor interpolation method to double the height and width, while the number of channels remains unchanged, the same below) to obtain the road damage feature map Y""4, perform matrix multiplication on the road damage feature map Y""4 and the road damage feature map Y"3, and input the result into the FCB module to obtain the road damage feature map Y"'3;

[0045] S1033-6, the road damage feature map Y"'3 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y""3, and the road damage feature map Y"2 is matrix multiplied, and the obtained result is input into the FCB module to obtain the road damage feature map Y"'2;

[0046] Then, the road damage feature map Y”’2 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y””2, which is matrix-multiplied with the road damage feature map Y”1. The result is input into the FCB module to obtain the road damage feature map Y”’1.

[0047] S1033-7. The road damage feature map Y”’1 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y””1, which is input into the Conv1×1+ReLU module. The result obtained is the final output road damage feature map.

[0048] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0049] 1. The present invention can quickly extract the features of road damage images by performing partial convolution on the convolution layers in the FCB module and the FCP module and improving the pooling layer, thereby enhancing the ability of the road damage warning model to handle complex road scenes and reducing the redundancy of road damage image features. In the CSAttention module, a channel attention mechanism is introduced, and the model is simple and runs fast.

[0050] 2. The present invention can obtain deep and shallow road damage feature maps by stacking modules, and then input them into the CSAttention module. Then, the features are combined through concat and matrix multiplication to finally obtain the road damage feature map. The feature extraction of small targets at damaged locations in road damage is more effective, the features of road damage images can be extracted more quickly, and feature redundancy can be reduced. The channel attention mechanism is introduced, the model is simple, and the running speed is fast.

[0051] 3. When preprocessing the road damage data set, the present invention performs a bilateral filtering operation. After the bilateral filtering process, the road damage features are more obvious and the overall image becomes smoother.

[0052] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a flow chart of the present invention;

[0055] Figure 2 Schematic diagram of the FCP module and the FCB module of the present invention;

[0056] Figure 3 Schematic diagram of the CSAttention module of the present invention;

[0057] Figure 4 It is a schematic diagram of feature extraction of the road damage warning model of the present invention. DETAILED DESCRIPTION

[0058] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0059] It should be noted that the terms "first", "second", "symmetrical", "array", etc. are only used to distinguish between description and position description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the definition of "first", "symmetrical", etc. can explicitly or implicitly include one or more of these features; similarly, when the quantity of certain features is not limited in the form of words such as "two" or "three", it should be noted that this feature also explicitly or implicitly includes one or more feature quantities;

[0060] In the present invention, unless otherwise clearly specified and limited, the terms such as "installation", "connection", "fixation" and the like should be understood in a broad sense; for example, it can be a fixed connection, a detachable connection, or an integral molding; it can be a mechanical connection, a direct connection, welding, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood based on the description and drawings combined with specific circumstances.

[0061] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0062] like Figure 1-Figure 4 As shown, an embodiment of the present invention provides a road damage warning method based on deep learning, comprising the following steps:

[0063] S101. Construct a road damage dataset or use an existing public road damage dataset:

[0064] The road damage dataset is collected by installing a camera on a car while driving on the road. The collected image data is manually annotated, and the damage location is selected and the type is annotated using an annotation tool.

[0065] S102, pre-process all images in the road damage dataset and perform bilateral filtering to obtain a processed road damage dataset:

[0066] After the road damage image is processed by bilateral filtering, the road damage features become more obvious and the image as a whole becomes smoother;

[0067] S103, constructing a road damage warning model, and inputting the preprocessed road damage image into the road damage warning model:

[0068] The road damage warning model is used to detect the damage location and damage type on the road;

[0069] S104, dividing the road damage data set, and then training and verifying the road damage warning model;

[0070] S105. After the road damage warning model is trained, the road damage warning model is used to detect road damage, and timely warning is issued for areas with road damage.

[0071] In this embodiment, specifically, the labeling tool adopts labelme.

[0072] In this embodiment, specifically, by directly calling the cv2.bilateralFilter(src,d,sigmaColor,sigmaSpace) function encapsulated in opencv through Python, the bilateral filtering operation can be conveniently performed on the image, where src is the grayscale or RGB image object to be filtered, d is the diameter of each pixel neighborhood used during filtering, sigmaColor is the δ parameter of the color space, and sigmaSpace is the δ parameter of the coordinate space.

[0073] In this embodiment, specifically, in S103, a road damage warning model is constructed, and the specific steps of inputting the pre-processed road damage image into the road damage warning model are as follows:

[0074] S1031. Design an FCP module and an FCB module. The FCP module is connected to an HPooling layer after the FCB module to quickly extract features of the road damage image, enhance the ability of the road damage warning model to handle complex road scenes, and reduce feature redundancy of the road damage image.

[0075] The FCP module is composed of a Conv layer, a ReLU layer, a LConv layer, a ReLU layer, a Conv layer, a ReLU layer, and a HPooling layer connected in series in sequence. The road damage feature map F1 is input into the Conv layer, and the result is input into the ReLU layer to obtain the road damage feature map F2. The road damage feature map F2 is then input into the LConv layer, and the result is input into the ReLU layer to obtain the road damage feature map F3. The road damage feature map F3 is then input into the Conv layer, and the result is input into the ReLU layer to obtain the road damage feature map F4. The road damage feature map F3 and the road damage feature map F4 are concat to obtain the road damage feature map F5, which is the output result of the FCB module. The road damage feature map F5 is then input into the HPooling layer to obtain the road damage feature map F6, which is the output result of the FCP module.

[0076] S1032. Designed the CSAttention module to achieve balanced perception of the visual field, which can perceive the characteristics of the entire space, taking into account global and local information, while also reducing interference with irrelevant areas in the road damage image;

[0077] S1033, constructing a feature extraction part in the road damage warning model, inputting a road damage image, generating a final output road damage feature map, taking into account global and local information, while also reducing interference with irrelevant areas in the road damage image, and making feature extraction of small targets at damaged locations in the road damage more effective;

[0078] S1034. Input the road damage feature map output by S1033 into the classifier and regressor respectively, output the predicted bounding box and target category, and finally remove the overlapping bounding boxes through non-maximum suppression, output the predicted bounding box and category, and finally realize the detection of the road damage warning model.

[0079] In this embodiment, specifically, in S1031, the specific operation steps are as follows:

[0080] S1031-1, input the road damage feature map F1 to a Conv layer with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1, to obtain an output result, where the output channel is half of the input channel, and then input the result to a ReLU layer to obtain a road damage feature map F2;

[0081] S1031-2, input F2 into the LConv layer, the LConv layer evenly takes 1 / 4 of the input channels of F2 for feature extraction (that is, each channel in this 1 / 4 of the channels has the same number of channels as the adjacent channels belonging to this 1 / 4), uses a convolution with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1 for feature extraction, and the remaining 3 / 4 of the input channels remain unchanged and are directly copied to the output result to obtain the output result, and then the result is input into the ReLU layer to obtain the road damage feature map F3;

[0082] S1031-3, input the road damage feature map F3 to a Conv layer with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1, to obtain an output result, and then input the result to a ReLU layer to obtain a road damage feature map F4;

[0083] S1031-4, concat the road damage feature maps F3 and F4 to obtain a road damage feature map F5, which is the final output result of the FCB module. Then, the road damage feature map F5 is input into the HPooling layer with a pooling window size of 4×4, a step size of 2, and a zero padding of 1 to obtain a road damage feature map F6, which is the final output result of the FCP module.

[0084] HPooling layer workflow:

[0085] The maximum value is obtained for each row in each pooling window, and then the average of the results is obtained. In addition, the maximum value is obtained for each column in the pooling window, and then the average of the results is obtained. The two average results are weighted to obtain the value of the pixel point on the corresponding output feature map.

[0086] In this embodiment, specifically, in S1031, the calculation formula is as follows:

[0087]

[0088] Where k is the size of the HPooling layer, a m,n is the pixel point in the mth row and nth column of the corresponding k×k pooling window in the input feature map (m=1,…,k,n=1,…,k), is the pixel point on the corresponding output feature map, and p is learned through model training, with the initial value set to 0.5.

[0089] In this embodiment, the road damage feature maps in the intermediate process are all used as a foundation for the final output road damage feature map. Conv layer is a convolution layer, ReLU is an activation function, LConv layer is an improved convolution layer (not an existing method), HPooling layer is an improved pooling layer (not an existing method), and concat is to connect two or more tensors together.

[0090] In this embodiment, specifically, the size of the feature map is assumed to be H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0091] For example, the road damage feature map F1 is 512×512×128 in size. It is input into the Conv layer with 64 convolution kernels of size 3×3, stride 1, and zero padding 1, and then input into the ReLU layer to obtain the road damage feature map F2 of size 512×512×64. Then it is input into the LConv layer, and the 1st, 5th, 9th, 13th, 17th, 21st, 25th, 29th, 33rd, 37th, 41st, 45th, 49th, 53rd, 57th, and 61st channels are taken. 16 convolution kernels of size 3×3, stride 1, and zero padding 1 are used for feature extraction. The remaining 48 channels are directly copied to the output result to obtain the output result, and then the result is input into the ReLU layer to obtain the road damage feature map F2. The road damage feature map F3 is 512×512×64 in size; F3 is input into the Conv layer with 64 convolution kernels of size 3×3, stride 1, and zero padding 1, and then input into the ReLU layer to obtain the road damage feature map F4 of size 512×512×64; the road damage feature maps F3 and F4 are concat to obtain the road damage feature map F5 of size 512×512×128, which is the final output result of the FCB module; F5 is input into the HPooling layer with a maximum pooling window size of 4×4, stride 2, and zero padding 1 to obtain the road damage feature map F6 of size 256×256×128, which is the final output result of the FCP module.

[0092] In this embodiment, specifically, in S1032, the specific steps are as follows:

[0093] S1032-1. Input the road damage feature map X1 into three Conv layers respectively, where the Conv layer is composed of 2C convolution kernels with a convolution kernel size of 1×1. Reduce the dimension of the output of the first Conv layer to obtain the road damage feature map X2. The Conv layer with a convolution kernel size of 1×1 changes the number of channels of the input feature map, which depends on the number of convolution kernels of the Conv layer. The number of convolution kernels of the Conv layer is equal to the number of channels of the output feature map. Reduce the dimension of the output of the second Conv layer and input it into the Softmax layer, then transpose the dimension of the result to obtain the road damage feature map X3. Reduce the dimension of the output of the third Conv layer and input it into the Softmax layer to obtain the road damage feature map X5.

[0094] S1032-2, performing matrix multiplication on the road damage feature map X2 and the road damage feature map X3 to obtain a road damage feature map X4;

[0095] S1032-3, performing matrix multiplication on the road damage feature map X4 and the road damage feature map X5, and then restoring the flattened shape to the shape of a two-dimensional feature map to obtain a road damage feature map X6;

[0096] S1032-4. Input the road damage feature map X6 into the Conv layer to obtain the road damage feature map X7.

[0097] In this embodiment, specifically, the size of the road damage feature map X1 is set to H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map. The road damage feature map X1 is input into a Conv layer with 2C convolution kernels of size 1×1, and the output result is of size H×W×2C. The result is reduced in dimension to obtain a road damage feature map X2 of size (H×W)×2C;

[0098] Then, the road damage feature map X1 is input into the Conv layer with 2C convolution kernels of size 1×1, and the output result is H×W×2C. It is reduced in dimension to obtain a result of size (H×W)×2C, and then input into the Softmax layer. Then, the dimension is transposed to obtain the road damage feature map X3 of size 2C×(H×W).

[0099] Then, the road damage feature map X1 is input into the Conv layer with 2C convolution kernels of size 1×1, and the output result is H×W×2C in size. The result is reduced in dimension to (H×W)×2C in size, and then input into the Softmax layer to obtain the road damage feature map X5 with a size of (H×W)×2C.

[0100] Perform matrix multiplication on the road damage feature map X2 and the road damage feature map X3 to obtain a road damage feature map X4 with a size of 2C×2C; perform matrix multiplication on the road damage feature map X4 and the road damage feature map X5, and then restore them from the flattened shape to the shape of the two-dimensional feature map to obtain a road damage feature map X6 with a size of H×W×2C;

[0101] The road damage feature map X6 is input into the Conv layer with C convolution kernels of size 1×1 to obtain the road damage feature map X7.

[0102] In this embodiment, specifically, dimensionality reduction is to flatten the remaining dimensions, that is, to merge the height and width dimensions into one dimension.

[0103] In this embodiment, specifically, in S1033, the specific steps are as follows:

[0104] S1033-1. Build a Conv layer and a ReLU layer with a convolution kernel size of 1×1 to form a Conv1×1+ReLU module, input the road damage image into the Conv1×1+ReLU module, and pass the output result into the FCP module to obtain the road damage feature map Y1;

[0105] S1033-2, stack four FCP modules in sequence, and the output results of the first three FCP modules are input into the next FCP module after passing through a Conv layer with a convolution kernel size of 1×1. The road damage feature maps output by the four FCP modules are recorded as Y1, Y2, Y3, and Y4 in sequence;

[0106] S1033-3, input the road damage feature maps Y1, Y2, Y3, and Y4 into four CSAttention modules respectively to obtain road damage feature maps Y'1, Y'2, Y'3, and Y'4;

[0107] S1033-4, concat the road damage feature maps Y1, Y2, Y3, and Y4 with the road damage feature maps Y'1, Y'2, Y'3, and Y'4, respectively, to obtain road damage feature maps Y"1, Y"2, Y"3, and Y"4;

[0108] S1033-5, input the road damage feature map Y"4 into the FCB module to obtain the road damage feature map Y"'4, then pass the road damage feature map Y"'4 through Conv1×1+Upsampling (a Conv layer with a convolution kernel size of 1×1 and 2 times upsampling, where the upsampling method uses the nearest neighbor interpolation method to double the height and width, while the number of channels remains unchanged, the same below) to obtain the road damage feature map Y""4, perform matrix multiplication on the road damage feature map Y""4 and the road damage feature map Y"3, and input the result into the FCB module to obtain the road damage feature map Y"'3;

[0109] S1033-6, the road damage feature map Y"'3 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y""3, and the road damage feature map Y"2 is matrix multiplied, and the obtained result is input into the FCB module to obtain the road damage feature map Y"'2;

[0110] Then, the road damage feature map Y”’2 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y””2, which is matrix-multiplied with the road damage feature map Y”1. The result is input into the FCB module to obtain the road damage feature map Y”’1.

[0111] S1033-7. The road damage feature map Y”’1 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y””1, which is input into the Conv1×1+ReLU module. The result obtained is the final output road damage feature map.

[0112] In this embodiment, specifically, the size of the road damage feature map is assumed to be H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0113] The road damage image (size 1024×1024×3) is input into the Conv1×1+ReLU module (where Conv1×1 is a Conv layer with 128 convolution kernels of size 1×1), and the output result is passed into the FCP module to obtain the road damage feature map Y1 with a size of 512×512×128;

[0114] Four FCP modules are stacked in sequence. The outputs of the first three FCP modules are passed through a Conv layer with a convolution kernel size of 1×1, and then the results are input into the next FCP module.

[0115] The road damage feature maps output by the four FCP modules are denoted as Y1, Y2, Y3, and Y4 respectively;

[0116] Among them, the size of the road damage feature map Y1 is 256×256×128, the size of the road damage feature map Y2 is 128×128×256, the size of the road damage feature map Y3 is 64×64×512, and the size of the road damage feature map Y4 is 32×32×1024;

[0117] The road damage feature maps Y1, Y2, Y3, and Y4 are input into four CSAttention modules respectively to obtain road damage feature maps Y'1, Y'2, Y'3, and Y'4;

[0118] Among them, the size of the road damage feature map Y'1 is 256×256×128, the size of the road damage feature map Y'2 is 128×128×256, the size of the road damage feature map Y'3 is 64×64×512, and the size of the road damage feature map Y'4 is 32×32×1024;

[0119] Concatenate the road damage feature maps Y1, Y2, Y3, and Y4 with the road damage feature maps Y'1, Y'2, Y'3, and Y'4 to obtain road damage feature maps Y"1, Y"2, Y"3, and Y"4;

[0120] Among them, the size of the road damage feature map Y'1 is 256×256×256, the size of the road damage feature map Y'2 is 128×128×512, the size of the road damage feature map Y'3 is 64×64×1024, and the size of the road damage feature map Y'4 is 32×32×2048;

[0121] The road damage feature map Y”4 is input into the FCB module to obtain the road damage feature map Y”'4, which is 32×32×2048 in size. The road damage feature map Y”'4 is then subjected to Conv1×1+Upsampling (where Conv1×1 is a Conv layer with 1024 convolution kernels of size 1×1, and 2 times upsampling) to obtain the road damage feature map Y””4, which is 64×64×1024 in size. The road damage feature map Y”3 is matrix multiplied, and the result is input into the FCB module to obtain the road damage feature map Y”'3, which is 64×64×1024 in size.

[0122] The road damage feature map Y”'3 is subjected to Conv1×1+Upsampling (Conv1×1 is a Conv layer with 512 convolution kernels of size 1×1) to obtain the road damage feature map Y””3, which is 128×128×512 in size. The road damage feature map Y2” is matrix multiplied with the road damage feature map, and the result is input into the FCB module to obtain the road damage feature map Y”’2, which is 128×128×512 in size.

[0123] The road damage feature map Y”’2 is then subjected to Conv1×1+Upsampling (Conv1×1 is a Conv layer with 256 convolution kernels of size 1×1) to obtain the road damage feature map Y””2, which is 256×256×256 in size. The road damage feature map Y”1 is matrix multiplied and the result is input into the FCB module to obtain the road damage feature map Y”’1, which is 256×256×256 in size.

[0124] The road damage feature map Y”’1 is subjected to Conv1×1+Upsampling (Conv1×1 is a Conv layer with 128 convolution kernels of size 1×1) to obtain the road damage feature map Y””1 with a size of 512×512×128, which is input into the Conv1×1+ReLU (Conv1×1 is a Conv layer with 128 convolution kernels of size 1×1) module. The result is the final output road damage feature map with a size of 512×512×128.

[0125] In this embodiment, specifically, the road damage data set preprocessed by S102 is divided into a training set, a validation set and a test set according to a certain ratio. The training set, the validation set and the test set are mutually exclusive data. The training set accounts for 70% of the total data set, and the validation set and the test set each account for 15% of the total data set.

[0126] Train the road damage warning model, input a batch of road damage training set data into the road damage warning model for training, and obtain the training loss value loss of the batch;

[0127] After all batches of data in the road damage training set are trained, the road damage verification set is input into the road damage warning model in batches to obtain the corresponding batch loss value batch_loss;

[0128] During training and validation, the road damage warning model will automatically learn and adjust parameters based on each loss and batch_loss situation;

[0129] When the training process is carried out for one or more rounds until the batch_loss value converges, the road damage warning model training is completed.

[0130] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A road damage warning method based on deep learning, characterized in that: The following steps are involved: S101. Construct a road damage dataset or use an existing public road damage dataset: The road damage dataset is collected by installing a camera on a car while driving on the road. The collected image data is manually annotated, and the damage location is selected and the type is annotated using an annotation tool. S102, pre-process all images in the road damage dataset and perform bilateral filtering to obtain a processed road damage dataset: After the road damage image is processed by bilateral filtering, the road damage features become more obvious and the image as a whole becomes smoother; S103, constructing a road damage warning model, and inputting the preprocessed road damage image into the road damage warning model: The road damage warning model is used to detect the damage location and damage type on the road; S104, dividing the road damage data set, and then training and verifying the road damage warning model; S105. After the road damage warning model is trained, the road damage warning model is used to detect road damage, and timely warns of damaged areas of the road; In the above S103, a road damage warning model is constructed, and the specific steps of inputting the pre-processed road damage image into the road damage warning model are as follows: S1031. Design an FCP module and an FCB module. The FCP module is connected to an HPooling layer after the FCB module to quickly extract features of the road damage image, enhance the ability of the road damage warning model to handle complex road scenes, and reduce feature redundancy of the road damage image. The FCP module is composed of a Conv layer, a ReLU layer, a LConv layer, a ReLU layer, a Conv layer, a ReLU layer, and a HPooling layer connected in series in sequence. The road damage feature map F1 is input into the Conv layer, and the result is input into the ReLU layer to obtain the road damage feature map F2. The road damage feature map F2 is then input into the LConv layer, and the result is input into the ReLU layer to obtain the road damage feature map F3. The road damage feature map F3 is then input into the Conv layer, and the result is input into the ReLU layer to obtain the road damage feature map F4. The road damage feature map F3 and the road damage feature map F4 are concat to obtain the road damage feature map F5, which is the output result of the FCB module. The road damage feature map F5 is then input into the HPooling layer to obtain the road damage feature map F6, which is the output result of the FCP module. The specific operation steps are as follows: S1031-1, input the road damage feature map F1 to a Conv layer with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1, to obtain an output result, where the output channel is half of the input channel, and then input the result to a ReLU layer to obtain a road damage feature map F2; S1031-2, input F2 into the LConv layer, the LConv layer evenly takes 1 / 4 of the input channels of F2 for feature extraction (that is, each channel in this 1 / 4 of the channels has the same number of channels as the adjacent channels belonging to this 1 / 4), uses a convolution with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1 for feature extraction, and the remaining 3 / 4 of the input channels remain unchanged and are directly copied to the output result to obtain the output result, and then the result is input into the ReLU layer to obtain the road damage feature map F3; S1031-3, input the road damage feature map F3 to a Conv layer with a convolution kernel size of 3×3, a step size of 1, and a zero padding of 1, obtain an output result, and then input the result to a ReLU layer to obtain a road damage feature map F4; S1031-4, concat the road damage feature maps F3 and F4 to obtain a road damage feature map F5, which is the final output result of the FCB module. Then, the road damage feature map F5 is input into the HPooling layer with a pooling window size of 4×4, a step size of 2, and a zero padding of 1 to obtain a road damage feature map F6, which is the final output result of the FCP module. HPooling layer workflow: Find the maximum value for each row in each pooling window, and then find the average of the results. Also find the maximum value for each column in the pooling window, and then find the average of the results. Weight these two average results to get the value of the pixel point on the corresponding output feature map. S1032. The CSAttention module is designed to achieve balanced perception of the visual field. It can perceive the characteristics of the entire space, taking into account global and local information, and also reduce interference with irrelevant areas in the road damage image. The specific steps are as follows: S1032-1, input the road damage feature map X1 into three Conv layers respectively, wherein the Conv layer is composed of 2C convolution kernels with a convolution kernel size of 1×1, and the result output by the first Conv layer is reduced in dimension to obtain the road damage feature map X2; the Conv layer with a convolution kernel size of 1×1 changes the number of channels of the input feature map, which depends on the number of convolution kernels of the Conv layer, and the number of convolution kernels of the Conv layer is equal to the number of channels of the output feature map; The output of the second Conv layer is reduced in dimension and input into the Softmax layer, and then the dimension of the result is transposed to obtain the road damage feature map X3; the output of the third Conv layer is reduced in dimension and input into the Softmax layer to obtain the road damage feature map X5; S1032-2, performing matrix multiplication on the road damage feature map X2 and the road damage feature map X3 to obtain a road damage feature map X4; S1032-3, performing matrix multiplication on the road damage feature map X4 and the road damage feature map X5, and then restoring the flattened shape to the shape of a two-dimensional feature map to obtain a road damage feature map X6; S1032-4, input the road damage feature map X6 into the Conv layer to obtain the road damage feature map X7; S1033, constructing a feature extraction part in the road damage warning model, inputting a road damage image, generating a final output road damage feature map, taking into account global and local information, while also reducing interference with irrelevant areas in the road damage image, and making feature extraction of small targets at damaged locations in the road damage more effective; S1034. Input the road damage feature map output by S1033 into the classifier and regressor respectively, output the predicted bounding box and target category, and finally remove the overlapping bounding boxes through non-maximum suppression, output the predicted bounding box and category, and finally realize the detection of the road damage warning model.

2. The road damage warning method based on deep learning according to claim 1 is characterized in that: In S1031, the calculation formula is as follows: Where k is the size of the HPooling layer, a m,n is the pixel point in the mth row and nth column of the corresponding k×k pooling window in the input feature map, where m=1,…,k,n=1,…,k, is the pixel point on the corresponding output feature map, and p is learned through model training, with the initial value set to 0.

5.

3. The road damage warning method based on deep learning according to claim 1 is characterized in that: The dimensionality reduction is to flatten the remaining dimensions, that is, to merge the height and width dimensions into one dimension.

4. The road damage warning method based on deep learning according to claim 1 is characterized in that: In S1033, the specific steps are as follows: S1033-1. Build a Conv layer and a ReLU layer with a convolution kernel size of 1×1 to form a Conv1×1+ReLU module, input the road damage image into the Conv1×1+ReLU module, and pass the output result into the FCP module to obtain the road damage feature map Y1; S1033-2, stack four FCP modules in sequence, and the output results of the first three FCP modules are input into the next FCP module after passing through a Conv layer with a convolution kernel size of 1×1. The road damage feature maps output by the four FCP modules are recorded as Y1, Y2, Y3, and Y4 in sequence; S1033-3, input the road damage feature maps Y1, Y2, Y3, and Y4 into four CSAttention modules respectively to obtain road damage feature maps Y'1, Y'2, Y'3, and Y'4; S1033-4, concat the road damage feature maps Y1, Y2, Y3, and Y4 with the road damage feature maps Y'1, Y'2, Y'3, and Y'4, respectively, to obtain road damage feature maps Y"1, Y"2, Y"3, and Y"4; S1033-5, input the road damage feature map Y"4 into the FCB module to obtain the road damage feature map Y"'4, then perform Conv1×1+Upsampling on the road damage feature map Y"'4 to obtain the road damage feature map Y""4, perform matrix multiplication on the road damage feature map Y""4 and the road damage feature map Y"3, and input the result into the FCB module to obtain the road damage feature map Y"'3; S1033-6, the road damage feature map Y"'3 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y""3, and the road damage feature map Y"2 is matrix multiplied, and the obtained result is input into the FCB module to obtain the road damage feature map Y"'2; Then, the road damage feature map Y”’2 is subjected to Conv1×1+Upsampling to obtain the road damage feature map Y””2, which is matrix-multiplied with the road damage feature map Y”1. The result is input into the FCB module to obtain the road damage feature map Y”’1. S1033-7, the road damage feature map Y"'1 is subjected to Conv1×1+Upsampling to obtain a road damage feature map Y""1, which is input into the Conv1×1+ReLU module. The result obtained is the final output road damage feature map; The Conv1×1+Upsampling refers to passing the feature map through a Conv layer with a convolution kernel size of 1×1, and then upsampling it by a factor of 2; the upsampling is to double the height and width by the nearest neighbor interpolation method, while the number of channels remains unchanged.

Images