A Road Defect Detection Method and System Based on Convolutional Neural Networks

Abstract

This invention belongs to the field of road construction technology and provides a road defect detection method and system based on convolutional neural networks. The method removes shadows from the road defect image using a shadow removal module based on generative adversarial networks. The road defect type is then detected based on the shadow-removed image and a target detection model. The target detection model is constructed as follows: a Yolov5 target detection network with integrated convolutional attention modules is used, performing attention mechanisms in both channel and spatial dimensions to extract feature maps of different dimensions. Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weightedly fuse the feature maps of different dimensions to obtain a fused feature map. This method solves the shortcomings of traditional road defect detection schemes and significantly improves detection accuracy.

Description

Technical Field

[0001] This invention belongs to the field of road construction technology, and in particular relates to a method and system for detecting road defects based on convolutional neural networks. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of road construction, the importance of road defect detection has become increasingly prominent. Timely and accurate acquisition of road defect information can save significant costs for road maintenance and reduce the likelihood of road traffic accidents.

[0004] Traditional road defect detection mainly relies on patrol personnel, who collect road defect data by parking for inspection, taking photos and recording data, and manual measurement.

[0005] However, it suffers from several problems: on the one hand, high labor costs, low detection efficiency, and poor security; on the other hand, the data is not objective and cannot effectively manage spatial information, failing to meet the requirements of modern road patrol and management. In recent years, computer vision methods and deep learning algorithms have been increasingly applied to the field of road defect detection, significantly improving the industry's level. However, current algorithms still have drawbacks such as being highly susceptible to noise and having poor model stability, and therefore require optimization. Summary of the Invention

[0006] To address at least one of the technical problems mentioned above, this invention provides a road defect detection method and system based on convolutional neural networks, which solves the shortcomings of traditional road defect detection schemes and significantly improves detection accuracy.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of the present invention provides a road defect detection method based on a convolutional neural network, comprising the following steps:

[0009] Acquire images of the road defects to be detected;

[0010] A shadow removal module based on generative adversarial networks removes shadows from images of road defects to be detected.

[0011] The road damage types are detected based on the image after shadow removal and the target detection model; wherein, the construction process of the target detection model is as follows: a Yolov5 target detection network with fused convolutional attention modules is used, and attention mechanisms are executed in the channel and spatial dimensions respectively to extract feature maps of different dimensions;

[0012] Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weight and fuse feature maps of different dimensions to obtain a fused feature map. Based on the fused feature map, feature recognition is performed to obtain the classification results of road defects.

[0013] A second aspect of the present invention provides a road defect detection system based on a convolutional neural network, comprising:

[0014] The data acquisition module is used to acquire images of road defects to be detected.

[0015] The shadow removal module is used to remove shadows from the image of road defects to be detected based on the generative adversarial network.

[0016] The road defect detection module is used to detect road defect types based on the image after shadow removal and the target detection model. The target detection model is constructed by using a Yolov5 target detection network with a fused convolutional attention module, performing attention mechanisms in the channel and spatial dimensions respectively, and extracting feature maps of different dimensions.

[0017] Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weight and fuse feature maps of different dimensions to obtain a fused feature map; based on the fused feature map, feature recognition is performed to obtain the classification results of road defects.

[0018] A third aspect of the present invention provides a computer-readable storage medium.

[0019] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the road defect detection method based on a convolutional neural network as described above.

[0020] A fourth aspect of the present invention provides a computer device.

[0021] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the road defect detection method based on a convolutional neural network as described above.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] This invention addresses the problem that the original YOLOv5 model has poor robustness in detecting road defects of varying sizes, particularly its over-focus on small objects, such as potholes with a diameter less than 30 mm, which are not considered road defects and thus increase workload. A method incorporating attention modules is proposed, implementing attention mechanisms in both the channel and spatial dimensions. This approach generally achieves higher detection confidence for correct categories compared to the original YOLOv5 model.

[0024] To address the issue that image shadows can easily interfere with the detection of road defects, especially cracks which are easily confused with tree branch shadows, this invention proposes a single-image shadow removal method based on channel attention and a generative adversarial network (GAN).

[0025] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a schematic flowchart of the road defect detection method based on convolutional neural networks according to an embodiment of the present invention;

[0028] Figures 2(a)-2(d) This is a sample database of road defect information according to embodiments of the present invention;

[0029] Figures 3(a)-3(d) These are example images of the shadow training dataset from an embodiment of the present invention;

[0030] Figures 4(a)-4(d) In this embodiment of the invention, the image to be tested is a shadowed image.

[0031] Figures 5(a)-5(d) This is the test result of the model after removing shadows in an embodiment of the present invention.

[0032] Figures 6(a)-6(j) This is a visualization result of the improved model training process according to an embodiment of the present invention. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0036] The overall idea of ​​this invention is:

[0037] First, based on the perspectives of vehicle-mounted cameras and drone aerial photography, a road damage image dataset was collected and labeled, and an image sample library containing information on four types of road damage was established. Second, based on the Yolov5 object detection algorithm, a shadow removal algorithm based on generative adversarial networks, an attention mechanism, and a feature fusion module were integrated to construct an object detection model with better performance.

[0038] This invention solves the shortcomings of traditional road defect detection schemes and significantly improves detection accuracy. The research results are mainly applied to the construction of information and intelligent road maintenance, so as to detect road defects in a timely manner, extend the service life of roads and related auxiliary facilities, and promote the innovation of the "Internet + road management" model.

[0039] Example 1

[0040] like Figure 1 As shown in the figure, this embodiment provides a road defect detection method based on convolutional neural networks, including the following steps:

[0041] Step 1: Obtain images of the road defects to be detected;

[0042] Step 2: Remove shadows from the road defect image based on a generative adversarial network-based shadow removal module;

[0043] Step 3: Based on the image after shadow removal and the target detection model, the road damage types are detected;

[0044] The construction process of the target detection model is as follows: a Yolov5 target detection network with fused convolutional attention modules is used, and attention mechanisms are executed in the channel and spatial dimensions respectively to extract feature maps of different dimensions;

[0045] Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weight and fuse feature maps of different dimensions to obtain a fused feature map.

[0046] Based on the three feature maps of different resolutions obtained by fusion, anchor boxes are used respectively. Regression calculations are performed on the anchor boxes based on the confidence threshold and non-maximum suppression (NMS) method. Finally, an output vector with class probability, confidence score and target bounding box is generated, realizing the classification and localization of road disease information.

[0047] As one or more embodiments, in step 1, a comprehensive and standardized dataset is the foundation for training the deep learning model. This embodiment addresses the road defect detection problem by establishing a sample library of road defect information from two perspectives and four categories; and for the image shadow processing problem, it compiles a highly targeted image shadow dataset.

[0048] like Figures 2(a)-2(d) As shown, the detection targets are divided into four categories of road defects: longitudinal cracks, transverse cracks, tortoise shell cracks, and road potholes.

[0049] The dataset is divided into two perspectives: vehicle-mounted video (HDV) and drone aerial photography (UAV), providing greater flexibility for road maintenance work.

[0050] In addition, to address the issue of shadow effects, this embodiment establishes an image shadow training dataset comprising four parts: images without shadows, images with shadows, shadow area masks, and shadow area edges, as follows: Figures 3(a)-3(d) As shown.

[0051] As one or more embodiments, in step 2, the shadow removal module of the generative adversarial network includes a shadow detector, a shadow detection discriminator, a shadow canceller, and a shadow cancellation discriminator;

[0052] The shadow detector and shadow canceller use the UNet++ network, which consists of upsampling, downsampling and multiple nodes. Each node is a residual block composed of a convolutional layer, a batch normalization layer, a Mish activation function and an scSE module.

[0053] The shadow detector has an additional structure after the UNet++ network. The additional structure consists of a 3×3 convolutional layer, a batch normalization layer, an LReLU activation function, and a Sigmoid activation function, which implements a range restriction on the shadow mask of the output, limiting the output to the range of 0 to 1.

[0054] The shadow detection discriminator stacks the labeled shadow mask, the input image, and the shadow mask output by the shadow detector into four channels to determine whether it is a shadow or a crack-type defect.

[0055] The shadow removal mechanism employs a ColorBlock structure appended to the UNet++ network. Since shadows are strongly influenced by light wavelength, and the light intensity obtainable by the camera varies depending on the target's color, training should focus on each color channel and the relationships between them. ColorBlock utilizes fully connected layers to estimate the weights of each channel, overcoming the limitation of ordinary convolutional layers in effectively training the relationships between channels. This allows for effective training of the physical characteristics of each wavelength, learning the differences between the input image and the actual ground image.

[0056] The shadow removal process of the shadow remover includes:

[0057] The shadow model represents the light intensity at any location and the light intensity in the shadowed area;

[0058] The difference between the input image and the ground reality image is obtained based on the light intensity at any location and the light intensity in the shadow area;

[0059] The image with shadows removed is obtained by eliminating shadows based on the difference between the input image and the ground reality image.

[0060] Specifically, the representation of light intensity at any location and light intensity in the shadow region based on the shadow model is as follows:

[0061] Based on the shadow model: I(x,λ)=L(x,λ)R(x,λ)

[0062] Where I is light intensity, L is illuminance, and R is reflectance. I, L, and R depend on the position x of a point on the image and the wavelength λ.

[0063] Therefore, we can conclude that:

[0064] The light intensity I at a point in the non-shaded area lit Represented as:

[0065] I lit (x,λ)=L d (x,λ)R(x,λ)+L a (x,λ)R(x,λ)

[0066] The light intensity I at a point in the shaded area shadow Represented as:

[0067] I shadow (x,λ)=L a (x,λ)R(x,λ)

[0068] In the formula, L d L represents the illuminance of direct lighting. a This indicates the illuminance of indirect lighting.

[0069] The difference between a point on the input image and the corresponding point on the ground reality image is represented as follows:

[0070] Δ=I gt -I input

[0071] =P(I lit (x,λ)-I shadow (x,λ))

[0072] ≈I lit (x,λ)-I shadow (x,λ)

[0073] =L d (x,λ)R(x,λ)

[0074] Wherein, function P represents the image processing of the camera image acquisition system, and I gt Represents a ground-based image (i.e., an image without shadows), I input This represents the input image (i.e., the image with shadows).

[0075] Since an image has only three color channels R, G, and B, λ in the above formula can be approximated as a function of R, G, and B. ColorBlock has the function of weighting each color channel by specific values, which can be understood as the illuminance of direct illumination for the three color channels of each image. Estimates.

[0076] Assuming the intensity of direct light from the light source is constant for the entire image, L d Regardless of position x, the result Δ can be expressed as:

[0077]

[0078] The shadow removal discriminator consists of a convolutional layer, a batch normalization layer, and a Mish activation function layer. These layers are connected in sequence, the stride of the convolutional layer is set, and the image after shadow removal is output.

[0079] The stride of the convolutional layer can be set according to actual needs. In this embodiment, the stride of the convolutional layer is set to 2 between every two convolutional layers.

[0080] The loss function of the shadow cancellation discriminator can be expressed as:

[0081]

[0082]

[0083] I input For the shaded image, M gt For the shadow mask image, Igt For a shadowless image, I output V is the deshadowed image output by the shadow remover. real V is a random value matrix with an average value of 0.5. fake It is a random value matrix with a mean of -0.5, which follows a Gaussian distribution.

[0084] The advantage of the above technology is that, in the process of target detection of road defects, in order to solve the problem that the detection effect is easily interfered with by image shadows, especially crack-type defects that are easily confused with tree branch shadows, a shadow removal network was designed based on the channel attention generative adversarial network single image shadow removal method (CANet) to reduce the probability of missed detection, false detection and false judgment.

[0085] As one or more embodiments, in step 3, the Convolutional Block Attention Module (CBAM) includes two sub-modules: Channel Attention (CAM) and Spatial Attention (SAM), which execute attention mechanisms in the channel and spatial dimensions, respectively.

[0086] In the process of detecting road defects, the original YOLOv5 model has poor robustness for detecting road defects of different sizes, especially for over-focusing on small objects, such as potholes with a diameter of less than 30 mm. These objects do not fall into the category of road defects and instead increase the workload.

[0087] The specific implementation of the attention mechanism in the channel and spatial dimensions includes:

[0088] When compressing the dimension of the feature map F0(H×W×C) extracted by a general convolutional layer, both average pooling and max pooling are used to obtain two one-dimensional feature maps F. 1,2 (1×1×C), F 1,2 Each feature is fed into a two-layer shared neural network (MLP). The features output by the MLP are summed element-wise and then activated by a sigmoid operation to generate the final channel attention feature Mc.

[0089] Element-wise multiplication is performed on the channel attention feature Mc and the input feature map F0 to obtain F3, which serves as the input feature of the SAM module. F3 is then subjected to global max pooling and global average pooling based on the channels, resulting in two one-dimensional feature maps F. 4,5 (H×W×1) The two feature maps are concatenated by channels, and a 7×7 convolution operation is used to reduce the dimensionality. Finally, a sigmoid activation operation is performed to generate the spatial attention feature Ms.

[0090] Finally, an element-wise multiplication operation is performed on the output Ms of the spatial attention module and the input feature map F3 to obtain the final generated features.

[0091] Spatial attention aims to enhance the feature representation of key regions. Essentially, it transforms the spatial information in the original image into another space through a spatial transformation module while retaining key information. It generates a weighted mask for each location and outputs a weighted result, thereby enhancing the specific target region of interest while weakening irrelevant background regions.

[0092] In this embodiment, in order to minimize changes to the network structure, that is, to make more use of the pre-trained parameters and to allow the model training process to converge as quickly as possible, the attention module is added outside the block.

[0093] In this embodiment, the C3 module, which is mainly responsible for extracting residual features in the YOLOv5 backbone network, is completely replaced with the "convolutional fusion attention + C3" module.

[0094] As one or more embodiments, step 3, specifically the weighted fusion of feature maps of different dimensions using an adaptive feature fusion method, includes:

[0095] Considering that simply stacking and overlaying the features cannot fully utilize the feature maps at different stages, this embodiment improves the original feature extraction module of Yolov5 in order to better fuse and extract the features output by the backbone network and fully represent features of different sizes.

[0096] Based on the idea of ​​bidirectional feature fusion, each layer of Adaptively Spatial Feature Fusion (ASFF) performs weighted fusion of stages of the original feature structure. The fusion of features from different stages adopts an attention mechanism to control the contribution of other stages to the features of this stage.

[0097] Algorithm verification

[0098] This embodiment is implemented based on the PyTorch deep learning framework and uses the GeForce RTX 2080Ti hardware platform. The experiment uses a Yolov5s pre-trained model. Because some pre-trained parameters are unavailable due to changes in the network structure, more training epochs are needed to avoid overfitting.

[0099] The model training strategies are shown in Table 1:

[0100] Table 1 Model Training Strategy

[0101]

[0102]

[0103] Multiple evaluation criteria were used in the experiment:

[0104] Accuracy (Accuracy, Acc):

[0105] Precision (P):

[0106] Recall (R):

[0107] Among them, TP is the number of positive classes that are determined to be positive, FP is the number of negative classes that are determined to be positive, FN is the number of positive classes that are determined to be negative, and TN is the number of negative classes that are determined to be negative.

[0108] Average Precision (AP):

[0109] AP is obtained based on the Precision-Recall curve, where r1, r2...rn are the Recall values ​​corresponding to the first interpolation point of the Precision interpolation segment arranged in ascending order.

[0110] Mean Average Precision (mAP):

[0111] mAP_0.5:0.95 represents the average value of IoU from 0.5 to 0.95, where mAP_0.5 represents the value of IoU when it is 0.5.

[0112] 1. Image shadow processing effect

[0113] The SRD+ pre-trained model was trained using a dataset consisting of 1055 tree branch shadow images and 1398 ISTD open-source images. Figures 4(a)-4(d) The image shown is shaded. Figures 5(a)-5(d) The image shows the test results of the model after removing the shadows.

[0114] Experiments show that the shadow removal algorithm model based on adversarial neural networks can efficiently remove shadows from images.

[0115] 2. The effects of the Yolov5 algorithm improvement

[0116] (1) To facilitate testing the model improvement effect, 3000 images from the vehicle's perspective were selected to form dataset 1 for training. A total of 25 experiments were designed for different improvement methods, and the training results are shown in Table 2:

[0117] Table 2 Training results for Dataset 1

[0118]

[0119] The experiment shows that YOLOv5 achieves the best performance improvement by adding the CBAM attention mechanism and replacing the ASFF feature extraction module, with a 3.5% increase in model training mAP.

[0120] (2) To improve the generalization performance of the model, a dataset of 14,055 images from multiple perspectives was selected for training. The training results are shown in Table 3, and the training process of the improved model is visualized as follows. Figures 6(a)-6(j) As shown:

[0121] Table 3 Training results for dataset 2

[0122] Methods metrics / mAP_0.5 metrics / mAP_0.5:0.95 Initial Yolov5 0.973 0.748 Improved Yolov5 0.987 0.794

[0123] Experiments show that the Yolov5 model using the improved ASFF+CBAM method improves mAP by 4.6% compared to the original Yolov5 algorithm.

[0124] (3) To adapt to actual road condition detection applications, 100 multi-view road defect images were collected to form dataset 3 for testing. The test results are shown in Table 4.

[0125] Table 4 shows the test results for dataset 3.

[0126] Methods Accuracy Original Yolov5 0.86 Improved Yolov5 0.94

[0127] Experiments show that the original YOLOv5 model tends to over-focus on small objects during detection, such as road potholes with a diameter of less than 30 mm. These objects do not constitute road defects and instead increase the workload. The improved YOLOv5 model solves this problem well. At the same time, the improved YOLOv5 model generally has a higher detection confidence for the correct category than the original YOLOv5 model.

[0128] Due to the overlap and complexity in the classification of road defects, the model encountered a small number of misclassifications during target detection.

[0129] 3. Introducing the Yolov5 detection model with image shadow processing

[0130] To test the performance improvement of the object detection model after shadow processing, a dataset of 100 images with shadows from different angles was created for testing. The test results are shown in Table 5.

[0131] Table 5. Test results for dataset 3

[0132] Methods Accuracy Yolov5 algorithm detects images before shadow removal. 0.83 Yolov5 algorithm for detecting shadow-removed images 0.90 Improved Yolov5 detection of images before shadow removal 0.86 Improved Yolov5 image after shadow removal 0.94

[0133] Experiments show that removing shadow interference significantly reduces the probability of false positives in road defect detection and can locate road defects more efficiently and accurately, resulting in a significant improvement in the detection of all four types of road defects.

[0134] In summary, the novel road defect detection algorithm proposed in this invention significantly improves detection accuracy compared to traditional road defect detection algorithms. Detection results show that the proposed solution can accurately identify cracks wider than 5 mm and potholes with a diameter greater than 50 mm, meeting the practical needs of current road surveys.

[0135] Example 2

[0136] This embodiment provides a road defect detection system based on convolutional neural networks, including:

[0137] The data acquisition module is used to acquire images of road defects to be detected.

[0138] The shadow removal module is used to remove shadows from the image of road defects to be detected based on the generative adversarial network.

[0139] The road defect detection module is used to detect road defect types based on the image after shadow removal and the target detection model. The target detection model is constructed by using a Yolov5 target detection network with a fused convolutional attention module, performing attention mechanisms in the channel and spatial dimensions respectively, and extracting feature maps of different dimensions.

[0140] Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weight and fuse feature maps of different dimensions to obtain a fused feature map; based on the fused feature map, feature recognition is performed to obtain the classification results of road defects.

[0141] Example 3

[0142] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the road defect detection method based on convolutional neural networks described above.

[0143] Example 4

[0144] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the road defect detection method based on convolutional neural networks as described above.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A road defect detection method based on convolutional neural networks, characterized in that, Includes the following steps: Acquire images of the road defects to be detected; A shadow removal module based on generative adversarial networks removes shadows from images of road defects to be detected. The types of road defects are obtained based on the image after shadow removal and the target detection model. The target detection model is constructed as follows: a Yolov5 target detection network with integrated convolutional attention modules is used, and attention mechanisms are executed in both the channel and spatial dimensions to extract feature maps of different dimensions. Specifically, the execution of attention mechanisms in both the channel and spatial dimensions includes: When compressing the dimensionality of the original feature map, both average pooling and max pooling are used to obtain two one-dimensional feature maps. The two one-dimensional feature maps are then fed into a two-layer shared neural network and summed to generate channel attention features. Multiply the channel attention features and the original feature map to obtain the third feature map. Perform global max pooling and global average pooling based on the channels to obtain two one-dimensional feature maps. Concatenate these two one-dimensional feature maps by channels and use convolution to reduce the dimensionality to generate spatial attention features. Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weight and fuse feature maps of different dimensions to obtain a fused feature map. Based on the fused feature map, feature recognition is performed to obtain the classification result of road defects. The specific steps of using the adaptive feature fusion method to weight and fuse feature maps of different dimensions include: Based on the idea of ​​bidirectional feature fusion, each layer of adaptive feature fusion performs weighted fusion of stages of the original feature structure. The fusion of features from different stages adopts an attention mechanism to control the contribution of other stages to the features of this stage.

2. The road defect detection method based on convolutional neural networks as described in claim 1, characterized in that, The shadow removal module based on generative adversarial networks removes shadows from the road defect image to be detected. The shadow removal module of the generative adversarial network includes a shadow remover, and the shadow remover's removal process includes: The shadow model represents the light intensity at any location and the light intensity in the shadowed area; The difference between the input image and the ground reality image is obtained based on the light intensity at any location and the light intensity in the shadow area; The image with shadows removed is obtained by eliminating shadows based on the difference between the input image and the ground reality image.

3. The road defect detection method based on convolutional neural networks as described in claim 2, characterized in that, The shadow remover uses a UNet++ network structure, consisting of upsampling, downsampling, and multiple nodes. Each node is a residual block composed of a convolutional layer, a batch normalization layer, a Mish activation function, and an scSE module. An additional ColorBlock structure is set after the UNet++ network, and the weights of each color channel of the image are estimated using a fully connected layer.

4. The road defect detection method based on convolutional neural networks as described in claim 2, characterized in that, The difference between the input image and the ground reality image is represented as follows: In the formula, Indicates an image without shadows. The function represents an image with shadows. P This refers to the image processing of the camera image acquisition system. For position x light intensity, The light intensity in the shaded area, R represents the illuminance of direct illumination, λ represents the reflectivity, and λ represents the wavelength.

5. The road defect detection method based on convolutional neural networks as described in claim 1, characterized in that, The types of road defects include four categories: longitudinal cracks, transverse cracks, tortoise-shell cracks, and potholes.

6. The road defect detection method based on convolutional neural networks as described in claim 1, characterized in that, The shadow removal module also includes a shadow detector that sets an additional structure after the UNet++ network to limit the range of the output shadow mask to between 0 and 1.

7. A road defect detection system based on convolutional neural networks, characterized in that, include: The data acquisition module is used to acquire images of road defects to be detected. The shadow removal module is used to remove shadows from the image of road defects to be detected based on the generative adversarial network. The road defect detection module is used to detect road defect types based on the image after shadow removal and the target detection model. The target detection model is constructed by using a Yolov5 target detection network with a fused convolutional attention module, performing attention mechanisms in the channel and spatial dimensions respectively, and extracting feature maps of different dimensions. Place The specific implementation of the attention mechanism in the channel and spatial dimensions includes: When compressing the dimensionality of the original feature map, both average pooling and max pooling are used to obtain two one-dimensional feature maps. The two one-dimensional feature maps are then fed into a two-layer shared neural network and summed to generate channel attention features. Multiply the channel attention features and the original feature map to obtain the third feature map. Perform global max pooling and global average pooling based on the channels to obtain two one-dimensional feature maps. Concatenate these two one-dimensional feature maps by channels and use convolution to reduce the dimensionality to generate spatial attention features. Based on the idea of ​​bidirectional feature fusion, an adaptive feature fusion method is used to weight and fuse feature maps of different dimensions to obtain a fused feature map. Based on the fused feature map, feature recognition is performed to obtain the classification result of road defects. The specific steps of using the adaptive feature fusion method to weight and fuse feature maps of different dimensions include: Based on the idea of ​​bidirectional feature fusion, each layer of adaptive feature fusion performs weighted fusion of stages of the original feature structure. The fusion of features from different stages adopts an attention mechanism to control the contribution of other stages to the features of this stage.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the road defect detection method based on convolutional neural networks as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the road defect detection method based on convolutional neural networks as described in any one of claims 1-6.

Images