Pavement crack and safety cone detection method and equipment oriented to abnormal weather

Abstract

The invention provides an abnormal weather-oriented pavement crack and safety cone detection method and device, and relates to the field of computer vision and image processing, and the method comprises the steps: inputting a to-be-detected synthetic image into a trained abnormal weather processing network, and obtaining a to-be-detected real image, and inputting a real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to respectively obtain a pavement crack detection result and a safety cone detection result. According to the invention, the abnormal weather processing network is constructed based on the combination of the GAN and the CNN, so that the problem that the result is excessively dim or saturated due to the confusion of rain stripes and image details in a conventional rain removal algorithm is solved, and the abnormal weather processing network is trained by using a semi-supervised learning method. And the rain removal effect of the image is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a method and device for detecting road cracks and safety cones in abnormal weather. Background Art

[0002] While the highway pavement is bearing the increasing traffic volume, it is also affected by various natural factors, and road problems such as pavement cracks and potholes gradually occur. Cracks are the prototype of many pavement diseases, so effective detection in the early stage of crack formation is an important guarantee for highway safety. In the process of repairing and maintaining the pavement in a non-completely closed state, in addition to the need for construction workers to wear safety equipment, it is also particularly important to reasonably place safety facilities on the pavement. Therefore, in the process of automatic road inspections by vehicles, drones, etc., it is of great significance to achieve high-precision real-time detection of pavement damage and the placement of safety facilities such as safety cones for safe construction. Images or videos taken in bad weather are easily interfered by scattering of rain, snow, and fog, and raindrops or snow particles of different sizes and shapes often adhere to glass windows or camera lenses when it rains or snows, which not only reduces the quality of the collected images, but also interferes with the subsequent computer vision tasks. Therefore, it is of great significance to take the correct method to remove the influence of abnormal weather before performing pavement crack detection and safety cone detection.

[0003] At present, the single image rain and snow removal algorithms are mainly divided into model-driven algorithms and data-driven algorithms. Model-driven algorithms use the prior knowledge of the image to design an optimization algorithm to solve the problem, thereby obtaining an image with a clean background. Data-driven algorithms build a neural network and use paired rain and snow streak labels and clean images without rain and snow to learn the nonlinear mapping from rain and snow to no rain and snow. Model-driven rain and snow removal algorithms have been gradually replaced by data-driven algorithms with better performance indicators in recent years due to their shortcomings such as poor generalization ability and high computational time. Yang et al. proposed a rain synthesis model and constructed a multi-task convolutional network JORDER, which uses paired rain streak labels and clean images without rain to learn the nonlinear mapping from rain to no rain, thereby realizing rain detection and removal. Fu et al. constructed a deep convolutional neural network to remove rain and snow streaks, used guided filtering to divide the image into a high-frequency detail layer and a low-frequency background layer, and then sent the high-frequency detail layer into the convolutional network for training to predict the rain and snow layer and the background layer, realizing the classification of rain and snow streaks and background information, and retaining object features.

[0004] The development of highway inspection has roughly gone through three stages: from traditional manual inspection, to semi-automatic inspection, and then to today's fully automated inspection based on machine vision. Among them, fully automated inspection technology makes up for the shortcomings of manual inspection, saves manpower, financial resources and time, greatly improves the evaluation efficiency, makes periodic inspection possible, and compared with semi-automatic inspection, its detection accuracy and speed have also been greatly improved. Fully automated inspection technology based on deep learning algorithms has become the mainstream method in the field of abnormal weather image restoration. Wang Hongmei et al. proposed a network framework built in combination with the SSD algorithm, which can detect various targets with high accuracy under complex backgrounds. Liu Hongjiang et al. thoroughly analyzed small target detection from multiple scales, and proposed relevant improvements for the problems of small targets in images with small coverage area, low resolution and unclear features; Song Xiaogang et al. proposed a real-time target detection algorithm YOLO-C for complex construction environments, which integrates the extracted low-level features with high-level features, enhances the global perception ability of the network, improves the detection accuracy and reduces the number of parameters, which is more suitable for target detection tasks in complex construction environments.

[0005] For abnormal weather processing, although model-driven and data-driven methods have made significant progress, they still face some challenges, such as excessive background smoothing or residual rain streaks, which affect the practicality and real-time performance of the restoration effect. For pavement crack and safety cone detection, Mandal V et al. compared the performance of several algorithms such as EfficientDet, CenterNet and YOLO series in pavement disease detection and found that the best performance was the YOLO series algorithm. However, in the face of the more complex road types and pavement environments, the existing YOLO algorithm still has some defects in pavement disease identification, such as weak detection capabilities for small-area, irregular-shaped diseases, difficult feature extraction, and insufficient efficiency and accuracy. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a method and device for detecting road cracks and safety cones in abnormal weather conditions, so as to overcome the problem that conventional rain removal algorithms may produce excessively dim or saturated results due to confusion between rain streaks and image details.

[0007] The present invention provides a method for detecting road cracks and safety cones in abnormal weather, comprising:

[0008] S1: Construct an abnormal weather processing network, train the abnormal weather processing network through a sample real image set and a sample synthetic image set, and obtain a trained abnormal weather processing network;

[0009] S2: construct a pavement crack image set and a safety cone image set through a sample real image set, input the pavement crack image set and the safety cone image set into an adaptive feature reconstruction network, and obtain a pavement crack feature set and a safety cone feature set;

[0010] S3: Construct a pavement crack detection network and a safety cone detection network, train the pavement crack detection network through the pavement crack feature set to obtain a trained pavement crack detection network; train the safety cone detection network through the safety cone feature set to obtain a trained safety cone detection network;

[0011] S4: Input the synthetic image to be detected into the trained abnormal weather processing network to obtain the real image to be detected, and input the real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to obtain the pavement crack detection results and the safety cone detection results respectively.

[0012] Preferred:

[0013] The abnormal weather processing network includes: feature conversion space module, matching network, GAN-CNN network, and image conversion space module;

[0014] The GAN-CNN network includes: convolutional neural network and generative adversarial network. The generative adversarial network includes generator and discriminator.

[0015] Preferably, step S1 specifically comprises:

[0016] S11: extract two sample real images from the sample real image set, use one of the sample real images as the guide rain image, extract a sample synthetic image from the sample synthetic image set as the synthetic rain image, and adjust the parameters of the matching network and the GAN-CNN network;

[0017] S12: extracting features from the synthetic rain image and the guided rain image through a feature conversion space module to obtain a first synthetic feature and a second synthetic feature;

[0018] S13: inputting the first synthetic feature and the second synthetic feature into a matching network, and calculating a matching loss of the matching network;

[0019] S14: The first synthetic feature passes through the matching network, the GAN-CNN network and the image conversion space module in sequence to obtain a derained image, and the perceptual loss and mean square error loss of the GAN-CNN network are calculated by the derained image and the sample real image;

[0020] S15: Repeat steps S12-S14 until the matching loss, perception loss and mean square error loss are all less than the corresponding preset values, and a trained abnormal weather processing network is obtained.

[0021] Preferred:

[0022] Matching loss L match The calculation formula is:

[0023] L match =L f +L c

[0024]

[0025] Among them, F s and F t Represent the first synthetic feature and the second synthetic feature respectively, M s and M t Respectively represent F s and F t The mean vector of represents the Frobenius norm, and d represents the feature dimension.

[0026] Preferred:

[0027] The perceptual loss of the generative adversarial network is calculated as:

[0028] perceptual loss=content loss+λ·adverserial loss

[0029] content loss=||VGG(I truth )-VGG(I clear )||2

[0030] adverse loss=∑[log(D(I truth ))+log(1-G(I clear ))]

[0031] Among them, I truth is the sample real image, I clear is a derained image, VGG represents a convolutional neural network, ||·||2 represents the L2 norm, and λ is the shared weight of the perceptual loss and the mean square error loss;

[0032] The calculation formula for the mean square error loss of the convolutional neural network is:

[0033] I=||VGG[f(I;θ)]-VGG(I truth )||2+λ·||f(I;θ)-I truth ||2

[0034] Among them, f (I; θ) = I clear Represents a derained image.

[0035] Preferred:

[0036] The pavement crack detection network includes: a first Backbone network, a first Neck network and a first Head network;

[0037] The first Backbone network includes a Conv module, a first CBS module, a first C3 module, a second CBS module, a second C3 module, a third CBS module, a third C3 module, a fourth CBS module, a first SPPF module and a fourth C3 module connected in sequence;

[0038] The first Neck network includes a fifth CBS module, a first Upsample module, a fifth C3 module, a sixth CBS module, a second Upsample module, a sixth C3 module, a seventh CBS module, a seventh C3 module, an eighth CBS module and an eighth C3 module connected in sequence;

[0039] The first Head network includes: a first Conv2d module, a second Conv2d module and a third Conv2d module;

[0040] The fourth C3 module, the fifth CBS module, the eighth C3 module and the first Conv2d module are connected in sequence;

[0041] The third C3 module, the fifth C3 module, the sixth CBS module, the seventh C3 module and the second Conv2d module are connected in sequence;

[0042] The second C3 module, the sixth C3 module and the third Conv2d module are connected in sequence.

[0043] Preferred:

[0044] The safety cone detection network includes: the second Backbone network, the second Neck network and the second Head network;

[0045] The second Backbone network includes a ninth CBS module, a first GhostConv module, a first C3Ghost module, a second GhostConv module, a second C3Ghost module, a third GhostConv module, a third C3Ghost module, a fourth GhostConv module, a fourth C3Ghost module and a second SPPF module connected in sequence;

[0046] The second Neck network includes a fifth GhostConv module, a third Upsample module, a fifth C3Ghost module, a sixth GhostConv module, a fourth Upsample module, a sixth C3Ghost module, a seventh GhostConv module, a seventh C3Ghost module, an eighth GhostConv module, and an eighth C3Ghost module connected in sequence;

[0047] The second Head network includes: a fourth Conv2d module, a fifth Conv2d module and a sixth Conv2d module;

[0048] The second SPPF module, the fifth GhostConv module, the eighth C3Ghost module and the fourth Conv2d module are connected in sequence;

[0049] The third C3Ghost module, the fifth C3Ghost module, the sixth GhostConv module, the seventh C3Ghost module and the fifth Conv2d module are connected in sequence;

[0050] The second C3Ghost module, the sixth C3Ghost module and the sixth Conv2d module are connected in sequence.

[0051] A storage medium stores instructions and data for implementing the road surface crack and safety cone detection method under abnormal weather conditions.

[0052] A road crack and safety cone detection device for abnormal weather conditions, comprising: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the road crack and safety cone detection method for abnormal weather conditions.

[0053] The present invention has the following beneficial effects:

[0054] 1. The present invention jointly constructs an abnormal weather processing network based on the generative adversarial network (GAN) and the convolutional neural network (CNN) to overcome the problem that the conventional rain removal algorithm will produce excessive dimness or saturation of the results due to the confusion between rain streaks and image details, and uses a semi-supervised learning method to train the abnormal weather processing network to improve the image rain removal effect.

[0055] 2. During the training of the abnormal weather processing network, the matching loss, perception loss and mean square error loss are all calculated to improve the model's rain removal performance on real rain images and ensure the consistency of the feature distribution of synthetic rain streaks with real rain streaks.

[0056] 3. More discriminative pavement crack features and safety cone features are obtained through the adaptive feature reconstruction network, making the pavement crack defect and safety cone sample data more balanced and increasing the diversity of abnormal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0058] Figure 2 A structural diagram of the network for abnormal weather processing;

[0059] Figure 3 This is the structural diagram of the GAN-CNN network;

[0060] Figure 4 Structural diagram of the network for adaptive feature reconstruction based on the twin network;

[0061] Figure 5 This is the structural diagram of the pavement crack detection network;

[0062] Figure 6 This is the structural diagram of the safety cone detection network;

[0063] Figure 7 This is a structural diagram of the device according to an embodiment of the present invention;

[0064] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0066] Reference Figure 1 The present invention provides a method for detecting road cracks and safety cones in abnormal weather, comprising:

[0067] S1: Construct an abnormal weather processing network, train the abnormal weather processing network through a sample real image set and a sample synthetic image set, and obtain a trained abnormal weather processing network;

[0068] Furthermore, the abnormal weather processing network adopts a semi-supervised learning strategy to convert the synthetic rain map and the real rain map into feature vectors in the feature transformation space, and accurately models the first-order and second-order statistical information of these feature maps to extract domain invariant features. The semi-supervised matching network is then input to minimize the feature differences between the synthetic and real rain maps; Figure 2 The structural diagram of the abnormal weather processing network is as follows: Figure 3 This is the structural diagram of the GAN-CNN network;

[0069] The abnormal weather processing network includes: feature conversion space module, matching network, GAN-CNN network, and image conversion space module;

[0070] The GAN-CNN network includes: convolutional neural network and generative adversarial network. The generative adversarial network includes generator and discriminator.

[0071] Specifically, the matching network consists of sixteen matching modules, each of which contains four 3×3 convolution kernels. The synthetic rain image and the real rain image are converted into feature vectors through this network. The feature output is as follows:

[0072] f0=Conv 3×3 (O)

[0073] y0=Conv 3×3 (O r )

[0074] Among them, O and O r are the inputs of the synthetic rain image and the real rain image respectively, and f0 and y0 are the corresponding feature outputs.

[0075] These features are processed by 1×1 convolution kernels and cascade operations, and the output of each matching module is as follows:

[0076] F i =M i (Conv 1×1 (Cat[f i-1 , f i-2 , …, f0]))

[0077] Y i =M i (Conv 1×1 (Cat[y i-1 ,y i-2 , …, y0]))

[0078] Among them, M i Represents the output of the i-th processing module. The outputs of these modules are combined through a densely connected network to enhance the fusion of information between deep and shallow layers. The specific operation is as follows:

[0079] F usion =Conv 1×1 (Cat[F i-1 , F i-2 , …, F0])

[0080] Y usion =Conv 1×1 (Cat[Y i-1 , Y i-2 , …, Y0])

[0081] Furthermore, step S1 is specifically as follows:

[0082] S11: extract two sample real images from the sample real image set, use one of the sample real images as the guide rain image, extract a sample synthetic image from the sample synthetic image set as the synthetic rain image, and adjust the parameters of the matching network and the GAN-CNN network;

[0083] Specifically, the synthetic rain image P and the guided rain image I are input into the abnormal weather processing network. The input image P and the guided image I are first compared and analyzed by the guided filter to generate a new output image Q. By reducing the gap between the output image Q and the original image P, the processing result is further optimized. The specific expression of the guided filter is as follows:

[0084]

[0085] Among them, μ k ,I i ,I j , σ k The relative position relationship between ε and ε plays a key role in determining the accuracy of the element layout in the image. The position information of these elements will be encoded into their own different matrices to realize the calculation of element adaptive weights and adjust the pixel values ​​of various regions of the image.

[0086] S12: extracting features from the synthetic rain image and the guided rain image through a feature conversion space module to obtain a first synthetic feature and a second synthetic feature;

[0087] S13: inputting the first synthetic feature and the second synthetic feature into a matching network, and calculating a matching loss of the matching network;

[0088] S14: The first synthetic feature passes through the matching network, the GAN-CNN network and the image conversion space module in sequence to obtain a derained image, and the perceptual loss and mean square error loss of the GAN-CNN network are calculated by the derained image and the sample real image;

[0089] Furthermore, the matching loss L match The calculation formula is:

[0090] L match =L f +L c

[0091]

[0092] Among them, F s and F t Represent the first synthetic feature and the second synthetic feature respectively, M s and M t Respectively represent Fs and F t The mean vector of represents the Frobenius norm, and d represents the feature dimension.

[0093] Furthermore, the element I is extracted from the image obtained by semi-supervised matching. base and I detail , where I detail =II base I base and I detail It is used as input to further train the generator and discriminator. The generative adversarial network uses perceptual loss as the loss function. The perceptual loss consists of content loss and adversarial loss, which is used to evaluate and optimize the processing performance of the model.

[0094] The perceptual loss of the generative adversarial network is calculated as:

[0095] perceptual loss=content loss+λ·adverserial loss

[0096] content loss=||VGG(I truth )-VGG(I clear )||2

[0097] adverserial loss=Σ[log(D(I truth ))+log(1-G(I clear ))]

[0098] Among them, I truth is the sample real image, I clear is a derained image, VGG represents a convolutional neural network, ||·||2 represents the L2 norm, and λ is the shared weight of the perceptual loss and the mean square error loss;

[0099] Furthermore, a convolutional neural network that shares the same architecture and weights as the generative adversarial network is used for further reconstruction to better preserve the color and texture of the image while eliminating the imperfections introduced by the perceptual loss. The generative adversarial network GAN and the convolutional neural network CNN are set to alternate training to optimize the performance of each network. At the same time, in order to quantify the difference between the generated image and the real image, a mean square error loss is defined, which includes two parts: one is the Euclidean distance between the pre-trained features of the VGG network, and the other is the mean square error MSE between the predicted image and the real image;

[0100] The calculation formula for the mean square error loss of the convolutional neural network is:

[0101] I=||VGG[f(I;θ)]-VGG(I truth )||2+λ·||f(I;θ)-I truth ||2

[0102] Among them, f (I; θ) = I clear Represents a derained image.

[0103] Specifically, the mean square error loss not only evaluates the pixel-level error, but also captures the high-level semantic differences of the image through the VGG network to ensure the visual authenticity and quality of the generated image. This design helps the network to effectively use the square of the L2 norm for optimization when encountering a local minimum. That is, it achieves:

[0104]

[0105] S15: Repeat steps S12-S14 until the matching loss, perception loss and mean square error loss are all less than the corresponding preset values, and a trained abnormal weather processing network is obtained.

[0106] S2: construct a pavement crack image set and a safety cone image set through a sample real image set, input the pavement crack image set and the safety cone image set into an adaptive feature reconstruction network, and obtain a pavement crack feature set and a safety cone feature set;

[0107] Furthermore, step S2 is specifically as follows:

[0108] Step S21, build a self-data set by collecting existing data such as Crake500, GAPs384, CFD, AEL, Craketree200, and acquiring real field data.

[0109] Step S22, according to step S21, obtain a higher quality synthetic data set. Use the data set to train the twin network model to learn how to extract useful features of the input image. Then, a part of the trained twin network (i.e., the twin network feature extraction part) is used as a general adaptive feature reconstruction network, such as Figure 4 As shown in the figure. The adaptive feature reconstruction network can obtain more discriminative features of defect anomaly targets, improve the network's ability to handle high-similarity categories and improve the accuracy under small samples. The feature extraction part of the twin network uses the same network, and the loss function is the difference between the predicted similarity and the label value.

[0110] Step S23, according to step S22, obtain the obvious features of the data. Use top-down global attention modulation and bottom-up local attention modulation to exchange multi-scale context to obtain richer semantic information and spatial detail encoding. Given a low-level feature X and a high-level feature Y, with C channels and a feature map of size H×W, the top-down and bottom-up attention modulation can be expressed as:

[0111]

[0112] in δ,BN,σ, They represent rectified linear unit (ReLU), batch normalization (BN), sigmoid function, and element-wise multiplication respectively.

[0113] Step S24: The context representation information can be enhanced through step S23. The multi-scale context is exchanged by using the top-down global attention modulation and the bottom-up local attention modulation to obtain the encoding of richer semantic information and spatial details.

[0114] Asymmetric context modulation is implemented for cross-layer feature fusion.

[0115]

[0116] in, For fusion features.

[0117] S3: Construct a pavement crack detection network and a safety cone detection network, train the pavement crack detection network through the pavement crack feature set to obtain a trained pavement crack detection network; train the safety cone detection network through the safety cone feature set to obtain a trained safety cone detection network;

[0118] Furthermore, step S3 is specifically as follows:

[0119] Step S31, YOLOv5 is used as the initial backbone network for road crack detection, and GhostNet is deployed on YOLOv5 as the initial backbone network for safety cone detection.

[0120] Step S32: According to step S31, the initial backbone network for road crack detection and safety cone detection is obtained, and the self-attention mechanism is introduced to replace the dot product similarity in the attention calculation with the position-aware similarity. Output The calculation is as follows:

[0121]

[0122] Among them, q, k, and v are all calculated projections from the input x, which can capture non-local information from the entire feature map. In order to overcome the disadvantage of this similarity measure that the amount of calculation is very large, the axial self-attention mechanism is introduced. For a given specific input feature map x, the original self-attention module can be decomposed into two independent self-attention modules by introducing position encoding and updating the width axis self-attention mechanism. The first module calculates self-attention on the height axis of the feature map, while the second module calculates on the width axis. In order to include position information in the calculation, a relative position encoding that can be learned during training is also introduced. The overall calculation can be expressed as:

[0123]

[0124] where r q 、r k 、r v is the position offset.

[0125] Step S33, according to step S32, the first-stage backbone network for pavement crack detection and safety cone detection is obtained. DIoU is used to replace the NMS standard of the first-stage backbone network. The penalty term that maximizes the overlap area by introducing the minimum bounding box in GIoU is modified to minimize the standardized distance between the center points of the two boxes, thereby accelerating the convergence process of the loss, considering not only the overlap area but also the center point distance. The DIoU calculation formula is as follows:

[0126]

[0127] where s i is the classification confidence, ε is the NMS threshold, M is the box with the highest confidence,

[0128] Step S34: According to step S24, data features are obtained. According to step S33, the final backbone network for pavement crack detection and safety cone detection is obtained. The required pavement crack prediction model and safety cone prediction model can be obtained through data features and backbone network training. When predicting, choose whether to turn on abnormal weather image processing according to the actual weather conditions. If turned on, image processing is first implemented through the abnormal weather processing model, and then target detection is implemented through the pavement crack detection model and triangular cone detection model respectively. If not turned on, target detection is directly implemented through the pavement crack detection model and triangular cone detection model.

[0129] Furthermore, the structure of the pavement crack detection network is as follows Figure 5 As shown;

[0130] The pavement crack detection network includes: a first Backbone network, a first Neck network and a first Head network;

[0131] The first Backbone network includes a Conv module, a first CBS module, a first C3 module, a second CBS module, a second C3 module, a third CBS module, a third C3 module, a fourth CBS module, a first SPPF module and a fourth C3 module connected in sequence;

[0132] The first Neck network includes a fifth CBS module, a first Upsample module, a fifth C3 module, a sixth CBS module, a second Upsample module, a sixth C3 module, a seventh CBS module, a seventh C3 module, an eighth CBS module and an eighth C3 module connected in sequence;

[0133] The first Head network includes: a first Conv2d module, a second Conv2d module and a third Conv2d module;

[0134] The fourth C3 module, the fifth CBS module, the eighth C3 module and the first Conv2d module are connected in sequence;

[0135] The third C3 module, the fifth C3 module, the sixth CBS module, the seventh C3 module and the second Conv2d module are connected in sequence;

[0136] The second C3 module, the sixth C3 module and the third Conv2d module are connected in sequence.

[0137] Furthermore, the structure of the safety cone detection network is as follows Figure 6 As shown;

[0138] The safety cone detection network includes: the second Backbone network, the second Neck network and the second Head network;

[0139] The second Backbone network includes a ninth CBS module, a first GhostConv module, a first C3Ghost module, a second GhostConv module, a second C3Ghost module, a third GhostConv module, a third C3Ghost module, a fourth GhostConv module, a fourth C3Ghost module and a second SPPF module connected in sequence;

[0140] The second Neck network includes a fifth GhostConv module, a third Upsample module, a fifth C3Ghost module, a sixth GhostConv module, a fourth Upsample module, a sixth C3Ghost module, a seventh GhostConv module, a seventh C3Ghost module, an eighth GhostConv module, and an eighth C3Ghost module connected in sequence;

[0141] The second Head network includes: a fourth Conv2d module, a fifth Conv2d module and a sixth Conv2d module;

[0142] The second SPPF module, the fifth GhostConv module, the eighth C3Ghost module and the fourth Conv2d module are connected in sequence;

[0143] The third C3Ghost module, the fifth C3Ghost module, the sixth GhostConv module, the seventh C3Ghost module and the fifth Conv2d module are connected in sequence;

[0144] The second C3Ghost module, the sixth C3Ghost module and the sixth Conv2d module are connected in sequence.

[0145] S4: Input the synthetic image to be detected into the trained abnormal weather processing network to obtain the real image to be detected, and input the real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to obtain the pavement crack detection results and the safety cone detection results respectively.

[0146] See also Figure 7 , Figure 7 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a road crack and safety cone detection device 401 for abnormal weather, a processor 402 and a storage medium 403.

[0147] A road surface crack and safety cone detection device 401 for abnormal weather conditions: The road surface crack and safety cone detection device 401 for abnormal weather conditions implements the road surface crack and safety cone detection method for abnormal weather conditions.

[0148] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the road surface crack and safety cone detection method under abnormal weather conditions.

[0149] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the road surface crack and safety cone detection method under abnormal weather conditions.

[0150] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0151] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.

[0152] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting road cracks and safety cones in abnormal weather, characterized in that: include: S1: Construct an abnormal weather processing network, train the abnormal weather processing network through a sample real image set and a sample synthetic image set, and obtain a trained abnormal weather processing network; S2: construct a pavement crack image set and a safety cone image set through a sample real image set, input the pavement crack image set and the safety cone image set into an adaptive feature reconstruction network, and obtain a pavement crack feature set and a safety cone feature set; S3: Construct a pavement crack detection network and a safety cone detection network, train the pavement crack detection network through the pavement crack feature set to obtain a trained pavement crack detection network; train the safety cone detection network through the safety cone feature set to obtain a trained safety cone detection network; S4: Input the synthetic image to be detected into the trained abnormal weather processing network to obtain the real image to be detected, and input the real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to obtain the pavement crack detection results and the safety cone detection results respectively.

2. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: The abnormal weather processing network includes: feature conversion space module, matching network, GAN-CNN network, and image conversion space module; The GAN-CNN network includes: convolutional neural network and generative adversarial network. The generative adversarial network includes generator and discriminator.

3. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: Step S1 is specifically as follows: S11: extract two sample real images from the sample real image set, use one of the sample real images as the guide rain image, extract a sample synthetic image from the sample synthetic image set as the synthetic rain image, and adjust the parameters of the matching network and the GAN-CNN network; S12: extracting features from the synthetic rain image and the guided rain image through a feature conversion space module to obtain a first synthetic feature and a second synthetic feature; S13: inputting the first synthetic feature and the second synthetic feature into a matching network, and calculating a matching loss of the matching network; S14: The first synthetic feature passes through the matching network, the GAN-CNN network and the image conversion space module in sequence to obtain a derained image, and the perceptual loss and mean square error loss of the GAN-CNN network are calculated by the derained image and the sample real image; S15: Repeat steps S12-S14 until the matching loss, perception loss and mean square error loss are all less than the corresponding preset values, and a trained abnormal weather processing network is obtained.

4. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 3, characterized in that: Matching loss L match The calculation formula is: L match =L f +L c Among them, F s and F t Represent the first synthetic feature and the second synthetic feature respectively, M s and M t Respectively represent F s and F t The mean vector of represents the Frobenius norm, and d represents the feature dimension.

5. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 3, characterized in that: The perceptual loss of the generative adversarial network is calculated as: perceptualloss=contentloss+λ·adverserialloss contentloss=||VGG(I truth )-VGG(I clear )||2 adverserial loss=∑[log(D(I truth ))+log(1-G(I clear ))] Among them, I truth is the sample real image, I clear is a derained image, VGG represents a convolutional neural network, ||·||2 represents the L2 norm, and λ is the shared weight of the perceptual loss and the mean square error loss; The calculation formula for the mean square error loss of the convolutional neural network is: I=||VGG[f(I;θ)]-VGG(I truth )||2+λ·||f(I;θ)-I truth ||2 Among them, f (I; θ) = I clear Represents a derained image.

6. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: The pavement crack detection network includes: a first Backbone network, a first Neck network and a first Head network; The first Backbone network includes a Conv module, a first CBS module, a first C3 module, a second CBS module, a second C3 module, a third CBS module, a third C3 module, a fourth CBS module, a first SPPF module and a fourth C3 module connected in sequence; The first Neck network includes a fifth CBS module, a first Upsample module, a fifth C3 module, a sixth CBS module, a second Upsample module, a sixth C3 module, a seventh CBS module, a seventh C3 module, an eighth CBS module and an eighth C3 module connected in sequence; The first Head network includes: a first Conv2d module, a second Conv2d module and a third Conv2d module; The fourth C3 module, the fifth CBS module, the eighth C3 module and the first Conv2d module are connected in sequence; The third C3 module, the fifth C3 module, the sixth CBS module, the seventh C3 module and the second Conv2d module are connected in sequence; The second C3 module, the sixth C3 module and the third Conv2d module are connected in sequence.

7. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: The safety cone detection network includes: the second Backbone network, the second Neck network and the second Head network; The second Backbone network includes a ninth CBS module, a first GhostConv module, a first C3Ghost module, a second GhostConv module, a second C3Ghost module, a third GhostConv module, a third C3Ghost module, a fourth GhostConv module, a fourth C3Ghost module and a second SPPF module connected in sequence; The second Neck network includes a fifth GhostConv module, a third Upsample module, a fifth C3Ghost module, a sixth GhostConv module, a fourth Upsample module, a sixth C3Ghost module, a seventh GhostConv module, a seventh C3Ghost module, an eighth GhostConv module, and an eighth C3Ghost module connected in sequence; The second Head network includes: a fourth Conv2d module, a fifth Conv2d module and a sixth Conv2d module; The second SPPF module, the fifth GhostConv module, the eighth C3Ghost module and the fourth Conv2d module are connected in sequence; The third C3Ghost module, the fifth C3Ghost module, the sixth GhostConv module, the seventh C3Ghost module and the fifth Conv2d module are connected in sequence; The second C3Ghost module, the sixth C3Ghost module and the sixth Conv2d module are connected in sequence.

8. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the road surface crack and safety cone detection method under abnormal weather conditions as described in any one of claims 1 to 7.

9. A road crack and safety cone detection device for abnormal weather conditions, characterized by: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the road crack and safety cone detection method for abnormal weather conditions as described in any one of claims 1 to 7.

Images