A method and device for detecting road cracks and safety cones in abnormal weather conditions

Abstract

This invention provides a method and device for detecting pavement cracks and safety cones in abnormal weather conditions, relating to the fields of computer vision and image processing. The method comprises the following steps: inputting a synthetic image to be detected into a trained abnormal weather processing network to obtain a real image to be detected; then inputting the real image to be detected into a trained pavement crack detection network and a trained safety cone detection network to obtain pavement crack detection results and safety cone detection results, respectively. This invention combines a generative adversarial network (GAN) and a convolutional neural network (CNN) to construct an abnormal weather processing network to overcome the problem of conventional rain removal algorithms causing excessive dimming or saturation of results due to the confusion between rain streaks and image details. The abnormal weather processing network is trained using a semi-supervised learning method to improve image rain removal effectiveness.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a method and device for detecting road cracks and safety cones in abnormal weather. Background Art

[0002] Highway pavements, while bearing increasing traffic volume and subject to various natural factors, are gradually developing road problems such as cracks and potholes. Cracks are the precursors of many pavement defects, so effective detection in their early stages is crucial for highway safety. During repair and maintenance operations on partially enclosed pavements, in addition to requiring workers to wear safety equipment, the proper placement of safety features is also crucial. Therefore, during automated road inspections by vehicles and drones, high-precision, real-time detection of pavement damage and the placement of safety features such as safety cones is crucial for safe construction. Images or videos captured in inclement weather are susceptible to scattering interference from rain, snow, and fog. Furthermore, raindrops and snow particles of varying sizes often adhere to glass windows or camera lenses, reducing image quality and interfering with subsequent computer vision tasks. Therefore, it is crucial to implement appropriate methods to mitigate the effects of abnormal weather before performing pavement crack and safety cone detection.

[0003] Currently, single-image rain and snow removal algorithms are mainly divided into model-driven and data-driven algorithms. Model-driven algorithms use prior knowledge of the image to design optimization algorithms for solution, thereby obtaining a clean background image. Data-driven algorithms build neural networks and utilize paired rain and snow streak labels and clean, rain-free images to learn a nonlinear mapping from rain to no rain. Due to shortcomings such as poor generalization and high computational time, model-driven rain and snow removal algorithms have been gradually replaced by data-driven algorithms with better performance metrics in recent years. Yang et al. proposed a rain synthesis model and constructed a multi-task convolutional network (JORDER). This network uses paired rain streak labels and clean, rain-free images to learn a nonlinear mapping from rain to no rain, thereby achieving rain detection and removal. Fu et al. constructed a deep convolutional neural network to remove rain and snow streaks. They used guided filtering to separate the image into a high-frequency detail layer and a low-frequency background layer. The high-frequency detail layer was then fed into a convolutional network for training to predict the rain and background layers, achieving classification of rain and snow streaks and background information while preserving object features.

[0004] Highway inspection has evolved through three stages: from traditional manual inspection to semi-automated inspection, and now to fully automated inspection based on machine vision. Fully automated inspection technology addresses the shortcomings of manual inspection, saving manpower, financial resources, and time, significantly improving evaluation efficiency, and enabling periodic inspection. Compared to semi-automated inspection, it also significantly improves detection accuracy and speed. Fully automated inspection technology based on deep learning algorithms has become a mainstream approach in abnormal weather image restoration. Wang Hongmei et al. proposed a network framework combined with the SSD algorithm, enabling high-precision detection of various targets in complex backgrounds. Liu Hongjiang et al. thoroughly analyzed small object detection at multiple scales and proposed improvements to address issues such as small image coverage, low resolution, and unclear features. Song Xiaogang et al. proposed YOLO-C, a real-time object detection algorithm for complex construction environments. By fusing extracted low-level features with high-level features, the algorithm enhances the network's global perception capabilities, improves detection accuracy, and reduces the number of parameters, making it more suitable for object detection tasks in complex construction environments.

[0005] Despite significant progress in model-driven and data-driven approaches to abnormal weather processing, challenges remain, such as over-smoothing of the background or residual rain streaks, which hinder the practicality and real-time performance of the restoration results. For pavement crack and safety cone detection, Mandal V et al. compared the performance of several algorithms, including EfficientDet, CenterNet, and the YOLO series, in pavement defect detection and found that the YOLO series performed best. However, given today's complex road types and pavement environments, the existing YOLO algorithm still has some shortcomings in pavement defect identification, such as weak detection capabilities for small, irregularly shaped defects, difficulty in feature extraction, and insufficient efficiency and accuracy. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and device for detecting road cracks and safety cones in abnormal weather conditions, so as to overcome the problem that conventional rain removal algorithms may produce excessively dim or saturated results due to the confusion between rain streaks and image details.

[0007] The present invention provides a method for detecting road cracks and safety cones in abnormal weather conditions, comprising:

[0008] S1: Construct an abnormal weather processing network and train it using a set of sample real images and a set of sample synthetic images to obtain a trained abnormal weather processing network.

[0009] S2: Construct a pavement crack image set and a safety cone image set from a sample real image set, input the pavement crack image set and the safety cone image set into an adaptive feature reconstruction network to obtain a pavement crack feature set and a safety cone feature set;

[0010] S3: Construct a pavement crack detection network and a safety cone detection network. Train the pavement crack detection network using the pavement crack feature set to obtain a trained pavement crack detection network. Train the safety cone detection network using the safety cone feature set to obtain a trained safety cone detection network.

[0011] S4: Input the synthetic image to be detected into the trained abnormal weather processing network to obtain the real image to be detected, and input the real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to obtain the pavement crack detection results and the safety cone detection results respectively.

[0012] Preferred:

[0013] The abnormal weather processing network includes: feature conversion space module, matching network, GAN-CNN network, and image conversion space module;

[0014] The GAN-CNN network includes: convolutional neural network and generative adversarial network, and the generative adversarial network includes generator and discriminator.

[0015] Preferably, step S1 is specifically as follows:

[0016] S11: Extract two sample real images from the sample real image set, use one of the sample real images as the guided rain image, extract a sample synthetic image from the sample synthetic image set as the synthetic rain image, and adjust the parameters of the matching network and the GAN-CNN network;

[0017] S12: extracting features from the synthetic rain image and the guided rain image using a feature conversion space module to obtain a first synthetic feature and a second synthetic feature;

[0018] S13: Inputting the first synthetic feature and the second synthetic feature into the matching network, and calculating the matching loss of the matching network;

[0019] S14: The first synthesized feature passes through the matching network, GAN-CNN network and image conversion space module in sequence to obtain a derained image. The perceptual loss and mean square error loss of the GAN-CNN network are calculated by using the derained image and the sample real image;

[0020] S15: Repeat steps S12-S14 until the matching loss, perception loss, and mean square error loss are all less than the corresponding preset values, and a trained abnormal weather processing network is obtained.

[0021] Preferred:

[0022] Matching loss L match The calculation formula is:

[0023] L match =L f +L c

[0024]

[0025] Among them, F s and F t Represent the first synthetic feature and the second synthetic feature respectively, M s and M t Respectively represent F s and F t The mean vector of represents the Frobenius norm, and d represents the feature dimension.

[0026] Preferred:

[0027] The perceptual loss of the generative adversarial network is calculated as:

[0028] perceptual loss=content loss+λ·adverserial loss

[0029] content loss=||VGG(I truth )-VGG(I clear )||2

[0030] adverse loss=∑[log(D(I truth ))+log(1-G(I clear ))]

[0031] Among them, I truth is the sample real image, I clear is a derained image, VGG represents a convolutional neural network, ||·||2 represents the L2 norm, and λ is the shared weight of the perceptual loss and the mean square error loss;

[0032] The calculation formula for the mean square error loss of the convolutional neural network is:

[0033] I=||VGG[f(I;θ)]-VGG(I truth )||2+λ·||f(I;θ)-I truth ||2

[0034] Among them, f (I; θ) = I clear Represents the derained image.

[0035] Preferred:

[0036] The pavement crack detection network includes: a first Backbone network, a first Neck network and a first Head network;

[0037] The first Backbone network includes a Conv module, a first CBS module, a first C3 module, a second CBS module, a second C3 module, a third CBS module, a third C3 module, a fourth CBS module, a first SPPF module and a fourth C3 module connected in sequence;

[0038] The first Neck network includes a fifth CBS module, a first Upsample module, a fifth C3 module, a sixth CBS module, a second Upsample module, a sixth C3 module, a seventh CBS module, a seventh C3 module, an eighth CBS module, and an eighth C3 module connected in sequence;

[0039] The first Head network includes: a first Conv2d module, a second Conv2d module and a third Conv2d module;

[0040] The fourth C3 module, the fifth CBS module, the eighth C3 module and the first Conv2d module are connected in sequence;

[0041] The third C3 module, the fifth C3 module, the sixth CBS module, the seventh C3 module and the second Conv2d module are connected in sequence;

[0042] The second C3 module, the sixth C3 module and the third Conv2d module are connected in sequence.

[0043] Preferred:

[0044] The safety cone detection network includes: the second Backbone network, the second Neck network and the second Head network;

[0045] The second Backbone network includes a ninth CBS module, a first GhostConv module, a first C3Ghost module, a second GhostConv module, a second C3Ghost module, a third GhostConv module, a third C3Ghost module, a fourth GhostConv module, a fourth C3Ghost module, and a second SPPF module connected in sequence;

[0046] The second Neck network includes a fifth GhostConv module, a third Upsample module, a fifth C3Ghost module, a sixth GhostConv module, a fourth Upsample module, a sixth C3Ghost module, a seventh GhostConv module, a seventh C3Ghost module, an eighth GhostConv module, and an eighth C3Ghost module, which are connected in sequence;

[0047] The second Head network includes: a fourth Conv2d module, a fifth Conv2d module and a sixth Conv2d module;

[0048] The second SPPF module, the fifth GhostConv module, the eighth C3Ghost module and the fourth Conv2d module are connected in sequence;

[0049] The third C3Ghost module, the fifth C3Ghost module, the sixth GhostConv module, the seventh C3Ghost module and the fifth Conv2d module are connected in sequence;

[0050] The second C3Ghost module, the sixth C3Ghost module and the sixth Conv2d module are connected in sequence.

[0051] A storage medium stores instructions and data for implementing the method for detecting road cracks and safety cones in abnormal weather.

[0052] A road crack and safety cone detection device for abnormal weather conditions includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the road crack and safety cone detection method for abnormal weather conditions.

[0053] The present invention has the following beneficial effects:

[0054] 1. This paper jointly constructs an abnormal weather processing network based on the generative adversarial network (GAN) and the convolutional neural network (CNN) to overcome the problem that conventional rain removal algorithms produce excessively dim or saturated results due to the confusion between rain streaks and image details. The abnormal weather processing network is trained using a semi-supervised learning method to improve the image rain removal effect.

[0055] 2. During the training of the abnormal weather processing network, matching loss, perception loss, and mean square error loss are calculated to improve the model's rain removal performance on real rain images and ensure that the characteristic distribution of synthetic rain streaks is consistent with that of real rain streaks.

[0056] 3. Through the adaptive feature reconstruction network, more discriminative pavement crack features and safety cone features are obtained, making the pavement crack defect and safety cone sample data more balanced and increasing the diversity of abnormal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0058] Figure 2 A structural diagram of the network for abnormal weather processing;

[0059] Figure 3 This is the structural diagram of the GAN-CNN network;

[0060] Figure 4 The structural diagram of the network for adaptive feature reconstruction based on the Siamese network;

[0061] Figure 5 This is the structural diagram of the pavement crack detection network;

[0062] Figure 6 This is the structural diagram of the safety cone detection network;

[0063] Figure 7 This is a structural diagram of the device according to an embodiment of the present invention;

[0064] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0066] Reference Figure 1 The present invention provides a method for detecting road cracks and safety cones in abnormal weather conditions, comprising:

[0067] S1: Construct an abnormal weather processing network and train it using a set of sample real images and a set of sample synthetic images to obtain a trained abnormal weather processing network.

[0068] Furthermore, the abnormal weather processing network uses a semi-supervised learning strategy to convert the synthetic rain images and real rain images into feature vectors in the feature transformation space, and accurately models the first-order and second-order statistical information of these feature images to extract domain-invariant features. The semi-supervised matching network is then input to minimize the feature differences between the synthetic and real rain images; Figure 2 This is the structural diagram of the abnormal weather processing network, such as Figure 3 This is the structural diagram of the GAN-CNN network;

[0069] The abnormal weather processing network includes: feature conversion space module, matching network, GAN-CNN network, and image conversion space module;

[0070] The GAN-CNN network includes: convolutional neural network and generative adversarial network, and the generative adversarial network includes generator and discriminator.

[0071] Specifically, the matching network consists of sixteen matching modules, each containing four 3×3 convolution kernels. The synthetic rain image and the real rain image are converted into feature vectors through this network. The feature output is as follows:

[0072] f0=Conv 3×3 (O)

[0073] y0=Conv 3×3 (O r )

[0074] Among them O and O r are the inputs of the synthetic rain image and the real rain image respectively, and f0 and y0 are the corresponding feature outputs.

[0075] These features are processed by 1×1 convolution kernel and cascade operation, and the output of each matching module is as follows:

[0076] F i =M i (Conv 1×1 (Cat[f i-1 , f i-2 ,…,f0]))

[0077] Y i =M i (Conv 1×1 (Cat[y i-1 ,y i-2 ,…,y0]))

[0078] Among them, M i The output of these modules is combined through a densely connected network to enhance the fusion of information between deep and shallow layers. The specific operation is as follows:

[0079] F usion =Conv 1×1 (Cat[F i-1 , F i-2 ,…,F0])

[0080] Y usion =Conv 1×1 (Cat[Y i-1 , Y i-2 ,…,Y0])

[0081] Furthermore, step S1 is specifically as follows:

[0082] S11: Extract two sample real images from the sample real image set, use one of the sample real images as the guided rain image, extract a sample synthetic image from the sample synthetic image set as the synthetic rain image, and adjust the parameters of the matching network and the GAN-CNN network;

[0083] Specifically, the synthetic rain image P and the guided rain image I are input into the abnormal weather processing network. The guided filter first compares and analyzes the input image P and the guided image I to generate a new output image Q. By reducing the gap between the output image Q and the original image P, the processing result is further optimized. The specific expression of the guided filter is as follows:

[0084]

[0085] Among them, μ k , I i , I j , σ k The relative position relationship between ε and ε plays a key role in determining the accuracy of the element layout in the image. The position information of these elements will be encoded into their own different matrices to realize the calculation of element adaptive weights and adjust the pixel values ​​of various regions of the image.

[0086] S12: extracting features from the synthetic rain image and the guided rain image using a feature conversion space module to obtain a first synthetic feature and a second synthetic feature;

[0087] S13: Inputting the first synthetic feature and the second synthetic feature into the matching network, and calculating the matching loss of the matching network;

[0088] S14: The first synthesized feature passes through the matching network, GAN-CNN network and image conversion space module in sequence to obtain a derained image. The perceptual loss and mean square error loss of the GAN-CNN network are calculated by using the derained image and the sample real image;

[0089] Furthermore, the matching loss L match The calculation formula is:

[0090] L match =L f +L c

[0091]

[0092] Among them, F s and F t Represent the first synthetic feature and the second synthetic feature respectively, M s and M t Respectively represent Fs and F t The mean vector of represents the Frobenius norm, and d represents the feature dimension.

[0093] Furthermore, the element I is extracted from the image obtained by semi-supervised matching. base and I detail , where I detail =II base I base and I detail It is used as input to further train the generator and discriminator. The generative adversarial network uses perceptual loss as the loss function. Perceptual loss consists of content loss and adversarial loss to evaluate and optimize the processing performance of the model.

[0094] The perceptual loss of the generative adversarial network is calculated as:

[0095] perceptual loss=content loss+λ·adverserial loss

[0096] content loss=||VGG(I truth )-VGG(I clear )||2

[0097] adverserial loss=Σ[log(D(I truth ))+log(1-G(I clear ))]

[0098] Among them, I truth is the sample real image, I clear is a derained image, VGG represents a convolutional neural network, ||·||2 represents the L2 norm, and λ is the shared weight of the perceptual loss and the mean square error loss;

[0099] Furthermore, a convolutional neural network (CNN) with the same architecture and weights as the generative adversarial network is used for further reconstruction to better preserve the image's color and texture while eliminating the imperfections introduced by perceptual loss. The generative adversarial network (GAN) and the convolutional neural network (CNN) are set to be trained alternately to optimize their respective network performance. To quantify the difference between the generated image and the real image, a mean squared error loss is defined, which consists of two parts: one is the Euclidean distance between the pre-trained features of the VGG network, and the other is the mean squared error (MSE) between the predicted image and the real image.

[0100] The calculation formula for the mean square error loss of the convolutional neural network is:

[0101] I=||VGG[f(I;θ)]-VGG(I truth )||2+λ·||f(I;θ)-I truth ||2

[0102] Among them, f (I; θ) = I clear Represents the derained image.

[0103] Specifically, the mean squared error loss not only evaluates pixel-level errors but also captures high-level semantic differences in images through the VGG network to ensure the visual authenticity and quality of the generated images. This design helps the network effectively optimize using the square of the L2 norm when encountering local minima. This achieves:

[0104]

[0105] S15: Repeat steps S12-S14 until the matching loss, perception loss, and mean square error loss are all less than the corresponding preset values, and a trained abnormal weather processing network is obtained.

[0106] S2: Construct a pavement crack image set and a safety cone image set from a sample real image set, input the pavement crack image set and the safety cone image set into an adaptive feature reconstruction network to obtain a pavement crack feature set and a safety cone feature set;

[0107] Furthermore, step S2 is specifically as follows:

[0108] In step S21, a self-built dataset is constructed by collecting existing data such as Crake500, GAPs384, CFD, AEL, and Craketree200, as well as obtaining real field data.

[0109] Step S22: According to step S21, a higher quality synthetic data set is obtained. The data set is used to train the Siamese network model to learn how to extract useful features of the input image. Then, a part of the trained Siamese network (i.e., the Siamese network feature extraction part) is used as a general adaptive feature reconstruction network, such as Figure 4 As shown in the figure, this adaptive feature reconstruction network can obtain more discriminative features for defect anomalies, improving the network's ability to handle highly similar categories and its accuracy under small sample sizes. The feature extraction portion of the twin network uses the same network, and the loss function is the difference between the predicted similarity and the label value.

[0110] In step S23, according to step S22, the data salient features are obtained. The top-down global attention modulation and the bottom-up local attention modulation are used to exchange multi-scale context to obtain richer semantic information and spatial detail encoding. Given a low-level feature X and a high-level feature Y, with C channels and a feature map of size H×W, the top-down and bottom-up attention modulation can be expressed as:

[0111]

[0112] in δ,BN,σ, They represent the rectified linear unit (ReLU), batch normalization (BN), sigmoid function, and element-wise multiplication, respectively.

[0113] In step S24, the context representation information can be enhanced through step S23. The multi-scale context is exchanged by using top-down global attention modulation and bottom-up local attention modulation to obtain richer semantic information and spatial detail encoding.

[0114] Asymmetric context modulation is implemented for cross-layer feature fusion.

[0115]

[0116] in, For fusion features.

[0117] S3: Construct a pavement crack detection network and a safety cone detection network. Train the pavement crack detection network using the pavement crack feature set to obtain a trained pavement crack detection network. Train the safety cone detection network using the safety cone feature set to obtain a trained safety cone detection network.

[0118] Furthermore, step S3 is specifically as follows:

[0119] In step S31, YOLOv5 is used as the initial backbone network for pavement crack detection, and GhostNet is deployed on YOLOv5 as the initial backbone network for safety cone detection.

[0120] Step S32: Based on step S31, the initial backbone network for road crack detection and safety cone detection is obtained, and the self-attention mechanism is introduced to replace the dot product similarity in the attention calculation with position-aware similarity. Output The calculation is as follows:

[0121]

[0122] Among them, q, k, and v are all calculated projections from the input x, which can capture non-local information from the entire feature map. In order to overcome the disadvantage of this similarity measure having a very large amount of computation, an axial self-attention mechanism is introduced. For a given specific input feature map x, the original self-attention module can be decomposed into two independent self-attention modules by introducing position encoding and updating the width axis self-attention mechanism. The first module calculates self-attention on the height axis of the feature map, while the second module calculates on the width axis. In order to include position information in the calculation, a relative position encoding that can be learned during training is also introduced. The overall calculation can be expressed as:

[0123]

[0124] where r q 、r k 、r v is the position offset.

[0125] In step S33, based on step S32, the first-stage backbone network for pavement crack detection and safety cone detection is obtained. The NMS criterion of the first-stage backbone network is replaced with DIoU. The penalty term for maximizing the overlap area by introducing a minimum bounding box in GIoU is modified to minimize the normalized distance between the center points of the two boxes, thereby accelerating the convergence of the loss. This considers not only the overlap area but also the center distance. The DIoU calculation formula is as follows:

[0126]

[0127] where s i is the classification confidence, ε is the NMS threshold, M is the box with the highest confidence,

[0128] Step S34: Data features are obtained according to step S24. According to step S33, the final backbone network for pavement crack detection and safety cone detection is obtained. Through data features and backbone network training, the required pavement crack prediction model and safety cone prediction model are obtained. During prediction, whether to enable abnormal weather image processing is determined based on actual weather conditions. If enabled, image processing is first performed using the abnormal weather processing model, followed by target detection using the pavement crack detection model and triangular cone detection model, respectively. If disabled, target detection is performed directly using the pavement crack detection model and triangular cone detection model.

[0129] Furthermore, the structure of the pavement crack detection network is as follows Figure 5 As shown;

[0130] The pavement crack detection network includes: a first Backbone network, a first Neck network and a first Head network;

[0131] The first Backbone network includes a Conv module, a first CBS module, a first C3 module, a second CBS module, a second C3 module, a third CBS module, a third C3 module, a fourth CBS module, a first SPPF module and a fourth C3 module connected in sequence;

[0132] The first Neck network includes a fifth CBS module, a first Upsample module, a fifth C3 module, a sixth CBS module, a second Upsample module, a sixth C3 module, a seventh CBS module, a seventh C3 module, an eighth CBS module, and an eighth C3 module connected in sequence;

[0133] The first Head network includes: a first Conv2d module, a second Conv2d module and a third Conv2d module;

[0134] The fourth C3 module, the fifth CBS module, the eighth C3 module and the first Conv2d module are connected in sequence;

[0135] The third C3 module, the fifth C3 module, the sixth CBS module, the seventh C3 module and the second Conv2d module are connected in sequence;

[0136] The second C3 module, the sixth C3 module and the third Conv2d module are connected in sequence.

[0137] Furthermore, the structure of the safety cone detection network is as follows Figure 6 As shown;

[0138] The safety cone detection network includes: the second Backbone network, the second Neck network and the second Head network;

[0139] The second Backbone network includes a ninth CBS module, a first GhostConv module, a first C3Ghost module, a second GhostConv module, a second C3Ghost module, a third GhostConv module, a third C3Ghost module, a fourth GhostConv module, a fourth C3Ghost module, and a second SPPF module connected in sequence;

[0140] The second Neck network includes a fifth GhostConv module, a third Upsample module, a fifth C3Ghost module, a sixth GhostConv module, a fourth Upsample module, a sixth C3Ghost module, a seventh GhostConv module, a seventh C3Ghost module, an eighth GhostConv module, and an eighth C3Ghost module, which are connected in sequence;

[0141] The second Head network includes: a fourth Conv2d module, a fifth Conv2d module and a sixth Conv2d module;

[0142] The second SPPF module, the fifth GhostConv module, the eighth C3Ghost module and the fourth Conv2d module are connected in sequence;

[0143] The third C3Ghost module, the fifth C3Ghost module, the sixth GhostConv module, the seventh C3Ghost module and the fifth Conv2d module are connected in sequence;

[0144] The second C3Ghost module, the sixth C3Ghost module and the sixth Conv2d module are connected in sequence.

[0145] S4: Input the synthetic image to be detected into the trained abnormal weather processing network to obtain the real image to be detected, and input the real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to obtain the pavement crack detection results and the safety cone detection results respectively.

[0146] See Figure 7 , Figure 7 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a road crack and safety cone detection device 401 for abnormal weather conditions, a processor 402 and a storage medium 403.

[0147] A road surface crack and safety cone detection device 401 for abnormal weather conditions: The road surface crack and safety cone detection device 401 for abnormal weather conditions implements the road surface crack and safety cone detection method for abnormal weather conditions.

[0148] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the road crack and safety cone detection method under abnormal weather conditions.

[0149] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the road surface crack and safety cone detection method under abnormal weather conditions.

[0150] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0151] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order and should be construed as identifiers.

[0152] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting road cracks and safety cones in abnormal weather conditions, characterized in that: include: S1: Construct an abnormal weather processing network and train it using a set of sample real images and a set of sample synthetic images to obtain a trained abnormal weather processing network. The abnormal weather processing network includes: a feature conversion space module, a matching network, a GAN-CNN network, and an image conversion space module. Step S1 is specifically as follows: S11: Extract two sample real images from the sample real image set, use one of the sample real images as the guided rain image, extract a sample synthetic image from the sample synthetic image set as the synthetic rain image, and adjust the parameters of the matching network and the GAN-CNN network; S12: extracting features from the synthetic rain image using a feature conversion space module to obtain a first synthetic feature; The feature conversion space module is used to extract features from the guided rain map to obtain the second synthetic feature; S13: Inputting the first synthetic feature and the second synthetic feature into the matching network, and calculating the matching loss of the matching network; S14: Using the first synthetic features, train the abnormal weather processing network and output a rain-free image; calculate the perceptual loss and mean square error loss of the GAN-CNN network using the rain-free image and the sample real image; S15: Repeat steps S12-S14 until the matching loss, perception loss, and mean square error loss are all less than corresponding preset values, thereby obtaining a trained abnormal weather processing network; Matching loss L match The calculation formula is: L match =L f +L c Among them, F s and F t Represent the first synthetic feature and the second synthetic feature respectively, M s and M t Respectively represent F s and F t The mean vector of represents the Frobenius norm, d represents the feature dimension; The perceptual loss of the generative adversarial network is calculated as: perceptualloss=contentloss+λ·adverserialloss contentloss=||VGG(I truth )-VGG(I clear )||2 adverserialloss=∑[log(D(I truth ))+log(1-G(I clear ))] Among them, I truth is the sample real image, I clear is a derained image, VGG represents a convolutional neural network, ||·||2 represents the L2 norm, and λ is the shared weight of the perceptual loss and the mean square error loss; The calculation formula for the mean square error loss of the convolutional neural network is: I=||VGG[f(I;θ)]-VGG(I truth )||2+λ·||f(I;θ)-I truth ||2 Among them, f (I; θ) = I clear represents the derained image; S2: Construct a pavement crack image set and a safety cone image set from a sample real image set, input the pavement crack image set and the safety cone image set into an adaptive feature reconstruction network to obtain a pavement crack feature set and a safety cone feature set; S3: Construct a pavement crack detection network and a safety cone detection network. Train the pavement crack detection network using the pavement crack feature set to obtain a trained pavement crack detection network. Train the safety cone detection network using the safety cone feature set to obtain a trained safety cone detection network. S4: Input the synthetic image to be detected into the trained abnormal weather processing network to obtain the real image to be detected, and input the real image to be detected into the trained pavement crack detection network and the trained safety cone detection network to obtain the pavement crack detection results and the safety cone detection results respectively.

2. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: The GAN-CNN network includes: convolutional neural network and generative adversarial network, and the generative adversarial network includes generator and discriminator.

3. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: The pavement crack detection network includes: a first Backbone network, a first Neck network and a first Head network; The first Backbone network includes a Conv module, a first CBS module, a first C3 module, a second CBS module, a second C3 module, a third CBS module, a third C3 module, a fourth CBS module, a first SPPF module and a fourth C3 module connected in sequence; The first Neck network includes a fifth CBS module, a first Upsample module, a fifth C3 module, a sixth CBS module, a second Upsample module, a sixth C3 module, a seventh CBS module, a seventh C3 module, an eighth CBS module, and an eighth C3 module connected in sequence; The first Head network includes: a first Conv2d module, a second Conv2d module and a third Conv2d module; The fourth C3 module, the fifth CBS module, the eighth C3 module and the first Conv2d module are connected in sequence; The third C3 module, the fifth C3 module, the sixth CBS module, the seventh C3 module and the second Conv2d module are connected in sequence; The second C3 module, the sixth C3 module and the third Conv2d module are connected in sequence.

4. The method for detecting road cracks and safety cones in abnormal weather conditions according to claim 1, characterized in that: The safety cone detection network includes: the second Backbone network, the second Neck network and the second Head network; The second Backbone network includes a ninth CBS module, a first GhostConv module, a first C3Ghost module, a second GhostConv module, a second C3Ghost module, a third GhostConv module, a third C3Ghost module, a fourth GhostConv module, a fourth C3Ghost module, and a second SPPF module connected in sequence; The second Neck network includes a fifth GhostConv module, a third Upsample module, a fifth C3Ghost module, a sixth GhostConv module, a fourth Upsample module, a sixth C3Ghost module, a seventh GhostConv module, a seventh C3Ghost module, an eighth GhostConv module, and an eighth C3Ghost module, which are connected in sequence; The second Head network includes: a fourth Conv2d module, a fifth Conv2d module and a sixth Conv2d module; The second SPPF module, the fifth GhostConv module, the eighth C3Ghost module and the fourth Conv2d module are connected in sequence; The third C3Ghost module, the fifth C3Ghost module, the sixth GhostConv module, the seventh C3Ghost module and the fifth Conv2d module are connected in sequence; The second C3Ghost module, the sixth C3Ghost module and the sixth Conv2d module are connected in sequence.

5. A road crack and safety cone detection device for abnormal weather conditions, characterized by: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the road crack and safety cone detection method for abnormal weather conditions as described in any one of claims 1 to 4.

Images