A road disease detection method, system, device and medium

Abstract

The application provides a road disease detection method, system, device and medium, the method comprises: acquiring a road disease inspection image, and forming a disease data set from the road disease inspection image; inputting the disease data set into a detection model for training; the detection model comprises a convolutional neural network, a feature fusion network and a contrast learning module; the disease data set is sequentially subjected to feature extraction by the convolutional neural network, feature fusion by the feature fusion network, and contrast learning by the contrast learning module; inputting a to-be-detected road disease inspection image into the trained detection model to obtain a road disease detection result. The application optimizes the contrast learning training process and can effectively detect and identify different road disease targets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision target detection, in particular to a road disease detection method, system, device and medium. BACKGROUND

[0002] As a common public facility, the road is damaged to varying degrees under the influence of weather, external force and other factors for a long time, thereby deriving various road disease conditions and causing potential traffic safety hazards. Timely detection of disease problems existing in the road and maintenance are of great help to prevent the further deterioration of the road condition and prevent various traffic safety problems, so that each country attaches great importance to road condition evaluation.

[0003] The traditional road disease detection is completed by manual work, which needs to consume a large amount of manpower and material resources, and is easy to cause the missed detection of diseases due to the fatigue of the detector. With the development of deep learning in the field of computer vision in recent years, we can fully tap the potential of deep learning technology in the field of intelligent transportation application, such as automatic driving, traffic condition analysis, traffic accident detection, etc. In the field of road disease identification application, there are also a lot of related research work. The road disease detection method can be divided into two categories, the image processing-based method and the deep network-based method. The image processing-based method uses the representative imaging features of diseases such as cracks and potholes in the photographed image to detect, such as Gabor filter, oriented gradient (HOG), local binary pattern (LBP), etc. Although these methods have achieved certain results on relatively simple data sets, when facing more complex road conditions and detection objects, the performance of the image processing-based method is often limited. The deep learning-based method uses the powerful modeling and feature extraction ability of the deep network to undertake more complex disease detection tasks, but the research in this aspect still needs to be developed.

[0004] For road disease detection, the detection object is often cracks, marking line pits and holes, etc. Unlike common target detection, since the road disease categories are more irregular in shape, and the target pixel space often only occupies a small part of the pixel area of the annotation box, the construction and classification of road disease features are more difficult, and in addition, the area positioning of road diseases such as cracks is also more difficult. SUMMARY

[0005] The purpose of the present application is to provide a road disease detection method, system, device and medium to effectively detect and identify different road disease targets.

[0006] In order to achieve the above purpose, in a first aspect, the embodiments of the present application provide a road disease detection method, which comprises:

[0007] acquire a road disease inspection image, and constitute a disease data set from the road disease inspection image;

[0008] input the disease data set into a detection model for training; the detection model comprises a convolutional neural network, a feature fusion network, and a contrast learning module; the disease data set is sequentially subjected to feature extraction by the convolutional neural network, feature fusion by the feature fusion network, and contrast learning by the contrast learning module;

[0009] input a road disease inspection image to be detected into the trained detection model to obtain a road disease detection result.

[0010] Further, the disease data set is sequentially subjected to feature extraction by the convolutional neural network, feature fusion by the feature fusion network, and contrast learning by the contrast learning module, comprising:

[0011] the feature fusion network comprises a feature pyramid network and a path aggregation network;

[0012] the disease data set is subjected to feature extraction by the convolutional neural network to obtain a first feature map of multiple scales and multiple levels;

[0013] the first feature map is subjected to feature fusion by the feature pyramid network and the path aggregation network to obtain a second feature map of multiple scales and multiple levels;

[0014] the second feature map is subjected to contrast learning at each scale.

[0015] Further, the second feature map is subjected to contrast learning at each scale, comprising:

[0016] the second feature map is subjected to contrast between target features, and similar features are pulled closer and dissimilar features are pulled farther apart.

[0017] Further, the second feature map is subjected to contrast learning at each scale, comprising:

[0018] a k-instance supervised target contrast loss is used to loss-optimize the contrast learning.

[0019] Further, the k-instance supervised target contrast loss is used to loss-optimize the contrast learning, comprising:

[0020] for a target feature vector in the second feature map, k-instances are randomly selected according to a disease category, a balanced positive and negative sample pair is constructed to calculate a dot product similarity, and a cross-entropy loss is calculated.

[0021] Further, the cross-entropy loss is calculated by using the following loss function formula:

[0022]

[0023]

[0024] in, For K instances of loss, The loss is the target feature loss, where N is the number of candidate targets in the batch. For target features, For batch removal The set of all target features, To and A collection of similar samples for A target feature in the set, where K is the number of targets to be selected.

[0025] Furthermore, the step of acquiring road defect inspection images and constructing a defect dataset from the road defect inspection images includes:

[0026] The disease dataset was labeled with bounding boxes and categories to identify road diseases, and data augmentation was performed by horizontal flipping, random cropping, scaling, rotation, color space conversion, and YOLOv5's Mosaic method.

[0027] Secondly, embodiments of the present invention provide a road defect detection system, the system comprising:

[0028] The dataset construction module is used to acquire road defect inspection images and construct a defect dataset from the road defect inspection images;

[0029] The model training module is used to input the disease dataset into the detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning;

[0030] The field detection module is used to input the inspection images of road defects to be detected into the trained detection model to obtain the road defect detection results.

[0031] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0032] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0033] This invention provides a method, system, device, and medium for road defect detection. The method includes: acquiring road defect inspection images and constructing a defect dataset from the images; inputting the defect dataset into a detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the defect dataset undergoes feature extraction via the convolutional neural network, feature fusion via the feature fusion network, and contrastive learning via the contrastive learning module; and the road defect inspection images to be detected are input into the trained detection model to obtain road defect detection results. This application optimizes the contrastive learning training process, enabling effective detection and identification of different road defect targets. Attached Figure Description

[0034] Figure 1 This is a schematic flowchart of a road defect detection method according to an embodiment of the present invention;

[0035] Figure 2 This is a comparison chart of model training indicators for a road defect detection method provided in an embodiment of the present invention;

[0036] Figure 3 This is a diagram illustrating the effect of a road defect detection method provided in an embodiment of the present invention;

[0037] Figure 4 This is a system block diagram of a road defect detection system provided in an embodiment of the present invention;

[0038] Figure 5 This is an internal structural diagram of the computer device in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and beneficial effects of this application clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described below are only part of the embodiments of the present invention and are used to illustrate the present invention, but are not intended to limit the scope of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] In one embodiment, such as Figure 1 As shown, the present invention provides a method for detecting road defects, the method comprising:

[0041] S11. Obtain road defect inspection images and construct a defect dataset from the road defect inspection images;

[0042] To ensure the obtained dataset fully represents its features, this embodiment performs bounding box annotation and category labeling on the road disease dataset to label road diseases. Data augmentation is then performed using horizontal flipping, random cropping, scaling, rotation, color space conversion, and YOLOv5's Mosaic method. This embodiment increases the number of training samples through data annotation and data augmentation, effectively improving the robustness of the detection model.

[0043] In one specific embodiment, the GRDDC2020 dataset was used for training and demonstration. The GRDDC dataset contains nine disease categories ('d00', 'd01', 'd0w0', 'd10', 'd11', 'd20', 'd40', 'd43', 'd44', 'd50'), collected from three different countries. In this embodiment, four categories with a relatively large number of images were selected for training: longitudinal cracks, transverse cracks, alligator cracks, and potholes (d00, d10, d20, d40). The training data contained 21,041 images. One-ninth of the images were used as the test set, and the remaining portion was divided into validation and training sets in a 1:9 ratio. Of course, this application can also use other road datasets, such as the NASA road dataset or the GRIP road dataset, to improve the training effect of the detection model, depending on actual needs.

[0044] S12. Input the disease dataset into the detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning.

[0045] To obtain multi-scale, multi-level feature maps, this embodiment uses a convolutional neural network to extract features from the disease dataset, resulting in a first feature map with multiple scales and levels. A feature pyramid network and a path aggregation network are then used to fuse the features of the first feature map, resulting in a second feature map with multiple scales and levels. Contrastive learning is then performed on the second feature map at each scale. The feature fusion network includes a feature pyramid network and a path aggregation network. Specifically, this embodiment utilizes the YOLOv5 network, a one-stage target detection method, for feature extraction. The target detection head constructs different prediction heads for features at different scales, predicting the target's location and classification information. Similar to the classification and regression branches, the contrastive learning branch accepts the model's neck features as input and learns from candidate region features in the feature map. Considering the differences in road disease size, this invention introduces branches for feature maps at multiple scales. This method allows the model to better adapt to disease targets of different sizes, and the changing background also makes the contrastive learning more robust. This embodiment, through feature extraction and feature fusion of the disease dataset, enables the model to better adapt to disease targets of different sizes.

[0046] Furthermore, to alter the distribution of target features, this embodiment compares the second feature map with target features, bringing similar features closer together and dissimilar features further apart. The intuitive explanation of the model optimization objective is as follows:

[0047]

[0048] in, Represents model parameters, This represents a measure of similarity between feature vectors. , Indicate target features, , Indicates features similar to the target. , This represents features that are different from the target. This embodiment achieves better distribution of disease features in the latent space by zooming in and out on the features.

[0049] To better balance the learning of different disease categories, this embodiment employs k-instance supervised target contrastive loss to optimize the contrastive learning. The loss function is calculated on feature maps at three scales to supervise multi-scale targets and multi-level feature information. The original supervised contrastive loss, SupCon loss, constructs positive sample pairs using all similar data. When the number of categories in the data is uneven, the category with more instances plays a decisive role in representation learning, which may lead to suboptimal results. This embodiment can generate a more uniform and discriminative feature distribution through k-instance supervised target contrastive loss.

[0050] To optimize the comparison between target features and reflect their relative response values ​​within a batch, this embodiment performs k-instance random selection on the target feature vector in the second feature map according to the disease category, constructs balanced positive and negative sample pairs, calculates the dot product similarity, and calculates the cross-entropy loss. This embodiment normalizes features within a batch, replacing cosine similarity with dot product similarity, which generates better feature representations within the batch and optimizes the contrastive learning process.

[0051] Preferably, this embodiment uses the following loss function formula for calculation:

[0052]

[0053]

[0054] in, For K instances of loss, The loss is the target feature loss, where N is the number of candidate targets in the batch. For target features, For batch removal The set of all target features, To and A collection of similar samples for A target feature in the set, where K is the number of targets to be selected.

[0055] Preferably, the overall loss of this application consists of classification loss, regression loss, confidence loss, and contrastive loss, wherein the classification loss and confidence loss both use BCE loss, and the regression loss uses CIoU loss. The contrastive loss first calculates the dot product similarity of the feature vectors, and then calculates the cross-entropy based on the similarity. The overall loss function is shown in the following formula, where... Set it to 0.5 to balance the scale of the damage.

[0056]

[0057] in, For the overall loss function, For classifying losses, To regress the loss, For confidence loss To compare the losses, This is the balance coefficient.

[0058] S13. Input the inspection image of the road defects to be detected into the trained detection model to obtain the road defect detection results.

[0059] In this embodiment, the CL-RDD (Contrastive Learning Road Defect Detection) method proposed in this application is compared with several existing road defect detection methods, such as... Figure 2 As shown, the mAP score of this application is 58.3 and the F1 score is 69.4. Compared with the existing models, this application has achieved better results in both mAP and F1 score. Figure 3 The demonstration shows the detection results of road defects. From left to right, the images show the ground truth labels, the original YOLOv5 model, and the detection results of this application. It can be seen that the method of this application effectively improves the accuracy of road defect detection. This embodiment can deploy the model to a terminal system, such as an in-vehicle computer device. During on-site road defect detection, image data is transmitted to the terminal system via the in-vehicle camera, and the model outputs the detection results, thereby better assisting in the road defect inspection task.

[0060] Compared with existing technologies, this invention introduces the contrastive learning method into the field of road defect detection by designing a multi-level target-level contrastive learning module and combining it with the YOLOv5 framework. It optimizes the learned features through contrastive training on road defect classes, thereby improving the road defect detection effect. Furthermore, this invention designs a k-instance supervised contrastive loss function for supervised contrastive learning scenarios. It optimizes the contrastive learning training process through batch normalization, dot product similarity, and the k-instance supervised contrastive function, improving aspects such as overcoming data imbalance and optimizing features.

[0061] Based on the aforementioned road defect detection method, this invention also provides a road defect detection system, such as... Figure 4 As shown, the system includes:

[0062] Dataset construction module 1 is used to acquire road defect inspection images and construct a defect dataset from the road defect inspection images;

[0063] Model training module 2 is used to input the disease dataset into the detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning;

[0064] The field detection module 3 is used to input the inspection images of road defects to be detected into the trained detection model to obtain the road defect detection results.

[0065] For specific limitations regarding a road defect detection system, please refer to the limitations of a road defect detection method described above, which will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0066] Figure 5 This diagram illustrates the internal structure of a computer device in one embodiment, which may specifically be a terminal or a server. The computer device includes a processor, memory, a network interface, a display, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input devices may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0067] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computing devices may include more or fewer components than shown in the diagram, or combine certain components, or have the same component arrangement.

[0068] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0069] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0070] This invention provides a method, system, device, and medium for road defect detection. It involves acquiring road defect inspection images and constructing a defect dataset from these images; inputting the defect dataset into a detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the defect dataset undergoes feature extraction via the convolutional neural network, feature fusion via the feature fusion network, and contrastive learning via the contrastive learning module; and the road defect inspection images to be detected are input into the trained detection model to obtain the road defect detection results. This application optimizes the contrastive learning training process, enabling effective detection and identification of different road defect targets.

[0071] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0072] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A method for detecting road defects, characterized in that, The method includes: Acquire road defect inspection images and construct a defect dataset from the road defect inspection images; The disease dataset is input into the detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning; The disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning, including: The feature fusion network includes a feature pyramid network and a path aggregation network; features are extracted from the disease dataset using a convolutional neural network to obtain a multi-scale, multi-level first feature map; features are fused from the first feature map using the feature pyramid network and the path aggregation network to obtain a multi-scale, multi-level second feature map; comparative learning is performed on the second feature map at each scale; The road defect inspection images to be detected are input into the trained detection model to obtain the road defect detection results.

2. The method for detecting road defects according to claim 1, characterized in that, The comparative learning of the second feature map at each scale includes: The second feature map is compared with the target features, bringing similar features closer together and dissimilar features further apart.

3. The method for detecting road defects according to claim 1, characterized in that, The comparative learning of the second feature map at each scale includes: The contrastive learning is optimized using k-instance supervised target contrastive loss.

4. The method for detecting road defects according to claim 3, characterized in that, The step of optimizing the contrastive learning using k-instance supervised target contrastive loss includes: For the target feature vector in the second feature map, k-instance random screening is performed according to the disease category, balanced positive and negative sample pairs are constructed to calculate the dot product similarity, and cross-entropy loss is calculated.

5. The method for detecting road defects according to claim 4, characterized in that, The cross-entropy loss is calculated using the following loss function formula: in, For K instances of loss, The loss is the target feature loss, where N is the number of candidate targets in the batch. For target features, For batch removal The set of all target features, To and A collection of similar samples for A target feature in the set, where K is the number of targets to be selected.

6. The method for detecting road defects according to claim 1, characterized in that, The step of acquiring road defect inspection images and constructing a defect dataset from these images includes: The disease dataset was labeled with bounding boxes and categories to identify road diseases, and data augmentation was performed by horizontal flipping, random cropping, scaling, rotation, color space conversion, and YOLOv5's Mosaic method.

7. A road defect detection system, characterized in that, The system includes: The dataset construction module is used to acquire road defect inspection images and construct a defect dataset from the road defect inspection images; The model training module is used to input the disease dataset into the detection model for training; the detection model includes a convolutional neural network, a feature fusion network, and a contrastive learning module; the disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning; The disease dataset is sequentially processed by the convolutional neural network for feature extraction, by the feature fusion network for feature fusion, and by the contrastive learning module for contrastive learning, including: The feature fusion network includes a feature pyramid network and a path aggregation network; features are extracted from the disease dataset using a convolutional neural network to obtain a multi-scale, multi-level first feature map; features are fused from the first feature map using the feature pyramid network and the path aggregation network to obtain a multi-scale, multi-level second feature map; comparative learning is performed on the second feature map at each scale; The field detection module is used to input the inspection images of road defects to be detected into the trained detection model to obtain the road defect detection results.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images