Pavement distress detection method and apparatus, and electronic device and storage medium

Abstract

Provided in the embodiments of the present application are a pavement distress detection method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring image information of a pavement; inputting the image information into a pre-established object detection model, and determining the type and a prediction box of a distress of the pavement; on the basis of the prediction box, determining dimension parameters of the distress; on the basis of the type and the dimension parameters, determining the degree of the distress; and on the basis of the type, the dimension parameters and the degree, outputting a detection report of the pavement. Thus, a distress of a pavement can be accurately detected, and the detection efficiency can be improved.

Description

Road disease detection method, device, electronic equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on April 2, 2024, with application number 202410391691.6 and invention name “Road Defect Detection Method, Device, Electronic Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the technical field of road damage detection, and in particular relates to a method, device, electronic equipment and storage medium for detecting road damage. Background Art

[0003] With the continuous advancement of urbanization and the growing transportation workload, roads, as an important part of infrastructure, bear enormous traffic pressure. However, during road use, due to the influence of various factors such as the natural environment, traffic load, and material aging, various road structure defects such as cracks, potholes, and subsidence will gradually appear. These defects not only affect road safety and comfort, but also reduce the service life of the road and increase maintenance costs. Therefore, road defect diagnosis is particularly important. With the continuous advancement and innovation of science and technology, road defect diagnosis technology is also constantly developing. In related technologies, manual inspections and field surveys are used to detect roads, which are inefficient and lack accuracy. Technical issues

[0004] In response to the above problems, the embodiments of the present application provide a method, device, electronic device and storage medium for detecting road defects, which can detect road defects more accurately and improve detection efficiency. Technical Solutions

[0005] The present invention provides a method for detecting road hazards, including:

[0006] Acquire image information of the road;

[0007] Inputting the image information into a pre-established target detection model to determine the type of disease in the road and the prediction box;

[0008] determining a size parameter of the disease based on the prediction frame;

[0009] determining the extent of the disease based on the type and the size parameters;

[0010] A detection report of the road is output based on the type, the size parameter, and the extent.

[0011] In some embodiments, the method further comprises:

[0012] Acquire a sample data set, where each sample data in the sample data set includes: a sample image and a type and a prediction box corresponding to the sample image;

[0013] An initial target detection model is trained based on the sample data set to obtain the target detection model.

[0014] In some embodiments, the target detection model includes: an improved YOLOv8s model, the improved YOLOv8s model includes: a backbone network, a neck network and a detection head, the backbone network includes: depthwise separable convolution and / or an improved C2F module, the improved C2F module is added with an attention mechanism, the neck network is used to connect the backbone network and the detection head, and the detection head adopts a decoupling head design.

[0015] In some embodiments, the neck network includes: a convolution module, a first feature enhancement module and a second feature enhancement module, the input of the convolution module is the output of the improved C2F module, the output of the convolution module is the input of the first feature enhancement module, the output of the first feature enhancement module is the input of the second feature enhancement model, and the output of the second feature enhancement model is the input of the detection head.

[0016] In some embodiments, the training of the initial target detection model based on the sample dataset to obtain the target detection model includes:

[0017] Divide the sample data set into training set and test set;

[0018] Training the initial target detection model based on the training set, and testing the trained initial target detection model using the test set to obtain an evaluation index;

[0019] When the evaluation index meets the optimal requirement, the trained initial target detection model is determined as the target detection model.

[0020] In some embodiments, the image information is captured by a collection device, and determining the size parameter of the disease based on the prediction frame includes:

[0021] Obtaining an intrinsic parameter matrix and an extrinsic parameter matrix of the acquisition device;

[0022] Determining the coordinates of the corner points of the prediction box in the world coordinate system based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the coordinates of the corner points of the prediction box in the pixel coordinate system;

[0023] The size parameters of the disease are calculated based on the coordinates of the corner points in the world coordinate system.

[0024] In some embodiments, determining the extent of the disease based on the type and size parameters comprises:

[0025] Acquiring a pre-established correspondence relationship, wherein the correspondence relationship includes: a correspondence relationship between degree and size parameter and type;

[0026] The type and the size parameter are matched with the size parameter and type in the corresponding relationship to determine the extent of the disease.

[0027] The present invention provides a road hazard detection device, comprising:

[0028] An acquisition module, used to acquire image information of the road;

[0029] A first detection module is used to input the image information into a pre-established target detection model to determine the type of disease in the road and a prediction frame;

[0030] A second detection module is used to determine the size parameters of the disease based on the prediction frame;

[0031] a third detection module for determining the extent of the disease based on the type and the size parameters;

[0032] An output module is used to output a detection report of the road based on the type, the size parameter and the degree.

[0033] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the computer program.

[0034] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0035] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the electronic device executes any one of the above methods.

[0036] The embodiments of the present application provide a road defect detection method, device, electronic device, and storage medium. These methods acquire image information of a road; input the image information into a pre-established target detection model to determine the type and prediction frame of the defects on the road; determine the size parameters of the defects based on the prediction frame; determine the extent of the defects based on the type and size parameters; and output a detection report for the road based on the type, size parameters, and extent. These methods can detect road defects more accurately and improve detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Hereinafter, the present application will be described in more detail based on embodiments with reference to the accompanying drawings.

[0038] FIG1 is a schematic diagram of a road hazard detection method according to the present invention;

[0039] FIG2 is a schematic diagram of an improved YOLOv8s model provided in an embodiment of the present application;

[0040] FIG3 is a schematic diagram of an improved C2F module provided in an embodiment of the present application;

[0041] FIG4 is a schematic diagram of a first feature enhancement module provided in an embodiment of the present application;

[0042] FIG5 is a schematic diagram of a second feature enhancement module provided in an embodiment of the present application;

[0043] FIG6 is a schematic diagram of a world coordinate system provided in an embodiment of the present application;

[0044] FIG7 is a schematic diagram of a camera coordinate system provided in an embodiment of the present application;

[0045] FIG8 is a schematic diagram of a perspective projection transformation provided by an embodiment of the present application;

[0046] FIG9 is a schematic diagram of a transformation from an image coordinate system to a pixel coordinate system provided in an embodiment of the present application;

[0047] FIG10 is a schematic diagram of the structure of a road hazard detection device provided in an embodiment of the present application;

[0048] FIG11 is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present application.

[0049] In the drawings, the same components use the same selection of welding guns, and the drawings are not drawn according to the actual scale. Modes for Carrying Out the Invention

[0050] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] If similar descriptions of "first\second\third" appear in the application documents, the following explanation will be added. In the following description, the terms "first\second\third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] To address the problems in the related art, the present invention provides a method for detecting road damage. The method can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The present invention does not impose any restrictions on the specific type of electronic device.

[0055] The functions implemented by the road hazard detection method provided in the embodiment of the present application can be implemented by calling program codes by a processor of an electronic device, wherein the program codes can be stored in a computer storage medium.

[0056] The present application provides a method for detecting road damage. FIG1 is a schematic diagram of a process flow for implementing the method for detecting road damage provided by the present application. As shown in FIG1 , the method for detecting road damage includes:

[0057] Step S101: Acquire image information of the road.

[0058] In the embodiments of the present application, photography can be performed using a camera, drone, or other image acquisition device. Furthermore, image information of the road can be acquired using technologies such as satellite imagery or ground-based radar. The acquired images can include various road surface defects, such as cracks, potholes, and damage. These images can be used to analyze and assess road conditions, enabling timely maintenance and repair work.

[0059] In the embodiment of the present application, when using a camera, the camera can be installed on a vehicle, and image information of the road can be collected by the vehicle moving. The image information can include pictures or videos.

[0060] In some embodiments, the electronic device may also obtain image information of the road through the Internet.

[0061] Step S102: input the image information into a pre-established target detection model to determine the type of disease in the road and the prediction box.

[0062] In an embodiment of the present application, image information can be input into a pre-established target detection model, and computer vision technology and deep learning models can be used to determine the type and prediction frame. After processing by the model, information on the type and prediction frame of each disease can be obtained.

[0063] In an embodiment of the present application, the type may include: one or more of: sink, longitudinal crack (crack_p), longitudinal crack patch (patch_crack_p), transverse crack (crack_t), transverse crack patch (patch_crack_t), net crack (net), and net crack patch (patch_net).

[0064] Step S103: determining the size parameters of the disease based on the prediction frame.

[0065] In the embodiment of the present application, after the prediction frame is determined, the pixel size is converted to the actual physical size. The size parameters of the disease may include: length, width, area, etc.

[0066] Step S104: determining the extent of the disease based on the type and the size parameters.

[0067] In the embodiments of the present application, determining the severity of a defect based on type and size parameters typically relies on professional road maintenance standards and specifications. Different types of defects may have different impacts on road safety and usability depending on their size parameters. For example, for a crack, its width and length may determine its impact on the road structure; for a pothole, its depth and diameter may determine its impact on vehicle drivability. Generally, the severity of the defect can be determined by comparing the size parameters of the defect with corresponding thresholds in professional road maintenance standards. For example, the standards may specify the degree of impact of cracks or potholes within different size ranges on road use, such as mild, moderate, or severe. By comparing the size parameters of the defect with these standards, the severity of the defect can be determined. Alternatively, the severity of the defect can be determined using expert systems or data-driven methods, combined with professional knowledge on type and severity. This allows for a more comprehensive consideration of the type and size parameters of the defect, as well as its impact on road safety and usability, leading to a more accurate determination of the severity of the defect.

[0068] Step S105: outputting a detection report of the road based on the type, the size parameter, and the degree.

[0069] In an embodiment of the present application, outputting a road inspection report based on type, size parameters, and extent can be achieved by integrating this information into a single report. This report may include the following:

[0070] Type: Describe each detected defect, such as cracks, potholes, road wear, etc.

[0071] Size parameters: Describe the size parameters of each disease, including length, width, depth, etc.

[0072] Severity: Assess the severity of each disease based on professional standards and specifications, such as mild, moderate, severe, etc.

[0073] Photos: Photos of the disease may be included in the report to provide a more visual representation of the disease detected.

[0074] Recommended maintenance measures: Based on the type, size parameters and degree, corresponding maintenance measures such as repair, re-paving, etc. are proposed.

[0075] In the embodiment of the present application, this information can be integrated into a complete inspection report so that road maintenance personnel can clearly understand the problems on the road and take appropriate maintenance measures. Such a report can help improve the efficiency and quality of road maintenance and ensure the safety and reliability of the road.

[0076] A road defect detection method provided in an embodiment of the present application obtains image information of a road; inputs the image information into a pre-established target detection model to determine the type and prediction frame of the defects in the road; determines the size parameters of the defects based on the prediction frame; determines the extent of the defects based on the type and the size parameters; and outputs a detection report for the road based on the type, the size parameters, and the extent. This method can detect road defects more accurately and improve detection efficiency.

[0077] In some embodiments, before step S102, the following steps may be included:

[0078] Step S1021 : obtaining a sample data set, where each sample data in the sample data set includes: a sample image and a type and a prediction box corresponding to the sample image.

[0079] In this embodiment of the present application, types can include: sink, longitudinal crack (crack_p), longitudinal crack patch (patch_crack_p), transverse crack (crack_t), transverse crack patch (patch_crack_t), net crack (net), and net crack patch (patch_net). In the sample image, the same type can correspond to different images, and these images can have different road surface textures, colors, and lighting conditions.

[0080] In an embodiment of the present application, a camera or other image acquisition device can be used to capture images of road defects in the real world. These images may include road cracks, potholes, road wear, and other types. Ensure that the image quality is good and can clearly display the defects. Annotate the captured images to mark the location and type of each defect. Specialized image annotation tools, such as LabelImg, CVAT, etc., can be used to manually annotate the location and type of each defect. The prediction box is usually a rectangular bounding box of the area where the defect is located. The annotated images and the corresponding annotation data are then organized into a data set. Ensure that each sample data includes an image and its corresponding type and prediction box. Finally, the organized data set can be stored in an electronic device, and can be stored and managed using a hard disk, cloud storage, etc. Once a data set containing sample images, types, and prediction boxes is obtained, these data sets can be used to train the target detection model. These data can be used to improve the performance of the model to more accurately detect road defects.

[0081] Step S1022: Training the initial target detection model based on the sample data set to obtain the target detection model.

[0082] In an embodiment of the present application, the sample data set can be preprocessed. The preprocessing may include operations such as scaling, cropping, and rotating the image, as well as formatting and processing the labeled data to make it suitable for the training requirements of the target detection model. An initial target detection model can then be selected, and the selected initial target detection model can be trained using the sample data set. Model training can be implemented using tools and APIs provided by deep learning frameworks (such as TensorFlow, PyTorch, etc.). During the training process, the model will learn how to accurately detect the type and location in the image through multiple iterations. During the training process, it may be necessary to tune the model's hyperparameters to achieve better performance. This may include adjusting hyperparameters such as the learning rate, batch size, and number of iterations. After training each epoch, the trained target detection model needs to be evaluated. The model that performs best on the validation set evaluation indicators is then used as the final target detection model, which can be deployed in practical applications to detect road damage, generate corresponding reports, or perform other related applications.

[0083] In some embodiments, step S1022 may be implemented by the following steps:

[0084] Step S1: Divide the sample data set into a training set and a validation set.

[0085] In the embodiment of the present application, the training set accounts for the majority of the total data set (eg, 70-80%), while the validation set accounts for a smaller portion (20-30%).

[0086] Step S2: training the initial target detection model based on the training set, and testing the trained initial target detection model using a validation set to obtain evaluation indicators.

[0087] In the embodiment of the present application, the initial target detection model is trained using the training set, and the model training and tuning are performed according to the steps mentioned above.

[0088] In the embodiment of the present application, after each epoch of training is completed, the trained target detection model is evaluated using the validation set to obtain an evaluation index.

[0089] In the embodiment of the present application, the evaluation indicators may include: one or more of precision, recall, average precision, and mean average precision. The evaluation indicators are calculated as follows:

[0090] Precision (P), also known as the precision rate, refers to the ratio of correctly predicted positives to all positive predictions, as shown in the formula:

[0091] Recall (R), also known as the recall rate, refers to the ratio of correctly predicted positive results to all actual positive results, as shown in the formula:

[0092] Average precision (AP) is calculated from precision and recall. A line graph of precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:

[0093] The mean average precision (mAP) refers to the average of the average precision AP of C different categories, as shown in the formula:

[0094] Step S3: When the evaluation index meets the optimal requirement, the trained initial target detection model is determined as the target detection model.

[0095] In the embodiments of the present application, if the performance of the model does not meet your requirements, you may need to adjust the structure, hyperparameters, etc. of the model and then retrain and evaluate the model.

[0096] In some embodiments, the target detection model includes: an improved YOLOv8s model, the improved YOLOv8s model can be called D-YOLOv8s, Figure 2 is a schematic diagram of an improved YOLOv8s model provided in an embodiment of the present application. As shown in Figure 2, the improved YOLOv8s model includes: a backbone network (backbone), a neck network (Neck) and a detection head (Head), the backbone network includes: depthwise separable convolution and / or an improved C2F module, the improved C2F module is added with an attention mechanism, the neck network is used to connect the backbone network and the detection head, and the detection head adopts a decoupling head design.

[0097] In the embodiment of the present application, the depthwise separable convolution (DWConv) reduces the amount of computation by using a different convolution kernel for each input channel. Compared with traditional convolution operations, it has the advantages of fewer parameters and less computation, which can shorten the calculation time.

[0098] In an embodiment of the present application, Figure 3 is a schematic diagram of an improved C2F module provided in an embodiment of the present application. As shown in Figure 3, the improved C2F module can be called C2fCBAM. The improved C2F module replaces BottleNeck in the C2f module with CBAMBottleneck, which can be more conducive to network learning important features.

[0099] In the embodiment of the present application, the CBAM (Convolutional Block Attention Module) attention mechanism is an attention mechanism module that combines channel and spatial. Starting from the two scopes of channel and spatial, it introduces two analysis dimensions of spatial attention and channel attention to realize the sequential attention structure from channel to space. Spatial attention can make the neural network pay more attention to the pixel areas in the image that play a decisive role in classification and ignore insignificant areas. Channel attention is used to process the distribution relationship of feature map channels. At the same time, allocating attention to the two dimensions enhances the effect of the attention mechanism on improving model performance. There are no large number of convolution structures inside the CBAM module, only a small number of pooling layers and feature fusion operations, which avoids the large amount of calculations brought by convolution multiplication, making its module complexity low and the amount of calculation small. Adding a CBAM module to a lightweight model can bring stable performance improvement. Compared with the small amount of increase in computation brought by it, the introduction of CBAM has a high cost performance.

[0100] In some embodiments, referring to Figure 2, the neck network includes: a convolution module, a first feature enhancement module, and a second feature enhancement module. The input of the convolution module is the output of the improved C2F module, the output of the convolution module is the input of the first feature enhancement module, the output of the first feature enhancement module is the input of the second feature enhancement model, and the output of the second feature enhancement model is the input of the detection head.

[0101] Figure 4 is a schematic diagram of a first feature enhancement module provided in an embodiment of the present application. As shown in Figure 4, the first feature enhancement module can be the ASFF2 module in Figure 2, primarily for adaptively fusing features at two scale levels. Figure 5 is a schematic diagram of a second feature enhancement module provided in an embodiment of the present application. As shown in Figure 5, the second feature enhancement module can be the ASFF3 module in Figure 2, primarily for adaptively fusing features at three scale levels. In this embodiment of the present application, the ASFF (Asymmetric Single-Stage Feature Fusion) module is a feature enhancement module designed to address the problem of inconsistent prediction targets within the same feature layer during multi-scale prediction in the FPN (Feature Pyramid Network). Compared to traditional feature pyramid networks, ASFF has the following advantages: Automatically learning feature fusion strategy: ASFF adaptively selects appropriate features for fusion based on targets of different scales through automatic learning. This adaptive feature fusion strategy can better handle targets of different scales and shapes, improving target detection performance. Reduced computational effort: ASFF reduces the computational effort and parameter count by eliminating unnecessary feature calculations and redundant information, thereby improving model efficiency and performance. This helps reduce the model's reliance on hardware resources, making it more efficient and reliable in practical applications.

[0102] In the embodiment of the present application, Table 1 is a schematic diagram of an improved YOLOv8s model improvement strategy provided in the embodiment of the present application, as shown in Figure 1.

[0103] Table 1 is a schematic diagram of an improved YOLOv8s model improvement strategy provided by an embodiment of the present application

[0104] Table 2 is a comparison table of a YOLOv8s model with different improvement strategies provided in an embodiment of the present application and YOLOv8s, as shown in Table 2:

[0105] By analyzing the table, we can find that:

[0106] Compared with the original YOLOv8s network, after using depthwise convolution (DW), although the number of model parameters and computational complexity are reduced, its precision P decreases by 0.1%, recall R decreases by 0.2%, and mAP0.5 decreases by 0.3%.

[0107] Compared with Improvement 1, Improvement 2 introduces the CBAM spatial attention mechanism. Although the number of parameters and computational complexity increases slightly, its precision P decreases by 0.2%, the recall R increases by 0.7%, and the mAP0.5 increases by 0.8%.

[0108] Compared with Improvement 2, Improvement 3 reduces the number of parameters and computational complexity by introducing the ASFF2 and ASFF3 weighted modules, while its precision P decreases by 0.6%, the recall R increases by 1.8%, and the mAP0.5 increases by 0.8%.

[0109] It can be seen that the improved D-YOLOv8s network can significantly improve the recognition and detection accuracy of road defects.

[0110] Table 3 is a performance comparison table of an improved YOLOv8s model and YOLOv8s provided in an embodiment of the present application. As shown in Table 3, Table 3 is a performance comparison table of an improved D-YOLOv8s and YOLOv8s provided in an embodiment of the present application.

[0111] By analyzing the table, we can find that:

[0112] (1) Compared with the original YOLOv8s network, the improved D-YOLOv8s network has a 0.9% decrease in precision P, a 2.3% increase in recall R, and a 1.3% increase in mAP. It can be seen that the improved D-YOLOv8s network can significantly improve the recognition and detection accuracy of road damage.

[0113] (2) For the original YOLOv8s network, the recognition accuracy of the four categories of horizontal cracks (crack_t), horizontal crack patches (patch_crack_t), net cracks (net), and net crack patches (patch_net) is not high, and they are difficult categories. The improved D-YOLOv8s network improves the mAP of these three categories by 0.3%, 2.4%, 4.3%, and 3.8%, respectively. It can be seen that the horizontal crack patches, net cracks, and net crack patches have both been greatly improved.

[0114] (3) Compared with the original YOLOv8s network, the improved D-YOLOv8s network can improve the detection accuracy of road defects with complex shapes, and at the same time, the problem of inconsistent scales is better solved.

[0115] In this embodiment of the present application, you can install the Triton Inference Server on the deployed server. Configure the Triton Inference Server to load the object detection model. Start the Triton Inference Server and have it listen on a specified port, waiting for inference requests. Deploy a client application developed using the Gunicorn framework that can send road damage images to the Triton Inference Server and receive and interpret inference results. Use the monitoring and management API to track model performance, resource usage, and service health.

[0116] When using the application, users can call an HTTP request, pass an actual collected image to the web service, then decode the image, perform pre-processing on it, and then pass the data to the Trition Inference Server in a certain format. It will predict the processed image and output the result. The web service receives the prediction result, parses it, and encapsulates it into a unified format, and finally returns it to the road damage detection system.

[0117] In some embodiments, in step S103, the image information is captured by a collection device, and the size parameter of the disease is determined based on the prediction frame, which can be achieved by the following steps:

[0118] Step S1031: Acquire the intrinsic parameter matrix and extrinsic parameter matrix of the acquisition device.

[0119] In this embodiment, intrinsic calibration can be achieved using the Zhang Zhengyou calibration algorithm. By acquiring checkerboard images at different angles, the camera intrinsic parameters are calibrated to obtain an intrinsic parameter matrix. The intrinsic parameter matrix may include: the intrinsic parameter matrix K: fx, fy, cx, cy, the intrinsic parameter matrix, and the distortion parameters: k1, k2, p1, p2, k3.

[0120] In an embodiment of the present application, external parameter calibration can restore the position of the camera in the world coordinate system by taking a picture facing the camera. First, the world coordinate system of the camera is defined to be located at the vertical point of the camera on the ground. The world coordinates of the checkerboard corners are obtained by the corner detection algorithm and the measured checkerboard distance. Then, the corners of the image are extracted to construct multiple point pairs associated with 3D world coordinates and 2D image coordinates. According to the camera projection model, it is constructed into a PNP solution problem. Solving PNP can obtain the rotation and translation vectors of the camera. Finally, the transformation matrix T and rotation matrix R of the camera coordinate system relative to the world coordinate system can be solved through Rodrigues transformation.

[0121] Step S1032 : determining the coordinates of the corner points of the prediction box in the world coordinate system based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the coordinates of the corner points of the prediction box in the pixel coordinate system.

[0122] In an embodiment of the present application, a transformation relationship from a world coordinate system to a pixel coordinate system may be pre-established, wherein the transformation relationship may be simplified as follows: world coordinate system--->camera coordinate system--->image coordinate system--->pixel coordinate system.

[0123] FIG6 is a schematic diagram of a world coordinate system provided in an embodiment of the present application, and FIG7 is a schematic diagram of a camera coordinate system provided in an embodiment of the present application. As shown in FIG6 and FIG7, Ow is the world coordinate system, Oc is the camera coordinate system, and the coordinates in the world coordinate system are expressed as [X w ,Y w ,Z w ,1], the coordinates in the camera coordinate system are expressed as [X c ,Y c ,Z c ,1], R represents the 3x3 rotation matrix of the camera coordinate system relative to the world coordinate system, and T is the 3x1 translation matrix. This transformation is a transformation between 3D space and 3D space. Both coordinate systems can be regarded as right-handed coordinate systems, with the following transformation relationship:

[0124] Next, the camera coordinate system is transformed into the image coordinate system by the perspective projection transformation. During this projection process, the 3D point loses its depth and realizes the transformation from 3D space to 2D space. Figure 8 is a schematic diagram of a perspective projection transformation provided by an embodiment of the present application. As shown in Figure 8, during this projection process, the 3D point loses its depth and realizes the transformation from 3D space to 2D space. According to the similarity property of triangles, the following projection formula can be obtained:

[0125] ΔABO c :ΔoCO c

[0126] ΔPBO c :ΔpCO c ;

[0127] Finally, there is the transformation from the image coordinate system to the pixel coordinate system. Figure 9 is a schematic diagram of a transformation from an image coordinate system to a pixel coordinate system provided by an embodiment of the present application. As shown in Figure 9, the origin of the image coordinate system O(u0, v0) is located at the projection center, and the origin of the pixel coordinate system is in the upper left corner. Assuming that the physical dimensions of each pixel in the u-axis and v-axis directions are dx and dy, the following transformation relationship can be obtained:

[0128] By sorting out the three transformation matrices of the above four coordinate systems, we can get the correspondence between the 3D world coordinate system points and the 2D pixel points.

[0129] Among them: the internal parameter matrix is:

[0130] The external parameter matrix is:

[0131] Thus, the transformation relationship from the world coordinate system to the pixel coordinate system can be obtained.

[0132] Step S1033: Calculate the size parameters of the defect based on the coordinates of the corner points in the world coordinate system.

[0133] In the embodiment of the present application, the outlines of the four corner points during the projection process are polygons of uncertain shapes. Therefore, the area of ​​the polygon can be solved by an iterative method, and the formula is as follows:

[0134] In some embodiments, determining the extent of the disease based on the type and size parameters comprises:

[0135] A pre-established correspondence is obtained, wherein the correspondence includes: a correspondence between the degree and the size parameter and the type; and the type and the size parameter are matched with the size parameter and the type in the correspondence to determine the degree of the disease.

[0136] In the embodiment of the present application, road damage can be classified into three levels: minor, moderate, and severe, based on highway maintenance specifications and taking into account factors such as width, length, and area. Minor damage has little impact on the performance of the road surface and generally does not require repair. Moderate and severe damage require timely repair to ensure road safety and smooth traffic.

[0137] In this embodiment, the length of a road defect is defined as the length of the diagonal of the rectangular frame, and the area is defined as the area of ​​the rectangular frame. Based on this, a road defect with a length greater than 5 meters or an area greater than 5 square meters is defined as severe; a road defect with a length between 1 and 5 meters or an area between 1 and 5 square meters is defined as moderate; and a road defect with a length less than 1 meter or an area less than 1 square meter is defined as mild.

[0138] The road damage detection method provided in the embodiments of the present application has the following significance:

[0139] Ensure traffic safety: Road defects such as cracks and potholes can pose a threat to road safety if not diagnosed and repaired promptly. Regular road defect diagnosis can identify these problems in advance and allow for timely repairs, ensuring traffic safety.

[0140] Extending road life: Road damage often indicates structural degradation and damage, which can affect the lifespan of roads. Timely diagnosis and repair of road damage can slow down structural degradation and extend road lifespan.

[0141] Improve road traffic efficiency: If a road is damaged, it may cause traffic to slow down or even cause congestion. Timely diagnosis and repair of the damage can restore the road to a good condition and improve road traffic efficiency.

[0142] Reduced repair costs: If road damage is not diagnosed and repaired promptly, it can worsen, requiring more repair work and expense. Timely diagnosis and repair of damage can reduce repair costs.

[0143] Promoting road maintenance and management: Road disease diagnosis is a key part of road maintenance and management. Regular disease diagnosis can help us understand the road's usage and existing problems, providing a basis for road maintenance and management, and promoting sustainable road development.

[0144] Based on the foregoing embodiments, an embodiment of the present application provides a road hazard detection device. The modules included in the device, and the units included in each module, can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; during implementation, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0145] An embodiment of the present application provides a road hazard detection device. FIG10 is a schematic structural diagram of a road hazard detection device provided in an embodiment of the present application. As shown in FIG10 , the road hazard detection device 1000 includes:

[0146] An acquisition module 1001 is used to acquire image information of a road;

[0147] The first detection module 1002 is used to input the image information into a pre-established target detection model to determine the type of disease in the road and the prediction frame;

[0148] A second detection module 1003 is configured to determine a size parameter of the disease based on the prediction frame;

[0149] A third detection module 1004 is configured to determine the extent of the disease based on the type and the size parameter;

[0150] The output module 1005 is configured to output a detection report of the road based on the type, the size parameter, and the degree.

[0151] In some embodiments, the road hazard detection device further comprises:

[0152] A sample acquisition module is used to acquire a sample data set, where each sample data in the sample data set includes: a sample image and a type and a prediction box corresponding to the sample image;

[0153] The training module is used to train the initial target detection model based on the sample data set to obtain the target detection model.

[0154] In some embodiments, the target detection model includes: an improved YOLOv8s model, the improved YOLOv8s model includes: a backbone network, a neck network and a detection head, the backbone network includes: depthwise separable convolution and / or an improved C2F module, the improved C2F module is added with an attention mechanism, the neck network is used to connect the backbone network and the detection head, and the detection head adopts a decoupling head design.

[0155] In some embodiments, the neck network includes: a convolution module, a first feature enhancement module and a second feature enhancement module, the input of the convolution module is the output of the improved C2F module, the output of the convolution module is the input of the first feature enhancement module, the output of the first feature enhancement module is the input of the second feature enhancement model, and the output of the second feature enhancement model is the input of the detection head.

[0156] In some embodiments, the training module includes:

[0157] A partitioning unit, used to divide the sample data set into a training set and a test set;

[0158] An evaluation unit, configured to train the initial target detection model based on the training set, and test the trained initial target detection model using a test set to obtain an evaluation index;

[0159] A determination unit is used to determine the trained initial target detection model as the target detection model when the evaluation index meets the optimal requirement.

[0160] In some embodiments, the second detection module includes:

[0161] An acquisition unit, configured to acquire an intrinsic parameter matrix and an extrinsic parameter matrix of the acquisition device;

[0162] a first calculation unit, configured to determine the coordinates of the corner points of the prediction box in the world coordinate system based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the coordinates of the corner points of the prediction box in the pixel coordinate system;

[0163] The second calculation unit is used to calculate the size parameters of the disease based on the coordinates of the corner point in the world coordinate system.

[0164] In some embodiments, determining the extent of the disease based on the type and size parameters comprises:

[0165] Acquiring a pre-established correspondence relationship, wherein the correspondence relationship includes: a correspondence relationship between degree and size parameter and type;

[0166] The type and the size parameter are matched with the size parameter and type in the corresponding relationship to determine the extent of the disease.

[0167] An embodiment of the present application provides an electronic device. FIG11 is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. As shown in FIG11 , the electronic device 600 includes: a processor 601, at least one communication bus 602, a user interface 603, at least one external communication interface 604, and a memory 605. The communication bus 602 is configured to enable communication between these components. The user interface 603 may include a display screen, and the external communication interface 604 may include a standard wired interface and a wireless interface. The processor 601 is configured to execute a program for the road detection method stored in the memory to implement the steps of the road detection method provided in the above embodiment.

[0168] In the embodiment of the present application, if the above-mentioned road detection method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0169] Accordingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the road detection method provided in the above embodiment are implemented.

[0170] An embodiment of the present application further provides a computer program product, which, when executed on a terminal device, enables the electronic device to execute any one of the above-mentioned road detection methods.

[0171] The description of the above electronic device and storage medium embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the computer device and storage medium embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0172] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0173] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0176] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0177] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROMs), magnetic disks, optical disks, and other media that can store program codes.

[0178] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a road detector to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program code, such as mobile storage devices, ROM, magnetic disks or optical disks.

[0179] The above are merely embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for detecting road damage, characterized in that: include: Acquire image information of the road; Inputting the image information into a pre-established target detection model to determine the type of disease in the road and the prediction box; determining a size parameter of the disease based on the prediction frame; determining the extent of the disease based on the type and the size parameters; A detection report of the road is output based on the type, the size parameter, and the extent.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a sample data set, where each sample data in the sample data set includes: a sample image and a type and a prediction box corresponding to the sample image; An initial target detection model is trained based on the sample data set to obtain the target detection model.

3. The method according to claim 2, characterized in that The target detection model includes: an improved YOLOv8s model, the improved YOLOv8s model includes: a backbone network, a neck network and a detection head, the backbone network includes: depthwise separable convolution and / or an improved C2F module, the improved C2F module is added with an attention mechanism, the neck network is used to connect the backbone network and the detection head, and the detection head adopts a decoupling head design.

4. The method according to claim 3, characterized in that The neck network includes: a convolution module, a first feature enhancement module and a second feature enhancement module. The input of the convolution module is the output of the improved C2F module, the output of the convolution module is the input of the first feature enhancement module, the output of the first feature enhancement module is the input of the second feature enhancement model, and the output of the second feature enhancement model is the input of the detection head.

5. The method according to claim 2, characterized in that The training of the initial target detection model based on the sample data set to obtain the target detection model includes: Divide the sample data set into training set and validation set; Training the initial target detection model based on the training set, and evaluating the trained initial target detection model using the validation set to obtain an evaluation index; When the evaluation index meets the optimal requirement, the trained initial target detection model is determined as the target detection model.

6. The method according to claim 1, characterized in that The image information is captured by a collection device, and the size parameter of the disease is determined based on the prediction frame, including: Obtaining an intrinsic parameter matrix and an extrinsic parameter matrix of the acquisition device; Determining the coordinates of the corner points of the prediction box in the world coordinate system based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the coordinates of the corner points of the prediction box in the pixel coordinate system; The size parameters of the disease are calculated based on the coordinates of the corner points in the world coordinate system.

7. The method according to claim 1, characterized in that Determining the extent of the disease based on the type and size parameters comprises: Acquiring a pre-established correspondence relationship, wherein the correspondence relationship includes: a correspondence relationship between degree and size parameter and type; The type and the size parameter are matched with the size parameter and type in the corresponding relationship to determine the extent of the disease.

8. A road disease detection device, characterized in that: include: An acquisition module, used to acquire image information of the road; A first detection module is used to input the image information into a pre-established target detection model to determine the type of disease in the road and a prediction frame; A second detection module is used to determine the size parameters of the disease based on the prediction frame; a third detection module for determining the extent of the disease based on the type and the size parameters; An output module is used to output a detection report of the road based on the type, the size parameter and the degree.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images