Vision-based highway pavement disease detection method and device, medium and product

Abstract

The invention discloses a highway pavement disease detection method and device based on vision, a medium and a product, and relates to the technical field of intelligent traffic, and the method comprises the steps: obtaining real pavement image data, inputting the real pavement image data to a pavement disease detection model, and obtaining a detection frame result and a corresponding disease category result; the determination process of the pavement disease detection model comprises the steps that a pavement image sample data set is constructed, and each sample comprises sample pavement image data and corresponding label data; constructing a target detection network which comprises a backbone network, a neck network and a head network; inputting each sample into a target detection network to obtain a predicted pavement disease detection result; and comparing the predicted pavement disease detection result with the label data, respectively calculating detection frame loss and disease category loss, and updating parameters of the target detection network according to the loss to obtain a pavement disease detection model. According to the method, the accuracy and the efficiency of pavement disease recognition are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a vision-based highway pavement disease detection method, device, medium and product. Background Art

[0002] In recent years, the focus of highway maintenance has gradually shifted from traditional passive maintenance to active, scientific preventive maintenance. As the most important or even the only mode of transportation and important infrastructure, highways are the leading conditions for safeguarding and improving rural people's livelihood, and are also the basic conditions for promoting economic development and inheriting and carrying forward regional ethnic characteristics. Therefore, how to carry out scientific and effective highway maintenance management to extend the service life of highways and ensure the normal use of highways has become the top priority of future highway maintenance management. Inspection, as the first priority of highway maintenance management, is an important means to promote refined highway maintenance management, a breakthrough for achieving sustainable and high-quality development of low-grade highways, and the most urgent shortcomings and hardware to be filled in highway maintenance management in the new era.

[0003] At present, highway inspection work mainly adopts the traditional method of manual inspection + traditional inspection vehicles, which has the following problems: (1) It consumes a lot of manpower and poses safety hazards: Inspection staff need to frequently get on and off vehicles to inspect and measure on the road surface. The inspection vehicles stop and go, which not only affects traffic, but also poses great safety hazards and consumes a lot of manpower and time costs; (2) Traditional methods have lagged behind today's advanced science and technology: The automation, informationization and intelligence levels of traditional inspection methods are low, and data analysis is affected by human factors, which is not conducive to objective and accurate evaluation of road damage, and the data reliability and traceability are insufficient. Therefore, it is imperative to develop a highway inspection method that can solve the above problems. Summary of the invention

[0004] The purpose of this application is to provide a vision-based highway pavement defect detection method, device, medium and product, which can significantly improve the accuracy and efficiency of pavement defect identification.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for detecting road pavement defects based on vision, comprising: acquiring real road surface image data;

[0007] Inputting the pavement image data into a pavement disease detection model to obtain a pavement disease detection result; the pavement disease detection result includes a detection frame result and a corresponding disease category result;

[0008] The process of determining the pavement disease detection model specifically includes:

[0009] Constructing a road surface image sample data set; each sample of the road surface image sample data set includes sample road surface image data and corresponding label data; the label data is a detection frame label and a corresponding disease category label;

[0010] Constructing a target detection network; the target detection network includes a backbone network, a neck network and a head network;

[0011] Input each sample of the pavement image sample data set into the target detection network to obtain a predicted pavement disease detection result; the predicted pavement disease detection result includes a predicted detection frame result and a corresponding predicted disease category result;

[0012] The predicted pavement defect detection result is compared with the label data, and the detection frame loss and the defect category loss are calculated respectively. According to the detection frame loss and the defect category loss, the parameters of the target detection network are updated to obtain an updated target detection network as the pavement defect detection model.

[0013] In a second aspect, the present application provides a vision-based highway pavement disease detection device, comprising: an image acquisition module, for acquiring real pavement image data;

[0014] The positioning module is used to locate the latitude and longitude information of the image acquisition module and trigger the image acquisition module to collect the current road surface image data according to the change of position;

[0015] An inertial navigation module, used to collect attitude information of the vehicle on which the image acquisition module is installed;

[0016] Edge computing modules for:

[0017] Execute the above-mentioned vision-based highway pavement disease detection method;

[0018] Get the latitude and longitude information of the image acquisition module;

[0019] Obtain the attitude information collected by the inertial navigation module and calculate the flatness of the road surface;

[0020] The latitude and longitude information of the image acquisition module, the road surface image data, the road surface disease detection results and the road surface flatness are associated to generate an inspection report.

[0021] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned vision-based highway pavement defect detection method.

[0022] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned vision-based highway pavement defect detection method.

[0023] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0024] The present application provides a vision-based highway pavement disease detection method, device, medium and product, which performs disease detection by using a pavement disease detection model trained based on a large number of pavement image sample data sets, so that the model has higher accuracy and reliability in identifying different types of pavement diseases. Compared with manual inspections, automated inspections reduce the impact of human factors on the results and improve the objectivity of the data; they can also reduce the working time of inspectors on the road and reduce the safety hazards caused by frequent boarding and alighting of vehicles. At the same time, through automated visual inspection methods, a large amount of pavement image data can be quickly processed, significantly reducing detection time and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is an application environment diagram of a vision-based highway pavement disease detection method in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a flow chart of a method for detecting road pavement defects based on vision provided in one embodiment of the present application;

[0028] Figure 3 A schematic diagram of a geometric scaling operation provided by an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a target detection network structure provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of functional modules of a visual-based highway pavement disease detection device provided in one embodiment of the present application;

[0031] Figure 6 for Figure 5 Schematic diagram of the actual installation effect of the highway pavement disease detection device. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] At present, a related technology adopts a deep learning method. On the basis of a multi-target detection algorithm, a detection algorithm specifically targeting road diseases has been developed, and quantitative identification of parameters such as the length, width, and area of ​​the disease has been performed. By describing the detailed information of the disease, intelligent evaluation and targeted maintenance of pavement diseases are realized. It is mainly described as using the model of the YOLO network as the detection framework. By learning a large amount of pavement disease image data, the model can learn more robust feature expressions, thereby increasing the generalization ability of the model and improving the accuracy of recognition. At the same time, the precise contour information of the disease is extracted, and through target recognition and image segmentation, multi-neural networks are used to improve the anti-interference ability of fine scratches, and the contour information of the identified pavement disease is output, thereby realizing quantitative identification of the disease. This related technology mainly proposes a detection method based on deep learning for traditional pavement disease detection technology, and realizes quantitative measurement of the disease by using multi-neural networks for disease recognition and segmentation.

[0034] Another related technology discloses a mobile multifunctional highway intelligent inspection system, including a mobile system, an inspection system, a communication module, a data processing module, a central control system and a power supply system; the inspection system mainly includes an inspection control mainboard, a positioning module, a camera, a speed measurement module, a temperature sensor and an ice measurement sensor; wherein the inspection control mainboard transmits the measured information data to the data processing system through the communication module, and the data processing system transmits the image, road surface temperature, road surface icing analysis results and vehicle speeding information obtained after processing the information data to the central control system. This related technology can automatically detect road spills, road surface diseases, road climate conditions, road surface temperature, and road surface icing conditions in real time all day and night; at the same time, it can also measure the speed of passing vehicles, monitor traffic conditions and monitor speeding violations, solving the problem of time-consuming and labor-intensive highway inspections in the past and low efficiency. Its application point is that it integrates multiple subsystems such as a mobile system, an inspection system, a communication module, a data processing system, a central control system, and a power supply system, with a high degree of integration, and can realize the all-weather inspection function of highways.

[0035] Although the above two technical solutions can realize the recognition of road surface defects, the defects are: (1) the multi-neural network structure will significantly affect the reasoning speed of image recognition, and will also have a certain impact on the recognition accuracy; (2) the system integration is high, and there are limitations in adding or reducing modules according to usage. Therefore, in order to solve the defects of the above two technical solutions, this application proposes a vision-based road surface defect detection method, device, medium and product.

[0036] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0037] The visual-based road pavement disease detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the acquired road surface image data to the server 104. After the server 104 receives the road surface image data, the server 104 inputs the road surface image data into the road surface disease detection model to obtain the road surface disease detection result. The road surface disease detection result includes the detection frame result and the corresponding disease category result. The server 104 can feedback the obtained road surface disease detection result to the terminal 102. In addition, in some embodiments, the vision-based highway pavement disease detection method can also be implemented separately by the server 104 or the terminal 102, such as the terminal 102 can directly perform detection and recognition processing on the road surface image data, or the server 104 can obtain the road surface image data from the data storage system and perform detection and recognition processing on the road surface image data.

[0038] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, and IoT devices. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0039] In an exemplary embodiment, Figure 2 As shown, a method for detecting road pavement defects based on vision is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 208. Among them:

[0040] Step 201, obtaining real road surface image data.

[0041] Step 202, input the pavement image data into the pavement disease detection model to obtain the pavement disease detection result; the pavement disease detection result includes the detection frame result and the corresponding disease category result.

[0042] Furthermore, the pavement image data is input into the pavement damage detection model to obtain the pavement damage detection results, which specifically include:

[0043] Step 2021, preprocessing the road surface image data to obtain target road surface image data; the preprocessing operation includes geometric scaling operation and translation operation; the geometric scaling operation is implemented by a matrix, such as Figure 3 As shown, the scaling factor (Scale) is the minimum value of the width ratio and height ratio between the target road image and the source road image (ie, the real road image); the translation operation is to translate the center of the source road image to the coordinate origin of the upper left corner.

[0044] Specifically, the preprocessing process can be completed through three matrices, namely, scaling matrix S, translation matrix O, and translation matrix T. These three matrices can be combined into a matrix M (M=TOS), where the M matrix is ​​called an affine transformation matrix, as shown in the following formula, where Origin represents the source road image, Dst represents the target road image, width represents the width, height represents the height, (x, y) is the coordinate of any pixel point on the source road image, and (x', y') is the coordinate of any pixel point on the target road image.

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] Through the above transformation, we can get the coordinates of any pixel value of the source road image corresponding to the position of the target road image, so the whole preprocessing process can be completed by the affine transformation matrix M. In addition to transforming the source road image to the target road image through M, we also need to consider mapping the target road image back to the source road image, that is, to find the inverse matrix M of the M matrix. -1 This step is mainly because the reasoning is performed on the target road image, that is, the detection frame is drawn on the target road image, and what is ultimately needed is the detection frame of the source road image. The conversion from the detection frame of the target road image to the detection frame of the source road image is done through the inverse matrix M-1 to be completed.

[0051] Inverse matrix M -1 As shown in the following formula, by using

[0052]

[0053] Then there is

[0054]

[0055] Step 2022, input the target road surface image data into the road surface disease detection model, obtain the disease category result and the initial detection frame result, and perform decoding operation and non-maximum suppression operation on the initial detection frame result to obtain the detection frame result.

[0056] Among them, the decoding process is through the affine transformation inverse matrix M -1 The detection frame on the target road image is mapped back to the source road image. Then the attributes of the detection frame need to be transformed from the coordinates represented by the width and height of the center point to the coordinates represented by the upper left corner and the lower right corner, and the decoding is completed. The process of non-maximum suppression is because many of the detection frames inferred by the road disease detection model overlap. By assuming that there are N frames, the probability value of each frame calculated by the classifier is S i , 1≤i≤N, specifically:

[0057] ① Set a set H for storing candidate boxes to be processed, and initialize it to include all N boxes; set a set M for storing optimal boxes, and initialize it to an empty set;

[0058] ② Sort all the boxes in set H, select the box m with the highest score, and move it from set H to set M;

[0059] ③ Traverse the boxes in set H and calculate the intersection and union ratio with m respectively. If it is higher than the set threshold, the box is considered to overlap with m and is removed from H;

[0060] ④ Return to ① and iterate until the set H is empty and the box in the set M is the final result.

[0061] Furthermore, the process of determining the pavement disease detection model specifically includes:

[0062] Step 301, construct a pavement image sample dataset, which includes a training dataset and a test dataset. The ratio of the training dataset to the test dataset is appropriately adjusted according to the predicted pavement disease detection results; each sample of the pavement image sample dataset includes sample pavement image data and corresponding label data; the label data is a detection box label and a corresponding disease category label.

[0063] Step 302, construct a target detection network, which includes a backbone network, a neck network and a head network. The target detection network structure is as follows: Figure 4 shown.

[0064] Step 303, input each sample of the road surface image sample data set into the target detection network to obtain the predicted road surface disease detection result; the predicted road surface disease detection result includes the predicted detection box result and the corresponding predicted disease category result, and the predicted detection box result includes the predicted detection box value and the corresponding probability value.

[0065] Furthermore, each sample of the pavement image sample data set is input into the target detection network to obtain the predicted pavement disease detection result, which specifically includes:

[0066] Step 3031: Input the sample road surface image data into the backbone network for feature extraction to obtain a first feature map.

[0067] The network uses the Darknet-53 network as the backbone. A convolution module (CBS) receives the sample road surface image data input, which is processed by four serially stacked combination modules. Each combination module is a convolution module plus a residual module (C2f, Convolution to Feature Map). Finally, a feature fusion module (SPPF) further fuses and enhances the extracted features. These features can be low-level features (such as edges, textures, etc.) or high-level features (such as object detection, segmentation, etc.), and output multi-scale feature representations. The reason for this design is that the network structure can be designed to be very deep, avoiding the problem of gradient disappearance during training, making the model easier to converge. In each of the two residual structures, a convolution layer Conv2D with a step size of 2 and a convolution kernel size of 3x3 is used instead of the pooling layer Maxpooling2D to complete the downsampling operation. Among them, the convolution module consists of a two-dimensional convolution layer, a two-dimensional batch normalization layer, and a SiLU activation function; the residual module consists of a convolution layer, a residual connection layer, an upsampling layer, and an activation function, which mainly involves gradually extracting features through a series of convolution layers, and converting the output feature map of the convolution module into a feature map containing higher-level abstract features. These feature maps reflect the different levels of image content. Usually, the convolution layer uses a 3x3 or larger filter for convolution operations to detect objects of different sizes and gradually reduce the size of the feature map; the residual connection layer is used to alleviate the problem of gradient disappearance; the upsampling layer is used in some cases to maintain a higher resolution and restore the lost spatial information.

[0068] Step 3032, input the first feature map into the neck network for feature enhancement to obtain a second feature map. In order to better utilize the features extracted by the backbone network, the neck network plays the role of feature fusion. The neck network includes a feature pyramid network (FPN) and a path aggregation network (PAN). The output result of the feature pyramid network is used to input the path aggregation network; the feature pyramid network is a top-down network structure, which transmits the strong semantic features of the high-level down to enhance the entire pyramid structure. The top is small in size and is obtained by multiple convolutions. The result is rich in semantic information, but only the semantic information is enhanced, and the positioning information is not transmitted. In order to convey the positioning features of the lower layer upward, the neck network is designed as a "double tower" structure, that is, a path aggregation network is added after the feature pyramid network to supplement the feature pyramid network. The path aggregation network is a bottom-up network structure, with a small number of convolutions at the bottom and a large size, and an increase in the number of convolutions upward, a smaller size, and richer semantic information. For the first feature map extracted by the backbone network, the neck network first uses the feature pyramid network to fuse the low-level features of the previous level through upsampling to obtain the predicted feature map output by the feature pyramid network, and the path aggregation network input is the predicted feature map output by the feature pyramid network. The feature pyramid network passes down the strong semantic features of the high-level, and the bottom-up path aggregation network supplements the fused high-level features and passes up the strong positioning features of the low-level. This "double tower" structural design can aggregate parameters of different detection layers from different backbone layers, and finally the entire neck network outputs the feature prediction results after enhancing different features.

[0069] Step 3033, the second feature map is input into the head network for target classification and detection, and the predicted road disease detection result is obtained. In the design of the head network, the input is divided into two paths to enter two convolution modules, and then passes through a Conv2d module respectively, and finally the predicted disease category result and the predicted detection box result are calculated respectively, that is, the current mainstream decoupled head structure (Decoupled-Head) is adopted to separate the classification head and the detection head.

[0070] In step 3034, the predicted pavement defect detection result is compared with the label data, and the detection frame loss and the defect category loss are calculated respectively. The parameters of the target detection network are updated according to the detection frame loss and the defect category loss until the loss function converges to obtain an updated target detection network, ensuring that the accuracy of the updated target detection network is ≥ 90% to serve as a pavement defect detection model.

[0071] In the design of the disease category loss VFL Loss, considering that in the road disease detection task, there will be fewer positive samples and more negative samples, and it is necessary to reduce the impact of negative samples on the Loss, so the loss function of VFL Loss is adopted, which has the feature that when the gt_IOU of the positive sample is very high, it will contribute more to the Loss, allowing the network to focus on higher quality samples, which is more beneficial to improving the accuracy of the model in the training stage. The calculation process of the disease category loss is:

[0072]

[0073] Among them, p is the probability value corresponding to the predicted disease category result, q is the intersection and union ratio of the probability value corresponding to the predicted disease category result and the disease category label, and γ and α are hyperparameters. When q>0, VFL has no hyperparameter for positive samples, that is, there is no attenuation; when q=0, VFL has hyperparameters for negative samples. The meaning of the γ parameter is to reduce the contribution of negative samples, and α is to prevent over-suppression. Generally speaking, it will reduce the contribution of negative samples.

[0074] The detection box loss uses the Distribution Focal Loss. The calculation process of the detection box loss is:

[0075] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 ) (9);

[0076] Among them, y is the detection box label value, y i+1 ,y i are the two predicted detection box values ​​adjacent to the detection box label value, S i+1 is with y i+1 The corresponding predicted detection box probability value, S i is with y i The corresponding predicted detection box probability value, By adjusting the representation of detection boxes and estimating the quality of these detection boxes, and using a generalized loss function to train the model, the average precision (AP) of the model can be further improved.

[0077] Implementing the above-mentioned vision-based highway pavement disease detection method, the present application performs disease detection through a pavement disease detection model obtained by training based on a large number of pavement image sample data sets, so that the model has higher accuracy and reliability in identifying different types of pavement diseases. Compared with manual inspections, automated detection reduces the impact of human factors on the results and improves the objectivity of the data; it can also reduce the working time of inspectors on the road and reduce the safety hazards caused by frequent boarding and alighting of vehicles. At the same time, through automated visual detection methods, a large amount of pavement image data can be processed quickly, significantly reducing detection time and improving work efficiency. Moreover, the deep learning algorithm in this application only uses a single neural network to achieve end-to-end detection and segmentation functions.

[0078] In addition, this application adopts an Anchor-free optimization method and abandons the traditional Anchor-based method. This is because it is necessary to improve the inference speed of the target detection network. The traditional Anchor-based method pre-sets boxes of different sizes and aspect ratios. These boxes will match positive and negative samples, thereby narrowing the search space and more accurately and simply returning gradients to train the network. However, matching will bring higher overhead, and the hyperparameters required for matching also require manual intervention and adjustment. Anchor-free does not need to set a priori anchor boxes (ie, detection boxes), and directly predicts specific points through the network to obtain anchor boxes. The following two rules are used to more concisely determine positive and negative samples:

[0079] ① Normally aligned anchors should be able to predict high classification scores and have accurate positioning;

[0080] ② Misaligned anchors should have low classification scores and be suppressed in the non-maximum suppression stage.

[0081] Based on the above two rules, in order to quantify the accuracy of the anchor boxes predicted by the network, an anchor alignment indicator is used to measure the level of anchor box matching, and this indicator is integrated into the sample allocation and loss function to dynamically optimize the prediction of each anchor. Since the classification score and the intersection-over-union ratio represent the prediction effects of the two tasks respectively, the matching uses a high-order combination of the classification score and the intersection-over-union ratio to measure the degree of matching, and is calculated using the following formula:

[0082] t=s α ×u β (10);

[0083] Among them, s and u are the classification score and intersection-over-union ratio respectively, and α and β are hyperparameters. From the formula, it can be seen that the optimization of the classification score and the intersection-over-union ratio is controlled at the same time to achieve matching and obtain the best matching anchor box.

[0084] The present application also provides an application scenario, which applies the above-mentioned vision-based highway pavement disease detection method. Specifically: the vision-based highway pavement disease detection method provided in this embodiment can be applied in a highway pavement disease detection scenario. Highway pavement disease detection includes a content pavement disease detection link and a pavement disease display link; the pavement disease detection link is used to obtain pavement disease detection results based on pavement image data; the pavement disease display link is used to display pavement disease detection results and positioning information, etc. The vision-based highway pavement disease detection method provided in this embodiment belongs to the pavement disease detection link in the highway pavement disease detection scenario.

[0085] Since most traditional inspection vehicles are heavy vehicles, they are expensive, inflexible in operation, and not suitable for the actual road conditions of low-grade roads, such as narrow rural roads with changeable road shapes, traditional large inspection vehicles are not suitable for daily inspections of low-grade roads, and it is difficult to carry out inspections. Based on the same inventive concept, the embodiment of the present application also provides a road surface disease detection device for implementing the above-mentioned vision-based road surface disease detection method.

[0086] In an exemplary embodiment, Figure 5 As shown, a visual-based highway pavement disease detection device is provided, comprising:

[0087] The image acquisition module is used to collect real road surface image data.

[0088] The positioning module is used to locate the longitude and latitude information of the image acquisition module and trigger the image acquisition module to collect the current road surface image data according to the change of position.

[0089] The inertial navigation module is used to collect the posture information of the vehicle on which the image acquisition module is installed.

[0090] Edge computing modules for:

[0091] Execute the vision-based highway pavement defect detection method described above; obtain the longitude and latitude information of the image acquisition module, obtain the attitude information collected by the inertial navigation module, and calculate the flatness of the road surface; associate the longitude and latitude information of the image acquisition module, the road surface image data, the road surface defect detection results and the road surface flatness to generate an inspection report.

[0092] Power module: used to supply power to the entire device.

[0093] The device designed in this application is based on the edge computing module as the core. Through the rich hardware interfaces of the edge computing module, it connects the necessary sensors to collect data and can easily expand other functions. At the same time, Darknet-53 and other similar architectures are used as the backbone network, thereby optimizing the ability of neural networks to extract and represent features. By integrating the detection and segmentation models into a unified framework, a set of architectures can be used to provide detection and segmentation functions at the same time, improving the overall reasoning speed and meeting the identification and analysis of more disease elements and highway facility elements.

[0094] The visual-based road surface disease detection device described in this application was tested in the field under good weather conditions and relatively high visibility. The actual loading effect diagram is shown in the figure. Figure 6 The actual test was conducted on national and provincial trunk roads and rural roads, with a cumulative mileage of more than 300 kilometers and more than 1,000 disease images collected. The identified disease types include six categories: horizontal and vertical cracks, potholes, bulges, subsidence, and looseness. The recognition accuracy exceeded 80%, and the reasoning time for a single photo was 10 to 20 ms.

[0095] In summary, the vision-based highway pavement disease detection device described in this application is superior to existing solutions in terms of installation method, equipment scalability and flexibility, while maintaining relatively high accuracy and real-time performance, and is effective in improving the efficiency of daily highway inspections.

[0096] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0097] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0098] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0099] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0100] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for detecting road pavement defects based on vision, characterized in that: The vision-based highway pavement disease detection method comprises: Obtain real road surface image data; Inputting the pavement image data into a pavement disease detection model to obtain a pavement disease detection result; the pavement disease detection result includes a detection frame result and a corresponding disease category result; The process of determining the pavement disease detection model specifically includes: Constructing a road surface image sample data set; each sample of the road surface image sample data set includes sample road surface image data and corresponding label data; the label data is a detection frame label and a corresponding disease category label; Constructing a target detection network; the target detection network includes a backbone network, a neck network and a head network; Input each sample of the pavement image sample data set into the target detection network to obtain a predicted pavement disease detection result; the predicted pavement disease detection result includes a predicted detection frame result and a corresponding predicted disease category result; The predicted pavement defect detection result is compared with the label data, and the detection frame loss and the defect category loss are calculated respectively. According to the detection frame loss and the defect category loss, the parameters of the target detection network are updated to obtain an updated target detection network as the pavement defect detection model.

2. The visual-based highway pavement disease detection method according to claim 1 is characterized in that: The pavement image data is input into a pavement damage detection model to obtain a pavement damage detection result, specifically including: Performing a preprocessing operation on the road surface image data to obtain target road surface image data; the preprocessing operation includes a geometric scaling operation and a translation operation; The target road surface image data is input into the road surface disease detection model to obtain the disease classification result and the initial detection frame result, and the initial detection frame result is decoded and non-maximum suppression is performed to obtain the detection frame result.

3. The visual-based highway pavement disease detection method according to claim 1 is characterized in that: Inputting each sample of the pavement image sample data set into the target detection network to obtain a predicted pavement disease detection result, specifically including: Inputting the sample road surface image data into the backbone network for feature extraction to obtain a first feature map; Inputting the first feature map into the neck network for feature enhancement to obtain a second feature map; The second feature map is input into the head network for target classification and detection to obtain the predicted road surface disease detection result.

4. The visual-based highway pavement disease detection method according to claim 3 is characterized in that: The backbone network includes a convolution module, four serially superimposed combination modules and a feature fusion module connected in sequence; The combined module includes a convolution module and a residual module, wherein the output result of the convolution module is used to input into the residual module; the convolution module is composed of a two-dimensional convolution layer, a two-dimensional batch normalization layer and an activation function, and the residual module is composed of a convolution layer, a residual connection layer, an upsampling layer and an activation function.

5. The visual-based highway pavement disease detection method according to claim 3 is characterized in that: The neck network includes a feature pyramid network and a path aggregation network, and the output result of the feature pyramid network is used to input into the path aggregation network; The feature pyramid network is a top-down network structure, and the feature pyramid network is used to enhance the semantic information of the first feature map; the path aggregation network is a bottom-up network structure, and the path aggregation network is used to enhance the positioning information of the first feature map.

6. The visual-based highway pavement disease detection method according to claim 1 is characterized in that: The calculation process of the damage category loss is as follows: Among them, p is the probability value corresponding to the predicted disease category result, q is the intersection and union ratio of the probability value corresponding to the predicted disease category result and the disease category label, and γ and α are hyper parameters.

7. The visual-based highway pavement disease detection method according to claim 1 is characterized in that: The calculation process of the detection box loss is: DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+1 ); Among them, y is the detection box label value, y i+1 ,y i are the two predicted detection box values ​​adjacent to the detection box label value, S i+1 is with y i+1 The corresponding predicted detection box probability value, S i is with y i The corresponding predicted detection box probability value.

8. A visual-based road surface disease detection device, characterized in that: The vision-based highway pavement disease detection device comprises: Image acquisition module, used to collect real road surface image data; The positioning module is used to locate the latitude and longitude information of the image acquisition module and trigger the image acquisition module to collect the current road surface image data according to the change of position; An inertial navigation module, used to collect attitude information of the vehicle on which the image acquisition module is installed; Edge computing modules for: Execute the visual-based highway pavement disease detection method according to any one of claims 1 to 7; Get the latitude and longitude information of the image acquisition module; Obtain the attitude information collected by the inertial navigation module and calculate the flatness of the road surface; The latitude and longitude information of the image acquisition module, the road surface image data, the road surface disease detection results and the road surface flatness are associated to generate an inspection report.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vision-based highway pavement defect detection method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vision-based highway pavement defect detection method described in any one of claims 1 to 7 is implemented.

Images