A calculation method and system for the size and severity of rutting, pothole, and subsidence on roads based on machine learning

Abstract

The present invention relates to a method and system for calculating the size and severity of rutting, bumps, and settlements on roads based on machine learning. The method includes the following steps: Step 1, install a high-definition camera, a central industrial control computer, and a line laser emitter device on the inspection vehicle, and the high-definition camera collects images according to the instructions of the central industrial control computer; Step 2, the neural network algorithm deployed on the central industrial control computer identifies and fits the coordinate points [u, v] of the laser line in the image, uses the road baseline algorithm based on machine learning deployed on the central industrial control computer to identify the road baseline, and performs post-processing calculations on the identified laser line and road baseline to obtain the three-dimensional disease size and severity; Step 3, calculate the disease length through the relationship between multiple consecutive pictures; Step 4, finally save the identified results locally and transmit them back to the cloud through the mobile network. This method improves the accuracy of detecting the depth, width, etc. of deformation diseases such as rutting, bumps, and settlements on roads.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional pavement disease detection, and in particular to a method and system for calculating the size and severity of rutting, bumping and subsidence of roads based on machine learning. Background Art

[0002] In recent years, with the application of machine learning and deep learning technologies in the field of traffic engineering, especially in the detection of road surface diseases and three-dimensional diseases (including rutting, bumping, subsidence, etc.) of road traffic, new lightweight road detection systems have partially replaced traditional detection methods, greatly improving the automation level and efficiency of road disease detection.

[0003] At present, although there have been certain progresses in the identification of deformation diseases such as rutting, bumping and subsidence, and quantitative analysis of the above deformation diseases can also be achieved, the method for calculating the reference line required in the analysis process has always affected the determination of the disease size and severity. The existing methods for quantitative detection of rutting, subsidence and bumping of roads mainly rely on methods such as road multi-functional detection vehicles, line structured light, 3D laser point cloud, etc. By relying on the above methods, a set of coordinate points of the road cross-section are obtained to characterize the change of the road cross-section, and then a straight line is generated by using linear fitting of this set of coordinate points as the reference line at this cross-section.

[0004] The above existing technical solutions have the following defects: The above method has a small error when the depth and width of the deformation disease are small. As the severity of the disease increases, the change range of the road cross-section curve is large. Only relying on the linear fitting method to calculate the cross-section reference line will cause a large actual error in the disease, and it is impossible to accurately judge the depth and width of the disease. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a method and system for calculating the size and severity of rutting, bumping and subsidence of roads based on machine learning. This method annotates the reference line corresponding to the existing cross-section curve, trains the reference line coordinate model by using the machine learning method, and predicts the reference line corresponding to the road cross-section in the road rutting and bumping detection algorithm, so as to more accurately obtain the size information such as the depth and width of deformation diseases such as road rutting and bumping, and thus more accurately judge the severity of the disease.

[0006] The above invention purpose of the present invention is achieved through the following technical solutions:

[0007] A method for calculating the size and severity of rutting, bumping and subsidence of roads based on machine learning includes the following steps:

[0008] Step 1: Install a high-definition camera, a central industrial control computer, and a line laser emitter device on the inspection vehicle. The high-definition camera collects images according to the instructions of the central industrial control computer.

[0009] Step 2: The neural network algorithm deployed in the central industrial control computer identifies and fits the coordinate points [u, v] of the laser line in the image, and uses the road reference line algorithm based on machine learning deployed in the central industrial control computer to identify the road reference line. Perform post-processing calculations on the identified laser line and road reference line to obtain the three-dimensional disease size and severity.

[0010] Step 3: Calculate the disease length through the relationship between multiple consecutive pictures.

[0011] Step 4: Finally, save the identified results locally and transmit them back to the cloud through the mobile network.

[0012] As a further technical solution of the present invention: In Step 1, a line laser is projected onto the ground by a line laser emitter, and the installed high-definition camera is used to collect the line laser in real time.

[0013] The power of the line laser emitter is 150 mw, the prism of the line laser emitter is a Powell prism, and the prism angle is between 90-120°.

[0014] As a further technical solution of the present invention: Before collection, use the laser line calibration method to calibrate the pictures to obtain the conversion coefficient between the picture pixels and the depth and width of the road disease.

[0015] As a further technical solution of the present invention: The laser line calibration method refers to selecting a relatively flat ground for calibration, and the calibration object is a rectangular cross-section profile or a wooden object. For calibration objects of different thicknesses, calculate the corresponding pixel differences in the image respectively, and perform equivalent conversion on the pixel differences and the actual thickness of the calibration object to obtain the disease depth conversion coefficient.

[0016] Calibrate the horizontal width of the picture, and perform equivalent conversion on the actual width value and the horizontal pixel value of the picture to obtain the disease width conversion coefficient.

[0017] As a further technical solution of the present invention: In Step 2, use the deep learning method to fit the laser line coordinates, so as to predict a set of cross-section coordinates [[x i1, y i1], [x i2, y i2],...] at the i-th cross-section of the road.

[0018] As a further technical solution of the present invention: In Step 2, the road reference line algorithm based on machine learning is specifically:

[0019] S1. Manually label the position tags of the reference line on the above cross-section as the labels for model training. The reference line is expressed by the start and end point coordinates, denoted as [[x i a , y i a ,[x i b , y i b ;

[0020] S2. Convert the cross-section coordinates [[x i1, y i1], [x i2, y i2],...] and the reference line coordinates [[x i a , y i a ,[x i b , y i b into one-dimensional vectors respectively as the model input and output;

[0021] S3. Adopt machine learning algorithms including but not limited to random forest regression, and train the machine learning model through a large number of samples of different road cross-sections, so as to realize the prediction of the reference line coordinates.

[0022] As a further technical solution of the present invention: utilize the above calibration conversion relationship to convert the pixel coordinate difference of the picture into the depth and width of the actual three-dimensional diseases of road rutting, heaving and subsidence.

[0023] As a further technical solution of the present invention: deploy the above model on edge devices or cloud services, and the reference line of the road cross-section can be predicted, which serves as the basis for analyzing dimensions such as disease depth and length.

[0024] As a further technical solution of the present invention: according to the information of disease depth and width, calculate and determine the definition of disease severity level based on the disease size calculation according to the disease severity level algorithm.

[0025] A system for calculating the size and severity of road rutting, heaving and subsidence based on machine learning, the system includes a central industrial control computer, a high-definition camera, and a line laser emitter, and the central industrial control computer is respectively connected to the high-definition camera and the line laser emitter;

[0026] The line laser emitter is installed at the bottom of the rear end of the inspection vehicle for projecting line laser on the ground;

[0027] The high-definition camera is installed at the top of the rear end of the inspection vehicle for real-time collecting the line laser according to the instructions of the central industrial control computer and uploading the image to the central industrial control computer;

[0028] The central industrial control computer is installed in the inspection vehicle to process and calculate the collected images, obtain the three-dimensional disease size and severity, calculate the disease length through the relationship between multiple consecutive pictures, and finally save the recognized results locally and transmit them back to the cloud through the mobile network.

[0029] In summary, the present invention includes at least one of the following beneficial technical effects:

[0030] The present invention discloses a method and system for calculating the size and severity of rutting, bumping, and settlement on roads based on machine learning. Compared with the traditional method of simply using linear fitting directly, this method uses machine learning to label and train the road cross-section baseline model, which can greatly improve the accuracy of detecting the depth, width, etc. of deformation diseases such as rutting, bumping, and settlement on roads. Therefore, it can be effectively used for three-dimensional disease detection of roads and technical condition assessment, and improve the intelligent level of road management and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall structure of the system of the present invention.

[0032] Figure 2 It is a schematic diagram of calibrating the line laser with a calibration object on a flat road surface.

[0033] Figure 3 It is a schematic diagram of the baseline obtained by the road baseline algorithm based on machine learning.

[0034] Figure 4 It is a flowchart for judging the disease severity level after calculating the disease size.

[0035] Figure 5 It is a flowchart for implementing the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application; obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0037] In the description of the present application, it should be noted that the orientation or positional relationship indicated by terms such as "upper", "lower", "inner", "outer", "top / bottom end", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be construed as a limitation on the present application. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0038] In the description of the present application, it should be noted that unless otherwise clearly specified and defined, terms such as "installation", "provided with", "sheathed / connected", "connection", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0039] Embodiment 1:

[0040] A method for calculating the size and severity of rutting, pothole, and subsidence on roads based on machine learning, comprising the following steps:

[0041] Step 1: Install a high-definition camera, a central industrial control computer, and a line laser emitter device on the inspection vehicle. The high-definition camera collects images according to the instructions of the central industrial control computer.

[0042] Step 2: The neural network algorithm deployed on the central industrial control computer identifies and fits the coordinate points [u, v] of the laser line in the image, and uses the road baseline algorithm based on machine learning deployed in the central industrial control computer to identify the road baseline. The identified laser line and road baseline are post-processed and calculated to obtain the three-dimensional disease size and severity.

[0043] Step 3: Then, calculate the disease length through the relationship between multiple consecutive pictures.

[0044] Step 4: Finally, save the identified results locally and transmit them back to the cloud through the mobile network.

[0045] In step 1, a line laser is projected on the ground by the line laser emitter, and the installed high-definition camera is used to collect the line laser in real time. For the system schematic diagram of the line structured light method, see Figure 1 ; the power of the line laser emitter is 150 mw, the prism of the line laser emitter is a Powell prism, and the prism angle is between 90 - 120°.

[0046] Before collection, use as Figure 2The laser line calibration method shown above calibrates the picture to obtain the conversion coefficients between the picture pixels and the depth and width of road diseases. The laser line calibration method means selecting a relatively flat ground for calibration, and the calibration objects are rectangular cross-section profiles and wooden objects. For calibration objects with different thicknesses, the corresponding pixel differences in the image are calculated respectively, and the pixel differences are equivalently converted with the actual thickness of the calibration object to obtain the disease depth conversion coefficient; for calibrating the horizontal width of the picture, the actual width value is equivalently converted with the horizontal pixel value of the picture to obtain the disease width conversion coefficient.

[0047] In step 2, the deep learning method is used to fit the laser line coordinates, so as to predict a set of cross-section coordinates [[x i1,y i1],[x i2,y i2],...] at the i-th cross-section of the road.

[0048] The above-mentioned road baseline algorithm based on machine learning refers to various machine learning algorithms such as trained random forests. By inputting the laser line coordinates predicted above, the starting and ending point coordinates of the baseline are preferably generated, thereby improving the accuracy of the road baseline.

[0049] In step 2, the road baseline algorithm based on machine learning is specifically as follows:

[0050] S1. Manually label the baseline position of the above cross-section as the label for model training. The baseline is expressed by the starting and ending point coordinates, denoted as [[x i a ,y i a ,[x i b ,y i b ;

[0051] S2. Convert the cross-section coordinates [[x i1,y i1],[x i2,y i2],...] and the baseline coordinates [[x i a ,y i a ,[xi b ,y i b into one-dimensional vectors respectively as the model input and output;

[0052] S3. Adopt machine learning algorithms including but not limited to random forest regression, and train the machine learning model through a large number of samples of different road cross-sections, so as to realize the prediction of the baseline coordinates; the road baseline algorithm can be seen in Figure 3 。

[0053] Using the above calibration conversion relationship, the pixel coordinate difference of the picture is converted into the depth and width of the actual three-dimensional diseases of road rutting, heaving and subsidence.

[0054] Deploying the above model on edge devices or cloud services can predict the road cross-section baseline, which serves as the basis for dimension analysis such as disease depth and length.

[0055] According to the information of disease depth and width, based on the Figure 4 disease severity level algorithm in it, calculate and determine the disease severity level definition after calculating based on disease dimensions. Finally, save the identified results locally and transmit them back to the cloud through the mobile network. See the overall implementation flowchart of the solution in Figure 5 .

[0056] The following further elaborates on the present invention in conjunction with embodiments:

[0057] The implementation manner of the method proposed in this embodiment is as shown in Figure 1 and the implementation process is summarized as:

[0058] (1) Install devices such as in-vehicle cameras, central industrial control computers, and laser emitters on the vehicle. Among them, the central industrial control computer is powered by the vehicle, and the in-vehicle camera is connected to the central industrial control computer, and the central industrial control computer sends image acquisition instructions;

[0059] (2) Before acquisition, use the Figure 2 laser line calibration method shown to calibrate the pictures to obtain the conversion coefficient between the pixel difference of the pictures and the actual road disease depth and width;

[0060] (3) After the system starts, the vehicle acquires images by the camera during the forward movement, and the convolutional neural network in the central industrial control computer fits the laser line to obtain the pixel coordinates of the laser line;

[0061] (4) Generate the baseline coordinates of the disease through the road baseline algorithm model deployed in the central industrial control computer;

[0062] (5) Calculate the pixel difference in the pictures, and use the above conversion coefficient to convert the difference into the road disease depth and width;

[0063] (6) Input the above disease information into the disease severity level determination algorithm to obtain the disease severity level;

[0064] (7) The system saves the detected disease data locally and uploads it to the cloud system through the mobile network.

[0065] Embodiment 2:

[0066] A road rut, bump, and settlement dimension and severity calculation system based on machine learning in Embodiment 1. The system includes a central industrial control computer, a high-definition camera, and a line laser emitter. The central industrial control computer is respectively connected to the high-definition camera and the line laser emitter;

[0067] The line laser emitter is installed at the bottom of the rear end of the inspection vehicle for projecting line laser on the ground;

[0068] The high-definition camera is installed at the top of the rear end of the inspection vehicle for collecting the line laser in real time according to the instructions of the central industrial control computer and uploading the images to the central industrial control computer;

[0069] The central industrial control computer is installed in the inspection vehicle for processing and calculating the collected images, obtaining the three-dimensional disease size and severity, then calculating the disease length through the relationship between multiple consecutive pictures, and finally saving the recognized results locally and transmitting them back to the cloud through the mobile network.

[0070] The implementation principle of the present invention is as follows: The present invention discloses a method and system for calculating the size and severity of rutting, pothole, and subsidence on roads based on machine learning. Compared with the traditional method of simply using linear fitting directly, this method uses machine learning to label and train the road cross-section baseline model, which can greatly improve the accuracy of detecting the depth, width, etc. of deformation diseases such as rutting, pothole, and subsidence on roads, and thus can be effectively used for three-dimensional disease detection and technical condition assessment of roads, improving the intelligent level of road management and maintenance.

[0071] The embodiments of the specific implementation manners are all preferred embodiments of the present invention, and do not limit the protection scope of the present invention accordingly. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for calculating the size and severity of road rutting subsidence based on machine learning, characterized in that: The following steps are involved: Step 1: Install a high-definition camera, a central industrial computer, and a line laser transmitter device on the inspection vehicle, wherein the high-definition camera collects images according to the instructions of the central industrial computer; Step 2: The neural network algorithm deployed in the central industrial computer identifies and fits the coordinate points [u, v] of the laser line in the image, and uses the machine learning-based road baseline algorithm deployed in the central industrial computer to identify the road baseline, and performs post-processing calculations on the laser line and road baseline obtained by the above identification to obtain the three-dimensional disease size and severity; Step 3: Calculate the length of the disease based on the relationship between multiple consecutive images; Step 4: Finally, the recognition results are saved locally and transmitted back to the cloud via the mobile network; In step 2, the laser line coordinates are fitted using a deep learning method to predict a set of section coordinates [[xi1 ,yi1 ] ,[xi2 ,yi2 ] ,...] at the road cross section i; In step 2, the road baseline algorithm based on machine learning is specifically: S1. Manually label the baseline position of the above section as a label for model training. The baseline is expressed by the coordinates of the start and end points, recorded as [[xia ,yia ] ,[xib ,yib ]]; S2, convert the cross-section coordinates [[xi1 ,yi1 ] ,[xi2 ,yi2 ] ,...] and the baseline coordinates [[xia ,yia ] ,[xib ,yib ]] into one-dimensional vectors as model input and output respectively; S3. Using machine learning algorithms including but not limited to random forest regression, the machine learning model is trained with a large number of samples of different road sections, so as to achieve the prediction of the baseline coordinates; Based on the information of the depth and width of the disease, according to the disease severity algorithm, the definition of the disease severity level calculated based on the disease size is calculated and determined.

2. The method for calculating the size and severity of road rutting subsidence based on machine learning according to claim 1 is characterized in that: In step 1, a line laser transmitter is used to project a line laser onto the ground, and the installed high-definition camera is used to collect the line laser in real time; The power of the line laser emitter is 150mw, the prism of the line laser emitter is a Powell prism, and the prism angle is between 90-120°.

3. The method for calculating the size and severity of road rutting subsidence based on machine learning according to claim 2 is characterized in that: Before data collection, the image is calibrated using the laser line calibration method to obtain the conversion coefficient between image pixels and the depth and width of road damage.

4. The method for calculating the size and severity of road rutting subsidence based on machine learning according to claim 3 is characterized in that: The laser line calibration method refers to selecting a relatively flat ground for calibration, selecting rectangular cross-section profiles and wood objects as calibration objects, calculating the corresponding pixel differences in the image for calibration objects of different thicknesses, and performing equivalent conversion between the pixel differences and the actual calibration object thickness to obtain the disease depth conversion coefficient; The horizontal width of the image is calibrated, and the actual width value is equivalently converted with the horizontal pixel value of the image to obtain the disease width conversion coefficient.

5. The method for calculating the size and severity of road rutting subsidence based on machine learning according to claim 1 is characterized in that: By using the calibration conversion relationship, the pixel coordinate difference of the image is converted into the actual depth and width of the three-dimensional road rutting subsidence disease.

6. The method for calculating the size and severity of road rutting subsidence based on machine learning according to claim 1, characterized in that: By deploying the above model on edge devices or cloud services, the road section baseline can be predicted, which serves as the basis for dimensional analysis such as the depth and length of the disease.

7. A system for calculating the size and severity of road rutting subsidence based on machine learning, characterized in that: The system includes a central industrial computer, a high-definition camera, and a line laser transmitter, wherein the central industrial computer is connected to the high-definition camera and the line laser transmitter respectively; The line laser transmitter is installed at the bottom of the rear end of the inspection vehicle to project the line laser on the ground; The high-definition camera is installed on the top of the rear end of the inspection vehicle and is used to collect the line laser in real time according to the instructions of the central industrial control computer and upload the image to the central industrial control computer; The central industrial control computer is installed in the inspection vehicle to process and calculate the collected images to obtain the three-dimensional disease size and severity, and then calculate the disease length through the relationship between multiple consecutive pictures. Finally, the recognition results are saved locally and transmitted back to the cloud through the mobile network; The neural network algorithm deployed in the central industrial computer identifies and fits the coordinate points [u, v] of the laser line in the image, identifies the road baseline using the machine learning-based road baseline algorithm deployed in the central industrial computer, and performs post-processing calculations on the identified laser line and road baseline to obtain the three-dimensional defect size and severity; The laser line coordinates are fitted using deep learning methods to predict a set of cross-section coordinates [[xi1 ,yi1 ] ,[xi2 ,yi2 ] ,...] at the road cross section i; The road baseline algorithm based on machine learning is specifically: S1. Manually label the baseline position of the above section as a label for model training. The baseline is expressed by the coordinates of the start and end points, recorded as [[xia ,yia ] ,[xib ,yib ]]; S2, convert the cross-section coordinates [[xi1 ,yi1 ] ,[xi2 ,yi2 ] ,...] and the baseline coordinates [[xia ,yia ] ,[xib ,yib ]] into one-dimensional vectors as model input and output respectively; S3. Using machine learning algorithms including but not limited to random forest regression, the machine learning model is trained with a large number of samples of different road sections, so as to achieve the prediction of the baseline coordinates; Based on the information of the depth and width of the disease, according to the disease severity algorithm, the definition of the disease severity level calculated based on the disease size is calculated and determined.

Images