Road disease lane positioning calculation method based on lane line identification

Abstract

In order to solve the problems of high cost, unsuitability for large-scale popularization, large calculation amount, few detection types and incapability of realizing real-time detection existing in current road disease detection, the invention provides a road disease lane positioning calculation method based on lane line identification, which integrates rapidness, high efficiency, accuracy and low cost. According to the technical scheme, the method is characterized by comprising the following steps that S1, a system is initialized; s2, loading a training weight; s3, acquiring and processing image data; s4, carrying out disease reasoning identification and CLRNet reasoning on the image data preprocessed in the S3; s5, lane reasoning: performing lane reasoning by using the disease reasoning identification data and the lane line position data in the step S4, and reasoning the position of the lane where the disease is located; and S6, processing a lane reasoning result.

Description

Technical Field

[0001] The present invention relates to the technical field of road disease management, and in particular to a method for calculating the lane position of road diseases based on lane line recognition. Background Art

[0002] With the accelerating pace of road construction, subway construction, and underground space development, the number of urban vehicles has increased sharply. At the same time, due to reasons such as increasing road traffic flow, pavement aging, subgrade settlement, rainwater scouring, pavement seepage, and underground pipeline leakage, phenomena such as local pavement unevenness, local potholes, pavement cracks, potholes, bumps, edge nibbling, peeling, chipping, pushing, and rotten edges often occur. In severe cases, problems such as road pavement cracking, deformation, settlement, and collapse will occur, which will affect the driving speed and driving comfort of vehicles to varying degrees. In some cases, the roadbed structure is damaged, which will affect driving safety and even cause traffic accidents, having a great impact on society. Whether the disease location can be accurately located in a timely manner, the disease category can be identified, and the cause of the disease can be correctly analyzed directly relates to whether road maintenance can be carried out correctly and in a timely manner, and directly affects the operation level of the road.

[0003] At present, the following several methods are usually adopted for road disease detection:

[0004] 1. "Three-dimensional integration" road detection method: This detection method combines technologies such as computers, communications, photogrammetry, and lasers, and comprehensively utilizes various technologies such as GPS positioning, mileage coding positioning, laser-assisted three-dimensional measurement, and digital imaging. Using advanced scientific equipment, it completes the automatic acquisition, processing, and analysis of pavement damage data. This method mainly uses shooting, measurement, and calculation to obtain a three-dimensional stereoscopic image of a certain section of highway, so as to intuitively judge the pavement diseases and damage degree, and further detect the health status of the highway pavement. This method can better reflect the pavement disease degree and can further promote the innovation of pavement disease detection methods in China to achieve the healthy development of highways.

[0005] The three-dimensional detection system is an advanced measurement and detection system for obtaining high-density spatial point clouds and image data. This system can quickly obtain continuous three-dimensional panoramic image data during high-speed movement. At present, there are mainly two means for pavement damage investigation and detection: One is manual visual inspection combined with measuring instruments for on-foot investigation, the disadvantage of which is low detection efficiency, and passing vehicles pose a threat to the safety of investigators; the other is to first quickly collect images by a detection vehicle, and then judge the development degree of diseases through manual reading. The disadvantage is that the location of the disease is not accurately positioned, and human factors have an impact on disease judgment, and misjudgment and missed judgment are likely to occur. Through the three-dimensional detection system, the area and depth of diseases in pavement damage investigation can be qualitatively and quantitatively detected.

[0006] 2. Road penetration radar detection technology

[0007] A ground-penetrating radar for roads uses high-frequency radio waves (500 MHz) in the form of broadband short pulses. It is sent into the soil layer through the transmitting device in the antenna on the road surface. When it encounters different interfaces, a reflection phenomenon occurs. It is received by another receiving device in the antenna and records the abnormal places in the road surface structure when returning, so as to detect the health status of each section of the road surface. The ground-penetrating radar technology is a non-destructive technology, and it integrates technologies such as electromagnetic waves, data conversion and acquisition, and wireless communication. It has the advantages of strong anti-interference ability, loose working conditions, high precision, short time-consuming, convenient and fast, etc. However, the disadvantage is that the cost is expensive and it is not suitable for large-scale promotion and use by maintenance companies.

[0008] 3. Road surface disease detection based on images

[0009] The road surface disease detection method based on image processing has become the main method of road surface disease detection technology. It can basically achieve detection automation and comprehensive evaluation of the road conditions of each section. The road surface disease detection based on images mainly includes the following several types: image enhancement processing, suppression processing, and image segmentation processing technology, etc. This technology mainly uses a computer to inspect the road surface section by section, collect data of each section, and then calculate based on the collected data to draw the health status of a certain section of the road surface, so as to obtain the specific disease degree of this section of the road. The disadvantages are large computational amount, few detection types, and inability to achieve real-time detection.

[0010] 4. Road surface disease positioning technology based on high-precision map technology

[0011] The road is an important part of transportation infrastructure assets. Scientific maintenance management of the road throughout its life cycle is the key factor to improve the road safety performance and extend its service life. With the development of technologies such as intelligent hardware technology, informatization technology, and artificial intelligence, high-precision detection of the road surface and establishment of a three-dimensional intuitive visualization model are carried out to accurately locate the positions of road diseases. Relying on high-precision maps to integrate static and dynamic data, the realization of "a single map for dynamic and static data of road traffic" has wide promotion value in road maintenance management, but the current application cost is quite large and it is not suitable for large-scale promotion.

[0012] The commonly used automatic detection methods for road diseases have their own advantages and disadvantages, and have also achieved a certain degree of automation in aspects such as road surface flatness detection, road surface radar detection, damage detection, rutting, bearing capacity detection, and anti-skid ability detection. However, there are significant differences in the detection results of various methods. Summary of the Invention

[0013] To solve the problems existing in the current road disease detection, such as high cost, unsuitability for large-scale promotion, large amount of calculation, few detection types, and inability to achieve real-time detection, the present invention proposes a road disease lane positioning calculation method based on lane line recognition that integrates speed, efficiency, accuracy, and low cost.

[0014] The technical solution of the present invention is: a road disease lane positioning calculation method based on lane line recognition, which is characterized by including the following steps:

[0015] S1: System initialization: Access the GPS data service, wireless communication service, and deep learning inference framework service. The deep learning inference framework service is a lane line recognition model with a CLRNet deep learning network;

[0016] S2: Load the training weights. The training weights are disease recognition models for deep learning pre-training of road diseases. The range of road diseases is: transverse cracks, longitudinal cracks, crocodile cracks, potholes, and expansion joints;

[0017] S3: Image data acquisition and processing: Connect a moving camera, obtain real-time video image data through the moving camera, perform real-time image analysis and edge calculation on the image data, and preprocess the image. The preprocessing process includes image denoising, light white balance, and image scaling;

[0018] S4: Perform disease inference recognition and CLRNet inference on the image data preprocessed in S3: Use the training weights accessed in S2 to perform disease inference recognition on the preprocessed image data to obtain disease inference recognition data. At the same time, use the image data processed by the deep learning inference framework service loaded in S1 to perform CLRNet inference, calculate lane line candidate points, perform lane line detection and analysis on the image, determine the lane line position, and obtain lane line position data;

[0019] S5: Lane inference: Use the disease inference recognition data and lane line position data in S4 to perform lane inference, infer the lane position where the road disease is located, thereby judging the disease on the lane, then number the road where the disease is located, record the GPS position information, and parse the specific road name information. At the same time, take a photo and overlay the road name, GPS position, device number, road surface anomaly type, and time information;

[0020] S6: Disposal of the lane inference result: Judge in step S5 whether it is a disease. If it is a disease, establish a data file number for the corresponding disease, save the disease image data containing GPS positioning information to generate a disease message, store the disease image data in a file, store the disease message in a database, and upload the disease image data and disease message to the cloud storage in real time through a wireless network; If it is not disease data, return to S3.

[0021] The pre-training of the road disease deep learning with the training weights in S2 includes the following steps:

[0022] S21: Sample training: The samples include positive samples and negative samples, and the sample content includes five road diseases: transverse cracks, longitudinal cracks, crocodile cracks, potholes, and expansion joints;

[0023] S22: Weight acquisition: Calculate the weights for each sample content;

[0024] S23: Weight update: Update the weights of the sample content.

[0025] The training of lane line detection and analysis of the image in S4 includes the following steps:

[0026] S41: Collect the road surface images from the in-vehicle perspective, and perform image cleaning to remove blurred images, images with angular inclination, and images without lane lines;

[0027] S42: Select the LabelMe annotation tool to annotate the lane lines in the image;

[0028] S43: Split the data set into a training set and a validation set;

[0029] S44: Add the TUSimple public data set;

[0030] S45: Select the CLRNet network to train the data set;

[0031] S46: Finally, obtain the data model for lane line recognition and detection.

[0032] The inference of the lane position where the road disease is located in S5 includes the following steps:

[0033] S51: Read the camera image and perform preprocessing;

[0034] S52: Load the CLRNet lane line detection data model;

[0035] S53: Calculate the lane line candidate points;

[0036] S54: Output the lane candidate points;

[0037] S55: Mark the candidate points in the image, and the positions of the candidate points determine the lane line positions.

[0038] The inference of the lane position where the road disease is located in S5 includes the following steps:

[0039] S61: Mark the positions of the road disease (M1), the center point position (D1) below the road disease, the position of the first lane line (L1), and the position of the second lane line (L2);

[0040] S62: Mark the positions of the intersection points of the horizontal line (H) where the center point position (D1) of the road disease is located with the first lane line (L1) and the second lane line (L2) as the first intersection point (X1) and the second intersection point (X2), respectively;

[0041] S63: If the abscissa of the center point position (D1) of the road disease is less than the abscissa of the first intersection point (X1), the lane number R1 where the road disease is located; if the abscissa of the center point position (D1) of the road disease is between or coincides with the first intersection point (X1) and the second intersection point (X2), the lane number R2 where the road disease is located; if the abscissa of the center point position (D1) of the road disease is greater than the abscissa of the second intersection point (X2), the lane number R3 where the road disease is located.

[0042] The effect of the present invention is:

[0043] The road disease lane positioning calculation method based on lane line recognition of the present invention obtains real-time video image data through a moving camera, performs real-time image analysis and edge calculation, preprocesses the image, can collect road surface video information in real time and store it, and incorporates it into the deep learning inference framework service for learning and inference in real time. This deep learning inference framework service is a lane line recognition model of the CLRNet deep learning network, and a training weight, that is, a disease recognition model, is loaded during the pre-training of this model. Since the disease recognition model as the training weight has undergone deep learning pre-training for road diseases, and this training weight limits the disease range, that is, transverse cracks, longitudinal cracks, block cracks, potholes, expansion joints; therefore, after loading this training weight, the disease recognition model using the training weight automatically recognizes road disease data such as transverse cracks, longitudinal cracks, block cracks, potholes, expansion joints, etc., to obtain disease inference recognition data. At the same time, using the lane line recognition model of the CLRNet deep learning network in the deep learning inference framework service, through the lane recognition technology of the CLRNet deep learning network, the lane position where the road disease is located is inferred to obtain lane line position data. The disease inference recognition data and the lane line position data are subjected to lane inference to obtain diseases on a specific lane, number the road where the disease is located, record the GPS position information, and parse the specific road name information. At the same time, a moving camera is used to take pictures and important information such as road name, GPS position, device number, road surface anomaly type, time, etc. is superimposed. After the lane inference confirms the disease, combined with the GPS high-precision positioning information, the disease image data is saved and a disease message is generated, and the disease data is uploaded to the cloud storage through the 4G network. In view of the coexistence of straight lanes and curved lanes, after practical tests, the data effects of the LaneNet network, GANet network, and SGNet network are not good in curved lanes. Therefore, the solution of the present invention uses the CLRNet deep learning network to achieve good results in curved lanes. At the same time, the present invention obtains real-time video image data in the field of on-vehicle road maintenance, adds scene photos, and helps to optimize the lane line detection algorithm; the present invention combines the lane inference method, lane detection, road disease detection, and GPS position positioning to quickly and accurately infer and determine the disease and its lane position.

[0044] Due to deep learning for up to five typical road diseases, a deep network model is established, which can perform real-time comparison and automatic analysis on the real-time acquired road video image data. It has the advantages of low cost, small computational load, suitability for large-scale promotion, and complete detection types, with advantages such as fast, efficient, accurate, and low cost. In short, the present invention has the following characteristics: 1. Fast detection: The pre-trained weight is deployed at the side end for computing and detecting diseases, with very good real-time performance; 2. Efficient and convenient: No additional hardware equipment is required, pure visual computing, and 3 lanes can be detected simultaneously, which is very efficient; 3. Low cost: High-precision map production requires a high data acquisition cost. Relatively speaking, the disease lane positioning method proposed by the present invention has a very low cost.

[0045] The following further describes the present invention in conjunction with the drawings and embodiments. Brief Description of the Drawings

[0046] Figure 1 It is a flowchart of the present invention;

[0047] Figure 2 It is a schematic diagram of the CLRNet network structure;

[0048] Figure 3 It is a detailed flowchart of the present invention;

[0049] Figure 4 It is a flowchart of lane line training;

[0050] Figure 5 Lane line recognition flowchart;

[0051] Figure 6 It is an effect diagram of lane line recognition;

[0052] Figure 7 Schematic diagram of reasoning for the lane position of road diseases. Detailed Implementation Manner

[0053] Figure 1 Among them, a lane position calculation method for road diseases based on lane line recognition includes the following steps:

[0054] S1: System initialization: Connect to the GPS data service, wireless communication service, and deep learning inference framework service. Then, the deep learning inference framework service is a lane line recognition model with a CLRNet deep learning network;

[0055] S2: Load the training weights, and the training weights are disease recognition models for pre-training deep learning of road diseases. The range of road diseases is: transverse cracks, longitudinal cracks, turtle cracks, potholes, and expansion joints;

[0056] S3: Image data acquisition and processing: Connect the moving camera, obtain real-time video image data through the moving camera, perform real-time image analysis and edge computing on the image data, and preprocess the image. The preprocessing process includes image denoising, light white balance, and image scaling;

[0057] S4: Perform disease inference recognition and CLRNet inference on the image data preprocessed in S3: Use the training weights accessed in S2 to perform disease inference recognition on the preprocessed image data to obtain disease inference recognition data; at the same time, use the deep learning inference framework service loaded in S1 to perform CLRNet inference on the preprocessed image data, calculate the candidate points of the lane lines, perform lane line detection and analysis on the image, determine the lane line position, and obtain the lane line position data; and further infer and recognize road diseases. For the schematic diagram of the CLRNet network structure, see Figure 2 . Given the coexistence of straight lanes and curved lanes, after practical tests, the data effects of the LaneNet network, GANet network, and SGNet network are not good in curved lanes. Therefore, the solution of the present invention adopts a deep learning network model with a CLRNet network. After verification, the disease inference recognition of curved lanes trained by the deep learning network model with a CLRNet network is not affected by curved lanes.

[0058] Figure 2 In, the CLRNet network consists of three parts, namely: (a) Refinement (b) RolGather and (c) Loss. The global context information is obtained through the RoIGather module, and the lane lines are optimized as a whole using the Line IoU loss.

[0059] S5: Lane inference: Use the disease inference recognition data and lane line position data in S4 to perform lane inference, infer the lane position where the road disease is located, thereby judging the disease on the lane, then number the road where the disease is located, record the GPS position information, and parse the specific road name information. At the same time, take a photo and overlay the road name, GPS position, device number, road surface anomaly type, and time information.

[0060] S6: Disposal of the lane inference result: According to step S5, judge whether it is a disease on the lane. If it is a disease, establish a data file number for the corresponding disease, save the disease image data containing GPS positioning information to generate a disease message, store the disease image data in a file, store the disease message in the message database, and upload the disease image data and disease message to the cloud storage in real time through a wireless network. If it is not a disease, return to S2 and continue to read the image data for analysis and judgment to end a road disease processing process (see Figure 3 ).

[0061] The deep learning pre-training for road diseases in S2 includes the following steps: Refer to Figure 3 the deep learning pre-training part of

[0062] S21: Sample training: The samples include positive samples and negative samples, and the sample content includes transverse cracks, longitudinal cracks, crocodile cracks, potholes, and expansion joints;

[0063] S22: Weight acquisition: Calculate the weights for each sample content;

[0064] S23: Weight update: Update the weights of the sample content.

[0065] Figure 4 In

[0066] S41: Collect on-vehicle perspective road surface images and perform image cleaning to remove blurred images, images with angular tilt, and images without lane lines;

[0067] S42: Select the LabelMe annotation tool to annotate the lane lines in the images;

[0068] S43: Split the dataset into a training set and a validation set;

[0069] S44: Add the TUSimple public dataset;

[0070] S45: Select the CLRNet network to train the dataset;

[0071] S46: Finally, obtain the data model for lane line recognition and detection.

[0072] Figure 5 In

[0073] S51: Read the camera image and perform preprocessing;

[0074] S52: Load the CLRNet lane line detection data model;

[0075] S53: Calculate the lane line candidate points;

[0076] S54: Output the lane candidate points;

[0077] S55: Mark the candidate points in the image, and the positions of the candidate points determine the lane line positions (Refer to Figure 5 , Figure 6 express the positions of the calculated lane line candidate points to determine the lane line positions).

[0078] Figure 7In step S5, inferring the lane position where the road disease is located includes the following steps:

[0079] S61: Mark the position M1 of the road disease, the position D1 of the center point below the road disease, the position of the first lane line L1, and the position of the second lane line L2. In this way, three lanes can be detected simultaneously, and the detection efficiency is very high.

[0080] S62: Mark the positions of the intersections of the horizontal line H where the center point D1 below the road disease is located with the first lane line L1 and the second lane line L2 as the first intersection point X1 and the second intersection point X2 respectively.

[0081] S63: If the abscissa of the center point D1 below the road disease is less than the abscissa of the first intersection point X1, the lane number R1 where the road disease is located; if the abscissa of the center point D1 below the road disease is between or coincides with the first intersection point X1 and the second intersection point X2, then the lane number R2 where the road disease is located; if the abscissa of the center point D1 below the road disease is greater than the abscissa of the second intersection point X2, then the lane number R3 where the road disease is located. In this way, an intuitive relationship between the three lanes and the disease is established, which is convenient for identifying the disease lane.

Claims

1. A method for calculating the location of road damage lanes based on lane line recognition, characterized by: The following steps are involved: S1: System initialization: access to GPS data service, wireless communication service, and deep learning reasoning framework service, wherein the deep learning reasoning framework service is a lane line recognition model with a CLRNet deep learning network; S2: Loading training weights, where the training weights are road disease recognition models pre-trained for deep learning of road diseases. The road disease ranges from transverse cracks, longitudinal cracks, cracks, potholes, and expansion joints. S3: Image data acquisition and processing: Connect the motion camera, acquire real-time video image data through the motion camera, perform image analysis and edge computing on the image data in real time, and pre-process the image. The pre-processing process includes image denoising, light white balance, and image scaling. S4: Perform disease reasoning and CLRNet reasoning on the image data preprocessed by S3: Use the training weights accessed by S2 to perform disease reasoning and recognition on the preprocessed image data to obtain disease reasoning and recognition data. At the same time, use the image data processed by the deep learning reasoning framework service loaded by S1 to perform CLRNet reasoning, calculate lane line candidate points, perform lane line detection and analysis on the image, determine the lane line position, and obtain lane line position data; S5: Lane reasoning: Use the disease reasoning identification data and lane line position data in S4 to perform lane reasoning, infer the lane position of the road disease, and thus determine the disease on the lane. Then number the road where the disease is located, record the GPS location information, and parse the specific name information of the road. At the same time, take a photo and superimpose the road name, GPS location, equipment number, road surface abnormality type and time information; S6: Handling of lane reasoning results: Step S5 determines whether it is a damage. If it is a damage, a data file number is established for the corresponding damage, and the damage image data containing GPS positioning information is saved to generate a damage message. The damage image data is stored in a file, and the damage message is stored in a message library. The damage image data and the damage message are uploaded to the cloud storage in real time through the wireless network; if it is not damage data, return to S3.

2. The method for calculating the location of road damage lanes based on lane line recognition according to claim 1 is characterized by: The training weights in S2 for road disease deep learning pre-training include the following steps: S21: Sample training: The samples include positive samples and negative samples, and the sample contents include five types of road diseases: transverse cracks, longitudinal cracks, cracks, potholes, and expansion joints; S22: Weight acquisition: Calculate the weight of each sample content; S23: Weight update: update the weight of the sample content.

3. The method for calculating the location of a road defect lane based on lane line recognition according to claim 1 or 2, characterized in that The training of lane line detection analysis on the image in S4 includes the following steps: S41: Collect road images from the vehicle's perspective and perform image cleaning to remove blurred images, angled images, and images without lane lines; S42: Select the LabelMe labeling tool to label the lane lines of the image; S43: Divide the data set into a training set and a validation set; S44: Add TUSimple public dataset; S45: Select the CLRNet network to train the data set; S46: Finally, the data model of lane line recognition detection is obtained.

4. The method for calculating the location of road damage lanes based on lane line recognition according to claim 3 is characterized by: The inference of the lane position of the road defect in S5 includes the following steps: S51: Read the camera image and perform preprocessing; S52: Load the CLRNet lane detection data model; S53: Calculate lane line candidate points; S54: output lane candidate points; S55: Mark candidate points in the image, and the positions of the candidate points determine the positions of the lane lines.

5. The method for calculating the location of road damage lanes based on lane line recognition according to claim 3 is characterized by: The inference of the lane position of the road defect in S5 includes the following steps: S61: Mark the road defect position (M1), the center point position of the lower edge of the road defect (D1), the first lane line (L1) and the second lane line (L2); S62: marking the intersections of the horizontal line (H) where the center point of the lower edge of the road defect (D1) is located and the first lane line (L1) and the second lane line (L2) as the first intersection point (X1) and the second intersection point (X2), respectively; S63: If the horizontal coordinate of the center point position (D1) of the lower side of the road damage is smaller than the horizontal coordinate of the first intersection point (X1), the lane where the road damage is located is numbered R1; if the horizontal coordinate of the center point position (D1) of the lower side of the road damage is between or coincides with the first intersection point (X1) and the second intersection point (X2), the lane where the road damage is located is numbered R2; if the horizontal coordinate of the center point position (D1) of the lower side of the road damage is larger than the horizontal coordinate of the second intersection point (X2), the lane where the road damage is located is numbered R3.

Images