Accurate assessment method of road marking wear for road maintenance

Abstract

This invention relates to a method for accurately assessing road marking wear for road maintenance. The method primarily involves extracting road markings from multiple images of a road to be interpreted, generating binary images of the road markings; then acquiring a high-resolution map and adjusting the marking vector layer of the high-resolution map according to the camera exterior orientation parameters of each image to be interpreted, thereby generating multiple reference binary images of the road markings that are substantially consistent with the images to be interpreted; finally, a first deep learning model is used to accurately assess the road marking wear rate for each image to be interpreted, its binary images of the road markings, and the reference binary images of the road markings.

Claims

1. A method for accurately assessing road marking wear for road maintenance, executed by at least one computer device with computing power, wherein the method comprises the following steps: (a) acquiring multiple images of a road to be interpreted, wherein each image to be interpreted includes at least the road surface and road markings thereon, and camera exterior orientation parameters; (b) extracting road markings from each image to be interpreted and generating a marking binary image; (c) adjusting the marking vector layer of a high-precision map according to the camera exterior orientation parameters of each image to be interpreted, and generating multiple reference marking binary images that are substantially consistent with the images to be interpreted, wherein the marking area of ​​each reference marking binary image is a complete and unworn marking; and (d) inputting each marking binary image and its corresponding reference marking binary image into a first deep learning model, wherein the first deep learning model evaluates the marking wear rate of the marking area of ​​each marking binary image and its corresponding reference marking binary image.

2. The method for accurately assessing road marking wear as described in claim 1 further includes step (e), extracting the road surface from each of the images to be interpreted and generating a road surface binary image, and in step (d), inputting these road surface binary images into the first deep learning model, wherein the first deep learning model assesses the wear rate of the markings located within the road surface range in the marking binary image based on the road surface range of the road surface binary image.

3. The method for accurately assessing road marking wear as described in claim 2 further includes step (f), reading a map of the road that includes step (a), marking the wear rate of each road segment assessed in step (d) on the map, and marking it on the map with different colors according to the road maintenance grade.

4. The method for accurately assessing road marking wear as described in claim 2, wherein: The images to be interpreted in step (a) above were captured by a mobile vehicle equipped with multiple cameras, a satellite navigation system, an inertial navigation system, and a LiDAR along the road. The camera's exterior orientation parameters include the conversion of the camera coordinate system and the LiDAR coordinate system to the global coordinate system of the satellite navigation system and the inertial navigation system, and are corrected by LiDAR point cloud stitching. The high-precision map in step (c) above contains different datum vector layers. These datum vector layers are combined to form the actual road markings. These datum vector layers are converted into area data according to the line width value in their attribute field based on their center lines, and then projected onto the image space according to the camera's exterior orientation parameters to generate a reference datum binary image that is substantially consistent with each of the images to be interpreted in step (a).

5. The method for accurately assessing road marking wear as described in claim 4, wherein the first deep learning model comprises: a convolutional neural network that receives binary images of the road marking, the road surface, and the reference marking from each of the images to be interpreted; and a second encoder that determines the depth, width, and resolution of the convolutional neural network by a certain ratio, so as to correspond to the number of convolutional layers, the number of channels in each layer, and the size of each of the images to be interpreted, respectively; wherein the convolutional neural network uses the second encoder to determine the proportion of the number of marking pixels retained on each of the images to be interpreted through two fully connected layers, and outputs the value of the road marking wear rate.

6. The method for accurately assessing road marking wear as described in claim 5, wherein in step (d), the first deep learning model calculates the percentage of road marking wear based on the proportion of the number of marking pixels remaining on each of the images to be interpreted, and expresses it using the marking wear rate formula, which is expressed as [(1 - actual number of marking pixels / ideal number of marking pixels) × 100%].

7. The method for accurately assessing road marking wear as described in request item 5, wherein: In step (b), a second deep learning model is used to extract road markings from the image to be interpreted and generate the corresponding binary image of the road markings. The second deep learning model includes: a convolutional neural network that receives road markings extracted from each of the images to be interpreted and generates the corresponding binary image of the road markings. The convolutional neural network includes a first encoder with multiple layers for extracting image features and a decoder for restoring the image feature positions. Connection points are added between the encoders and the decoders to retain more details of the feature images of different sizes of each image to be interpreted; and a second encoder that determines the depth, width, and resolution of the convolutional neural network by a certain ratio to correspond to the number of convolutional layers, the number of channels in each layer, and the size of each input image to be interpreted. In step (e), a third deep learning model is used to extract the road surface from the image to be interpreted and generate a binary image of the road surface. The third deep learning model includes: a convolutional neural network that receives the extracted road surface from each of the images to be interpreted and generates the corresponding binary image of the road surface. The convolutional neural network of the third deep learning model includes a first encoder with multiple layers for extracting image features and a decoder for restoring the image feature positions. Connection points are added between the encoders and the decoders to retain more details of the feature images of different sizes of each image to be interpreted; and a second encoder that determines the depth, width, and resolution of the convolutional neural network of the third deep learning model by a certain ratio to correspond to the number of convolutional layers, the number of channels in each layer, and the size of each input image to be interpreted.

8. The method for accurately assessing road marking wear as described in claim 7, wherein: Each convolutional neural network uses an inverse residual block (MBConv Block) to first increase the number of input channels of the input image to be interpreted using a 1×1 sliding window convolution, then performs depthwise convolution on each channel, then performs average pooling on each channel, then multiplies back with multiple fully connected layers as coefficients, and finally changes the number of channels using a 1×1 tensor convolution and adds the input image to be interpreted through these connection points; and each second encoder uses an EfficientNet-B7 encoder.

9. The method for accurately assessing road marking wear as described in claim 8, wherein: The convolutional neural networks of the second and third deep learning models respectively receive each of the images to be interpreted, process them into 512×512 pixel RGB images, and output 512×512 pixel binary images as extracted marking binary images and road surface binary images; and the convolutional neural network of the first deep learning model receives three 512×512 pixel marking binary images, road surface binary images and reference marking binary images of each of the images to be interpreted.

10. A method for accurately assessing road marking wear as described in any of claims 7 to 9, wherein: The second deep learning model in step (b) is trained for paving mark extraction through the following steps (b1) and (b2); wherein: in step (b1), a large number of road images of different roads are obtained, and the RGB images of each road image and their binary images with paving marks are input into the second deep learning model for the first training; and in step (b2), multiple road images of the road in step (a) are obtained, and the RGB images of each road image and their binary images with paving marks and binary images are input into the second deep learning model for the second training; and the third deep learning model in step (e) is trained for paving mark extraction through the following steps (c1) and (c2); wherein: In step (c1), a large number of road images of different roads are obtained, and the RGB images of each road image and the binary images with road surface annotations are input into the third deep learning model for the first training; and in step (c2), multiple road images of the road in step (a) are obtained, and the RGB images of each road image and the binary images with road surface annotations are input into the third deep learning model for the second training.

11. The method for accurately assessing road marking wear as described in claim 10, wherein: In steps (b1) and (b2), the RGB image and its road marking binary image of each road image are further cropped to include only the road area, and then cropped into multiple sub-images within the cropped area. The boundaries of these sub-images are connected or overlapped, so that each road image generates multiple sets of RGB images and its road marking binary images containing the road. In steps (c1) and (c2), the RGB image and its road surface binary image of each road image are further cropped to include only the road area, and then cropped into multiple sub-images within the cropped area. The boundaries of these sub-images are connected or overlapped, so that each road image generates multiple sets of RGB images and its road surface binary images containing the road.

12. The method for accurately assessing road marking wear as described in claim 5, wherein the first deep learning model is trained by generating a number of simulated images using a data augmentation method, wherein the data augmentation method includes the following steps: (d1) obtaining a road image and generating an ideal road marking binary image and a road surface binary image after annotation; (d2) using different visual noise as masks and performing masking processing on the ideal road marking binary image to obtain road marking binary images with different wear corresponding to different masks; and (d3) generating a reference road marking binary image from the high-precision map to combine the simulated images of road markings with different wear and road markings with different offsets.

13. The method for accurately assessing road marking wear as described in any one of claims 1 to 9, wherein the images to be interpreted in step (a) are selected from multiple road images taken consecutively, and the road markings in any two adjacent images to be interpreted overlap.

Images