A method, device, equipment, medium and product for extracting highway markings

Abstract

The present invention discloses a method, device, equipment, medium and product for extracting highway markings, relating to the technical field of point cloud data analysis. After obtaining the point cloud data collected from the target highway, the method first uses a deep learning-based road surface point cloud segmentation network to extract the road surface point cloud data, and then successively performs point cloud rasterization processing and binarization processing to obtain a road surface intensity binarized image. Then, foreground connected component analysis processing is performed on the binarized image to obtain multiple clustering objects, and for each clustering object, according to the geometric information of the corresponding minimum bounding rectangle, it is judged whether the marking type of the corresponding object is a solid line type marking. If not, an edge template matching algorithm is also used to further identify the corresponding non-solid marking type. Finally, all clustering objects with determined marking types are summarized to obtain the marking extraction result of the target highway, so that the extraction result has high reliability and high stability.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data analysis, and particularly relates to a method, device, equipment, medium and product for extracting highway markings. Background Art

[0002] Markings are typical elements for the digitization and informatization of highway assets, and are also an important basis for realizing highway alignment extraction and road modeling. Therefore, extracting the accurate position of the markings and their semantic recognition results can provide important support for the digital operation, maintenance and intelligent management of highways.

[0003] In recent years, three-dimensional point cloud data obtained by airborne LiDAR (Light Laser Detection and Ranging) has been widely used in real-scene three-dimensional city modeling and more refined DEM (Digital Elevation Model) generation. The mobile laser scanning (MLS) system can efficiently, reliably and low-cost obtain three-dimensional point cloud data with high density, high precision and many details in the road scene, and has become an effective data source for road marking semantic recognition and update.

[0004] Highway markings are highly reflective materials attached to asphalt or concrete roads. Compared with road surface point clouds, the intensity value of the markings becomes a significant feature to distinguish them from the road surface. According to semantic knowledge (such as shape) and the intensity attributes of MLS point clouds, the existing MLS point cloud road marking extraction methods are mainly divided into two categories: (1) methods based on two-dimensional images; (2) methods based on three-dimensional point clouds.

[0005] Currently, the method based on two-dimensional images will inevitably cause information loss in the process of converting unstructured point cloud data into raster images, and it is difficult to accurately extract complex types of markings. The method based on three-dimensional point clouds has developed to introduce deep learning methods. However, the deep learning-based method not only requires a large number of samples, but also faces challenges such as a wide variety of scene elements, scale differences and sample imbalance in large-scale MLS point cloud scenes. In addition, most of the existing MLS point cloud road marking extraction schemes focus on the extraction and classification of road markings in vehicle-mounted point clouds, and few studies not only digitize the extracted road markings, but also conduct degradation analysis on the markings for maintenance needs. Summary of the Invention

[0006] The object of the present invention is to provide a method, device, computer device, computer-readable storage product and computer program product for extracting highway markings, so as to solve the problems existing in the existing MLS point cloud road marking extraction scheme, such as easy information loss, difficulty in accurately extracting complex types of markings, and challenges in facing large-scale MLS point cloud scenarios, including a wide variety of scene elements, scale differences, and sample imbalance.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, a method for extracting highway markings is provided, including:

[0009] Obtain point cloud data collected by a mobile laser scanning device for a target highway;

[0010] Use a deep learning-based road surface point cloud segmentation network to perform road surface point cloud extraction processing on the point cloud data to obtain road surface point cloud data;

[0011] Perform point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity feature image;

[0012] Perform binarization processing on the road surface intensity feature image to obtain a road surface intensity binary image;

[0013] Perform foreground connected component analysis processing on the road surface intensity binary image to obtain a plurality of clustering objects;

[0014] For each clustering object among the plurality of clustering objects, according to the geometric information of the corresponding minimum bounding rectangle, determine whether the marking type of the corresponding clustering object is a solid line marking. If not, further use an edge template matching algorithm to identify the corresponding non-solid marking type, where the non-solid marking type refers to other marking types except solid line markings;

[0015] Summarize all clustering objects with determined marking types to obtain the marking extraction result for the target highway.

[0016] Based on the above invention content, a new solution for extracting highway markings driven by point cloud data is provided. That is, after obtaining the point cloud data collected from the target highway by a mobile laser scanning device, first, a deep learning-based road surface point cloud segmentation network is used to extract the road surface point cloud data. Then, through point cloud rasterization processing and binarization processing in sequence, a road surface intensity binary image is obtained. Then, foreground connected component analysis processing is performed on the binary image to obtain multiple clustering objects. For each clustering object, according to the geometric information of the corresponding minimum bounding rectangle, it is judged whether the marking type of the corresponding object is a solid line marking. If not, an edge template matching algorithm is also used to further identify the corresponding non-solid marking type. Finally, all clustering objects with determined marking types are summarized to obtain the marking extraction result for the target highway. In this way, the advantages of existing methods based on two-dimensional images and three-dimensional point clouds can be combined, avoiding information loss, accurately extracting complex types of markings, and being able to easily challenge the situation of a large variety of scene elements, scale differences, and sample imbalance in the face of large-scale MLS point cloud scenarios, making the extraction result highly reliable and stable, and facilitating practical application and promotion.

[0017] In a possible design, a deep learning-based road surface point cloud segmentation network is used to perform road surface point cloud extraction processing on the point cloud data to obtain road surface point cloud data, including:

[0018] Perform noise filtering preprocessing on the point cloud data to obtain denoised point cloud data;

[0019] Use a deep learning-based road surface point cloud segmentation network to perform road surface point cloud extraction processing on the denoised point cloud data to obtain road surface point cloud data, where the road surface point cloud segmentation network is a certain network with the highest road surface point cloud segmentation accuracy among the PointNet network, PointNet++ network, and RandLA-Net network pre-trained based on a road surface point sample annotation set.

[0020] In a possible design, perform point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity feature image, including:

[0021] Project each road surface point in the road surface point cloud data onto a rasterized two-dimensional road surface;

[0022] For each grid on the two-dimensional road surface, if there is no projected road surface point in the corresponding grid, set the pixel value of the corresponding pixel point to zero, and if there is at least one projected road surface point in the corresponding grid, set the pixel value of the corresponding pixel point to the average laser intensity of the at least one road surface point;

[0023] Obtain a road surface intensity feature image according to the pixel values of all pixel points.

[0024] In a possible design, the pavement strength feature image is binarized to obtain a pavement strength binarized image, including:

[0025] The pavement strength feature image is divided into blocks to obtain a plurality of sub-images;

[0026] For each sub-image in the plurality of sub-images, the maximum cross-entropy threshold segmentation algorithm is used to determine the corresponding optimal pixel threshold;

[0027] For each of the sub-images, the pixel points in the corresponding image with pixel values greater than the corresponding optimal pixel threshold are used as foreground pixel points, and the pixel points in the corresponding image with pixel values less than or equal to the corresponding optimal pixel threshold are used as background pixel points, to obtain a binarized sub-image that aggregates all foreground pixel points and all background pixel points;

[0028] The binarized sub-images are subjected to image stitching processing to obtain a pavement strength binarized image.

[0029] In a possible design, for each clustering object in the plurality of clustering objects, according to the geometric information of the corresponding minimum bounding rectangle, it is determined whether the marking type of the corresponding clustering object is a solid line marking, including:

[0030] For a certain clustering object in the plurality of clustering objects, a corresponding minimum bounding rectangle is generated, and the geometric information of the minimum bounding rectangle is calculated, wherein the geometric information includes the rectangle width , the rectangle aspect ratio and the object-rectangle area ratio , and the object-rectangle area ratio refers to the area ratio of the clustering object to the minimum bounding rectangle;

[0031] It is determined whether the geometric information of the minimum bounding rectangle of the certain clustering object satisfies the following conditions:

[0032]

[0033] In the formula, represents the standard width of the solid line of the target highway, represents a preset aspect ratio threshold, represents a preset area ratio threshold, represents a preset allowable difference threshold;

[0034] If it is determined that the geometric information of the minimum bounding rectangle of the certain clustering object satisfies the conditions, it is determined that the marking type of the certain clustering object is a solid line marking, otherwise it is determined that the marking type of the certain clustering object is not a solid line marking;

[0035] And / or, for each clustering object among the multiple clustering objects whose marking type is not a solid marking type, the edge template matching algorithm is used to further identify the corresponding non-solid marking type, including:

[0036] For any clustering object among the multiple clustering objects whose marking type is not a solid marking type, according to the corresponding minimum bounding rectangle, the corresponding rectangular sub-image is intercepted from the pavement intensity binary image;

[0037] Obtain all template instance images of non-solid markings, where the non-solid markings refer to highway markings of non-solid marking types, the non-solid marking types refer to other marking types except solid marking types, and the marking types of all non-solid markings are different from each other. The template instance image is a binary image with template marking instance pixel points as foreground pixel points;

[0038] For each non-solid marking among all non-solid markings, use the similarity measurement method to compare all foreground edge pixel points of the corresponding template instance image and the rectangular sub-image, and obtain the corresponding similarity measurement index value;

[0039] Determine a certain non-solid marking with the largest similarity measurement index value from all non-solid markings;

[0040] Judge whether the largest similarity measurement index value reaches a preset index threshold. If so, use the non-solid marking type of the certain non-solid marking as the marking type of the any clustering object.

[0041] In a possible design, summarize all clustering objects with determined marking types to obtain the marking extraction result of the target highway, including:

[0042] For any clustering object determined to be a certain non-solid marking type, according to the corresponding minimum bounding rectangle, intercept the corresponding rectangular binary sub-image from the pavement intensity binary image;

[0043] Obtain the template instance image corresponding to the certain non-solid marking type, where the template instance image is a binary image with template marking instance pixel points as foreground pixel points;

[0044] Use the similarity measurement method to compare all foreground edge pixel points of the template instance image and the rectangular binary sub-image, and obtain the marking contour integrity degradation index value;

[0045] If the value of the marking profile integrity degradation index is greater than or equal to a preset first degradation index threshold, it is determined that the first marking maintenance strategy for any one of the clustering objects is not to maintain, otherwise it is determined that the first marking maintenance strategy for any one of the clustering objects is to maintain;

[0046] Take the certain non-real marking type, the rectangular binary sub-image, the template instance image, the marking profile integrity degradation index value, the first marking maintenance strategy, and / or the geographical location corresponding to any one of the clustering objects as the marking extraction result for the target highway and for any one of the clustering objects;

[0047] And / or, summarize all clustering objects with the determined marking types to obtain the marking extraction result for the target highway, including:

[0048] For any one of the clustering objects determined to be a certain non-real marking type, intercept the corresponding rectangular binary sub-image from the pavement strength binary image according to the corresponding minimum bounding rectangle;

[0049] Obtain the template instance image corresponding to the certain non-real marking type, where the template instance image is a binary image with template marking instance pixel points as foreground pixel points;

[0050] According to all pixel values of the template instance image and the rectangular binary sub-image, calculate the marking radiation consistency degradation index value according to the following formula :

[0051]

[0052] In the formula, represents the pixel point in the template instance image, represents the radiation range, represents the radiation pixel point in the rectangular binary sub-image and corresponding to the pixel point ; represents the pixel point 's pixel value, represents the radiation pixel point 's pixel value;

[0053] If the value of the marking radiation consistency degradation index is greater than or equal to a preset second degradation index threshold, it is determined that the second marking maintenance strategy for any one of the clustering objects is not to maintain, otherwise it is determined that the second marking maintenance strategy for any one of the clustering objects is to maintain;

[0054] Take the certain non-solid line type, the rectangular binary sub-image, the template instance image, the line radiation consistency degradation index value, the second line maintenance strategy, and / or the geographical location corresponding to any one of the clustering objects as the line extraction result for the target highway and for any one of the clustering objects.

[0055] In a second aspect, a highway line extraction device is provided, including a point cloud data acquisition unit, a road surface point cloud extraction unit, a point cloud rasterization processing unit, an image binarization processing unit, a connected region analysis processing unit, a line type recognition unit, and a line extraction summary unit that are sequentially communicatively connected;

[0056] The point cloud data acquisition unit is configured to acquire point cloud data collected by a mobile laser scanning device for a target highway;

[0057] The road surface point cloud extraction unit is configured to perform road surface point cloud extraction processing on the point cloud data by using a deep learning-based road surface point cloud segmentation network to obtain road surface point cloud data;

[0058] The point cloud rasterization processing unit is configured to perform point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity feature image;

[0059] The image binarization processing unit is configured to perform binarization processing on the road surface intensity feature image to obtain a road surface intensity binarized image;

[0060] The connected region analysis processing unit is configured to perform foreground connected region analysis processing on the road surface intensity binarized image to obtain a plurality of clustering objects;

[0061] The line type recognition unit is configured to, for each clustering object among the plurality of clustering objects, determine whether the line type of the corresponding clustering object is a solid line type according to the geometric information of the corresponding minimum bounding rectangle. If not, further identify the corresponding non-solid line type by using an edge template matching algorithm, where the non-solid line type refers to other line types except the solid line type;

[0062] The line extraction summary unit is configured to summarize all clustering objects with determined line types to obtain a line extraction result for the target highway.

[0063] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a transceiver that are sequentially communicatively connected, where the memory is configured to store a computer program, the transceiver is configured to send and receive messages, and the processor is configured to read the computer program and execute the highway line extraction method as described in the first aspect or any possible design in the first aspect.

[0064] Fourthly, the present invention provides a computer-readable storage product, on which instructions are stored. When the instructions run on a computer, the method for extracting highway markings as described in the first aspect or any possible design in the first aspect is executed.

[0065] Fifthly, the present invention provides a computer program product, including a computer program or instructions. When the computer program or the instructions are executed by a computer, the method for extracting highway markings as described in the first aspect or any possible design in the first aspect is implemented.

[0066] Beneficial effects of the above solutions:

[0067] (1) The present invention provides a new solution for extracting highway markings driven by point cloud data. That is, after obtaining the point cloud data collected by a mobile laser scanning device for a target highway, first, a deep learning-based road surface point cloud segmentation network is used to extract the road surface point cloud data. Then, through point cloud rasterization processing and binary processing in sequence, a road surface intensity binary image is obtained. Then, foreground connected component analysis is performed on the binary image to obtain multiple clustering objects. For each clustering object, according to the geometric information of the corresponding minimum bounding rectangle, it is judged whether the marking type of the corresponding object is a solid line marking. If not, an edge template matching algorithm is further used to identify the corresponding non-solid marking type. Finally, all clustering objects with determined marking types are summarized to obtain the marking extraction result for the target highway. In this way, the advantages of existing methods based on two-dimensional images and three-dimensional point clouds can be combined, avoiding information loss, accurately extracting complex types of markings, and easily challenging the situation of various scene elements, scale differences, and sample imbalance in the face of large-scale MLS point cloud scenarios, making the extraction result highly reliable and stable, and facilitating practical application and promotion.

[0068] (2) Degradation analysis can also be performed on the identified markings, which is convenient for highway maintenance departments to efficiently locate the markings with wear or partial loss and implement maintenance. Description of the Drawings

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0070] Figure 1 It is a schematic flowchart of the method for extracting highway markings provided by the embodiment of the present application.

[0071] Figure 2 This is a comparison example diagram of some point cloud data before and after annotation provided in the embodiment of the present application, where: Figure 2 (a) shows an example image of some point cloud data before annotation. Figure 2 (b) in the figure shows an example image after some point cloud data is annotated.

[0072] Figure 3 This is an example diagram of a road surface strength characteristic image obtained based on road surface point cloud data provided in an embodiment of the present application, wherein: Figure 3 (a) shows an example of road point cloud data. Figure 3 (b) in the figure shows an example of a road surface strength feature image.

[0073] Figure 4 This is an example diagram of obtaining a pavement strength binary image based on a pavement strength feature image provided in an embodiment of the present application.

[0074] Figure 5 The template example images of the short thick dashed line marking, the long thin dashed line marking, the straight arrow type marking, the left arrow type marking, the right arrow type marking and the straight-right turn arrow type marking provided in the embodiment of the present application, wherein: Figure 5 (a) shows an example of a template instance image of a short thick dashed line. Figure 5 (b) shows an example of a template instance image of a long and thin dashed line. Figure 5 (c) shows an example of a template instance image of a straight arrow-shaped marking line. Figure 5 (d) shows an example of a template instance image of a left arrow-shaped marking line. Figure 5 (e) in the figure shows an example of a template instance image of a right arrow type marking line. Figure 5 (f) shows an example of a template instance image of a straight-right turn arrow type marking line.

[0075] Figure 6 This is an example diagram of the line extraction results and local details of the Duwen Expressway section provided in an embodiment of the present application.

[0076] Figure 7 This is an example diagram of the line extraction results and local details of the Ya'an-Xichang Expressway section provided in an embodiment of the present application.

[0077] Figure 8 An example diagram of a marking extraction result including a marking profile integrity degradation index value and a first marking maintenance strategy provided in an embodiment of the present application.

[0078] Figure 9 An example diagram of a marking extraction result including a marking radiation consistency degradation index value and a second marking maintenance strategy provided in an embodiment of the present application.

[0079] Figure 10 This is a schematic structural diagram of the highway marking extraction device provided by the embodiment of the present application.

[0080] Figure 11 This is a schematic structural diagram of the computer device provided by the embodiment of the present application. Detailed implementation manners

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the present invention in combination with the accompanying drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the accompanying drawings is only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these embodiments without creative efforts. It should be noted here that the description of these embodiment modes is used to help understand the present invention, but does not constitute a limitation to the present invention.

[0082] It should be understood that although terms such as first and second etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object can be called the second object, and similarly, the second object can be called the first object, without departing from the scope of the exemplary embodiments of the present invention.

[0083] It should be understood that for the term "and / or" that may appear herein, it is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, B exists alone, or A and B exist simultaneously, etc. three situations; and for example, A, B and / or C can represent any one of A, B and C or any combination of them; for the term " / and" that may appear herein, it is a description of another association object relationship, indicating that two relationships can exist. For example, A / and B can represent: A exists alone or A and B exist simultaneously, etc. two situations; in addition, for the character " / " that may appear herein, generally it represents that the front and rear associated objects are an "or" relationship.

[0084] Embodiment

[0085] As Figure 1 shown, the highway marking extraction method provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device with certain computing resources, such as an electronic device such as a data processing server. As Figure 1 shown, the highway marking extraction method can include, but is not limited to, the following steps S1 to S7.

[0086] S1. Obtain the point cloud data collected by the mobile laser scanning device for the target highway.

[0087] In the step S1, the target highway is a section of highway for which road markings are to be extracted. The mobile laser scanning device can specifically be an existing vehicle-mounted laser scanning device, so that during the process of the vehicle equipped with the vehicle-mounted laser scanning device driving through the target highway, the vehicle front road surface area or the vehicle rear road surface area is scanned by the vehicle-mounted laser scanning device, thereby obtaining the point cloud data (i.e., MLS point cloud data), or it can also be an existing airborne lidar device.

[0088] S2. Use a deep learning-based road surface point cloud segmentation network to perform road surface point cloud extraction processing on the point cloud data to obtain road surface point cloud data.

[0089] In the step S2, it is considered that highways are different from general urban roads: usually, there are no prominent features such as road embankments at the edges of highways, making it difficult to extract highway boundaries through sudden changes in road surface elevation. Therefore, it is necessary to introduce a deep learning method for point clouds to perform road surface point cloud extraction. Preferably, use a deep learning-based road surface point cloud segmentation network to perform road surface point cloud extraction processing on the point cloud data to obtain road surface point cloud data, including but not limited to the following steps S21 to S22.

[0090] S21. Perform noise filtering preprocessing on the point cloud data to obtain denoised point cloud data.

[0091] In the step S21, the specific process of the noise filtering preprocessing includes but not limited to: first, for each point in the point cloud data, if the total number of other points in the spherical space with the corresponding point as the center of the sphere and a preset value (such as 0.05M) as the spherical radius is less than the preset point number threshold (such as 10), then the corresponding point is regarded as a noise point and removed; then use the movement trajectory data of the mobile laser scanning device (i.e., the movement trajectory data of a vehicle or a drone, etc.) to remove a large number of points outside the road in the remaining point cloud data to obtain denoised point cloud data.

[0092] S22. Use a deep learning-based road surface point cloud segmentation network to perform road surface point cloud extraction processing on the denoised point cloud data to obtain road surface point cloud data, where the road surface point cloud segmentation network can be but not limited to a certain network with the highest road surface point cloud segmentation accuracy among the PointNet network, PointNet++ network, RandLA-Net network, etc. that are pre-trained based on a road surface point sample annotation set.

[0093] In the step S22, the PointNet network is the earliest deep network model that attempts to directly use unstructured point clouds as learning primitives without learning through indirect representations of point clouds (such as two-dimensional multi-views or three-dimensional voxels, etc.); it mainly solves problems such as the disorder, permutation invariance, and rotation invariance of point clouds through modules such as symmetric function aggregation, multi-layer perceptron (MLP), and spatial transformer networks (STN); although the PointNet network can overcome the above problems and directly learn from the original point cloud to achieve point cloud classification and segmentation, this network only processes each independent point and cannot capture the neighborhood information of points, restricting its receptive field. The PointNet++ network is a hierarchical version of the PointNet network. It uses the farthest point sampling method for downsampling step by step, increasing the utilization of neighborhood information, effectively using context semantic information, and interpolating semantic labels to the original input point cloud through the k-nearest neighbor (KNN) classification algorithm; the PointNet++ network not only solves the problem of uneven point cloud sampling but also makes the learned network more robust by hierarchically learning local features. The RandLA-Net network is a lightweight network for large-scale point cloud processing. This network uses the random point sampling method to replace the farthest point sampling method of the PointNet++ network, captures and retains local geometric features through the local feature aggregation module, and achieves significant improvements in storage and computing. The road surface point sample annotation set is the per-point sample annotation result of known point cloud data based on the segmentation categories (specifically, two categories: road surface points and non-road surface points), and some annotation results are as shown in Figure 2 as shown in (b) of. Using the road surface point sample annotation set as the learning sample, train the PointNet network, the PointNet++ network, and the RandLA-Net network respectively (the specific training process is the existing conventional calibration modeling process) and compare the test results. Finally, select the network with the best road surface point cloud segmentation accuracy to perform road surface point cloud extraction processing on the denoised point cloud data to obtain road surface point cloud data. For example, the comparison results of using the PointNet network, the PointNet++ network, and the RandLA-Net network trained under the same sample size and parameter conditions to perform road surface point cloud segmentation on a certain test data are shown in Table 1 below:

[0094] Table 1. Comparison results of using PointNet, PointNet++, and RandLA-Net for road surface point cloud segmentation of a certain test data

[0095] Network type Number of input points Pavement point extraction accuracy (%) PointNet network 8192 97.86 PointNet++ network 8192 99.62 RandLA network 65536 99.27

[0096] Based on the above comparison results, it can be seen that the extraction accuracy of road surface points (i.e., the segmentation accuracy of road surface point clouds) by the PointNet++ network reaches 99.62%, which is better than 97.86% of the PointNet network and 99.27% of the RandLA-Net network. Therefore, in this embodiment, the PointNet++ network can be selected to perform road surface point cloud extraction processing on the denoised point cloud data.

[0097] S3. Perform point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity feature image.

[0098] In the step S3, the point cloud rasterization processing is a process of using point cloud attribute information as the image gray value and generating a feature image according to the three-dimensional point cloud space range and the image resolution. Preferably, performing point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity feature image includes, but is not limited to, the following steps S31 to S33.

[0099] S31. Project each road surface point in the road surface point cloud data onto a rasterized two-dimensional road surface.

[0100] In the step S31, assume that a certain road surface point in the road surface point cloud data is , where represents the three-dimensional coordinates of the certain road surface point, and represents the laser intensity value at the certain road surface point; projecting the certain road surface point onto the two-dimensional road surface, the projected planar position can be obtained as , where represents the scale coefficient relationship between the certain road surface point and the projection point; further, the grid to which the projection point of the certain road surface point belongs can be represented by the following formula:

[0101]

[0102] In the formula, represents the corresponding pixel coordinates of the grid, represents the size of the grid (its size determines the size and detail level of the generated image; considering the actual point cloud interval collected is 0.04M, the size can be designed as 0.08M for example), represents the minimum abscissa in all projection positions, represents the minimum ordinate in all projection positions.

[0103] S32. For each grid on the two-dimensional road surface, if there is no projected road surface point in the corresponding grid, the pixel value of the corresponding pixel point is set to zero; if there is at least one projected road surface point in the corresponding grid, the pixel value of the corresponding pixel point is set to the average laser intensity of the at least one road surface point.

[0104] S33. Obtain the road surface intensity feature image according to the pixel values of all pixel points.

[0105] Based on the above steps S31 - S33, for example, according to the Figure 3 road surface point cloud data shown in (a) below, the road surface intensity feature image shown in (b) below can be obtained. Figure 3

[0106] S4. Perform binarization processing on the road surface intensity feature image to obtain the road surface intensity binary image.

[0107] In the above step S4, considering that during the process of point cloud data acquisition, the laser intensity value not only depends on the material of the measured object, but is also affected by factors such as scanning distance and incident angle, it is difficult to accurately extract road markings only by single-threshold segmentation. Therefore, in this embodiment, the idea of divide and conquer is preferably adopted to first divide the road surface intensity feature image into blocks, so that the incident angle and span of each sub-image after division are greatly reduced, thereby effectively overcoming the problems of different scanning modes or large intensity differences. That is, preferably, the binarization processing of the road surface intensity feature image to obtain the road surface intensity binary image includes but is not limited to the following steps S41 - S44.

[0108] S41. Perform block processing on the road surface intensity feature image to obtain a plurality of sub-images.

[0109] In the above step S41, the block size of the block processing can be preset. For example, a sub-image is obtained by dividing every 3M along the route direction.

[0110] S42. For each sub-image in the plurality of sub-images, use the maximum cross-entropy threshold segmentation algorithm to determine the corresponding optimal pixel threshold.

[0111] In the above step S42, the maximum cross-entropy threshold segmentation algorithm is used to unsupervised select an optimal threshold to maximize the sum of the entropies of the background and foreground parts: Given a specific threshold , , represents the total number of pixel points. For the two image regions and segmented by this threshold, the corresponding probability densities are estimated according to the following formula:

[0112] ​

[0113]

[0114] Wherein, represents the probability of background pixels segmented based on the threshold and represents the cumulative probability of background pixels segmented based on the threshold and represents the cumulative probability of foreground pixels segmented based on the threshold , and the sum of the two is 1. The entropy corresponding to the background and the entropy corresponding to the foreground are respectively expressed as follows:

[0115]

[0116] Under the above threshold, the total entropy of the image is . Finally, calculate the total entropy of the image under all segmentation thresholds, find the maximum entropy, and use the segmentation threshold corresponding to the maximum entropy as the final optimal threshold of the pixels.

[0117] S43. For each of the sub-images, use the pixel points in the corresponding image whose pixel values are greater than the corresponding optimal pixel threshold as foreground pixel points, and use the pixel points in the corresponding image whose pixel values are less than or equal to the corresponding optimal pixel threshold as background pixel points, to obtain a binary sub-image that aggregates all foreground pixel points and all background pixel points.

[0118] S44. Perform image stitching processing on all binary sub-images to obtain a binary image of the road surface intensity.

[0119] Based on the above steps S41 to S44, for example, according to the road surface intensity feature image shown in (b) of Figure 3 , a binary image of the road surface intensity as shown in Figure 4 can be obtained.

[0120] S5. Perform foreground connected component analysis processing on the binary image of the road surface intensity to obtain multiple clustering objects.

[0121] In the step S5, since in the binary image of the road surface intensity, foreground pixels are considered to be the markings to be extracted, but the above marking pixels have not been objectified and different types of markings cannot be further identified, so this step needs to use connected component analysis to achieve clustering of different marking objects. In addition, the specific process of the foreground connected component analysis processing is a prior art and will not be elaborated here.

[0122] S6. For each of the multiple clustering objects, based on the geometric information of the corresponding minimum bounding rectangle, determine whether the marking type of the corresponding clustering object is a solid marking line. If not, further identify the corresponding non-solid marking type using an edge template matching algorithm, where the non-solid marking type refers to other marking types except the solid marking line type.

[0123] In step S6, specifically, for each of the multiple clustering objects, based on the geometric information of the corresponding minimum bounding rectangle, determine whether the marking type of the corresponding clustering object is a solid marking line, including but not limited to the following steps S611 to S613.

[0124] S611. For a certain clustering object among the multiple clustering objects, generate the corresponding minimum bounding rectangle and calculate the geometric information of the minimum bounding rectangle, where the geometric information includes but is not limited to the rectangle width , the rectangle aspect ratio and the object-rectangle area ratio , and the object-rectangle area ratio refers to the area ratio of the clustering object to the minimum bounding rectangle.

[0125] In step S611, the rectangle width , the rectangle aspect ratio and the object-rectangle area ratio can be calculated conventionally according to the shape of the minimum bounding rectangle and the shape of the clustering object. In addition, the geometric information may also include but is not limited to the rectangle length .

[0126] S612. Determine whether the geometric information of the minimum bounding rectangle of the certain clustering object satisfies the following conditions:

[0127]

[0128] In the formula, represents the standard width of the solid line of the target highway, represents a preset aspect ratio threshold, represents a preset area ratio threshold, represents a preset allowable difference threshold.

[0129] In step S612, the standard width of the solid line can be obtained in advance based on highway road specifications. is a constraint on the aspect ratio of the minimum bounding rectangle of the marking candidate area. For the aspect ratio of the solid marking line, a relatively large threshold should be set, for example, set to 8M; represents the limitation on the clustering rectangularity; It reflects the allowable difference in width between the clustered object and the design dimension. and are both constraint coefficients, which are related to the density of the clustered point cloud and the integrity of the point cloud. Generally, they can be set to 0.6, 0.2.

[0130] S613. If it is determined that the geometric information of the minimum circumscribed rectangle of a certain clustered object satisfies the condition, then it is determined that the marking type of the certain clustered object is a solid line marking; otherwise, it is determined that the marking type of the certain clustered object is not a solid line marking.

[0131] In step S6, the non-solid marking type may include, but is not limited to, short and thick dashed line markings, long and thin dashed line markings, straight arrow markings, left arrow markings, right arrow markings, straight-left turn arrow markings, straight-right turn arrow markings, left turn U-turn markings, and / or right turn U-turn markings, etc. Specifically, for each clustered object among the multiple clustered objects whose marking type is not a solid line marking, the edge template matching algorithm is used to further identify the corresponding non-solid marking type, including but not limited to the following steps S621 to S625.

[0132] S621. For any clustered object among the multiple clustered objects whose marking type is not a solid line marking, according to the corresponding minimum circumscribed rectangle, the corresponding rectangular sub-image is intercepted from the pavement strength binary image.

[0133] S622. Obtain the template instance images of all non-solid markings, where the non-solid markings refer to the highway markings of non-solid marking types, the non-solid marking types refer to other marking types except solid line markings, the marking types of all non-solid markings are different from each other, and the template instance images are binary images with the template marking instance pixel points as foreground pixel points.

[0134] In step S622, all non-solid markings include, but are not limited to, short and thick dashed line markings, long and thin dashed line markings, straight arrow markings, left arrow markings, right arrow markings, straight-left turn arrow markings, straight-right turn arrow markings, left turn U-turn markings, and / or right turn U-turn markings, etc. Their template instance images can be designed in advance based on the highway road specifications, as Figure 5 shown.

[0135] S623. For each non-solid marking among all non-solid markings, use the similarity measurement method to compare all foreground edge pixel points of the corresponding template instance image and the rectangular sub-image, and obtain the corresponding similarity measurement index value.

[0136] In the step S623, the similarity measurement method is used to calculate the sum of the dot products of all the normalized gradient vectors of the template instance image, and search for the image (i.e., search for the rectangular sub-image) at all points in the model data set, and the score of each edge point in the search image is calculated as follows:

[0137]

[0138] In the formula, and respectively represent the gradient values of the template instance image in the x direction and the y direction, and respectively represent the gradient values of the search image in the x direction and the y direction at the corresponding points, represents the total number of edge points, and thus the similarity measurement index value can be obtained based on the above scores of all points by conventional calculation (such as taking the average). In addition, if there is a perfect match between the template instance image and the search image, the similarity measurement index value will be 1, specifically corresponding to the visible object part in the search image, and if there is no object in the search image, the similarity measurement index value will be 0.

[0139] S624. Determine a certain non-real marking line with the maximum similarity measurement index value from all the non-real marking lines.

[0140] S625. Determine whether the maximum similarity measurement index value reaches a preset index threshold. If so, use the non-real marking line type of the certain non-real marking line as the marking line type of any clustering object.

[0141] In the step S625, the index threshold can be determined based on the results of multiple tests. If it is determined that the maximum similarity measurement index value is less than the index threshold, it can be determined that any clustering object is a non-marking line. Based on the foregoing steps S621 - S625, the influence of marking line missing and intensity noise can be overcome to a certain extent, making the subsequent extraction result more robust to the original data noise.

[0142] S7. Aggregate all the clustering objects with the determined marking line types to obtain the marking line extraction result for the target highway.

[0143] In the step S7, the marking line extraction result specifically includes but is not limited to the marking line type, rectangular sub-image, template instance image, and / or geographical location corresponding to the clustering objects with the determined marking line types, etc., which can provide important support for the digital operation, maintenance, and intelligent management of highways.

[0144] To verify the effectiveness of the freeway marking extraction method described based on the foregoing steps S1 to S7, in this embodiment, point cloud data of two freeways located at both ends of the Duwen Freeway and the Yaxi Freeway in Sichuan Province (their lengths are 3.964 km and 12.854 km respectively, where the former involves an urban suburban scene with overpasses and the latter involves a mountain and canyon area scene) are respectively collected to evaluate the above-mentioned method; the mobile measurement acquisition system used for collecting the foregoing point cloud data integrates devices such as a laser scanner, a panoramic camera, an IMU, a GNSS, an odometer, and computing control. Among them, the specific parameters of the laser scanner are shown in Table 2 below:

[0145] Table 2. Data Acquisition Parameters of Vehicle-mounted Laser Scanner

[0146]

[0147] Based on the foregoing point cloud data, after implementing the above-mentioned freeway marking extraction method using a C++ program, the marking extraction results and local detail maps of the two sections of freeways as shown in Figure 6 and Figure 7 can be obtained. To quantitatively evaluate the recognition results, in this embodiment, the following precision , recall and F1 score are also used as evaluation indicators:

[0148]

[0149]

[0150]

[0151] In the formula, represents the number of correctly recognized markings, represents the number of incorrectly recognized markings, represents the number of unrecognized markings. Solid-line markings are counted according to the recognition length (M or meter), and non-solid-line markings are counted according to the number of categories. The evaluation index data of the marking extraction results of the two sections of freeways obtained based on the foregoing evaluation indicators are shown in Table 3 and Table 4 below:

[0152] Table 3. Marking Recognition Accuracy of Duwen Freeway Section (GT represents the true value of marking recognition)

[0153]

[0154] Table 4. Marking Recognition Accuracy of Yaxi Freeway Section (GT represents the true value of marking recognition)

[0155]

[0156] It can be seen from the evaluation results shown in Table 3 and Table 4 above that the average recall rate and precision rate of the solid and non-solid markings on the Duwen Expressway section are 96.2% and 93.6% respectively; similarly, the average recall rate and precision rate of the solid and non-solid markings on the Yaxi Expressway section are 94.7% and 94.3% respectively; the accuracy rate, recall rate and F1 score of the recognition of various markings in the two expressway sections all reach more than 91% and the differences are small, indicating that the method in this paper has high reliability and high stability.

[0157] In step S7, in order to quantitatively describe the degradation of the markings, this embodiment also constructs a marking contour integrity degradation index and / or a marking radiation consistency degradation index respectively to perform degradation analysis on the recognized markings, so as to characterize the health status of the markings from two dimensions of the geometric appearance and / or radiation information of the markings. Specifically, all clustering objects with the determined marking types are summarized to obtain the marking extraction result of the target expressway, including but not limited to the following steps S711 to S715.

[0158] S711. For any clustering object determined to be a certain non-solid marking type, according to the corresponding minimum circumscribed rectangle, intercept the corresponding rectangular binary sub-image from the pavement strength binary image.

[0159] S712. Obtain a template instance image corresponding to the certain non-solid marking type, where the template instance image is a binary image with the template marking instance pixel points as foreground pixel points.

[0160] S713. Use a similarity measurement method to compare all foreground edge pixel points of the template instance image and the rectangular binary sub-image to obtain the marking contour integrity degradation index value.

[0161] In step S713, the marking contour integrity degradation index value is the similarity measurement index value obtained in step S623, so its specific process can refer to the aforementioned step S623 and will not be elaborated here. In addition, the value range of the marking contour integrity degradation index value is [0, 1], and the lower its value, the more serious the wear or loss of the marking edge and the more maintenance is required.

[0162] S714. If the marking contour integrity degradation index value is greater than or equal to a preset first degradation index threshold, determine that the first marking maintenance strategy for the arbitrary clustering object is not to maintain, otherwise determine that the first marking maintenance strategy for the arbitrary clustering object is to maintain.

[0163] In step S714, the first degradation index threshold can be specifically set based on the results of multiple tests under various conditions, for example, set to 0.618.

[0164] S715. Take the certain non-solid marking type, the rectangular binary sub-image, the template instance image, the marking contour integrity degradation index value, the first marking maintenance strategy, and / or the geographical location corresponding to any one of the clustering objects as the marking extraction result for the target highway and for any one of the clustering objects.

[0165] In step S715, for example, the marking extraction result including non-solid marking type, rectangular binary sub-image, template instance image, marking contour integrity degradation index value (denoted by SEM in the figure), first marking maintenance strategy (a tick in the figure indicates no maintenance, a cross indicates need for maintenance), and geographical location is as Figure 8 shown, which can facilitate the highway maintenance department to efficiently locate the markings with wear or partial loss and implement maintenance.

[0166] In step S7, specifically, summarize all clustering objects with determined marking types to obtain the marking extraction result for the target highway, including but not limited to the following steps S721 - S725.

[0167] S721. For any one of the clustering objects determined to be a certain non-solid marking type, intercept the corresponding rectangular binary sub-image from the pavement strength binary image according to the corresponding minimum bounding rectangle.

[0168] S722. Obtain the template instance image corresponding to the certain non-solid marking type, where the template instance image is a binary image with template marking instance pixel points as foreground pixel points.

[0169] S723. Calculate the marking radiation consistency degradation index value according to all pixel values of the template instance image and the rectangular binary sub-image according to the following formula :

[0170]

[0171] In the formula, represents the pixel point in the template instance image, represents the radiation range, represents the radiation pixel point in the rectangular binary sub-image and corresponding to the pixel point ; represents the pixel point ; represents the pixel value of the pixel point ;

[0172] In the step S723, the calculation principle of the marking radiation consistency degradation index is known from the existing literature: as an important indication of road safety, the backward radiation information of the marking material has a high correlation with the point cloud intensity value. Therefore, the radiation information of non-solid markings can be calculated through the gray-scale similarity with the corresponding template, and then the above calculation formula can be obtained. In addition, the value range of the marking radiation consistency degradation index is [0, 1]. The lower the value, the more serious the internal wear of the marking, or it indicates that the current marking lacks a high-brightness indication function, which will pose a safety hazard to drivers' night driving.

[0173] S724. If the value of the marking radiation consistency degradation index is greater than or equal to the preset second degradation index threshold, it is determined that the second marking maintenance strategy for any one of the clustering objects is not to be maintained; otherwise, it is determined that the second marking maintenance strategy for any one of the clustering objects is to be maintained.

[0174] In the step S724, the second degradation index threshold can be specifically set based on the results of multiple tests under various conditions, for example, set to 0.618.

[0175] S725. Take the certain non-solid marking type, the rectangular binarized sub-image, the template instance image, the marking radiation consistency degradation index value, the second marking maintenance strategy, and / or the geographical location corresponding to any one of the clustering objects as the marking extraction result for the target highway and for any one of the clustering objects.

[0176] In the step S725, for example, the marking extraction result including the non-solid marking type, the rectangular binarized sub-image, the template instance image, the marking radiation consistency degradation index value (denoted by SAM in the figure), the second marking maintenance strategy (a tick in the figure indicates not to be maintained, and a cross indicates to be maintained), and the geographical location is as Figure 9 shown, which can facilitate the highway maintenance department to efficiently locate the markings with wear or partial loss and implement maintenance. In addition, if there is a conflict between the second marking maintenance strategy and the first marking maintenance strategy, the exclusive OR logic method can be used to determine the final marking maintenance strategy; for example, if the second marking maintenance strategy is not to be maintained and the first marking maintenance strategy is to be maintained, or the second marking maintenance strategy is to be maintained and the first marking maintenance strategy is not to be maintained, then it is determined that the final marking maintenance strategy is to be maintained.

[0177] In the step S7, based on the degradation analysis result of the recognized markings, the marking degradation statistical result of the entire target highway can also be conventionally obtained, so as to provide clear guidance for the fine and efficient marking repair of the highway maintenance department or provide a relative reference for reflecting the change of marking degradation at different stages.

[0178] Based on the freeway marking extraction method described in the foregoing steps S1 to S7, a new freeway marking extraction solution driven by point cloud data is provided. That is, after obtaining the point cloud data collected by a mobile laser scanning device for the target freeway, first use a deep learning-based road surface point cloud segmentation network to extract the road surface point cloud data, and then sequentially perform point cloud rasterization processing and binarization processing to obtain a road surface intensity binarized image. Then, perform foreground connected component analysis processing on the binarized image to obtain multiple clustering objects. For each clustering object, according to the geometric information of the corresponding minimum bounding rectangle, determine whether the marking type of the corresponding object is a solid marking. If not, further use an edge template matching algorithm to identify the corresponding non-solid marking type. Finally, summarize all clustering objects with determined marking types to obtain the marking extraction result for the target freeway. In this way, the advantages of existing methods based on two-dimensional images and three-dimensional point clouds can be combined, avoiding information loss, accurately extracting complex types of markings, and easily challenging the situation of a large variety of scene elements, scale differences, and sample imbalance in the face of large-scale MLS point cloud scenarios, making the extraction result highly reliable and stable, and facilitating practical application and promotion. In addition, degradation analysis can be performed on the identified markings, which is convenient for highway maintenance departments to efficiently locate markings with wear or partial loss and implement maintenance.

[0179] As Figure 10 shown, in the second aspect of this embodiment, a virtual device for implementing the freeway marking extraction method described in the first aspect is provided, including a point cloud data acquisition unit, a road surface point cloud extraction unit, a point cloud rasterization processing unit, an image binarization processing unit, a connected component analysis processing unit, a marking type recognition unit, and a marking extraction summary unit that are sequentially communicatively connected;

[0180] The point cloud data acquisition unit is used to acquire the point cloud data collected by a mobile laser scanning device for the target freeway;

[0181] The road surface point cloud extraction unit is used to perform road surface point cloud extraction processing on the point cloud data by using a deep learning-based road surface point cloud segmentation network to obtain road surface point cloud data;

[0182] The point cloud rasterization processing unit is used to perform point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity feature image;

[0183] The image binarization processing unit is used to perform binarization processing on the road surface intensity feature image to obtain a road surface intensity binarized image;

[0184] The connected component analysis processing unit is used to perform foreground connected component analysis processing on the road surface intensity binarized image to obtain multiple clustering objects;

[0185] The marking type recognition unit is configured to, for each clustering object among the multiple clustering objects, determine whether the marking type of the corresponding clustering object is a solid marking line according to the geometric information of the corresponding minimum circumscribed rectangle. If not, the edge template matching algorithm is further used to identify the corresponding non-solid marking type, where the non-solid marking type refers to other marking types except the solid marking line type;

[0186] The marking extraction and summarization unit is configured to summarize all clustering objects with determined marking types to obtain the marking extraction result of the target highway.

[0187] For the working process, working details and technical effects of the foregoing device provided in the second aspect of this embodiment, reference may be made to the highway marking extraction method described in the first aspect, which will not be elaborated herein.

[0188] As Figure 11 shown, a computer device for executing the highway marking extraction method described in the first aspect is provided in the third aspect of this embodiment, including a memory, a processor, and a transceiver that are sequentially communicatively connected, where the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the highway marking extraction method described in the first aspect. Specifically, by way of example, the memory may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first in first out memory (FIFO), and / or a first in last out memory (FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may further include, but is not limited to, a power module, a display screen, and other necessary components.

[0189] For the working process, working details and technical effects of the foregoing computer device provided in the third aspect of this embodiment, reference may be made to the highway marking extraction method described in the first aspect, which will not be elaborated herein.

[0190] In the fourth aspect of this embodiment, a computer-readable storage product storing instructions for the highway marking extraction method described in the first aspect is provided. That is, instructions are stored on the computer-readable storage product, and when the instructions are run on a computer, the highway marking extraction method described in the first aspect is executed. Among them, the computer-readable storage product refers to a carrier for storing data, and may include, but is not limited to, computer-readable storage media such as floppy disks, optical discs, hard disks, flash memories, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0191] For the working process, working details, and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment, reference may be made to the highway marking extraction method described in the first aspect, which will not be elaborated herein.

[0192] In the fifth aspect of this embodiment, a computer program product is provided, including a computer program or instructions. When the computer program or the instructions are executed by a computer, the highway marking extraction method described in the first aspect is implemented. Among them, the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0193] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for extracting highway markings, characterized in that: include: Acquire point cloud data collected by mobile laser scanning equipment on the target highway; Using a road point cloud segmentation network based on deep learning to perform road point cloud extraction processing on the point cloud data to obtain road point cloud data; Performing point cloud rasterization processing on the road surface point cloud data to obtain a road surface strength characteristic image; Binarizing the road surface strength characteristic image to obtain a road surface strength binary image; Performing foreground connected area analysis on the road strength binary image to obtain multiple clustering objects; For each cluster object in the plurality of cluster objects, judging whether the marking type of the corresponding cluster object is a solid marking type according to the geometric information of the corresponding minimum circumscribed rectangle, and if not, further identifying the corresponding non-solid marking type by using an edge template matching algorithm, wherein the non-solid marking type refers to other marking types except the solid marking type; Summarize all cluster objects of determined road marking types to obtain road marking extraction results for the target highway; Summarize all cluster objects of determined line marking types to obtain line marking extraction results for the target highway, including: for any cluster object determined as a certain non-real line marking type, cut out a corresponding rectangular binary sub-image from the road surface intensity binary image according to the corresponding minimum circumscribed rectangle; obtain a template instance image corresponding to the certain non-real line marking type, wherein the template instance image is a binary image with template line marking instance pixels as foreground pixels; use a similarity metric to compare all foreground edge pixels of the template instance image and the rectangular binary sub-image to obtain a line marking contour integrity degradation index value; if the line marking contour integrity degradation index value is greater than or equal to a preset first degradation index threshold, then the first line marking maintenance strategy for any one of the cluster objects is determined to be no maintenance, otherwise the first line marking maintenance strategy for any one of the cluster objects is determined to require maintenance; the certain non-real line marking type, the rectangular binarized sub-image, the template instance image, the line marking contour integrity degradation index value, the first line marking maintenance strategy and / or the geographical location corresponding to any one of the cluster objects are used as the line marking extraction result for the target highway and for any one of the cluster objects; And / or, summarizing all cluster objects of the determined line marking type to obtain the line marking extraction result of the target highway, including: for any cluster object that has been determined as a certain non-real line marking type, according to the corresponding minimum circumscribed rectangle, intercepting a corresponding rectangular binary sub-image from the road surface intensity binary image; obtaining a template instance image corresponding to the certain non-real line marking type, wherein the template instance image is a binary image with the template line marking instance pixel points as foreground pixel points; according to all pixel values ​​of the template instance image and the rectangular binary sub-image, calculating the line marking radiation consistency degradation index value SAM according to the following formula (x,y) : Wherein, (x', y') represents a pixel point in the template instance image, (x, y) represents a radiation range, (x+x', y+y') represents a radiation pixel point in the rectangular binary sub-image and corresponding to the pixel point (x', y'), T(x', y') represents a pixel value of the pixel point (x', y'), and I(x+x', y+y') represents a pixel value of the radiation pixel point (x+x', y+y'); if the marking radiation consistency degradation index value is greater than or equal to a preset second degradation index threshold, then the second marking maintenance strategy for the any one cluster object is determined to be no maintenance, otherwise, the second marking maintenance strategy for the any one cluster object is determined to require maintenance; the certain non-real marking type, the rectangular binary sub-image, the template instance image, the marking radiation consistency degradation index value, the second marking maintenance strategy and / or the geographical location corresponding to the any one cluster object are used as the marking extraction result for the target highway and for the any one cluster object.

2. The highway marking extraction method according to claim 1, characterized in that: A road point cloud segmentation network based on deep learning is used to perform road point cloud extraction processing on the point cloud data to obtain road point cloud data, including: Performing noise filtering preprocessing on the point cloud data to obtain denoised point cloud data; A road point cloud segmentation network based on deep learning is used to perform road point cloud extraction processing on the denoised point cloud data to obtain road point cloud data, wherein the road point cloud segmentation network is a network with the highest road point cloud segmentation accuracy among the PointNet network, PointNet++ network and RandLA-Net network pre-trained based on a road point sample annotation set.

3. The highway marking extraction method according to claim 1, characterized in that: The road surface point cloud data is subjected to point cloud rasterization processing to obtain a road surface strength characteristic image, including: Projecting each road surface point in the road surface point cloud data onto a rasterized two-dimensional road surface; For each grid on the two-dimensional road surface, if there is no projected road surface point in the corresponding grid, the pixel value of the corresponding pixel point is set to zero, and if there is at least one projected road surface point in the corresponding grid, the pixel value of the corresponding pixel point is set to the average laser intensity of the at least one road surface point; According to the pixel values ​​of all pixels, a road surface intensity characteristic image is obtained.

4. The highway marking extraction method according to claim 1, characterized in that: Binarizing the road surface strength characteristic image to obtain a road surface strength binary image includes: Performing block processing on the road surface strength characteristic image to obtain a plurality of sub-images; For each sub-image in the plurality of sub-images, using a maximum cross entropy threshold segmentation algorithm to determine a corresponding optimal pixel threshold; For each of the sub-images, taking the pixel points in the corresponding image whose pixel values ​​are greater than the corresponding pixel optimal threshold as foreground pixel points, and taking the pixel points in the corresponding image whose pixel values ​​are less than or equal to the corresponding pixel optimal threshold as background pixel points, to obtain a binarized sub-image that summarizes all foreground pixel points and all background pixel points and corresponds to the image; All binary sub-images are stitched together to obtain a road surface strength binary image.

5. The highway marking extraction method according to claim 1, characterized in that: For each cluster object in the plurality of cluster objects, judging whether the marking type of the corresponding cluster object is a solid line marking according to geometric information of the corresponding minimum circumscribed rectangle includes: For a certain cluster object among the multiple cluster objects, a corresponding minimum bounding rectangle is generated, and geometric information of the minimum bounding rectangle is calculated, wherein the geometric information includes a rectangle width b, a rectangle aspect ratio σ1, and an object rectangle area ratio σ2, wherein the object rectangle area ratio σ2 refers to an area ratio between the cluster object and the minimum bounding rectangle; Determine whether the geometric information of the minimum bounding rectangle of the cluster object meets the following conditions: In the formula, solid width represents the standard width of the solid line of the target highway, Indicates the preset aspect ratio threshold. Indicates the preset area ratio threshold, Indicates the preset allowable difference threshold; If it is determined that the geometric information of the minimum circumscribed rectangle of the certain cluster object meets the condition, then the line type of the certain cluster object is determined to be a solid line type line, otherwise it is determined that the line type of the certain cluster object is not a solid line type line; And / or, for each cluster object whose marking type is not a solid line marking type in the plurality of cluster objects, further identifying the corresponding non-solid marking type using an edge template matching algorithm, including: For any cluster object whose marking type is not a solid line among the plurality of cluster objects, a corresponding rectangular sub-image is obtained from the road surface strength binary image according to the corresponding minimum circumscribed rectangle; Acquire template instance images of all non-solid marking lines, wherein the non-solid marking lines refer to highway marking lines of non-solid marking type, the non-solid marking line type refers to marking line types other than solid line marking line, the marking line types of all non-solid marking lines are different from each other, and the template instance image is a binary image with template marking line instance pixel points as foreground pixel points; For each non-solid marking line among all the non-solid marking lines, using a similarity measurement method to compare the corresponding template instance image with all foreground edge pixels of the rectangular sub-image to obtain a corresponding similarity measurement index value; Determine a non-solid marked line having a maximum similarity measurement index value from all the non-solid marked lines; It is determined whether the maximum similarity measurement index value reaches a preset index threshold value. If so, the non-solid marking line type of the certain non-solid marking line is used as the marking line type of any clustering object.

6. A highway marking extraction device, characterized in that: It includes a point cloud data acquisition unit, a road surface point cloud extraction unit, a point cloud rasterization processing unit, an image binarization processing unit, a connected area analysis processing unit, a marking type identification unit and a marking extraction summary unit which are sequentially connected in communication; The point cloud data acquisition unit is used to acquire point cloud data collected by a mobile laser scanning device on a target highway; The road surface point cloud extraction unit is used to use a road surface point cloud segmentation network based on deep learning to perform road surface point cloud extraction processing on the point cloud data to obtain road surface point cloud data; The point cloud rasterization processing unit is used to perform point cloud rasterization processing on the road surface point cloud data to obtain a road surface intensity characteristic image; The image binarization processing unit is used to perform binarization processing on the road surface strength characteristic image to obtain a road surface strength binarization image; The connected region analysis processing unit is used to perform foreground connected region analysis processing on the road strength binary image to obtain multiple cluster objects; The marking type identification unit is used to determine, for each cluster object in the plurality of cluster objects, whether the marking type of the corresponding cluster object is a solid line marking according to the geometric information of the corresponding minimum circumscribed rectangle, and if not, further identify the corresponding non-solid marking type by using an edge template matching algorithm, wherein the non-solid marking type refers to other marking types except the solid line marking type; The line marking extraction and summarization unit is used to summarize all cluster objects of determined line marking types to obtain a line marking extraction result for the target highway; Summarize all cluster objects of determined line marking types to obtain line marking extraction results for the target highway, including: for any cluster object determined as a certain non-real line marking type, cut out a corresponding rectangular binary sub-image from the road surface intensity binary image according to the corresponding minimum circumscribed rectangle; obtain a template instance image corresponding to the certain non-real line marking type, wherein the template instance image is a binary image with template line marking instance pixels as foreground pixels; use a similarity metric to compare all foreground edge pixels of the template instance image and the rectangular binary sub-image to obtain a line marking contour integrity degradation index value; if the line marking contour integrity degradation index value is greater than or equal to a preset first degradation index threshold, then the first line marking maintenance strategy for any one of the cluster objects is determined to be no maintenance, otherwise the first line marking maintenance strategy for any one of the cluster objects is determined to require maintenance; the certain non-real line marking type, the rectangular binarized sub-image, the template instance image, the line marking contour integrity degradation index value, the first line marking maintenance strategy and / or the geographical location corresponding to any one of the cluster objects are used as the line marking extraction result for the target highway and for any one of the cluster objects; And / or, summarizing all cluster objects of the determined line marking type to obtain the line marking extraction result of the target highway, including: for any cluster object that has been determined as a certain non-real line marking type, according to the corresponding minimum circumscribed rectangle, intercepting a corresponding rectangular binary sub-image from the road surface intensity binary image; obtaining a template instance image corresponding to the certain non-real line marking type, wherein the template instance image is a binary image with the template line marking instance pixel points as foreground pixel points; according to all pixel values ​​of the template instance image and the rectangular binary sub-image, calculating the line marking radiation consistency degradation index value SAM according to the following formula (x,y) : Wherein, (x', y') represents a pixel point in the template instance image, (x, y) represents a radiation range, (x+x', y+y') represents a radiation pixel point in the rectangular binary sub-image and corresponding to the pixel point (x', y'), T(x', y') represents a pixel value of the pixel point (x', y'), and I(x+x', y+y') represents a pixel value of the radiation pixel point (x+x', y+y'); if the marking radiation consistency degradation index value is greater than or equal to a preset second degradation index threshold, then the second marking maintenance strategy for the any one cluster object is determined to be no maintenance, otherwise, the second marking maintenance strategy for the any one cluster object is determined to require maintenance; the certain non-real marking type, the rectangular binary sub-image, the template instance image, the marking radiation consistency degradation index value, the second marking maintenance strategy and / or the geographical location corresponding to the any one cluster object are used as the marking extraction result for the target highway and for the any one cluster object.

7. A computer device, characterized in that: The invention comprises a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the highway marking extraction method as described in any one of claims 1 to 5.

8. A computer readable storage product, characterized in that The computer-readable storage product stores instructions, and when the instructions are run on a computer, the highway marking extraction method as described in any one of claims 1 to 5 is executed.

9. A computer program product comprising a computer program or instructions, characterized in that The computer program or the instruction, when executed by a computer, implements the highway marking extraction method as claimed in any one of claims 1 to 5.

Images