Complete marking damage detection method, device and medium based on marking geometric position

Abstract

The present invention provides a complete road marking damage detection method, device and medium based on the geometric position of the road marking, which relates to the field of intelligent transportation technology, including: S1, obtaining a frame to be identified that is collected by a vehicle-mounted camera device during a road inspection process; S2, performing road marking detection on the frame to be identified according to the road marking recognition model, and determining and obtaining the geometric position of the road markings in the frame to be identified; S3, taking out a left road marking set composed of the corresponding geometric positions of the road markings on the left lane line of the target and a right road marking set composed of the corresponding geometric positions of the road markings on the right lane line of the target from the frame to be identified according to the geometric positions of the road markings in the frame to be identified; S4, determining the minimum circumscribed quadrilateral corresponding to the target road marking on the frame to be identified according to the geometric positions of the road markings in the left road marking set and the right road marking set, and extending the minimum circumscribed quadrilateral forward and backward to obtain the corresponding complete road marking area, and performing damage identification on the complete road marking area on the frame to be identified.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a complete road marking damage detection method, device and medium based on the geometric position of the road marking. Background Art

[0002] Road markings are an important part of traffic safety facilities and are the key to ensuring traffic safety. However, with the increase of years of use, continuous vehicle running over and weather factors, road markings will become damaged, blurred or missing, thus affecting the driver or intelligent driving judgment and increasing traffic accidents.

[0003] At present, road management departments often choose manual inspection and traditional graphics algorithms to detect road marking damage, but both methods have problems, as follows:

[0004] (1) Manual inspection is easily affected by subjective factors. Different standards for judging damage levels can easily lead to misdetection. In addition, manual inspection wastes a lot of manpower and takes a long time, making it difficult to cover all road markings, resulting in low detection efficiency.

[0005] (2) Traditional graphics algorithms mainly refer to the recognition of road markings through edge detection and color threshold segmentation. This recognition method performs poorly in complex environments and is easily affected by lighting changes, shadows, stains and weather, making the detection results unstable. Summary of the invention

[0006] The purpose of the present invention is to provide a complete road marking damage detection method, device and medium based on the geometric position of the road marking. The method mainly involves arranging a vehicle-mounted camera device on the roof of a vehicle. During road inspection, the geometric position of the road marking in the collected road surface image is identified frame by frame, and the road marking is extended based on the geometric position of the road marking to obtain a complete road marking area. This ensures that the present invention can identify damage to the complete road marking area. While improving the detection efficiency, it can also improve the accuracy of identifying the road marking damage level.

[0007] In order to solve the above technical problems, the present invention adopts the following solutions:

[0008] A complete marking damage detection method based on marking geometry position, comprising:

[0009] S1, obtaining frames to be identified collected by a vehicle-mounted camera device during a road inspection process;

[0010] S2, obtaining a marking recognition model for detecting markings, performing marking detection on the frame to be identified according to the marking recognition model, and determining and obtaining the geometric position of the markings in the frame to be identified;

[0011] S3, extracting from the frame to be identified a left marking set consisting of the marking geometric positions corresponding to the marking geometric positions on the target left lane line and a right marking set consisting of the marking geometric positions corresponding to the target right lane line from the frame to be identified;

[0012] S4. Determine the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified based on the geometric positions of the markings in the left marking set and the right marking set, and extend the minimum circumscribed quadrilateral forward and backward to obtain the corresponding complete marking area, and perform damage identification on the complete marking area on the frame to be identified.

[0013] Furthermore, the vehicle-mounted camera device is a camera device installed on the roof of the inspection vehicle. During the road inspection process of the inspection vehicle, the vehicle-mounted camera device can collect road information images in real time to form a video, and the video is extracted frame by frame to obtain all frames to be identified frame by frame.

[0014] Furthermore, the road marking recognition model is a convolutional neural network YOLOP model, and the road marking detection module in the convolutional neural network YOLOP model detects the outline of the road markings in the frame to be recognized. At the same time, the road markings are segmented on the frame to be recognized according to the position of the road marking outline obtained by the detection. The geometric position of the road markings obtained by the road marking recognition model includes the road marking outline information and the lane line information where the road markings are located.

[0015] Furthermore, in S3, the lane line information where the marking is located is obtained according to the geometric position of the marking in the frame to be identified, and the left and right lane lines closest to the center point are searched respectively in the frame to be identified, and the left lane line and the right lane line are found as the target left lane line and the target right lane line respectively, and then the target left lane line and the target right lane line are matched with the lane line where the marking is located, and the lane line where the marking is located with the same geometric position as the target left lane line is added to the left marking set, and the lane line where the marking is located with the same geometric position as the target right lane line is added to the right marking set.

[0016] Furthermore, the S4 includes the following steps:

[0017] S41, obtaining marking contour information according to the geometric positions of markings in the left marking set and the right marking set, and finding the markings with the largest contour area and the markings closest to the center point of the frame to be identified in both the left marking set and the right marking set according to the marking contour information;

[0018] S42, judging the marking line with the largest contour area in the left marking line set and the right marking line set and the marking line closest to the center point of the frame to be identified, and obtaining the target marking line according to the judgment result;

[0019] S43, obtaining the marking contour information of the target marking according to the marking geometric position, and performing fitting with the marking contour information of the target marking to obtain the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified;

[0020] S44, obtaining the coordinate position of the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified, and calculating the geometric width of the minimum circumscribed quadrilateral according to the coordinate position;

[0021] S45, obtaining the road facility standard of the lane where the target marking is located, extending the minimum circumscribed quadrilateral according to the road facility standard and the geometric width of the minimum circumscribed quadrilateral to obtain a corresponding complete marking area, and performing damage identification on the complete marking area on the frame to be identified.

[0022] Further, in S42, the marking line with the largest contour area in the left marking line set and the right marking line set and the marking line closest to the center point of the frame to be identified are judged, and the process of obtaining the target marking line according to the judgment result is as follows:

[0023] Obtain the marking line with the largest contour area in the left marking line set and the right marking line set, and determine the area ratio between the contour area of ​​other marking lines in the left marking line set and the right marking line set and the maximum contour area according to the marking line contour information. If the area ratios are all less than the preset threshold, select the marking line with the largest contour area as the target marking line; if any area ratio is greater than the preset threshold, select the marking line closest to the center point of the frame to be identified as the target marking line.

[0024] Furthermore, in S45, the process of extending the minimum circumscribed quadrilateral according to the road facility standard and the geometric width of the minimum circumscribed quadrilateral is as follows:

[0025] Based on the standard marking width of road facilities, the difference between the geometric width of the minimum circumscribed quadrilateral and the reference is calculated, and the difference is extended layer by layer in the X-axis and Y-axis directions of each vertex of the minimum circumscribed quadrilateral.

[0026] Furthermore, in S4, damage identification is performed on the complete marking area on the frame to be identified by using a convolutional neural network Swin-Transformer model to identify damage to the complete marking area on the frame to be identified, and the damage level of the complete marking area is detected.

[0027] A complete marking damage detection device based on marking geometric position, comprising:

[0028] a memory for non-transitory storage of computer readable instructions;

[0029] The processor is used to execute the computer-readable instructions, and the computer-readable instructions, when executed by the processor, implement the complete marking damage detection method based on the marking geometric position.

[0030] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, a complete marking damage detection method based on marking geometric position is implemented.

[0031] Beneficial effects of the present invention:

[0032] The present invention provides a complete road marking damage detection method, device and medium based on the geometric position of road markings. By arranging a vehicle-mounted camera device on the roof of a vehicle, the geometric position of road markings in the collected road surface image is identified frame by frame during road inspection. The geometric position of road markings can not only meet the requirements of road marking damage detection, but also locate damaged road markings, which is convenient for road management departments to maintain road markings.

[0033] Furthermore, the complete marking damage detection method is mainly extended based on the geometric position of the marking to obtain a complete marking area, ensuring that the present invention identifies damage to the complete marking area, while improving the detection efficiency and the accuracy of marking damage level identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the process of the complete marking damage detection method in Example 1 of the present invention;

[0035] Figure 2 Schematic diagram of the structure of the marking line recognition model in Example 1 of the present invention;

[0036] Figure 3 A schematic diagram of a road marking in Embodiment 1 of the present invention;

[0037] Figure 4 A schematic diagram of obtaining a region containing complete marking lines from a left marking line set in Example 1 of the present invention;

[0038] Figure 5 It is a schematic diagram of obtaining a region containing complete marking lines from a right marking line set in Example 1 of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0041] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0042] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness.One of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0043] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0044] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0045] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments:

[0046] Example 1

[0047] In this embodiment, a complete marking damage detection method based on the marking geometric position is proposed, such as Figure 1 As shown, the complete marking damage detection method comprises the following steps:

[0048] S1, obtaining frames to be identified collected by a vehicle-mounted camera device during a road inspection process;

[0049] S2, obtaining a marking recognition model for detecting markings, performing marking detection on the frame to be identified according to the marking recognition model, and determining and obtaining the geometric position of the markings in the frame to be identified;

[0050] S3, extracting from the frame to be identified a left marking set consisting of the marking geometric positions corresponding to the marking geometric positions on the target left lane line and a right marking set consisting of the marking geometric positions corresponding to the target right lane line from the frame to be identified;

[0051] S4. Determine the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified based on the geometric positions of the markings in the left marking set and the right marking set, and extend the minimum circumscribed quadrilateral forward and backward to obtain the corresponding complete marking area, and perform damage identification on the complete marking area on the frame to be identified.

[0052] In one embodiment, the vehicle-mounted camera device is a camera device installed on the roof of the inspection vehicle. During the road inspection process of the inspection vehicle, the vehicle-mounted camera device can collect road information images in real time to form a video, and the video is extracted frame by frame to obtain all frames to be identified. The purpose is to enable the vehicle-mounted camera device to easily achieve full coverage of the area through the road inspection process of the inspection vehicle, reduce the waste of labor costs, and improve the efficiency of road marking detection.

[0053] Specifically, during the road inspection process, considering the principle of larger near and smaller far, we choose to collect road surface images in real time within an interval of 5 meters to form video files, where each video file contains 200 frames of images. Frames are extracted from the video to obtain frames to be identified, and their resolution is 1920*1200.

[0054] In one embodiment, the marking line recognition model is a convolutional neural network YOLOP model, which is a model obtained through improvement and training. The marking line recognition model is mainly used to output the geometric position of the marking line.

[0055] Because YOLOP is a single-stage network, it contains a shared encoder and three decoders for specific tasks. The encoder contains Backbone (image feature extraction module) and Neck (network). Backbone is used to extract the features of the input image; the decoder includes Detect head (target head detection module), Drivable area segment head (drivable area segmentation module), and Lane line segment head (lane line segmentation module).

[0056] In order to better extract image features during downsampling, such as Figure 2 As shown in the figure, firstly, SpatialAttention (attention mechanism module) is introduced in Backbone. The basic idea is to perform global average pooling and global maximum pooling on the input feature map, concatenate the generated feature vectors, and then generate an attention feature map through a 1*1 small convolution. The attention feature map is multiplied with the input feature map, so that the features of the important areas are enhanced, so as to improve the performance and accuracy of the model under conditions such as lighting, shadows, and marking wear. The specific steps include:

[0057] (1) The input feature map is , where C is the number of channels and H * W is the size in space;

[0058] (2) Perform global maximum pooling and global average pooling on the input feature map F to generate two feature maps

[0059] The maximum pooling formula is: ;

[0060] in, Represents the maximum value of k channels at the position (i, j) of the feature map;

[0061] The average pooling formula is: ;

[0062] in, Represents the average value of k channels at the position (i, j) of the feature map;

[0063] The result is two feature maps of size H*W and ; Where H is the feature map height and W is the feature map width;

[0064] (3) Concatenate the two feature maps together according to the channel dimension to form a new feature map. The formula is: ;

[0065] (4) Apply a 1x1 convolution to the concatenated feature map to generate a spatial attention map, the formula of which is: , where σ is the Sigmoid function, which is used to map the output to between [0,1];

[0066] (5) Multiply the spatial attention map by the original input map bit by bit to generate an attention-enhanced map that highlights important areas. The formula is: ,in, To enhance the combined feature map, is the weight factor, is the original combined feature.

[0067] In addition, the original Drivable area segment head is removed and only the Lane line segment head is retained to improve the performance of the convolutional neural network YOLOP model. The specific change steps are:

[0068] (1) Adjust the output layer of the model: only keep the output layer related to lane line detection, namely Lane line segmenthead, in order to obtain the geometric position of the lane line in the frame to be identified through the convolutional neural network YOLOP model;

[0069] (2) Adjusting the damage function: Based on the YOLOP loss function, only the loss function of the lane segmentation module is retained. The cross entropy loss function does not distinguish between simple and difficult samples. When the proportion of easily distinguishable negative samples is large, the training process is easily affected by negative samples, thereby drowning out positive samples and causing large losses. Lane line pixels account for a small proportion of the entire image and are difficult samples. Therefore, the cross entropy loss function is adjusted to the FocalLoss loss function. Adding FocalLoss can effectively handle sample imbalance and allow the model to focus on learning difficult samples, thereby improving detection accuracy.

[0070] Specifically, the formula of the FocalLoss loss function is as follows:

[0071] ;in, is the weight factor; The value range of is 0~1, which is the probability that the model predicts that it belongs to the prospect; Tune hyperparameters for weights, is the modulation factor introduced to reduce the loss ratio of easy-to-classify samples, thereby increasing the loss ratio of difficult-to-classify samples. When it tends to 1, it means that the sample is easy to divide. The value of tends to 0, which reduces the loss ratio of easy-to-divided samples. It tends to 0, which means that false detection occurs. The value of tends to 1, and the impact on the overall loss is not significant.

[0072] Based on the above convolutional neural network YOLOP model, when the frame to be identified is processed, the model segmentation result of the lane segmentation module can be obtained. According to the model segmentation result, the geometric position of the marking on the frame to be identified is obtained. One marking geometric position corresponds to one marking. After segmentation, the marking may not be a complete marking. In addition, the model output result is a tensor. Through the slicing operation, the valid prediction area ll_predict can be extracted from the tensor, and the invalid area added due to padding is removed. Then, the valid prediction area ll_seg_mask is interpolated by bilinear interpolation, and a downsampled result ll_seg_mask is generated. Among them, the bilinear interpolation method can ensure the smoothness of the prediction result; the maximum value of each pixel is extracted from the downsampled mask ll_seg_mask, and the part of the dimension of the tensor that is 1 is removed after the mask is converted to an integer type to ensure that the shape of ll_seg_mask is consistent with the image.

[0073] In one embodiment, since the present invention uses a vehicle-mounted camera device disposed on the roof of the inspection vehicle to photograph the road surface, the road markings are located on both sides of the vehicle-mounted camera device, and different lanes have different road markings, such as Figure 3 As shown in FIG. 1 , there are usually two lane lines on both sides of a lane, that is, road markings, and the contour output result of the captured road surface image may have multiple contours due to the different number of lanes, false detection, changes in the position of the markings, and changes in the curvature of the markings. In order to solve these problems, specifically, the target lane lines are limited to two, and the two road markings of the lane where the inspection vehicle is located are taken for detection. Then, in S3, the lane line information where the markings are located is obtained according to the geometric position of the markings in the frame to be identified, and the frame to be identified is searched from the center point to the left and right respectively to find the left vehicle closest to the center point. The lane line and the right lane line are used as the target left lane line and the target right lane line respectively, and then the target left lane line and the target right lane line are matched with the lane line where the marking is located, and the lane line where the marking is located and the target left lane line The same marking geometric position is added to the left marking set, and the lane line where the marking is located and the target right lane line The same marking geometric position is added to the right marking set, that is, according to the marking geometric information of the output result of the marking recognition model and the preset image center point The left side is the left marking line set and the right side is the right marking line set The geometric features are equivalent to obtaining the contour sets of the left and right marking lines respectively.

[0074] In one embodiment, the S4 includes the following steps:

[0075] S41, obtaining marking contour information according to the geometric positions of markings in the left marking set and the right marking set, and finding the markings with the largest contour area and the markings closest to the center point of the frame to be identified in both the left marking set and the right marking set according to the marking contour information;

[0076] S42, judging the marking line with the largest contour area in the left marking line set and the right marking line set and the marking line closest to the center point of the frame to be identified, and obtaining the target marking line according to the judgment result;

[0077] In S42, the marking line with the largest contour area in the left marking line set and the right marking line set and the marking line closest to the center point of the frame to be identified are judged, and the process of obtaining the target marking line according to the judgment result is as follows:

[0078] Obtain the marking line with the largest contour area in the left marking line set and the right marking line set, and determine the area ratio between the contour area of ​​other marking lines in the left marking line set and the right marking line set and the maximum contour area according to the marking line contour information. If the area ratios are all less than the preset threshold, select the marking line with the largest contour area as the target marking line; if there is an area ratio greater than the preset threshold, select the marking line closest to the center point of the frame to be identified as the target marking line;

[0079] Specifically, the area ratio of the contour area of ​​other marking lines in the left marking set and the right marking set to the contour with the largest area is calculated. After analyzing a large amount of image data, the threshold can be set to 0.75. If the area ratio between the contour area of ​​other marking lines in the left marking set and the contour with the largest area is less than 0.75, the marking line with the contour with the largest area is selected as the target marking line; if the area ratio between the contour area of ​​other marking lines in the left marking set and the contour with the largest area is greater than 0.75, the marking line closest to the center point of the frame to be identified is selected as the target marking line;

[0080] S43, obtaining the marking contour information of the target marking according to the marking geometric position, and performing fitting with the marking contour information of the target marking to obtain the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified;

[0081] Specifically, the marking line contour information includes the marking line edge, which is the outer edge line of the area. Fitting is performed based on the marking line edge to obtain the minimum circumscribed quadrilateral of the area, which can reduce the influence of pixels around the marking line on the marking line damage recognition. In addition, the marking line contour calculated according to the output result of the marking line recognition model will have different angles, making the marking line contour relatively irregular. In order to obtain a complete marking line, the marking line contour is selected to be fitted to obtain the minimum circumscribed quadrilateral to fit the marking line shape.

[0082] S44, obtaining the coordinate position of the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified, and calculating the geometric width of the minimum circumscribed quadrilateral according to the coordinate position;

[0083] Specifically, the contour of the line marking calculated based on the output of the line marking recognition model is a part of the actual line marking and cannot completely cover the complete line marking. The width of the line markings in different sections is subject to road facility standards. After calculating the geometric width of the line marking contour, the line marking contour can be extended according to the road facility standards to obtain the complete line marking.

[0084] S45, obtaining the road facility standard of the lane where the target marking is located, extending the minimum circumscribed quadrilateral according to the road facility standard and the geometric width of the minimum circumscribed quadrilateral to obtain a corresponding complete marking area, and performing damage identification on the complete marking area on the frame to be identified;

[0085] Specifically, the marking width of the road facility standard is used as the benchmark. The road facility standard should be the "Highway Traffic Safety Facilities Design Specification" and the "Highway Traffic Safety Facilities Design Detailed Rules". Both specifications have provisions on the geometric dimensions of the markings. According to the above provisions, the difference between the geometric width of the minimum circumscribed quadrilateral and the benchmark is calculated, and the difference is extended in layers toward the X-axis and Y-axis directions of each vertex of the minimum circumscribed quadrilateral. The area containing the complete markings can be cut out from the frame to be identified with the extended coordinates, such as Figure 4 , Figure 5 As shown, Figure 4 Pasted in is the area containing the complete markings obtained from the left marking set, Figure 5 The area containing the complete marking line obtained from the right marking line set is pasted in the middle, and is extended based on the geometric position of the marking line to obtain the complete marking line area, so as to ensure that the present invention can identify damage to the complete marking line area, and at the same time improve the detection efficiency, it can also improve the accuracy of identifying the marking line damage level.

[0086] In one embodiment, the convolutional neural network Swin-Transformer model is used to perform damage recognition on the complete marking area on the frame to be recognized, and the damage level of the complete marking area is detected, specifically:

[0087] SA1. The image containing the markings is obtained through the first stage, and the obtained image is cleaned to remove the dark and blurred images in the tunnel, and the marking sample image dataset is obtained;

[0088] SA2. Determine the model detection result according to the image to be identified and the trained excellent, good, inferior and poor marking line appearance grade classification model. Specifically, the appearance grade classification model is obtained by improving and training the Swin-Transformer network model: the network model includes an input module, a feature extraction module and an output module. The input module enhances the image by restricting the adaptive histogram equalization of the image, improves the contrast of the image, obtains more image details and suppresses useless noise in the image, and reduces the influence of light and shadow. Among them, the feature extraction module adopts the Swin-Transformer structure. Swin-Transformer is a hierarchical Transformer structure used in the field of computer vision. Specifically, first, the input image size is 416*416*3, that is, H=416, W=416, and then, the patch is divided and the patch size is set to 8*8. After division, 52*52 small squares of size 8*8*3 will be obtained. After flattening, it will be a cube of 52*52*192. Add a linear The embedding layer is used for linear transformation, and the final image size is 52*52*384. The data is passed into SwinTransformer and processed by Swin Transformerblock to obtain the output result. In SwinTransformer block, the input data is first normally paid attention within the window range (W-MSA, W stands for Window), and then after a series of modules such as LN and MLP, a moving window attention is performed again (SW-MSA, SW stands for Shifted Window), and then the final output result is obtained. The output result is subjected to Patch Merging, and the input data size becomes 28*28*768. PatchMerging is to splice the pixels at the same position in a given pixel set of size 8*8 together to obtain four Feature maps, and the four Feature maps are concat spliced. Patchmerging is very similar to the pooling operation in CNN, except that the pooling operation is replaced by taking patches at fixed positions. The output result of PatchMerging is processed again by Swin Transformer block, and the output result is processed again by Patch Merging. The input data size becomes 14*14*1536, and the output size is 14*14*1536 after being processed by Swin Transformer block again. Among them, the output feature of 14*14*1536 is the feature vector used in subsequent downstream classification tasks;

[0089] Referring to the Highway Technical Condition Assessment Standard (JTG 5210-2018), road marking defects should be defined as missing or damaged road markings. In order to achieve the classification of road marking technical condition levels, the road marking appearance is defined as excellent as no defects, no wear, no covering, and no dirt on the road markings. The road marking appearance is good, which is defined as slight defects, slight wear, slight covering, or slight dirt on the road markings, which does not affect the driver's identification. The road marking appearance is poor, which is defined as partial defects, partial wear, partial covering, or partial dirt on the road markings, which poses certain safety hazards. The road marking appearance is poor, which is defined as serious defects, serious wear, serious covering, or serious dirt on the road markings, which affects traffic safety and requires immediate maintenance. The data set after data processing is divided into training set, validation set, and test set, which are passed into the improved Swin Transformer model and trained to obtain the road marking appearance grade classification model.

[0090] SA3, train the damage type classification model according to the wear, defect, cover and dirt of the secondary and poor categories. Specifically, according to the data set divided in SA2, divide the pictures from the secondary and poor categories into training set, verification set and test set according to the wear, defect, cover and dirt, and then pass them into the same Swin Transformer model and train the model to obtain the marking damage category classification model;

[0091] SA4. Determine the specific defect classification of inferior and poor. Specifically, the image to be tested is passed into the marking line positioning model to obtain a new image containing only the marking line, the new image is passed into the marking line appearance grade classification model to obtain the marking line appearance grade, and the image with inferior and poor marking line appearance grade is passed into the marking line damage category classification model to obtain the specific marking line damage classification;

[0092] SA5. Generate a detailed report based on the inspection and rating results. The report includes statistics on the appearance grade of each kilometer of markings, the total appearance grade of each 100 kilometers of markings, the appearance grade of each line in different urban areas, and the total appearance grade of each line.

[0093] In summary, the present invention proposes a complete road marking damage detection method based on the geometric position of the road marking. By setting a vehicle-mounted camera device on the roof, during the road inspection process, the collected road surface image is subjected to road marking detection frame by frame through the improved YOLOP model to obtain the geometric position of the road marking, and then the road marking geometric position is extended to obtain the complete road marking area, and then the convolutional neural network Swin-Transformer model is used to classify the complete road marking area, so as to achieve accurate judgment of road markings, which has the following advantages:

[0094] (1) Based on computer vision and deep learning technology, it replaces the traditional manual road marking judgment and fills the gap in automatic road marking appearance recognition and road marking damage classification;

[0095] (2) Based on computer vision and deep learning technology, a model cascade network, namely the convolutional neural network Swin-Transformer model, is used to refine the appearance grade of road markings. Not only can it be divided into excellent, good, inferior, and poor, but it can also be divided into the classification of inferior and poor in terms of wear, defect, covering, and dirt;

[0096] (3) The overall detection architecture based on cascade, attention mechanism and convolutional neural network Swin-Transformer model can adapt to detection in complex environments (shadows, occlusions);

[0097] (4) Improve the YOLOP model by introducing the SpatialAttention mechanism in Backbone and adding FocalLoss to the loss function, which greatly improves the line detection capability. Specifically, adding the SpatialAttention mechanism can suppress the influence of shadows on the detection results to a certain extent and improve the generalization ability of the model. Adding FocalLoss can reduce the impact of unbalanced samples on the model effect.

[0098] (5) Introduce the convolutional neural network Swin-Transformer model into the line detection, adjust the patch size to 8*8, and reduce the number of Swin Transformer block modules;

[0099] (6) A two-level training strategy is provided for the hierarchical model, which improves the overall detection capability and accuracy.

[0100] Example 2

[0101] A complete marking damage detection device based on marking geometric position, comprising:

[0102] a memory for non-transitory storage of computer readable instructions;

[0103] The processor is used to execute the computer-readable instructions, and the computer-readable instructions, when executed by the processor, implement the complete marking damage detection method based on the marking geometric position.

[0104] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, a complete marking damage detection method based on marking geometric position is implemented.

[0105] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A complete marking damage detection method based on marking geometric position, characterized in that: The complete marking damage detection method comprises the following steps: S1, obtaining frames to be identified collected by a vehicle-mounted camera device during a road inspection process; S2, obtaining a marking recognition model for detecting markings, performing marking detection on the frame to be identified according to the marking recognition model, and determining and obtaining the geometric position of the markings in the frame to be identified; S3, extracting from the frame to be identified a left marking set consisting of the marking geometric positions corresponding to the marking geometric positions on the target left lane line and a right marking set consisting of the marking geometric positions corresponding to the target right lane line from the frame to be identified; S4, determining the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified according to the geometric positions of the markings in the left marking set and the right marking set, and extending the minimum circumscribed quadrilateral to obtain the corresponding complete marking area, and performing damage identification on the complete marking area on the frame to be identified; The S4 includes the following steps: S41, obtaining marking contour information according to the geometric positions of markings in the left marking set and the right marking set, and finding the markings with the largest contour area and the markings closest to the center point of the frame to be identified in both the left marking set and the right marking set according to the marking contour information; S42, judging the marking line with the largest contour area in the left marking line set and the right marking line set and the marking line closest to the center point of the frame to be identified, and obtaining the target marking line according to the judgment result; In S42, the marking line with the largest contour area in the left marking line set and the right marking line set and the marking line closest to the center point of the frame to be identified are judged, and the process of obtaining the target marking line according to the judgment result is as follows: Obtain the marking line with the largest contour area in the left marking line set and the right marking line set, and determine the area ratio between the contour area of ​​other marking lines in the left marking line set and the right marking line set and the maximum contour area according to the marking line contour information. If the area ratios are all less than the preset threshold, select the marking line with the largest contour area as the target marking line; if there is an area ratio greater than the preset threshold, select the marking line closest to the center point of the frame to be identified as the target marking line; S43, obtaining the marking contour information of the target marking according to the marking geometric position, and performing fitting with the marking contour information of the target marking to obtain the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified; S44, obtaining the coordinate position of the minimum circumscribed quadrilateral corresponding to the target marking on the frame to be identified, and calculating the geometric width of the minimum circumscribed quadrilateral according to the coordinate position; S45, obtaining the road facility standard of the lane where the target marking is located, extending the minimum circumscribed quadrilateral according to the road facility standard and the geometric width of the minimum circumscribed quadrilateral to obtain a corresponding complete marking area, and performing damage identification on the complete marking area on the frame to be identified; In S45, the process of extending the minimum circumscribed quadrilateral according to the road facility standard and the geometric width of the minimum circumscribed quadrilateral is as follows: Based on the standard marking width of road facilities, the difference between the geometric width of the minimum circumscribed quadrilateral and the reference is calculated, and the difference is extended layer by layer in the X-axis and Y-axis directions of each vertex of the minimum circumscribed quadrilateral.

2. A complete marking damage detection method based on marking geometric position according to claim 1, characterized in that: The vehicle-mounted camera device is a camera device installed on the roof of the inspection vehicle. During the road inspection process of the inspection vehicle, the vehicle-mounted camera device can collect road information images in real time to form a video, and the video is frame-by-frame extracted to obtain all frames to be identified.

3. A complete marking damage detection method based on marking geometric position according to claim 1, characterized in that: The road marking recognition model is a convolutional neural network YOLOP model. The road marking detection module in the convolutional neural network YOLOP model detects the outline of the road markings in the frame to be recognized. At the same time, the road markings are segmented on the frame to be recognized according to the position of the road marking outline obtained by the detection. The geometric position of the road markings obtained by the road marking recognition model includes the road marking outline information and the lane line information where the road markings are located.

4. A complete marking damage detection method based on marking geometric position according to claim 3, characterized in that: In S3, the lane line information where the marking is located is obtained according to the geometric position of the marking in the frame to be identified, and the left and right lane lines closest to the center point are searched respectively on the frame to be identified, and the left lane line and the right lane line closest to the center point are found as the target left lane line and the target right lane line respectively, and then the target left lane line and the target right lane line are matched with the lane line where the marking is located, and the lane line where the marking is located with the same geometric position as the target left lane line is added to the left marking set, and the lane line where the marking is located with the same geometric position as the target right lane line is added to the right marking set.

5. The complete marking damage detection method based on marking geometric position according to claim 1 is characterized in that: In S4, damage identification is performed on the complete marking area on the frame to be identified. Specifically, damage identification is performed on the complete marking area on the frame to be identified by using a convolutional neural network Swin-Transformer model to detect and obtain the damage level of the complete marking area.

6. A complete marking damage detection device based on marking geometric position, characterized in that: include: a memory for non-transitory storage of computer readable instructions; A processor is used to execute the computer-readable instructions, and when the computer-readable instructions are executed by the processor, a complete marking damage detection method based on the marking geometric position according to any one of claims 1 to 5 is implemented.

7. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, a complete marking damage detection method based on marking geometric position according to any one of claims 1 to 5 is implemented.

Images