A road accessory defect detection method and system based on deep learning

Abstract

The application belongs to the technical field of deep learning, and discloses a road accessory defect detection method and system based on deep learning. Guardrail target detection and classification: a PointRend model of a deep convolution network is used to realize accurate detection classification and precise segmentation of guardrails and anti-glare plates; guardrail deformation and fracture detection: when image target detection is performed and it is inferred that there is a guardrail in the image, the damage state of the neat boundary of the guardrail is further judged, and the damage state of the guardrail includes bending deformation and gap fracture; anti-glare plate missing and deformation detection: after image target detection, it is inferred that there is an anti-glare plate in the image, and the damage state of the anti-glare plate is further judged. The application can more conveniently obtain the damage state of the road accessory along the way, and can greatly reduce the working strength and working efficiency of maintenance personnel in combination with the correlation of the detection vehicle related equipment and the road stake number.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, and particularly relates to a road accessory defect detection method and system based on deep learning. BACKGROUND

[0002] With the gradual improvement of the highway network, the highway mileage will be equipped with a large number of road accessories. The road accessories damaged due to various reasons such as human factors, traffic accidents and falling rocks are mostly found by road maintenance personnel on patrol, which is low in efficiency and slow in time.

[0003] With the continuous development of artificial intelligence, deep learning is used more and more widely. However, for the problem of defect and damage detection of highway guardrails and baffle plates, the existing technology still has low patrol efficiency, high labor cost in real-time detection, classification, defect identification and positioning of highway guardrails and baffle plates. SUMMARY

[0004] In order to overcome the problems of low patrol efficiency and high labor cost in the related art for damaged road accessories, the present application provides a road accessory defect detection method and system based on deep learning.

[0005] The technical solution is as follows: a road accessory defect detection method based on deep learning, comprising:

[0006] S1, using a deep convolutional network PointRend model to realize image target detection classification and accurate segmentation of guardrails and anti-glare plates;

[0007] S2, when the image in the image target detection exists a guardrail, judging the damage state of the boundary of the whole guardrail, the damage state of the guardrail including bending deformation and gap rupture;

[0008] S3, when the image target detection is inferred as existing an anti-glare plate in the image, judging the damage state of the anti-glare plate to realize anti-glare plate missing deformation detection.

[0009] In step S1, the image target detection classification and accurate segmentation of guardrails and anti-glare plates are realized, comprising:

[0010] S101, extracting part of the image data from the collected highway image data, labeling the guardrails and anti-glare plates in the image, and setting target area calibration and classification label through image labeling software;

[0011] S102, training the guardrails and anti-glare plates with the labeled data as training data, and detecting and classifying;

[0012] S103, the high-speed road side image sequence collected by the data acquisition system is stored in a fixed folder, and in the actual measurement stage, the model interface is called to sequentially import the road side image.

[0013] In step S102, the guardrails and anti-dazzle plates are trained with the labeled data as training data, including: using a deep convolutional network PointRend model to train the guardrails and anti-dazzle plates on the highway, to detect and locate and segment the guardrails and anti-dazzle plate targets.

[0014] Further, the deep convolutional network PointRend model comprises:

[0015] (1) Point selection: adaptively selecting points in the image plane to predict segmentation labels;

[0016] (2) Point-by-point feature representation: extracting the characteristic features of the selected difficult points, calculating the sub-pixel information along the channel dimension by linear interpolation, and predicting the segmentation;

[0017] (3) Point head: using a multi-layer perceptron for point-by-point segmentation prediction, the multi-layer perceptron shares weights on all points, and the multi-layer perceptron predicts segmentation labels for each point, and is trained by a segmentation loss of the task.

[0018] In step S103, in the actual measurement stage, the model interface is called to sequentially import the road side image, including: using the trained deep convolutional network PointRend model to infer whether the image contains a guardrail or an anti-dazzle plate, if the target exists, the accurate position information of the target is given in the form of the outer polygon vertex sequence of the target area and the binary mask; at the same time, the deep convolutional network PointRend model returns the target classification label information of the picture; the deep convolutional network PointRend model identifies and classifies the target in the input picture and feeds back the result information;

[0019] The result of the road side image after being inferred by the deep convolutional network PointRend model is detected according to the specific target classification in the image, and the guardrail damage detection and the anti-dazzle plate damage detection are performed.

[0020] In step S2, the damage state of the boundary of the guardrail as a whole is determined, including:

[0021] S201, extracting the guardrail Mask region, traversing and searching the Mask region along the column direction, taking the pixel in the first row as the initial candidate edge point for each column, taking the first point as the upper boundary candidate point and the second point as the lower boundary candidate point when there is a boundary point in this column; if the guardrail Mask region is broken, there is only one initial candidate boundary point in this column, which is the upper boundary point by default;

[0022] S202, in the case of data positive difference noise error, using the least square linear fitting method to fit the upper and lower boundary line respectively;

[0023] S203, in the rail linear fitting, when the candidate boundary pixels exist abnormal outliers produced by fracture or bending, the optimal rail boundary in point is selected by using random sample consensus (RANSAC) algorithm, the optimal boundary straight line is fitted by using least square fitting, and whether there is fracture and bending is determined according to the distance from all candidate edge points to the fitted boundary.

[0024] In step S203, when the distance from the candidate edge point to the straight line is greater than the high threshold T1, it is considered that there is a fracture damage through the guardrail in the image column corresponding to the pixel, so as to determine the fracture area position; when the distance from the candidate edge point to the straight line is between the low threshold T2 and the high threshold T1, it is considered that it is the pixel on the curved boundary, so as to determine the bending area of the guardrail.

[0025] In step S3, the damage state of the anti-glare plate is judged to realize the anti-glare plate missing deformation detection, including:

[0026] (1) the convex hull algorithm is processed on the mask area generated by the depth convolution network PointRend model to obtain all the key points of the convex hull; the distances from the four corners of the image to the key points of the convex hull are calculated to find the farthest convex hull key points from each vertex;

[0027] (2) according to the affine transformation property, the determined quadrilateral is subjected to affine transformation to achieve the effect of a front rectangle;

[0028] (3) the inclined deformation disease detection of the damage type of the anti-glare plate is performed, if the pixel block appears in the region, the pixel block is insufficient, and the anti-glare plate is damaged, and the inclined deformation damage type disease occurs.

[0029] In step (1), the distances from the four corners of the image to the key points of the convex hull are calculated to find the farthest convex hull key points from each vertex, including:

[0030] The opencv convex hull algorithm is called to draw and return the convex hull key point coordinates:

[0031] PS={P1:(x1,y1),P2:(x2,y2)…P n :(x n ,y n )}

[0032] In the formula, PS is the coordinate set of the top corner to the convex hull key point, P1:(x1,y1) is the coordinate of the first top corner relative to the convex hull key point, and P n :(x n ,y n ) is the coordinate of the nth top corner relative to the convex hull key point.

[0033] Take four vertices A(0, 0), B(w, 0), C(0, -h), D(w, h) as reference points, w is the picture width, h is the picture height;

[0034] Take the image vertex A as the reference point, calculate the vertex A to all the key points on the convex hull, the farthest convex hull key point A' is one of the vertices of the quadrilateral, determine all the vertices of the quadrilateral, the four points are the circumscribed quadrilateral of Mask, the expression is:

[0035]

[0036] In the formula, d A is the distance between two points;

[0037] In step (2), the four points of the quadrilateral vertex (x i ,y i ) after the perspective transformation are (x′ i ,y′ i ); Set the coordinates of the transformed rectangle vertices as (0, 0), (0, W), (H, 0), (H, W); Where W, H are the width and length of the rectangle set by human, the coordinate affine transformation formula is as follows:

[0038]

[0039] In the formula, a, b, c are the coefficients of the fitting straight line after the affine transformation of the vertex coordinates relative to the horizontal coordinate of the affine transformation, the constant, the inverse height of the picture, d, e, f are the coefficients of the fitting straight line after the affine transformation of the vertex coordinates relative to the vertical coordinate of the affine transformation, the constant, the inverse height of the picture; (x, y) is the coordinate of the quadrilateral vertex, (x′, y′) is the quadrilateral vertex after perspective transformation;

[0040] Inverse transform the matching coordinate data matrix to estimate the affine transformation matrix, and perform affine transformation on the original antiglare plate region to correct to a front rectangular shape; Determine the horizontal coordinate position of each antiglare plate through non-maximum suppression for the corrected antiglare plate Mask;

[0041] In step (3), the damage type of the antiglare plate is detected, including:

[0042] S3.1, zero the six-degree-of-freedom platform, select six antiglare plates as the detection object, and the vector composed of the spatial lengths of the six antiglare plates is G is the order, and the pose corresponding to the six-degree-of-freedom platform is:

[0043]

[0044] In the formula, g1 is the pose coordinate corresponding to the vector of the spatial length of the six free platforms and six anti-dazzle plates, s1, s2, s3, s4, s5 and s6 are respectively the vector values of the spatial length of the first anti-dazzle plate, the spatial length of the second anti-dazzle plate, the spatial length of the third anti-dazzle plate, the spatial length of the fourth anti-dazzle plate, the spatial length of the fifth anti-dazzle plate and the spatial length of the sixth anti-dazzle plate; I is the attitude angle of the first anti-dazzle plate, J is the attitude angle of the second anti-dazzle plate, K is the attitude angle of the third anti-dazzle plate, is the attitude angle of the fourth anti-dazzle plate, is the attitude angle of the fifth anti-dazzle plate, is the attitude angle of the sixth anti-dazzle plate.

[0045] S3.2, a coordinate system 2 is established on the anti-dazzle plate section 1 of the expressway, and the origin coordinates and attitude angles are obtained through the pose change program in the detection software, and there are six parameters, which are denoted as:

[0046]

[0047] The pose relationship change matrix of the coordinate system 2 and the operating platform coordinate system 1 is:

[0048]

[0049] In the formula, O is the tilt deformation change matrix, the upper left subscript indicates the serial number of the fixed coordinate system at this time, the lower left subscript indicates the serial number of the moving coordinate system at this time, and 0 indicates the global coordinate system.

[0050] S3.3, a coordinate system 3 is established on the docking interface on the platform of the anti-dazzle plate section 2 of the expressway, and the origin coordinates and attitude angles are obtained through the pose change program, and there are six parameters, which are denoted as:

[0051]

[0052] S3.4, the values of g2 in the coordinate system 2 and the values of g3 in the coordinate system 2 are sent to the detection software of the upper computer, and the kinematics calculation program in the alignment machine software is used to calculate the pose to be reached by the coordinate system 2 after adjustment, so that the height variance value in the alignment target is minimized, h k The selection is determined by the following optimization method:

[0053]

[0054] In the formula, h1 and h2 are respectively the heights of the anti-dazzle plate sections 1 and 2 of the expressway relative to the ground, δh 12 is the height difference of the parallel lines of the highest end faces of the anti-dazzle plate sections 1 and 2 of the expressway.

[0055] Another object of the present application is to provide a deep learning-based road accessory defect detection system that implements the deep learning-based road accessory defect detection method, the system comprising:

[0056] A guardrail target detection and classification module uses a deep convolutional network PointRend model to accurately detect and classify and precisely segment guardrails and anti-glare plates;

[0057] A guardrail deformation and fracture detection module is used to determine the damage state of the straight boundary of the guardrail when the guardrail is detected by image target detection and inferred to exist in the image, and the guardrail damage state includes bending deformation and gap fracture;

[0058] An anti-glare plate missing and deformation detection module is used to determine the damage state of the anti-glare plate when the anti-glare plate is detected by image target detection and inferred to exist in the image.

[0059] In combination with all the above technical solutions, the present application has the beneficial effects that the present application uses high-resolution cameras to collect data, relies on the big data processing advantages of deep supercomputing workstations, uses intelligent video image processing and deep learning technology to perform real-time detection, classification, defect identification and positioning on highway guardrails and baffles. The present application uses complete algorithm achievements integrated into road detection equipment to more conveniently obtain the damage conditions of roadside highway accessory facilities, and in combination with the correlation of detection vehicle related equipment and road stake numbers, can greatly reduce the working intensity and working efficiency of maintenance personnel. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure;

[0061] Figure 1 is a flowchart of the deep learning-based road accessory defect detection method provided by the embodiments of the present application;

[0062] Figure 2 is a comparison chart of the least squares and RANSAC fitting straight line effects provided by the embodiments of the present application;

[0063] Figure 3 is a principle diagram for determining the circumscribed quadrilateral of Mask provided by the embodiments of the present application;

[0064] Figure 4 is a damage detection chart of the inclined deformation of the anti-glare plate provided by the embodiments of the present application. DETAILED DESCRIPTION

[0065] In order to make the above objectives, characteristics and advantages of the present application more apparent, concrete embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to provide a sufficient understanding of the present application. However, the present application can be implemented in many other different ways than those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application, and therefore the present application is not limited to the specific implementations disclosed below.

[0066] As shown in Embodiment 1, Figure 1 The deep learning-based road accessory defect detection method provided by the embodiments of the present application includes:

[0067] S1, using a deep convolutional network PointRend model to realize image target detection classification and accurate segmentation of guardrails and anti-glare plates;

[0068] Specifically, it includes:

[0069] S101, extract part of the image data from a large number of collected highway side image data, and manually mark the guardrails and anti-glare plates in the image. Specifically, through professional image marking software, the image is target region calibrated and classified label set.

[0070] S102, train the guardrails and anti-glare plates with the labeled data as training data, and detect and classify them;

[0071] For example, the deep convolutional network PointRend model not only improves the smoothness in the segmentation process, realizes accurate segmentation of image details, but also greatly saves resource consumption, and the computing power requirement is only 2.6% of Mask R-CNN. Under the current trend of deploying artificial intelligence algorithm models on the side, the design idea of the deep convolutional network PointRend model framework can be divided into three parts:

[0072] (1) Point Selection: Select a few 'difficult points' for prediction to avoid excessive calculation of all pixels in high-resolution output. The core idea of PointRend is to adaptively select points in the image plane to predict segmentation labels. In theory, these points should be widely distributed in high-frequency areas (such as the edges of the image).

[0073] (2) Point-wise feature representation: the selected 'hard' points are characterized by fine-grained features, which are computed by linear interpolation, then sub-pixel information is encoded along the channel dimension, and segmentation is predicted. PointRend constructs point-wise features at selected points by combining fine-grained features and coarse prediction. In terms of fine-grained features, a feature vector is extracted for each sampling point in the CNN feature map to present fine segmentation details. Although fine-grained features can resolve details, they have two shortcomings: (a) they do not contain specific region information, which may predict different labels at the same point for instance segmentation tasks. (b) the feature map used for fine-grained features may only contain relatively low-level information. This requires coarse prediction to be supplemented to provide more global context. Taking instance segmentation as an example, coarse prediction can be the output of the 7x7 lightweight mask head in Mask R-CNN.

[0074] (3) Point head: a small neural network used to predict labels based on point-wise feature representation. For point-wise feature representation of each selected point, PointRend uses a simple multi-layer perceptron for point-wise segmentation prediction. The multi-layer perceptron shares weights across all points (all regions). And since the multi-layer perceptron predicts segmentation labels for each point, it can be trained by task-specific segmentation loss.

[0075] In the present application, the PointRend model is used to achieve accurate detection and positioning and precise segmentation and extraction of guardrails and anti-glare board targets on expressways.

[0076] S103, the expressway side image sequence obtained by the data acquisition system is stored in a fixed folder. In the actual measurement stage, the model interface is called to sequentially import the roadside images. The trained PointRend model can infer whether the image contains a guardrail or an anti-glare board. If the target exists, the accurate position information of the target is given in the form of the sequence of the vertices of the circumscribed polygon of the target region and the binary mask. At the same time, the PointRend model returns the target classification label information of the picture. The PointRend model predicts the input picture, that is, identifies and classifies the target in the picture, and feeds back the result information.

[0077] The result of the roadside image after inference by the PointRend model will be processed separately in the subsequent anomaly detection according to the specific target classification in the figure, and guardrail damage detection and anti-dazzle plate damage detection will be performed respectively.

[0078] Exemplarily, the accurate position information of the target in the form of the sequence of the vertices of the circumscribed polygon of the target area and the binary mask includes:

[0079] For the binary image mask with the target, the findContours method in opencv is used for contour detection, the minAreaRect method is used to calculate the minimum circumscribed rectangle of the contour, and the boxPoints method is used to calculate the minimum circumscribed rectangle to obtain the four-point coordinates of the minimum circumscribed rectangle of the contour. Therefore, the accurate position information of the target can be obtained.

[0080] S2, when the image target detection in the image exists the guardrail, the damage state of the boundary of the guardrail is judged, and the damage state of the guardrail includes bending deformation and gap breakage;

[0081] When the image target detection in step S1 and the inference of the existence of the guardrail in the image are performed, the damage state of the boundary of the guardrail is further judged. The bending deformation and the gap breakage damage mainly manifest as abnormal contour distortion and loss. Specifically, it includes:

[0082] S201, the guardrail Mask area is extracted, and then the Mask area is searched along the column direction. Each column takes the pixel in the first row as the initial candidate edge point. When there is a boundary point in this column, the first point found is taken as the upper boundary candidate point, and the second point is taken as the lower boundary candidate point. If the guardrail Mask is broken, there is only one initial candidate boundary point in this column, which is the upper boundary point by default.

[0083] S202, then the least square straight line fitting method is used to perform straight line fitting on the upper and lower boundaries respectively. The basic idea of the least square method is as follows: using the given n candidate boundary points, a best fitting straight line is sought:

[0084] y=Ax+B

[0085] In the formula, y is the vertical coordinate value of the fitting straight line, A is the coefficient of the fitting straight line, x is the horizontal coordinate value, and B is a constant;

[0086] The best fitting straight line should pass through or be close to all candidate points as much as possible, and the essence is to find the best value of parameters a and b. The principle of the least square method is as follows: given a set of points {(x i ,y i )|1≤i≤n}, the error square sum E is:

[0087]

[0088] When the straight line makes the error square sum minimum, the partial derivative of the straight line parameter is solved, and the parameters a, b satisfy:

[0089]

[0090] Through this method, the values of the straight line parameters A and B can be estimated, that is, the fitting straight line can be obtained. It only considers the optimal straight line fitting under the condition of data positive difference noise error.

[0091] In the guardrail straight line fitting, when the candidate boundary pixels exist abnormal "wild points" caused by breakage or bending, the random sampling consensus (RANSAC) algorithm is used to select the optimal guardrail boundary inlier, and the least square fitting optimal boundary straight line is used; then whether the breakage and bending exist is determined according to the distance of all candidate edge points to the fitted boundary.

[0092] When the distance of the candidate edge point to the straight line is greater than the high threshold T1, it is considered that there is a breakage damage through the guardrail in the image column corresponding to the pixel, so that the breakage area position is determined. When the distance of the candidate edge point to the straight line is between the low threshold T2 and the high threshold T1, it is considered that it is a pixel on the curved boundary, so that the guardrail bending area is determined.

[0093] It can be understood that in the guardrail straight line fitting, when the candidate boundary pixels exist abnormal "wild points" caused by breakage or bending, the random sampling consensus (RANSAC) algorithm is used to realize the boundary fitting in the "wild point" environment, and the error inevitably appeared by using the least square fitting is solved.

[0094] Therefore, the random sampling consensus (RANSAC) algorithm is used to realize the boundary fitting in the "wild point" environment. It can estimate the optimal model parameters from a set of observation data containing "wild points" through an iterative manner. In principle, the more the iteration times are, the more the optimal model is obtained with high probability. The core idea is randomness and assumption. The randomness is used to reduce the calculation, and the cycle times are used to utilize the probability of correct data. The so-called assumption is that the randomly sampled data are considered to be correct, and other points satisfying the transformation relationship are obtained by calculation, and then the voting mechanism is used to select the fitting straight line with the most votes as the optimal fitting of the boundary.

[0095] Among them, the difference between the random sampling consensus RANSAC and the least square is that the least square method tries to adapt to all points including "wild points". On the contrary, RANSAC can obtain a model calculated by only inlier points, and the probability is high enough. Figure 2 Comparison chart of least square and RANSAC fitting straight line effect;

[0096] S3, missing anti-glare board deformation detection: after step S2 image target detection and inferred that there is an anti-glare board in the image, further judge the anti-glare board damage state.

[0097] Common anti-glare board damage state has anti-glare board loss and tilt deformation. For the case of anti-glare board loss, the distance relationship between anti-glare boards is judged. Specifically:

[0098] (1) On the mask generated by the PointRend model, the convex hull algorithm is processed to obtain all the key points of the convex hull. The distance from the four corners of the image to the key points of the convex hull is calculated, and the farthest convex hull key point from each vertex is found;

[0099] Wherein, in opencv, the convex hull algorithm is an algorithm for finding the smallest convex polygon of a point set, which can help extract the contour features in the image. For binary images, the convex hull algorithm usually involves the following steps:

[0100] Image preprocessing: first, the image needs to be converted to a binary image. This is usually done through thresholding to distinguish between foreground and background. The mask generated by PointRend is a binary image.

[0101] Contour detection: in the binary image, the findContours function is used to detect the contours in the image. These contours are the boundaries of the foreground objects in the image.

[0102] Convex hull calculation: for each detected contour, the convexHull function is used to calculate its convex hull. The convex hull is a set of points that form a smallest convex polygon that can contain all the points in the contour.

[0103] For example, as Figure 3 The principle diagram for determining the circumscribed quadrilateral of the mask is shown in the red box: the convex hull drawn by calling the opencv convex hull algorithm, and according to the returned convex hull key point coordinates:

[0104] PS={P1:(x1,y1),P2:(x2,y2)…P n :(x n ,y n )}

[0105] Wherein, PS is the coordinate set of the top corner to the convex hull key point, P1:(x1,y1) is the coordinate of the first top corner relative to the convex hull key point, P n :(x n ,y n ) is the coordinate of the nth top corner relative to the convex hull key point;

[0106] Take four vertices A(0, 0), B(w, 0), C(0, -h), D(w, h) as reference points, w is the picture width, h is the picture height;

[0107] Take the image vertex A as the reference point, calculate the distance from vertex A to all key points on the convex hull, and the farthest convex hull key point A' is one of the vertices of the quadrilateral. Determine all vertices of the quadrilateral, and the four points are the circumscribed quadrilateral of Mask, and the expression is:

[0108]

[0109] In the formula, d A is the distance between two points;

[0110] (2) According to the affine transformation property, the above determined quadrilateral is subjected to a perspective transformation, so that it reaches the front rectangular effect. Specifically, assuming that the vertices of the quadrilateral are (x i ,y i ), the four points after perspective transformation are (x′ i ,y′ i ). In this project, the coordinates of the transformed rectangle are set as (0, 0), (0, W), (H, 0), (H, W), where W and H are the reasonable rectangle width and length set by man. Then the coordinate affine transformation formula is as follows:

[0111]

[0112] In the formula, a, b, and c are the coefficients of the fitting straight line after the affine transformation of the vertex coordinates with respect to the horizontal coordinate of the affine transformation, the constant, the inverse height of the picture, d, e, and f are the coefficients of the fitting straight line after the affine transformation of the vertex coordinates with respect to the vertical coordinate of the affine transformation, the constant, the inverse height of the picture; (x, y) is the coordinate of the quadrilateral vertex, and (x′, y′) is the quadrilateral vertex after perspective transformation;

[0113] The affine transformation matrix is estimated by matrix inverse transformation of 4 pairs of matching coordinate data. Then the original anti-glare plate region is subjected to affine transformation, and is corrected to a front rectangular shape. After this processing, the relative distance and size between the anti-glare plates have consistency.

[0114] The horizontal coordinate position of each anti-glare plate is determined by non-maximum suppression for the corrected anti-glare plate Mask.

[0115] Where, non-maximum suppression, NMS (nonmaximum suppression). The nms is used in the Canny operator in Opencv;

[0116] Canny operator is one of the most classical and advanced algorithms in image edge detection algorithm. Compared with Sobel and Prewitt operators, Canny algorithm is more excellent. Canny algorithm proposes two improvements: non-maximum suppression based on edge gradient direction and double threshold processing.

[0117] Canny algorithm is based on three basic objectives:

[0118] (i) low error rate, all edges should be found while there are no false responses.

[0119] (ii) edge points are well positioned, and the already positioned edges must be as close to the true edges as possible.

[0120] (iii) single edge point response, when and only when there is a single edge point position, the detector should not indicate multiple pixel edges.

[0121] The steps of Canny algorithm are as follows:

[0122] (a) Gaussian smoothing is performed on the input image to reduce the error rate.

[0123] (b) Calculate the gradient amplitude and direction to estimate the edge strength and direction at each point.

[0124] The calculation process of gradient amplitude generally uses Sobel operator horizontal and vertical operators and input image convolution to calculate d x ,d y ;

[0125]

[0126] d x =∫(x,y)*SobelX(x,y)

[0127] d y =∫(x,y)*SobelY(x,y)

[0128] In the formula, SobelX is the horizontal operator, SobelY is the vertical operator, and (x, y) is the coordinate composed of horizontal and vertical;

[0129] Get the amplitude of the image gradient:

[0130]

[0131] In the formula, M(x, y) is the amplitude of the image gradient;

[0132] (c) According to the gradient direction, non-maximum suppression is performed on the gradient amplitude.

[0133] (d) Use double threshold processing and connect edges.

[0134] (3) Anti-glare board damage type tilt deformation disease detection, such as Figure 4 shown, Figure 4 yellow block in the figure, illustrates that if the area where the pixel block should appear, there is a pixel block that is not enough size, then this is suspected to be damaged anti-glare board, that is, anti-glare board damage type disease. For example, the normal anti-glare board is set every 30 cm, which is reflected in the figure as every 30 pixels there is a white block, and the block height gradually changes according to the projection rules. By analogy, there should be a full-size block in the red box area, but only a yellow block that is not full-size appears, so this place can be listed as a suspected damage to the anti-glare board.

[0135] For example, in step (3), the anti-glare board damage type tilt deformation disease detection specifically includes:

[0136] S3.1, the six-degree-of-freedom platform is zeroed, and six anti-glare boards are selected as detection objects, and the vector formed by the spatial lengths of the six anti-glare boards is G is the order, and the pose corresponding to the six-degree-of-freedom platform is:

[0137]

[0138] In the formula, g1 is the pose coordinates corresponding to the six-degree-of-freedom platform and the vector formed by the spatial lengths of the six anti-glare boards, s1, s2, s3, s4, s5, and s6 are respectively the vector values formed by the spatial lengths of the first anti-glare board, the second anti-glare board, the third anti-glare board, the fourth anti-glare board, the fifth anti-glare board, and the sixth anti-glare board; I is the first anti-glare board attitude angle, J is the second anti-glare board attitude angle, K is the third anti-glare board attitude angle, is the fourth anti-glare board attitude angle, is the fifth anti-glare board attitude angle, is the sixth anti-glare board attitude angle.

[0139] S3.2, a coordinate system 2 is established on the anti-glare board section of the highway, and the origin coordinates and attitude angles, a total of 6 parameters, are obtained through the pose change program in the detection software, and are denoted as:

[0140]

[0141] The pose relationship change matrix of the coordinate system 2 and the operating platform coordinate system 1 is:

[0142]

[0143] In the formula, O is a skew deformation change matrix, the upper left subscript represents the serial number of the fixed coordinate system at this time, the lower left subscript represents the serial number of the moving coordinate system at this time, and 0 represents the global coordinate system:

[0144] S3.3, a coordinate system 3 is established on the butt joint of the platform of the highway anti-dazzle plate section 2, and through a pose change program, 6 parameters of the origin coordinates and attitude angles are obtained, which are denoted as:

[0145]

[0146] S3.4, the values of g2 in the coordinate system 2 and the values of g3 in the coordinate system 2 are sent to the detection software of the upper computer, and a kinematics calculation program in the alignment machine software calculates the pose to be reached by the coordinate system 2 after adjustment, so that the height variance value in the alignment target is minimized, h k The selection is determined by the following optimization method:

[0147]

[0148] In the formula, h1 and h2 are the heights of the highway anti-dazzle plate sections 1 and 2 relative to the ground, respectively, and δh 12 is the height difference of the parallel lines of the highest end faces of the highway anti-dazzle plate sections 1 and 2.

[0149] Embodiment 2 provides a road accessory defect detection system based on deep learning, comprising:

[0150] A guardrail target detection and classification module is used to realize accurate detection, classification and precise segmentation of guardrails and anti-dazzle plates by using a deep convolutional network PointRend model.

[0151] A guardrail deformation and fracture detection module is used to further judge the damage state of the straight boundary of the guardrail when the guardrail is detected and inferred to exist in the image.

[0152] An anti-dazzle plate missing and deformation detection module is used to further judge the damage state of the anti-dazzle plate when the anti-dazzle plate is detected and inferred to exist in the image.

[0153] The present application can be integrated into a portable device of a road monitoring vehicle, and used in combination with the equipment in the detection vehicle, so as to convert the damage information of the identified accessory along the way into a stake number and feed back to the road maintenance personnel, so as to facilitate the road maintenance personnel to count the damage of the road accessory, and replace the related facilities in time.

[0154] The above merely describes the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any modification, equivalent replacement and improvement within the technical range disclosed by the present application and within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A road ancillary facilities defect detection method based on deep learning, characterized in that: The method includes: S1. Use the deep convolutional network PointRend model to achieve image target detection, classification and segmentation of guardrails and anti-glare panels; S2. When a guardrail exists in the image during the image target detection, the damage state of the entire guardrail boundary is determined. The damage state of the guardrail includes bending deformation and notch fracture. S3. When it is inferred that an anti-glare plate exists in the image after image target detection, the damage state of the anti-glare plate is determined to implement deformation detection of the anti-glare plate missing; In step S3, the damage state of the anti-glare plate is determined to detect the missing deformation of the anti-glare plate, including: (1) Perform convex hull algorithm processing on the mask area generated by the deep convolutional network PointRend model to obtain all the key points of the convex hull; calculate the distance from the four vertex corners of the image to the convex hull key points, and find the convex hull key point farthest from each vertex; (2) According to the properties of affine transformation, the determined quadrilateral is subjected to transmission transformation to achieve a frontal rectangular effect; (3) Detection of tilt deformation damage of the anti-glare plate. If a pixel block smaller than the specified size appears in the area where the pixel block appears, the anti-glare plate is damaged and a tilt deformation damage type disease has occurred. In step (3), the anti-glare plate is damaged by tilt deformation, including: S3.

1. Return the six-degree-of-freedom platform to zero and select six anti-glare panels as the test objects. The vector formed by the spatial lengths of the six anti-glare panels is G is the order, and the corresponding posture of the six-degree-of-freedom platform is: Wherein, g1 is the pose coordinate corresponding to the vector formed by the six-degree-of-freedom platform and the six anti-glare panel space lengths; s1, s2, s3, s4, s5, and s6 are respectively the vector value formed by the first anti-glare panel space length, the vector value formed by the second anti-glare panel space length, the vector value formed by the third anti-glare panel space length, the vector value formed by the fourth anti-glare panel space length, the vector value formed by the fifth anti-glare panel space length, and the vector value formed by the sixth anti-glare panel space length; I is the attitude angle of the first anti-glare panel, J is the attitude angle of the second anti-glare panel, and K is the attitude angle of the third anti-glare panel. It is the attitude angle of the 4th anti-glare panel. It is the attitude angle of the 5th anti-glare plate. It is the attitude angle of the 6th anti-glare plate; S3.

2. Establish coordinate system 2 on highway anti-glare board section 1. Obtain the origin coordinates and attitude angles through the posture change program in the detection software. A total of 6 parameters are recorded as: The pose relationship change matrix between coordinate system 2 and operating platform coordinate system 1 is: Where O is the tilt deformation change matrix, the upper left script indicates the serial number of the fixed coordinate system during this change, the lower left script indicates the serial number of the moving coordinate system during this change, and 0 indicates the global coordinate system: S3.

3. Establish coordinate system 3 on the docking interface of the platform of highway anti-glare plate section 2. Through the posture change program, obtain its origin coordinates and posture angles, a total of 6 parameters, recorded as: S3.

4. Send the values ​​of g2 and g3 in coordinate system 2 to the detection software of the host computer. Use the kinematic solution program in the detection software of the host computer to calculate the position that coordinate system 2 will reach after adjustment, so as to minimize the height variance value in the alignment target, h k The selection of is determined by the following optimization method: Where h1 and h2 are the heights of highway anti-glare board sections 1 and 2 relative to the ground, respectively, and δh 12 It is the height difference between the parallel lines of the highest end surfaces of highway anti-glare plate sections 1 and 2.

2. The road ancillary facilities defect detection method based on deep learning according to claim 1, characterized in that: In step S1, the image target detection, classification and accurate segmentation of the guardrail and anti-glare plate are achieved, including: S101, extracting part of the image data from the collected highway image data, annotating the guardrails and anti-glare panels in the image, and using image annotation software to calibrate the target area and set classification labels for the image; S102, using the labeled data as training data to train guardrails and anti-glare panels, and detect and classify them; S103: The data acquisition system acquires a sequence of highway roadside images and stores them in a fixed folder. During the measurement phase, the model interface is called to sequentially import the roadside images.

3. The road ancillary facilities defect detection method based on deep learning according to claim 2, characterized in that: In step S102, the annotated data is used as training data to train guardrails and anti-glare panels, including: using a deep convolutional network PointRend model to train guardrails and anti-glare panels on highways, and detecting, locating, and segmenting and extracting guardrail and anti-glare panel targets.

4. The road ancillary facilities defect detection method based on deep learning according to claim 3 is characterized in that: Deep convolutional network PointRend model, including: (1) Point selection: adaptively selecting points in the image plane to predict segmentation labels; (2) Point-by-point feature representation: extract the points selected for the predicted segmentation label, encode the sub-pixel information along the channel dimension through linear interpolation, and predict the segmentation; (3) Point head: Use a multi-layer perceptron to perform point-by-point segmentation prediction. The multi-layer perceptron shares weights across all points and predicts a segmentation label for each point, trained using the segmentation loss of the task.

5. The road ancillary facilities defect detection method based on deep learning according to claim 2, characterized in that: In step S103, during the measurement phase, the model interface is called to sequentially import roadside images. This includes: using the trained deep convolutional network PointRend model to infer whether the image contains guardrails or anti-glare panels. If a target exists, the target's accurate location information is provided in the form of a vertex sequence of the circumscribed polygon of the target area and a binary mask; the deep convolutional network PointRend model is also returned to perform target classification labeling on the image; the deep convolutional network PointRend model recognizes and classifies the target in the input image and provides feedback on the result information; The results of the roadside image inference by the deep convolutional network PointRend model are used to perform guardrail damage detection and anti-glare plate damage detection according to the specific target classification in the image.

6. The road ancillary facilities defect detection method based on deep learning according to claim 1, characterized in that: In step S2, the damage state of the entire guardrail boundary is determined, including: S201: Extract the guardrail mask area and perform a traversal search along the column direction. The pixels in the first row of each column are used as the initial candidate edge points. If there are boundary points in this column, the first traversed point is used as the upper boundary candidate point, and the second point is used as the lower boundary candidate point. If there is a break in the guardrail mask area, there is only one initial candidate boundary point in this column, which is the upper boundary point by default. S202, when searching the mask area along the column direction, if there is a break in the guardrail mask area, a least squares straight line fitting method is used to fit the upper and lower boundaries respectively; S203: In the guardrail straight line fitting, when there are abnormal wild points caused by breakage or bending in the candidate boundary pixels, the random sampling consensus RANSAC algorithm is used to select the optimal guardrail boundary inner point, and the least squares method is used to fit the optimal boundary line. The distance from all candidate edge points to the fitted boundary is used to determine whether there is a breakage or bending.

7. The road ancillary facilities defect detection method based on deep learning according to claim 6, characterized in that: In step S203, when the distance from the candidate edge point to the straight line is greater than the high threshold T1, it is considered that there is a fracture damage that penetrates the guardrail in the image column corresponding to the pixel, thereby determining the position of the fracture area; when the distance from the candidate edge point to the straight line is between the low threshold T2 and the high threshold T1, it is considered to be a pixel on the curved boundary, thereby determining the curved area of ​​the guardrail.

8. The road ancillary facilities defect detection method based on deep learning according to claim 1, characterized in that: In step (1), the distances from the four vertex corners of the image to the convex hull key points are calculated, and the convex hull key points farthest from each vertex are found, including: Call the OpenCV convex hull algorithm to draw and return the coordinates of the convex hull key points: PS={P1:(x1,y1),P2:(x2,y2)…P n :(x n ,y n )} Where PS is the coordinate set of the vertex to the convex hull key point, P1: (x1, y1) is the coordinate of the first vertex relative to the convex hull key point, P n :(x n ,y n ) is the coordinate of the nth vertex relative to the convex hull key point; Take the four vertices A(0,0), B(w,0), C(0,-h), and D(w,h) as reference points, where w is the image width and h is the image height; Taking the image vertex A as the reference point, calculate all key points from vertex A to the convex hull. The convex hull key point A′ with the farthest distance is a vertex of the quadrilateral. Determine all the vertices of the quadrilateral. The four points are the circumscribed quadrilateral of the Mask. The expression is: Where, d A is the distance between two points; In step (2), for the quadrilateral vertices (x i ,y i )The four points after the transmission transformation are (x′ i , y′ i ); Set the coordinates of the transformed rectangle vertices to (0, 0), (0, W), (H, 0), (H, W); where W and H are the manually set width and length of the rectangle, respectively. The coordinate affine transformation formula is as follows: Where a, b, c are the coefficients of the fitted line relative to the horizontal coordinate of the affine transformation of the vertex coordinates after the affine transformation, constants, and reverse heights of the image; d, e, f are the coefficients of the fitted line relative to the vertical coordinate of the affine transformation of the vertex coordinates after the affine transformation, constants, and reverse heights of the image; (x, y) are the coordinates of the vertices of the quadrilateral, and (x′, y′) are the vertices of the quadrilateral after the transmission transformation; The inverse transformation of the matrix of the matching coordinate data is performed to estimate the affine transformation matrix, and the original anti-glare plate area is affine transformed to correct it to a frontal rectangular shape; the horizontal coordinate position of each anti-glare plate is determined by non-maximum suppression of the corrected anti-glare plate Mask.

9. A road ancillary facilities defect detection system based on deep learning, characterized in that: The system implements the road ancillary facility defect detection method based on deep learning as described in any one of claims 1 to 8, and the system includes: The guardrail target detection and classification module uses the deep convolutional network PointRend model to detect, classify, and segment guardrails and anti-glare panels; The guardrail deformation and fracture detection module is used to determine the damage state of the guardrail's neat boundary when it is inferred that a guardrail exists in the image through image target detection. Guardrail damage states include bending deformation and notch fracture. The anti-glare plate missing deformation detection module is used to determine the damage status of the anti-glare plate when it is inferred that the anti-glare plate exists in the image after image target detection.

Images