Road subsidiary facility defect detection method and system based on deep learning

Abstract

The invention belongs to the technical field of deep learning, and discloses a road ancillary facility defect detection method and system based on deep learning. Guardrail target detection and classification are carried out, and accurate detection classification and accurate segmentation of guardrails and anti-glare panels are realized by using a deep convolutional network Point Rend model; guardrail deformation and fracture detection: when image target detection is carried out and the guardrail is deduced to exist in the image, the damage state of the tidy boundary of the guardrail is further judged, and the guardrail damage state comprises bending deformation and notch fracture; and anti-glare panel missing deformation detection: further judging the damage state of the anti-glare panel when image target detection is performed and the anti-glare panel is deduced to exist in the image. According to the invention, the damage condition of the ancillary facilities of the road along the way can be more conveniently obtained, and the work intensity and the work efficiency of maintenance personnel can be greatly reduced by combining the correlation of the detection vehicle related equipment and the road stake number.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular, relates to a road ancillary facility defect detection method and system based on deep learning. Background Art

[0002] As the highway network is gradually improved, a large number of road ancillary facilities will be deployed on the expressway. Road ancillary facilities damaged by various reasons such as human factors, traffic accidents and rockfalls are mostly discovered by road maintenance personnel, which is inefficient and time-consuming.

[0003] With the continuous development of artificial intelligence, deep learning is being used more and more widely. However, for the problem of defect and damage detection of highway guardrails and barriers, the existing technology for real-time detection, classification, defect identification and positioning of highway guardrails and barriers still has low inspection efficiency and high labor input costs. Summary of the invention

[0004] In order to overcome the problems of low inspection efficiency and high manpower input for damaged road ancillary facilities existing in the related art, the disclosed embodiment of the present invention provides a road ancillary facilities defect detection method and system based on deep learning.

[0005] The technical solution is as follows: A road ancillary facility defect detection method based on deep learning, comprising:

[0006] S1. Use the deep convolutional network PointRend model to achieve image target detection, classification and accurate segmentation of guardrails and anti-glare panels;

[0007] S2. When there is a guardrail in the image in the image target detection, the damage state of the entire boundary of the guardrail is determined. The damage state of the guardrail includes bending deformation and notch fracture;

[0008] S3. When the image target is detected and it is inferred that there is an anti-glare plate in the image, the damage state of the anti-glare plate is determined to realize the deformation detection of the missing anti-glare plate.

[0009] In step S1, the image target detection, classification and accurate segmentation of the guardrail and anti-glare plate are realized, including:

[0010] S101, extracting part of the image data from the collected highway image data, marking the guardrails and anti-glare panels in the image, and calibrating the target area and setting classification labels for the image through image annotation software;

[0011] S102, using the labeled data as training data to train guardrails and anti-glare panels, and detect and classify them;

[0012] S103, a sequence of highway roadside images is acquired by the data acquisition system and stored in a fixed folder. During the actual measurement phase, the model interface is called to import the roadside images sequentially.

[0013] In step S102, the annotated data is used as training data to train guardrails and anti-glare panels, including: using a deep convolutional network PointRend model to train guardrails and anti-glare panels on highways, and detecting, locating, and segmenting and extracting guardrail and anti-glare panel targets.

[0014] Furthermore, the deep convolutional network PointRend model includes:

[0015] (1) Point selection: adaptively selecting points in the image plane to predict segmentation labels;

[0016] (2) Point-by-point feature representation: extract the representation features of the selected difficult points, encode the sub-pixel information along the channel dimension through linear interpolation calculation, and predict the segmentation;

[0017] (3) Point head: Use a multi-layer perceptron to perform point-by-point segmentation prediction. The multi-layer perceptron shares weights across all points and predicts a segmentation label for each point, trained using the segmentation loss of the task.

[0018] In step S103, during the actual measurement phase, the model interface is called to sequentially import roadside images, including: using the trained deep convolutional network PointRend model to infer whether the image contains guardrails or anti-glare panels, and if there is a target, the accurate location information of the target is given in the form of a vertex sequence of the circumscribed polygon of the target area and a binary mask; at the same time, the deep convolutional network PointRend model is returned to perform target classification label information on the image; the deep convolutional network PointRend model recognizes and classifies the target in the input image, and feeds back the result information;

[0019] The results of the roadside image inference by the deep convolutional network PointRend model are used to perform guardrail damage detection and anti-glare plate damage detection according to the specific target classification in the image.

[0020] In step S2, the damage state of the entire boundary of the guardrail is determined, including:

[0021] S201, extract the guardrail mask area, traverse and search the mask area along the column direction, and use the pixels in the first row of each column as the initial point candidate edge point. When there is a boundary point in this column, the first traversed point is used as the upper boundary candidate point, and the second point is used as the lower boundary candidate point. If there is a break in the guardrail mask area, there is only one initial candidate boundary point in this column, which is the upper boundary point by default.

[0022] S202, in the case of positive noise error in the data, using the least squares straight line fitting method to perform straight line fitting on the upper and lower boundaries respectively;

[0023] S203, in the guardrail straight line fitting, when there are abnormal wild points caused by breakage or bending in the candidate boundary pixels, the random sampling consistent RANSAC algorithm is used to select the optimal guardrail boundary inner point, and the least squares method is used to fit the optimal boundary line. The distance from all candidate edge points to the fitting boundary is used to determine whether there is a break or bend.

[0024] In step S203, when the distance from the candidate edge point to the straight line is greater than the high threshold T1, it is considered that there is a fracture damage that penetrates the guardrail in the image column corresponding to the pixel, thereby determining the position of the fracture area; when the distance from the candidate edge point to the straight line is between the low threshold T2 and the high threshold T1, it is considered to be a pixel on the curved boundary, thereby determining the curved area of ​​the guardrail.

[0025] In step S3, the damage state of the anti-glare plate is determined to realize the anti-glare plate missing deformation detection, including:

[0026] (1) Perform convex hull algorithm processing on the mask area generated by the deep convolutional network PointRend model to obtain all the key points of the convex hull; calculate the distance from the four vertex corners of the image to the convex hull key points, and find the convex hull key point farthest from each vertex;

[0027] (2) According to the properties of affine transformation, the determined quadrilateral is subjected to transmission transformation to achieve a frontal rectangular effect;

[0028] (3) Detection of tilt deformation damage of the anti-glare panel. If a pixel block smaller than the size appears in the area where the pixel block appears, the anti-glare panel is damaged and a tilt deformation damage type disease has occurred.

[0029] In step (1), the distances from the four vertex corners of the image to the convex hull key points are calculated, and the convex hull key points farthest from each vertex are found, including:

[0030] Call the opencv convex hull algorithm to draw and return the coordinates of the convex hull key points:

[0031] PS={P1:(x1,y1),P2:(x2,y2)…P n :(x n ,y n )}

[0032] Where PS is the coordinate set from the vertex to the convex hull key point, P1:(x1,y1) is the coordinate of the first vertex relative to the convex hull key point, P n :(x n ,y n ) is the coordinate of the nth vertex relative to the key point of the convex hull;

[0033] Take four vertices A(0,0), B(w,0), C(0,-h), and D(w,h) as reference points, w is the image width, and h is the image height;

[0034] Take the image vertex A as the reference point, calculate all the key points from vertex A to the convex hull, the farthest convex hull key point A′ is a vertex of the quadrilateral, determine all the vertices of the quadrilateral, and the four points are the circumscribed quadrilateral of the Mask. The expression is:

[0035]

[0036] Where, d A is the distance between two points;

[0037] In step (2), for the quadrilateral vertices (x i ,y i ) after the transmission transformation is (x′ i ,y′ i ); Set the coordinates of the transformed rectangle vertices to (0,0), (0,W), (H,0), (H,W); where W and H are the manually set width and length of the rectangle, respectively. The coordinate affine transformation formula is as follows:

[0038]

[0039] Where a, b, c are the coefficients of the fitted line relative to the horizontal coordinate of the affine transformation of the vertex coordinates, constants, and reverse heights of the image respectively; d, e, and f are the coefficients of the fitted line relative to the vertical coordinate of the affine transformation of the vertex coordinates, constants, and reverse heights of the image respectively; (x, y) are the coordinates of the vertices of the quadrilateral, and (x′, y′) are the vertices of the quadrilateral after the transmission transformation;

[0040] The inverse transformation of the matrix of the matching coordinate data is calculated to estimate the affine transformation matrix, and the original anti-glare plate area is affine transformed to correct it to a frontal rectangular shape; the horizontal coordinate position of each anti-glare plate is determined by non-maximum suppression of the corrected anti-glare plate Mask;

[0041] In step (3), the anti-glare panel is inspected for damage-type tilt deformation, including:

[0042] S3.1. Return the six-degree-of-freedom platform to zero and select six anti-glare panels as the test objects. The vector formed by the spatial lengths of the six anti-glare panels is G is the order, and the corresponding posture of the six-degree-of-freedom platform is:

[0043]

[0044] Wherein, g1 is the posture coordinate corresponding to the vector formed by the six-degree-of-freedom platform and the six anti-glare panel space lengths; s1, s2, s3, s4, s5, and s6 are respectively the vector value formed by the first anti-glare panel space length, the vector value formed by the second anti-glare panel space length, the vector value formed by the third anti-glare panel space length, the vector value formed by the fourth anti-glare panel space length, the vector value formed by the fifth anti-glare panel space length, and the vector value formed by the sixth anti-glare panel space length; I is the attitude angle of the first anti-glare panel, J is the attitude angle of the second anti-glare panel, and K is the attitude angle of the third anti-glare panel. It is the attitude angle of the 4th anti-glare panel. The 5th anti-glare panel attitude angle, is the attitude angle of the sixth anti-glare panel;

[0045] S3.2. Coordinate system 2 is established on the anti-glare board section 1 of the expressway. The origin coordinates and attitude angles are obtained through the posture change program in the detection software. There are 6 parameters in total, which are recorded as:

[0046]

[0047] The pose relationship change matrix between coordinate system 2 and operating platform coordinate system 1 is:

[0048]

[0049] Where O is the tilt deformation change matrix, the upper left script indicates the serial number of the fixed coordinate system during this change, the lower left script indicates the serial number of the moving coordinate system during this change, and 0 indicates the global coordinate system:

[0050] S3.3. A coordinate system 3 is established on the docking interface of the platform of the highway anti-glare board section 2. Through the posture change program, a total of 6 parameters including the origin coordinates and attitude angles are obtained, which are recorded as:

[0051]

[0052] S3.4, send the value of g2 in coordinate system 2 and the value of g3 in coordinate system 2 to the detection software of the host computer, and use the kinematic solution program in the alignment software to calculate the position and posture that coordinate system 2 will reach after adjustment, so as to minimize the height variance value in the alignment target, h k The selection of is determined by the following optimization method:

[0053]

[0054] Where h1 and h2 are the heights of highway anti-glare board sections 1 and 2 relative to the ground, respectively, and δh 12 It is the height difference between the parallel lines of the highest end surfaces of highway anti-glare plate sections 1 and 2.

[0055] Another object of the present invention is to provide a road ancillary facility defect detection system based on deep learning, which implements the road ancillary facility defect detection method based on deep learning, and the system comprises:

[0056] The guardrail target detection and classification module uses the deep convolutional network PointRend model to achieve accurate detection, classification and precise segmentation of guardrails and anti-glare panels;

[0057] The guardrail deformation and fracture detection module is used to determine the damage state of the neat border of the guardrail when it is inferred that there is a guardrail in the image through image target detection. The damage state of the guardrail includes bending deformation and notch fracture;

[0058] The anti-glare plate missing deformation detection module is used to determine the damage state of the anti-glare plate when it is inferred that the anti-glare plate exists in the image after image target detection.

[0059] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows: the present invention uses high-resolution cameras to collect data, relies on the big data processing advantages of deep supercomputing workstations, and uses intelligent video image processing and deep learning technology to perform real-time detection, classification, defect identification and positioning of highway guardrails and baffles. The present invention integrates a set of algorithm results into road detection equipment, which can more conveniently obtain the damage of ancillary facilities along the road. Combined with the relevant equipment of the detection vehicle and the association of road pile numbers, it can greatly reduce the work intensity and work efficiency of maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0061] Figure 1 is a flow chart of a road ancillary facility defect detection method based on deep learning provided by an embodiment of the present invention;

[0062] Figure 2 This is a comparison diagram of the least squares and RANSAC straight line fitting effects provided by an embodiment of the present invention;

[0063] Figure 3 is a schematic diagram of a circumscribed quadrilateral for determining a Mask provided by an embodiment of the present invention;

[0064] Figure 4 This is a tilt deformation disease detection diagram of the damage type of the anti-glare plate provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0066] Embodiment 1, as Figure 1 As shown, the road ancillary facilities defect detection method based on deep learning provided by an embodiment of the present invention includes:

[0067] S1. Use the deep convolutional network PointRend model to achieve image target detection, classification and accurate segmentation of guardrails and anti-glare panels;

[0068] Specifically include:

[0069] S101, extracting part of the image data from a large amount of collected highway roadside image data, and annotating the guardrails and anti-glare panels in the image by manual marking. Specifically, professional image annotation software is used to calibrate the target area of ​​the image and set classification labels.

[0070] S102, using the labeled data as training data to train guardrails and anti-glare panels, and detect and classify them;

[0071] For example, the deep convolutional network PointRend model not only improves the smoothness of the segmentation process and achieves accurate segmentation of image details, but also greatly saves resource consumption, with computing power requirements of only 2.6% of Mask R-CNN. Under the current trend of deploying artificial intelligence algorithm models on the edge, the design ideas of the deep convolutional network PointRend model framework can be specifically divided into three parts:

[0072] (1) Point Selection: Select a few "difficult points" for prediction to avoid over-computing all pixels in high-resolution output. The core idea of ​​PointRend is to adaptively select points in the image plane to predict segmentation labels. In theory, these points should be widely distributed in high-frequency areas (such as the edges of the image).

[0073] (2) Point-wise feature representation: extract the representation features of the selected "difficult points", which are calculated by linear interpolation, then encode sub-pixel information along the channel dimension and predict segmentation. PointRend constructs point-wise features at selected points by combining low-level features (fine-grained features) and high-level features (coarse prediction). In terms of fine-grained features, feature vectors are extracted for each sample point in the CNN feature map to present fine segmentation details. Although fine-grained features can resolve details, they also have two shortcomings: (a) they do not contain specific area information. For instance segmentation tasks, different labels may be predicted at the same point. (b) The feature map used for fine-grained features may only contain relatively low-level information. This requires coarse segmentation prediction to supplement and provide more global context. Taking instance segmentation as an example, coarse segmentation prediction can be the output of the 7×7 lightweight mask head in Mask R-CNN.

[0074] (3) Point head: A small neural network that predicts labels based on the point-by-point feature representation. For each selected point, PointRend uses a simple multilayer perceptron to make point-by-point segmentation predictions. The multilayer perceptron shares weights across all points (all regions). And, since the multilayer perceptron predicts segmentation labels for each point, it can be trained with a task-specific segmentation loss.

[0075] The present invention uses the PointRend model to achieve accurate detection, positioning, and precise segmentation and extraction of guardrails and anti-glare panels on highways.

[0076] S103, the data acquisition system acquires a sequence of highway roadside images and stores them in a fixed folder. In the actual measurement phase, the model interface is called to import the roadside images sequentially. The trained PointRend model can infer whether the image contains guardrails or anti-glare panels. If there is a target, the accurate location information of the target is given in the form of a sequence of circumscribed polygon vertices and a binary mask of the target area. At the same time, the PointRend model is returned to the target classification label information of the image, where the PointRend model predicts the input image, that is, it identifies and classifies the targets in the image and feeds back the result information.

[0077] The results of the roadside image inferred by the PointRend model will be processed separately in the subsequent anomaly detection according to the specific target classification in the image, and guardrail damage detection and anti-glare plate damage detection will be performed respectively.

[0078] Exemplarily, providing the accurate location information of the target in the form of a vertex sequence of the circumscribed polygon of the target area and a binary mask includes:

[0079] For binary image masks with targets, use the findContours method in opencv to detect contours, use the minAreaRect method to calculate the minimum bounding rectangle of the contour, and use the boxPoints method to calculate the minimum bounding rectangle to obtain the four-point coordinates of the minimum bounding rectangle of the contour. Therefore, the accurate location information of the target can be obtained.

[0080] S2. When there is a guardrail in the image in the image target detection, the damage state of the entire boundary of the guardrail is determined. The damage state of the guardrail includes bending deformation and notch fracture;

[0081] When the image target is detected in step S1 and it is inferred that there is a guardrail in the image, the damage state of the neat boundary of the guardrail is further determined. Bending deformation and notch fracture damage are mainly manifested as abnormal contour distortion and loss. Specifically, it includes:

[0082] S201, extract the guardrail Mask area, and then traverse and search the Mask area along the column direction. The pixel in the row of each column is used as the initial point candidate edge point. When there is a boundary point in this column, the first traversed point is used as the upper boundary candidate point, and the second point is used as the lower boundary candidate point. If the guardrail Mask is broken, there is only one initial candidate boundary point in this column, which is the upper boundary point by default.

[0083] S202, then use the least squares straight line fitting method to fit the upper and lower boundaries respectively. The basic idea of ​​the least squares method is: using the given n candidate boundary points, find an optimal fitting straight line:

[0084] y=Ax+B

[0085] In the formula, y is the ordinate value of the fitted line, A is the coefficient of the fitted line, x is the abscissa value, and B is a constant;

[0086] The best fitting line should pass through or be close to all candidate points as much as possible. Its essence is to find the best values ​​of parameters a and b. The principle of the least squares method is as follows: given a set of points {(x i ,y i )|1≤i≤n}, the error sum of squares E is:

[0087]

[0088] When the straight line minimizes the sum of squared errors, the partial derivative of the straight line parameters is taken, and the parameters a and b satisfy:

[0089]

[0090] This method can be used to estimate the values ​​of the straight line parameters A and B, and then obtain the fitted straight line. It only considers the optimal straight line fit under the condition of positive noise error of the data.

[0091] S203, in the guardrail straight line fitting, when there are abnormal "wild points" caused by breaks or bends in the candidate boundary pixels, the random sampling consistent RANSAC algorithm is used to select the optimal guardrail boundary inner point, and the least squares method is used to fit the optimal boundary line; then, the distance from all candidate edge points to the fitted boundary is used to determine whether there are breaks and bends.

[0092] When the distance from the candidate edge point to the straight line is greater than the high threshold T1, it is considered that there is a fracture damage that penetrates the guardrail in the image column corresponding to the pixel, thereby determining the location of the fracture area. When the distance from the candidate edge point to the straight line is between the low threshold T2 and the high threshold T1, it is considered to be a pixel on the curved boundary, thereby determining the curved area of ​​the guardrail;

[0093] It can be understood that in the guardrail straight line fitting, when there are abnormal "wild points" caused by breaks or bends in the candidate boundary pixels, the random sampling consensus (RANSAC) algorithm is used to achieve boundary fitting in the "wild point" environment to solve the errors that are bound to occur when using least squares fitting.

[0094] To this end, the present invention uses the random sampling consensus (RANSAC) algorithm to achieve boundary fitting in a "wild point" environment. It can estimate the optimal model parameters from a set of observation data sets containing "wild points" in an iterative manner. In principle, the more iterations there are, the greater the probability of obtaining the optimal model. The core idea is randomness and assumptions. Randomness is used to reduce calculations, and the number of cycles is the probability of using correct data. The so-called assumption means that the randomly extracted data are considered correct, and other points are calculated based on this to obtain other points that satisfy the transformation relationship, and then the voting mechanism is used to select the fitting line with the most votes as the optimal fit of the boundary.

[0095] Among them, the difference between random sampling consistent RANSAC and least squares is that the least squares method tries to adapt to all points including "wild points". On the contrary, RANSAC can derive a model that only uses local points to calculate the model, and the probability is high enough. Figure 2 Comparison of the effects of least squares and RANSAC fitting straight lines;

[0096] S3, anti-glare plate missing deformation detection: after the image target detection in step S2 and it is inferred that there is an anti-glare plate in the image, further determine the damage state of the anti-glare plate.

[0097] Common damage states of anti-glare panels include loss of anti-glare panels and tilted deformation. In the case of anti-glare panel loss, the distance relationship between anti-glare panels is used for judgment. Specifically:

[0098] (1) Perform convex hull algorithm processing on the mask generated by the PointRend model to obtain all the key points of the convex hull. Calculate the distance from the four vertex points of the image to the convex hull key points respectively, and find the convex hull key point farthest from each vertex;

[0099] Among them, in OpenCV, the convex hull algorithm is an algorithm for finding the minimum convex polygon of a point set, which can help extract contour features in an image. For binary images, the convex hull algorithm usually involves the following steps:

[0100] Image preprocessing: First, the image needs to be converted into a binary image. This is usually achieved through a threshold operation to distinguish the foreground from the background. The Mask generated by PointRend is a binary image.

[0101] Contour Detection: In a binary image, use the findContours function to detect contours in the image. These contours are the boundaries of foreground objects in the image.

[0102] Convex hull calculation: For each detected contour, use the convexHull function to calculate its convex hull. The convex hull is a set of points that form a minimum convex polygon that can contain all the points in the contour.

[0103] For example, Figure 3 Determine the circumscribed quadrilateral of the Mask as shown in the schematic diagram: the red box is the convex hull drawn by calling the opencv convex hull algorithm, and the coordinates of the key points of the convex hull are returned:

[0104] PS={P1:(x1,y1),P2:(x2,y2)…P n :(x n ,y n )}

[0105] Where PS is the coordinate set from the vertex to the convex hull key point, P1:(x1,y1) is the coordinate of the first vertex relative to the convex hull key point, P n :(x n ,y n ) is the coordinate of the nth vertex relative to the key point of the convex hull;

[0106] Take four vertices A(0,0), B(w,0), C(0,-h), and D(w,h) as reference points, w is the image width, and h is the image height;

[0107] Take the image vertex A as the reference point, calculate all the key points from vertex A to the convex hull, the farthest convex hull key point A′ is a vertex of the quadrilateral, determine all the vertices of the quadrilateral, and the four points are the circumscribed quadrilateral of the Mask. The expression is:

[0108]

[0109] Where, d A is the distance between two points;

[0110] (2) According to the properties of affine transformation, the above-determined quadrilateral is subjected to a transmission transformation to achieve a frontal rectangular effect. Specifically, it is assumed that the vertices (x i ,y i ), the four points after the transmission transformation are (x′ i ,y′ i ). In this project, the coordinates of the transformed rectangle vertices are set to (0,0), (0,W), (H,0), (H,W), where W and H are the reasonable width and length of the rectangle set manually. Then the coordinate affine transformation formula is as follows:

[0111]

[0112] Where a, b, c are the coefficients of the fitted line relative to the horizontal coordinate of the affine transformation of the vertex coordinates, constants, and reverse heights of the image respectively; d, e, and f are the coefficients of the fitted line relative to the vertical coordinate of the affine transformation of the vertex coordinates, constants, and reverse heights of the image respectively; (x, y) are the coordinates of the vertices of the quadrilateral, and (x′, y′) are the vertices of the quadrilateral after the transmission transformation;

[0113] The affine transformation matrix is ​​estimated by using the matrix inverse transformation of the four pairs of matching coordinate data. Then the original anti-glare plate area is affine transformed to correct it to the front rectangular shape, so that the relative distance and size between the anti-glare plates are consistent after processing.

[0114] The horizontal coordinate position of each anti-glare plate is determined by non-maximum suppression of the corrected anti-glare plate Mask.

[0115] Among them, non-maximum suppression, NMS (nonmaximum suppression). The Canny operator in Opencv uses nms;

[0116] The Canny operator is one of the most classic and advanced algorithms in image edge detection algorithms. Compared with the Sobel and Prewitt operators, the Canny algorithm is more excellent. The Canny algorithm proposes two improvements: non-maximum suppression based on edge gradient direction, and double-threshold hysteresis threshold processing.

[0117] The Canny algorithm is based on three basic goals:

[0118] (i) Low error rate, all edges should be found with no spurious responses.

[0119] (ii) Edge points will be well localized and the localized edges must be as close to the true edges as possible.

[0120] (iii) Single edge point response: The detector should not indicate multiple pixel edges at locations where there is only a single edge point.

[0121] The steps of Canny algorithm are as follows:

[0122] (a) Gaussian smoothing is performed on the input image to reduce the error rate.

[0123] (b) Calculate the gradient magnitude and direction to estimate the edge strength and direction at each point.

[0124] The calculation process of the gradient magnitude generally uses the Sobel operator horizontal and vertical operators to convolve with the input image to calculate d x ,d y ;

[0125]

[0126] d x =∫(x,y)*SobelX(x,y)

[0127] d y =∫(x,y)*SobelY(x,y)

[0128] Where SobelX is the horizontal operator, SobelY is the vertical operator, and (x, y) is the horizontal and vertical coordinates.

[0129] Get the magnitude of the image gradient:

[0130]

[0131] Where M(x,y) is the magnitude of the image gradient;

[0132] (c) According to the gradient direction, non-maximum suppression is performed on the gradient amplitude.

[0133] (d) Double thresholding and edge connection.

[0134] (3) Detection of tilt deformation damage of anti-glare panels, such as Figure 4 As shown, Figure 4 The yellow block in the figure indicates that if a pixel block that is less than the size appears in the area where the pixel block should appear, then the anti-glare board is suspected to be damaged, that is, the anti-glare board has a damage type of disease. For example, under normal circumstances, an anti-glare board is set every 30 cm, which is reflected in the figure that there is a white block every 30 pixels, and the height of the block changes gradually according to the projection rule. Similarly, there should be a full-size block in the red frame area, but only a yellow block that is less than the size appears, then this place can be listed as a suspected damage location of the anti-glare board.

[0135] Exemplarily, in step (3), the detection of the tilt deformation damage of the anti-glare plate specifically includes:

[0136] S3.1. Return the six-degree-of-freedom platform to zero and select six anti-glare panels as the test objects. The vector formed by the spatial lengths of the six anti-glare panels is G is the order, and the corresponding posture of the six-degree-of-freedom platform is:

[0137]

[0138] Wherein, g1 is the posture coordinate corresponding to the vector formed by the six-degree-of-freedom platform and the six anti-glare panel space lengths; s1, s2, s3, s4, s5, and s6 are respectively the vector value formed by the first anti-glare panel space length, the vector value formed by the second anti-glare panel space length, the vector value formed by the third anti-glare panel space length, the vector value formed by the fourth anti-glare panel space length, the vector value formed by the fifth anti-glare panel space length, and the vector value formed by the sixth anti-glare panel space length; I is the attitude angle of the first anti-glare panel, J is the attitude angle of the second anti-glare panel, and K is the attitude angle of the third anti-glare panel. It is the attitude angle of the 4th anti-glare panel. The 5th anti-glare panel attitude angle, is the attitude angle of the sixth anti-glare panel;

[0139] S3.2. Coordinate system 2 is established on the anti-glare board section 1 of the expressway. The origin coordinates and attitude angles are obtained through the posture change program in the detection software. There are 6 parameters in total, which are recorded as:

[0140]

[0141] The pose relationship change matrix between coordinate system 2 and operating platform coordinate system 1 is:

[0142]

[0143] Where O is the tilt deformation change matrix, the upper left script indicates the serial number of the fixed coordinate system during this change, the lower left script indicates the serial number of the moving coordinate system during this change, and 0 indicates the global coordinate system:

[0144] S3.3. A coordinate system 3 is established on the docking interface of the platform of the highway anti-glare board section 2. Through the posture change program, a total of 6 parameters including the origin coordinates and attitude angles are obtained, which are recorded as:

[0145]

[0146] S3.4, send the value of g2 in coordinate system 2 and the value of g3 in coordinate system 2 to the detection software of the host computer, and use the kinematic solution program in the alignment software to calculate the position and posture that coordinate system 2 will reach after adjustment, so as to minimize the height variance value in the alignment target, h k The selection of is determined by the following optimization method:

[0147]

[0148] Where h1 and h2 are the heights of highway anti-glare board sections 1 and 2 relative to the ground, respectively, and δh 12 It is the height difference between the parallel lines of the highest end surfaces of highway anti-glare plate sections 1 and 2.

[0149] Embodiment 2, the present invention provides a road ancillary facility defect detection system based on deep learning, comprising:

[0150] Guardrail target detection and classification module, which uses the deep convolutional network PointRend model to accurately detect, classify and segment guardrails and anti-glare panels;

[0151] The guardrail deformation and fracture detection module is used to further determine the damage state of the neat border of the guardrail when it is inferred that there is a guardrail in the image through image target detection. The damage state of the guardrail includes bending deformation and notch fracture;

[0152] The anti-glare plate missing deformation detection module is used to further determine the damage status of the anti-glare plate when it is inferred that the anti-glare plate exists in the image after image target detection.

[0153] The present invention can be integrated into the portable hardware device of the road monitoring vehicle and used in combination with the equipment in the detection vehicle to convert the damage information of the ancillary facilities identified along the way into pile numbers and feed them back to the road maintenance personnel, so as to facilitate the road maintenance personnel to count the damage of the road ancillary facilities and replace the relevant facilities in time.

[0154] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A road ancillary facilities defect detection method based on deep learning, characterized in that: The method includes: S1. Use the deep convolutional network PointRend model to achieve image target detection, classification and accurate segmentation of guardrails and anti-glare panels; S2. When there is a guardrail in the image in the image target detection, the damage state of the entire boundary of the guardrail is determined. The damage state of the guardrail includes bending deformation and notch fracture; S3. When the image target is detected and it is inferred that there is an anti-glare plate in the image, the damage state of the anti-glare plate is determined to realize the deformation detection of the missing anti-glare plate.

2. The road ancillary facilities defect detection method based on deep learning according to claim 1 is characterized in that: In step S1, the image target detection, classification and accurate segmentation of the guardrail and anti-glare plate are realized, including: S101, extracting part of the image data from the collected highway image data, marking the guardrails and anti-glare panels in the image, and calibrating the target area and setting classification labels for the image through image annotation software; S102, using the labeled data as training data to train guardrails and anti-glare panels, and detect and classify them; S103, a sequence of highway roadside images is acquired by the data acquisition system and stored in a fixed folder. During the actual measurement phase, the model interface is called to import the roadside images sequentially.

3. The road ancillary facilities defect detection method based on deep learning according to claim 2 is characterized in that: In step S102, the annotated data is used as training data to train guardrails and anti-glare panels, including: using a deep convolutional network PointRend model to train guardrails and anti-glare panels on highways, and detecting, locating, and segmenting and extracting guardrail and anti-glare panel targets.

4. The road ancillary facilities defect detection method based on deep learning according to claim 3 is characterized in that: Deep convolutional network PointRend model, including: (1) Point selection: adaptively selecting points in the image plane to predict segmentation labels; (2) Point-by-point feature representation: extract the representation features of the selected difficult points, encode the sub-pixel information along the channel dimension through linear interpolation calculation, and predict the segmentation; (3) Point head: Use a multi-layer perceptron to perform point-by-point segmentation prediction. The multi-layer perceptron shares weights across all points and predicts a segmentation label for each point, trained using the segmentation loss of the task.

5. The road ancillary facilities defect detection method based on deep learning according to claim 2 is characterized in that: In step S103, during the actual measurement phase, the model interface is called to sequentially import roadside images, including: using the trained deep convolutional network PointRend model to infer whether the image contains guardrails or anti-glare panels, and if there is a target, the accurate location information of the target is given in the form of a vertex sequence of the circumscribed polygon of the target area and a binary mask; at the same time, the deep convolutional network PointRend model is returned to perform target classification label information on the image; the deep convolutional network PointRend model recognizes and classifies the target in the input image, and feeds back the result information; The results of the roadside image inference by the deep convolutional network PointRend model are used to perform guardrail damage detection and anti-glare plate damage detection according to the specific target classification in the image.

6. The road ancillary facilities defect detection method based on deep learning according to claim 1, characterized in that: In step S2, the damage state of the entire boundary of the guardrail is determined, including: S201, extract the guardrail mask area, traverse and search the mask area along the column direction, and use the pixels in the first row of each column as the initial point candidate edge point. When there is a boundary point in this column, the first traversed point is used as the upper boundary candidate point, and the second point is used as the lower boundary candidate point. If there is a break in the guardrail mask area, there is only one initial candidate boundary point in this column, which is the upper boundary point by default. S202, in the case of positive noise error in the data, using the least squares straight line fitting method to perform straight line fitting on the upper and lower boundaries respectively; S203, in the guardrail straight line fitting, when there are abnormal wild points caused by breakage or bending in the candidate boundary pixels, the random sampling consistent RANSAC algorithm is used to select the optimal guardrail boundary inner point, and the least squares method is used to fit the optimal boundary line. The distance from all candidate edge points to the fitting boundary is used to determine whether there is a break or bend.

7. The road ancillary facilities defect detection method based on deep learning according to claim 6 is characterized in that: In step S203, when the distance from the candidate edge point to the straight line is greater than the high threshold T1, it is considered that there is a fracture damage that penetrates the guardrail in the image column corresponding to the pixel, thereby determining the position of the fracture area; when the distance from the candidate edge point to the straight line is between the low threshold T2 and the high threshold T1, it is considered to be a pixel on the curved boundary, thereby determining the curved area of ​​the guardrail.

8. The road ancillary facilities defect detection method based on deep learning according to claim 1, characterized in that: In step S3, the damage state of the anti-glare plate is determined to realize the anti-glare plate missing deformation detection, including: (1) Perform convex hull algorithm processing on the mask area generated by the deep convolutional network PointRend model to obtain all the key points of the convex hull; calculate the distance from the four vertex corners of the image to the convex hull key points, and find the convex hull key point farthest from each vertex; (2) According to the properties of affine transformation, the determined quadrilateral is subjected to transmission transformation to achieve a frontal rectangular effect; (3) Detection of tilt deformation damage of the anti-glare panel. If a pixel block smaller than the size appears in the area where the pixel block appears, the anti-glare panel is damaged and a tilt deformation damage type disease has occurred.

9. The road ancillary facilities defect detection method based on deep learning according to claim 8, characterized in that: In step (1), the distances from the four vertex corners of the image to the convex hull key points are calculated, and the convex hull key points farthest from each vertex are found, including: Call the opencv convex hull algorithm to draw and return the coordinates of the convex hull key points: PS={P1:(x1,y1),P2:(x2,y2)…P n :(x n ,y n )} Where PS is the coordinate set from the vertex to the convex hull key point, P1:(x1,y1) is the coordinate of the first vertex relative to the convex hull key point, P n :(x n ,y n ) is the coordinate of the nth vertex relative to the key point of the convex hull; Take four vertices A(0,0), B(w,0), C(0,-h), and D(w,h) as reference points, w is the image width, and h is the image height; Take the image vertex A as the reference point, calculate all the key points from vertex A to the convex hull, the farthest convex hull key point A′ is a vertex of the quadrilateral, determine all the vertices of the quadrilateral, and the four points are the circumscribed quadrilateral of the Mask. The expression is: Where, d A is the distance between two points; In step (2), for the quadrilateral vertices (x i ,y i ) after the transmission transformation is (x′ i ,y′ i ); Set the coordinates of the transformed rectangle vertices to (0,0), (0,W), (H,0), (H,W); where W and H are the manually set width and length of the rectangle, respectively. The coordinate affine transformation formula is as follows: Where a, b, c are the coefficients of the fitted line relative to the horizontal coordinate of the affine transformation of the vertex coordinates, constants, and reverse heights of the image after the affine transformation of the vertex coordinates; d, e, f are the coefficients of the fitted line relative to the vertical coordinate of the affine transformation of the vertex coordinates, constants, and reverse heights of the image after the affine transformation of the vertex coordinates; (x, y) are the coordinates of the vertices of the quadrilateral, and (x′, y′) are the vertices of the quadrilateral after the transmission transformation; The inverse transformation of the matrix of the matching coordinate data is calculated to estimate the affine transformation matrix, and the original anti-glare plate area is affine transformed to correct it to a frontal rectangular shape; the horizontal coordinate position of each anti-glare plate is determined by non-maximum suppression of the corrected anti-glare plate Mask; In step (3), the anti-glare panel is inspected for damage-type tilt deformation, including: S3.

1. Return the six-degree-of-freedom platform to zero and select six anti-glare panels as the test objects. The vector formed by the spatial lengths of the six anti-glare panels is G is the order, and the corresponding posture of the six-degree-of-freedom platform is: Wherein, g1 is the posture coordinate corresponding to the vector formed by the six-degree-of-freedom platform and the six anti-glare panel space lengths; s1, s2, s3, s4, s5, and s6 are respectively the vector value formed by the first anti-glare panel space length, the vector value formed by the second anti-glare panel space length, the vector value formed by the third anti-glare panel space length, the vector value formed by the fourth anti-glare panel space length, the vector value formed by the fifth anti-glare panel space length, and the vector value formed by the sixth anti-glare panel space length; I is the attitude angle of the first anti-glare panel, J is the attitude angle of the second anti-glare panel, and K is the attitude angle of the third anti-glare panel. It is the attitude angle of the 4th anti-glare panel. The 5th anti-glare panel attitude angle, is the attitude angle of the sixth anti-glare panel; S3.

2. Coordinate system 2 is established on the anti-glare board section 1 of the expressway. The origin coordinates and attitude angles are obtained through the posture change program in the detection software. There are 6 parameters in total, which are recorded as: The pose relationship change matrix between coordinate system 2 and operating platform coordinate system 1 is: Where O is the tilt deformation change matrix, the upper left script indicates the serial number of the fixed coordinate system during this change, the lower left script indicates the serial number of the moving coordinate system during this change, and 0 indicates the global coordinate system: S3.

3. A coordinate system 3 is established on the docking interface of the platform of the highway anti-glare board section 2. Through the posture change program, a total of 6 parameters including the origin coordinates and attitude angles are obtained, which are recorded as: S3.4, send the value of g2 in coordinate system 2 and the value of g3 in coordinate system 2 to the detection software of the host computer, and use the kinematic solution program in the alignment software to calculate the position and posture that coordinate system 2 will reach after adjustment, so as to minimize the height variance value in the alignment target, h k The selection of is determined by the following optimization method: Where h1 and h2 are the heights of highway anti-glare board sections 1 and 2 relative to the ground, respectively, and δh 12 It is the height difference between the parallel lines of the highest end surfaces of highway anti-glare plate sections 1 and 2.

10. A road ancillary facilities defect detection system based on deep learning, characterized in that: The system implements the road ancillary facility defect detection method based on deep learning as described in any one of claims 1 to 9, and the system comprises: The guardrail target detection and classification module uses the deep convolutional network PointRend model to achieve accurate detection, classification and precise segmentation of guardrails and anti-glare panels; The guardrail deformation and fracture detection module is used to determine the damage state of the neat border of the guardrail when it is inferred that there is a guardrail in the image through image target detection. The damage state of the guardrail includes bending deformation and notch fracture; The anti-glare plate missing deformation detection module is used to determine the damage state of the anti-glare plate when it is inferred that the anti-glare plate exists in the image after image target detection.

Images