Road defect detection method based on deep learning

Through deep learning technology, combined with image preprocessing and multi-scale feature extraction, the problems of low efficiency and poor accuracy of traditional road defect detection methods are solved, and accurate identification and classification of different types of road defects are achieved, and detailed damage level judgment is provided. , improves the accuracy and robustness of detection.

CN118918551AActive Publication Date: 2024-11-08CHENGDU HONGYUNDING TECH CO LTD

Patent Information

Application Number
CN202410949985.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Traditional road defect detection methods are inefficient and costly, have poor real-time performance and identification accuracy problems, making them difficult to adapt to road defects of different scales and types, and lack in-depth distinction between defect types and damage level judgments.

Method used

Using deep learning-based road defect detection method, the road defect detection of all-round image data preprocessing, multi-scale feature extraction, object detection and classification algorithms can be used to identify and classify different types of road defects, and calculate the degree of damage, using image preprocessing and improved Weighted average filtering and jump structure noise reduction technology combines attention mechanism and multi-scale feature extraction to achieve accurate identification and classification of road defects.

Benefits of technology

Improve the accuracy and robustness of road defect detection, reduce noise impact, provide more comprehensive feature representation and more accurate classification results, can finely quantify road surface damage, and divide defects into damage levels, support Intelligent traffic management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918551A_ABST
    Figure CN118918551A_ABST
Patent Text Reader

Abstract

The invention provides a road defect detection method based on deep learning. The method comprises the steps that noise frames are screened by calculating inter-frame similarity, weighted pixels are used for replacement, an improved weighted average filter is used for smooth processing and multi-frame noise reduction, and a new image is obtained; extracting a feature pyramid of an input image by using an improved multi-scale cavity convolution structure, and extracting features in each layer by applying a multi-scale G-LBP operator to obtain an image multi-scale feature map; designing a target detection algorithm to recognize defect positions and bounding boxes on the feature map, and designing a convolution attention module to detect road defect targets from different scales; constructing a road defect classifier, and classifying detected targets according to pothole defects and crack defects; calculating the damage degree according to the defect type and grading to obtain the grade of each road defect; according to the invention, different types of obstacles on the road can be effectively identified and classified, errors and cost of manual detection are reduced, and decision support is provided for road maintenance and management.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and intelligent transportation systems, and in particular to a road defect detection method based on deep learning, which is used to automatically identify and classify different types of obstacles on the road to reduce the errors and costs of manual detection and provide support for road maintenance and traffic safety management. Background Art

[0002] Computer vision is solving more and more practical applications. Among them, the accuracy of road defect detection in the field of smart transportation depends on efficient image preprocessing technology. There are many types of road defects, and their shapes and sizes make detection difficult. Effective preprocessing of road defect images can play a vital role in identifying and classifying different types of defects and improving the accuracy of judging the degree of damage of road defects. With the continuous advancement of urbanization, the increase in road use frequency and traffic density has made road defect detection and maintenance more important. The timely discovery and repair of road defects are of great significance to ensuring traffic safety, improving road service life and optimizing traffic management. However, traditional road defect detection methods mainly rely on manual inspections or formulaic image detection, which is inefficient, costly, and has certain safety hazards. With the rapid development of computer vision and artificial intelligence technology, road defect detection technology based on deep learning can automatically detect and classify different types of road defects from road video streams, providing key information support for road maintenance and traffic management.

[0003] Traditional road defect detection methods still have problems with poor real-time performance and recognition accuracy. Usually, traditional road defect detection methods use a single preprocessing method for different defect images, and the processed images still have noise that affects subsequent recognition; at the same time, image defect features are affected by acquisition, and traditional feature extraction methods are single and difficult to adapt to road defects of different scales and types; and traditional defect classification methods are simple and lack in-depth distinction of defect types; in addition, traditional methods usually do not classify detected defects and judge the damage level, and cannot provide users with more effective intelligent information support. Summary of the invention

[0004] The purpose of the present invention is to provide a deep learning road defect detection method, which uses all-round image data preprocessing and efficient target recognition algorithms to identify and classify different types of road defects, determine the degree of damage and upload it to a server simultaneously.

[0005] The road defect detection method based on deep learning includes the following steps:

[0006] S1: Based on the original image data, an image preprocessor is constructed to perform multi-frame smoothing and noise reduction preprocessing operations based on time sequence;

[0007] S2: Based on the improved multi-scale feature extraction algorithm, feature value extraction is performed on the processed image to obtain different low, medium and high-level features of current road surface cracks, potholes and other defects;

[0008] S3: Build a target detection algorithm to identify road defects in the feature map;

[0009] S4: Construct a road defect classifier to classify detected targets into pothole defects and crack defects;

[0010] S5: Calculate the damage degree according to different road defect categories and classify the damage levels;

[0011] S6: Transmit the identified road defect information back to the vehicle display for early warning, and upload the relevant information to the cloud for analysis and statistics;

[0012] Further, the step S1 specifically includes:

[0013] S101: For each video stream to be processed, decompose the video into individual frames;

[0014] S102: intercepting k frames of continuous images, selecting the middle frame as a key frame, filtering the noise frame according to the similarity between frames, and performing pixel weighted replacement on the noise frame;

[0015] S103: performing a smoothing operation on the received image using an improved adaptive weighted average filtering method;

[0016] S104: Arrange the smoothed images in time sequence and perform time sequence noise reduction processing based on the jump structure;

[0017] Further, the step S102 specifically includes:

[0018] First, the similarity between each frame image and the key frame image is calculated, where the similarity calculation formula is:

[0019]

[0020] Among them, i, Represent the current frame and the middle frame image of the video, μ i , Represent the mean values ​​of the pixels of the two frames, Represent the variance of the pixels of the two frames, cov(·) is the covariance function, σ i is the standard deviation of the Gaussian function of the i-th frame image, C1 and C2 are stability constants;

[0021] Then, a similarity threshold τ is set to filter out noise frames whose similarity to the key frame is less than the threshold τ;

[0022] Furthermore, the pixel weight of the noise frame image is calculated, where the pixel weight calculation formula is:

[0023]

[0024] Among them, Rmse(·) is the root mean square error function, and R(·) is the range function;

[0025] Finally, the noise frame pixels are weighted replaced according to the weights, where the pixel weighted replacement formula is:

[0026]

[0027] Among them, I w (x, y) is the weighted pixel value, I i (x,y),I j (x, y) represents the pixel value of the i-th noise image and the j-th normal frame image closest to it at the coordinate (x, y), w i is the weight of the i-th frame image, i is the current time point;

[0028] Further, the step S103 specifically includes:

[0029] (1) For a given filtering window S xy , calculate S xy The minimum gray value Z of each pixel (x, y) in min , maximum gray value Z max And the average gray value Z of all pixels mean , and set the noise judgment threshold T;

[0030] (2) Determine whether the pixel mean of the filter window is a noise point. If T min <(Z min +Z max ) / Z mean <T max If true, then the mean Z mean is a normal point, otherwise, expand the window and continue to search for the normal mean;

[0031] (3) Determine Z min w (x,y) <Z max Whether the condition is met, if so, the pixel value of the point is output, otherwise the Euclidean distance is used to calculate the difference between each pixel of the input image and the pixels in its neighborhood;

[0032] (4) Calculate the pixel weight according to the difference value. The calculation formula of the pixel weight is:​

[0033]

[0034] Among them, W(x,y) represents the weight of the pixel point (x,y), Indicates the pixel at this point and the center pixel (c x ,c y ) difference value;

[0035] (5) Multiply the weight by the corresponding pixel value to calculate the filtered pixel value. The calculation method of the filtered pixel value is:

[0036]

[0037] Among them, FP(x,y) represents the pixel value after filtering, I w (x,y) represents the pixel value in the original image;

[0038] (6) Repeat the steps until all pixels are traversed;

[0039] (7) When there are still noisy pixels after the window reaches the maximum size, the noisy pixels are replaced by Gaussian filtering;

[0040] Further, the step S104 specifically includes:

[0041] First, k frames of continuous images are passed into the encoder, and then downsampled through three convolution kernels of different sizes to extract feature information, and the encoded feature Z is obtained. E It is expressed as:

[0042]

[0043] Then, the encoded feature Z E Divide every n into a group in chronological order, and divide into k-n+1 groups in total, as the input of the bidirectional recurrent neural network combined with the residual in the skip structure, and learn the forward and reverse time series correlation in the time series;

[0044] Finally, the k-n+1 groups of time series are input into the decoder for upsampling, and matched and spliced ​​with the corresponding positions in the encoder to obtain the final denoised image;

[0045] Among them, the steps of learning the bidirectional recurrent neural network combined with the residual are:

[0046] First, calculate the forward and reverse reset gates at time t respectively and propagation gate

[0047]

[0048]

[0049] in, The forward and reverse reset gates respectively determine to what extent the hidden state information of the previous moment is reset, thereby allowing the network to forget irrelevant information; are the forward and reverse propagation gates, respectively, which control the extent to which the new candidate hidden state information is updated to the current hidden state; chx(·) and shx(·) are the hyperbolic cosine activation function and hyperbolic sine activation function used for gating degree scaling in the reset gate and propagation gate, respectively; W o , U o , b o , W s , U s , b s is the gating parameter of the model's forward and reverse propagation, W o and W s Represents the weight matrix between the reset gate and propagation gate layers, U o and U s denote the time step matrices of the reset gate and the propagation gate, respectively, and b o and b s They represent the bias matrices of the reset gate and the propagation gate respectively;

[0050] Then, the candidate hidden states for the forward and reverse directions are calculated separately

[0051]

[0052] in, They are the candidate hidden states for the forward and reverse directions, respectively, and contain the information of the current input and the hidden state at the previous moment; W f , W b , U f , U b , b f , b b are the model forward and reverse gating parameters, W f , W b is the weight matrix for forward and reverse learning, U f , U b is the time step matrix of forward and reverse learning, b f , b b is the bias matrix for forward and reverse learning; H is the H-Swith(·) activation function, and the H-Swith activation function formula is as follows:

[0053]

[0054] ReLU6=min(ReLU,6)

[0055] Finally, according to the propagation gate and the candidate hidden state, the current forward and reverse hidden states are Concatenate into the final hidden state h t , and with the input encoding feature Z E Make residual connection to get the final time series feature Z R :

[0056]

[0057] Z R =Z E +h t

[0058] Among them, h t is the final hidden state, which is the concatenation of the forward and reverse hidden states; Z R is the final time series feature, which is the input encoding feature Z E and the hidden state h t The residual connection of [:] is the concatenation operation, and ⊙ is the element-by-element multiplication.

[0059] Further, the step S2 specifically includes:

[0060] S201: Apply the improved multi-expansion rate fusion dilated convolution operation on the input image to obtain the preliminary feature map C1-C of the input image n And the corresponding sub-feature maps D1-D at different depths n ;

[0061] S202: In feature map C1-C n A cross attention layer is introduced between each layer, and the correlation between its sub-feature maps is learned through the designed AN network;

[0062] S203: feature map C1-C n Different sub-feature graphs D * Perform multi-scale fusion to obtain the final feature pyramid P;

[0063] S204: Apply a multi-scale G-LBP operator on each layer of the obtained feature pyramid P to extract features and obtain a feature map;

[0064] Further, the step S201 specifically includes:

[0065] First, n convolutional layers of different sizes are used to extract multi-scale features of the image, and n preliminary feature maps C1-C n ;

[0066] Then, in the feature map C1-C nThe dilated convolution operation with multiple dilation rates is used to perform convolution respectively to obtain the sub-feature maps D1-D corresponding to different depths. n ;

[0067] Among them, the hole convolution operation with multiple dilation rates is defined as follows:

[0068] The expansion rate of the i-th convolution layer Rate i Represents the distance expansion rate between adjacent elements in the convolution kernel of this layer, Rate i Defined as:

[0069] Rate i =f(i)

[0070] Among them, f(i) is the hierarchical function, f(i) = 2 i-1 ;

[0071] The actual effective convolution size V of the i-th layer convolution kernel size is k i It can be expressed as:

[0072] V i =(k-1)*Rate i +1

[0073] The operation of the i-th layer image convolution can be expressed as:

[0074]

[0075] Among them, I(i,j) and K(i,j) are the pixel values ​​at the (i,j) coordinates of the image and the convolution kernel respectively;

[0076] Further, the step S202 specifically includes:

[0077] First, calculate the attention weights of different feature maps:

[0078] W = softmax(AN(D i ,D j ))

[0079] Among them, D i , D j Represents different sub-feature graphs;

[0080] Then, apply the weights to the category feature map to obtain the fused feature map D:

[0081] D=W*D i

[0082] Finally, the fused feature map D is combined with the original category sub-feature map D i Merge to get a new feature map D * :

[0083] D * =Merge(D,D i )

[0084] Among them, D * represents the final sub-feature map, Merge(·) represents the concatenation operation;

[0085] Further, the step S204 specifically includes:

[0086] (1) Divide the image into n m×m grids, each of which includes a different radius r and a different sampling point p;

[0087] (2) Calculate the MC-LBP operator eigenvalue M where r is r1 and p is p1 in each local grid. The calculation formula of the eigenvalue M is as follows:

[0088]

[0089] Among them, M p,r (x, y) represents the MC-LBP operator eigenvalue, p is the number of sampling points, r is the radius, (x, y) is the coordinate of the center point, a i is the angle information of the i-th sampling point, s(·) is the threshold function, and the threshold function formula is as follows:

[0090]

[0091] Among them, θ i is the angle of the i-th sampling point, g(x,y), g(x+rcos(θ i ),y+rsin(θ i )) are the grayscale values ​​of the center pixel and the sampling point respectively;

[0092] (3) According to each eigenvalue M in the local grid p,r (x,y), update the global eigenvalue M of the i-th local grid i , eigenvalue M i The calculation formula is as follows:

[0093]

[0094] Where p represents the number of sampling points;

[0095] (4) According to the updated local grid eigenvalue M i , calculate the MC-LBP operator eigenvalue T where r is r2 and p is p2 in the whole image m , eigenvalue T m The calculation formula is as follows:

[0096]

[0097] T m =softmax(M * p,r (x,y)

[0098] Among them, M * p,r (x, y) is the global MC-LBP operator eigenvalue, g c is the gray value of the center pixel, M i is the gray value of the local grid where the sampling point is located; T m is the normalized global MC-LBP operator eigenvalue;

[0099] (5) Perform a convolution operation on the Gabor filter and each pixel position of the image to obtain the response value at that position. The specific method of extracting features by the Gabor filter is as follows:

[0100] T g =softmax(conv(G,I))

[0101] Among them, T g represents the normalized Gabor feature, G represents the Gabor filter, and I represents the input image;

[0102] (6) Initialize Q-table and select the initial attention weight ξ;

[0103] (7) Based on the current attention weight ξ, explore how to increase or decrease the weight ξ with a certain probability;

[0104] (8) Define a reward signal Q to guide the model to learn how to select feature weights. The calculation formula of the reward signal Q is defined as:

[0105]

[0106] Where Q(s,a) is the Q value of executing action a in state s, α is the learning rate, r is the immediate reward after executing action a, γ is the discount factor, which indicates the importance of future rewards, s' is the next state transferred to after executing action a, and Q(s,a) is updated according to the current reward signal r and the expected future reward maxQ(s,a), guiding the model to choose the best action in a given state;

[0107] (9) The features T extracted by MC-LBP operator and Gabor filter m and T g Weighted fusion to obtain G-LBP feature T * :

[0108] T * =ξ*T m+(1-ξ)*T g

[0109] Among them, T * represents the G-LBP feature, T m 、T g They are the normalized MC-LBP and Gabor features respectively;

[0110] Further, the step S3 specifically includes:

[0111] S301: randomly rotating and truncating the input feature map and adjusting it to a feature map of the same size to complete data augmentation;

[0112] S302: Build a deep learning-driven object detection module to accurately identify and locate road surface defects on the feature map;

[0113] S303: further filtering the overlapping detection frames by a non-maximum suppression algorithm to ensure that each defect is detected only once;

[0114] Further, the step S302 specifically includes:

[0115] First, through the convolutional attention module, objects of different scales are detected on the obtained feature map;

[0116] (1) Calculate the channel attention weights CA between feature maps of each scale in different time frames i and spatial attention weight SA i ;

[0117] (2) For different feature maps, use the attention weight to adjust the original feature map. The adjustment formula is as follows;

[0118]

[0119] In the formula, F i Represent the adjusted feature map and the original feature map respectively; CA i ,SA i Represent channel and spatial attention weights respectively;

[0120] (3) Add the adjusted feature maps. The addition formula is as follows:

[0121]

[0122] In the formula, F * is the final feature map channel; ε1 and ε2 are weights for adjusting the fusion of different feature maps, and they satisfy:

[0123] ε1+ε2≤1

[0124] (4) Setting multiple prior boxes with different aspect ratios as the basis for subsequent prediction of bounding boxes;

[0125] (5) Flatten the regional features corresponding to each prior box into a vector, and predict the offset of the bounding box for the generated prior box in the fully connected layer of regression prediction;

[0126] Among them, the default prior box generated at position (i, j) on the feature map can be expressed as:

[0127] B i,j =(x,y,w,h)

[0128] Among them, B i,j is the coordinate centered at (x, y), w and h are the length and width of the prior box;

[0129] Offset prediction k It can be expressed as:

[0130] l k =(Δx, Δy, Δw, Δh)

[0131] Among them, Δx, Δy represent the offset of the horizontal and vertical coordinates of the center coordinates of the prior frame, and Δw, Δh represent the offset of the width and height scaling ratio of the prior frame respectively;

[0132] The center coordinates and the actual offsets for width and height can be expressed as:

[0133] t x =(xx a ) / w a

[0134] t y =(yy a ) / h a

[0135] t w =ln(w / w a )

[0136] t h =ln(h / h a )

[0137] Among them, t x ,t y Indicates the actual offset of the center coordinates of the dimension, t w ,t h It is expressed as the actual offset of the length and width of the prior box, x a ,y a Represent the actual center coordinates, w a ,h a Represent the length and width of the actual prior frame respectively;

[0138] The center coordinates and predicted offsets for width and height can be expressed as:

[0139] t' x =(x-Δx) / Δw

[0140] t' y =(y-Δy) / Δh

[0141] t' w =ln(w / Δw)

[0142] t' h =ln(h / Δh)

[0143] Among them, t' x ,t' y Expressed as the predicted offset of the center coordinate, t' w ,t' h is the predicted offset of the prior frame length and width;

[0144] The position error is defined as follows:

[0145]

[0146] Where N is the number of samples and HL(·) is the Loss function, which is defined as follows:

[0147]

[0148] Among them, δ is the threshold parameter of the loss function;

[0149] Further, the step S4 specifically includes:

[0150] S401: construct a road defect classifier, classify the road defects in the detection frame output by the target detection module into pothole defects and crack defects, and output the classification result R1;

[0151] S402: passing the classified result R1 into the neural network, performing a Dropout operation to prevent overfitting, and outputting the classified result R2;

[0152] S403: The two classification results R1 and R2 are passed to the voting machine and the weights are set The final road defect classification result R is obtained by using the soft voting mechanism * ;

[0153] Further, the step S401 specifically includes:

[0154] (1) Calculate the distance between each sample point and other sample points and their corresponding weights. The weight calculation formula is as follows:

[0155]

[0156] Where k is the number of K nearest neighbors of sample i, S(i) is the set of k nearest neighbor points of sample i, and d i,j is the distance between sample points i and j;

[0157] (2) Calculate the local density of each sample point based on its weight and the number of neighbors. The local density of the point indicates the density of the point in the feature space. Calculate the local density of the weighted neighbors The formula is as follows:

[0158]

[0159] (3) Calculate the relative distance η between each pair of sample points and other samples with higher density than it i , the relative distance calculation formula is as follows:

[0160]

[0161] (4) Calculate the sample decision value θ i , and select the cluster center set, assign cluster labels, and the sample decision value calculation formula is as follows:

[0162]

[0163] (5) Select k samples with the highest decision values ​​as the initial cluster centers;

[0164] (6) Calculate the similarity between samples and construct a similarity matrix. The similarity between samples is defined as follows:

[0165]

[0166] Among them, sim i,j and w i,j are the similarity and similarity weight between sample points i and j respectively;

[0167] (7) Traverse the samples that have been assigned cluster labels, find the unassigned samples with the greatest similarity, and assign the unassigned samples to the clusters where the assigned samples are located. Repeat this operation until all samples have been assigned, or the similarity between the assigned samples and the unassigned samples is 0;

[0168] (8) If there are samples that have not been assigned, assign the remaining samples to the cluster that contains the samples with higher density and closest distance to it;

[0169] (9) The sample decision value θ in the cluster i The highest sample is used as the new cluster center. Repeat the above steps until the cluster center no longer changes, and output the classification result R1;

[0170] Further, the step S5 specifically includes:

[0171] S501: According to the classification result R * Calculate the damage extent of pothole defects and crack defects respectively;

[0172] S502: Classify the damage levels of pothole defects and crack defects;

[0173] The calculation formulas for pothole and crack damage degrees are as follows:

[0174]

[0175] Among them, p a The number of pixels quantified for the pothole defect target, p s is the number of pixels quantified in the entire road pothole defect image, a is the number of pixels in the smallest circumscribed rectangle of the crack; by using the pothole damage degree D p The severity of pothole image damage is measured numerically by using D n To measure the severity of crack image damage;

[0176] The damage levels for potholes and cracks are classified as follows:

[0177] Road pothole damage level: D p ≤C1 is a light pothole; C1≤D p ≤C2 is a medium-sized pothole; C2≤D p For heavy potholes;

[0178] Road crack defect damage level: D a ≤C3 is a light seam defect; C3≤D a For heavy duty seams;

[0179] Further, the step S6 specifically includes:

[0180] S601: using a classifier to identify the obstacle type and location of potholes or crack defects on the road, and transmitting them back to the display of the vehicle in a flashing manner through different graphics and colors;

[0181] S602: Upload the road defect information and the driver's actions to the cloud server.

[0182] The beneficial effects of the present invention include:

[0183] (1) The image preprocessing work of the present invention is based on the idea of ​​video stream data processing. First, the collected video data is intercepted into continuous image frames each time as input, which not only reduces the amount of calculation of image preprocessing and retains the order relationship and timing information between image frames, but also uses the image data information of the frames before and after a certain moment as a reference to better eliminate various noises on the collected image, thus providing a solid foundation for subsequent road defect detection work;

[0184] (2) The present invention proposes a new noise judgment method: first, it judges whether a certain frame of the whole image is a noise image, and then judges whether a certain point in the image is a noise point. The noise reduction processing is performed by using inter-frame weighted replacement and improved weighted average filtering respectively. The influence of noise on road defect detection can be effectively reduced at different levels, and the accuracy and robustness of the detection algorithm can be improved.

[0185] (3) The present invention proposes a learning method for continuous grouping of video streams. While using a carefully designed activation function for nonlinear mapping, the time series correlation before and after the video stream grouping is used and combined with the residual idea for time series denoising, which can better remove image noise and improve the accuracy of image recognition.

[0186] (4) The present invention makes full use of the advantages of the attention mechanism and designs a variety of different attention mechanisms to be applied to different steps in the road defect detection process. This can better complete this work from different angles: learning feature maps at different levels, fusing different reverse trait values, and identifying targets of different scales. The machine imitates the human attention mechanism instead of manual work, and can learn in an automated manner, ensuring the comprehensiveness and accuracy of road defect detection.

[0187] (5) In the process of detecting road defects based on deep learning, the present invention applies multi-scale circular operators MC-LBP and Gabor filters to extract features on each layer of the feature pyramid, designs an attention mechanism based on reinforcement learning to fuse the features extracted by MC-LBP and Gabor, can effectively capture the local texture information and direction information in the road image to provide a more comprehensive feature representation, dynamically adjust the feature weight according to the feature importance of the defect, so that the model can allocate different attention to features of different scales and levels, can better obtain the comprehensive features of road defects, and further improve the accuracy of image recognition and classification;

[0188] (6) In the process of constructing a road defect classifier, the present invention can effectively capture the subtle differences between pothole defects and crack defects by calculating the distance and weight between sample points, as well as the local density and relative distance, and provide more accurate classification results. In addition, the classified results are passed to the neural network and the two classification results are fused using a soft voting mechanism, which can comprehensively consider the decisions of multiple classifiers and further improve the robustness and accuracy of the model;

[0189] (7) In the road defect detection method based on deep learning, the present invention designs a damage degree calculation method, namely, pothole damage degree and crack damage degree, which can accurately quantify the damage of the road surface. In addition, the present method innovatively divides the damage levels of pothole defects and crack defects, making the detection results more precise, providing auxiliary decision support for the maintenance of the road in the later stage, allowing the staff to repair the road at the lowest cost, and has significant technical advantages and application value in the field of road defect detection.

[0190] Other advantages, objectives and features of the present invention will be described in detail in the subsequent description. Through in-depth study of the following, those skilled in the art will be able to clearly recognize these advantages and features and obtain valuable teachings from the practice of the present invention. The objectives and other advantages of the present invention can be realized and reflected in the following description and the aforementioned claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0191] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0192] Figure 1 A schematic diagram of a road defect detection method based on deep learning according to the present invention;

[0193] Figure 2 This is an example diagram of normal image filtering using the improved adaptive weighted average filtering of the present invention;

[0194] Figure 3 This is an example diagram of the improved adaptive weighted average filtering noise image filtering of the present invention;

[0195] Figure 4 This is a schematic diagram of the time series denoising of a road defect image according to the present invention;

[0196] Figure 5 A schematic diagram of obtaining road defect characteristics according to the present invention;

[0197] Figure 6 It is a schematic diagram of MC-LBP feature extraction of the present invention;

[0198] Figure 7A schematic diagram of the working process of the road defect classifier of the present invention;

[0199] Figure 8 This is an example diagram of the road crack defect grade classification of the present invention;

[0200] Fig. 9 This is an example diagram of the classification of road pothole defect levels according to the present invention. DETAILED DESCRIPTION

[0201] The preferred embodiments of the present invention are described in detail below. It should be clear that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0202] Considering the defects of traditional road defect detection methods, the present invention makes full use of image preprocessing technology and the attention mechanism of deep learning. With the help of all-round image data preprocessing and efficient target recognition algorithms, different types of road defects are identified and classified, and the degree of damage is determined and uploaded to the server simultaneously. Among them, image preprocessing is a prerequisite for accurate target recognition. It can calculate the similarity between frames, filter out noise frames, perform pixel weighted replacement on them, and use adaptive weighted average filtering to smooth the entire video stream after excluding noise frames; learn the forward and reverse correlation of the video stream time through the encoder, and perform temporal denoising of structural jumps; the use of an all-round denoising mechanism can greatly reduce the negative impact of different noises generated during the acquisition of different defects, including salt and pepper noise, Gaussian noise, etc. on subsequent defect recognition.

[0203] Attention mechanism is an important component in deep learning. It can improve the expressiveness of the model through multi-scale feature extraction attention mechanism combined with reinforcement learning, learn how to select feature weights, so as to achieve more effective feature extraction, while ignoring irrelevant background information; through the cross attention layer in the feature pyramid network, high-level features can pay attention to the details in low-level features, and low-level features can pay attention to the contextual information in high-level features, further enhancing the expressiveness of the model. Through the target detection attention mechanism, the channel attention weight and spatial attention weight between different time frames are calculated for each scale of feature map, so as to better detect targets of different scales; different attention mechanisms used in different periods have unique advantages in road defect detection; using different attention mechanisms, the weight of feature maps can be adaptively adjusted, which enhances the sensitivity of the model to defect features and reduces the interference of background noise, which not only improves the detection accuracy and robustness of the model, but also enhances the model's ability to understand complex scenes.

[0204] like Figure 1 As shown, a road defect detection method based on deep learning of the present invention comprises the following steps:

[0205] Step S1: construct an image preprocessor based on the original image data, and perform preprocessing operations of multi-frame smoothing and noise reduction based on the time sequence;

[0206] Step S2: Based on the improved multi-scale feature extraction algorithm, feature value extraction is performed on the processed image to obtain different low, medium and high-level features of defects such as cracks and potholes on the current road surface;

[0207] Step S3: constructing a target detection algorithm to identify the target of road defects in the feature map;

[0208] Step S4: construct a road defect classifier to classify the detected targets into pothole defects and crack defects;

[0209] Step S5: Calculate the damage degree according to different road defect categories and classify the damage levels;

[0210] Step S6: Transmit the identified road defect information back to the vehicle display for early warning, and upload the relevant information to the cloud for analysis and statistics.

[0211] It should be noted that any process or method description in the flowchart of the present invention or described in other ways herein may be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations, in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0212] like Figures 2 to 9 As shown, the specific steps of the above method will be further described below through a specific embodiment.

[0213] The step S1 comprises the following sub-steps:

[0214] Step S101, collecting video image data, using a 25fps vehicle-mounted camera for real-time collection, and intercepting each second of video and decomposing it into individual frames;

[0215] In step S101, each collected video frame image data includes the following data information: {Video frame: D frame , pixel information: D color , Image resolution: D resolution , frame rate: D rate , compression encoding format: D encoding}, the acquired image width D weight = 256 pixels, height D height=256 pixels;

[0216] Step S102, intercepting 25 consecutive frames of images within one second, and selecting the middle frame of the video stream: the 13th frame is used as the key frame, filtering the noise frames according to the similarity between the frames, and performing pixel weighted replacement on the noise frames;

[0217] In step S102, the specific process of performing pixel weighted replacement on the noise frame is as follows:

[0218] (1) Calculate the similarity between each of the 24 non-key frame images and the key frame. The similarity calculation formula is:

[0219]

[0220] Among them, i, Represent the current frame and the middle frame of the video, and are 1, 2, ..., 12, 14, ..., 25 and 13 respectively; μ i , Represent the mean of two frames of images, Represent the variance of two frames of images, cov(·) is the covariance function, σ i is the standard deviation of the Gaussian function of the i-th frame image, C1 and C2 are stability constants, which are 0.05 respectively; the similarities between the 24 non-key frame images and the key frame are as follows: the similarities are as follows: [0.79, 0.93, 0.91, 0.90, 0.88, 0.92, 0.88, 0.97, 0.98, 0.87, 0.95, 0.90, 0.91, 0.98, 0.81, 0.78, 0.78, 0.96, 0.95, 0.97, 0.99, 0.95, 0.88, 0.95];

[0221] (2) Filter the image frames whose similarity is less than the threshold value τ = 0.8 as noise frames, among which the 1st, 17th and 18th frames whose similarity with the key frame is less than the threshold value 0.8 are judged as noise frames;

[0222] (3) Calculate the pixel weight of the filtered noise frame. The pixel weight calculation formula is:

[0223]

[0224] Among them, Rmse(·) is the root mean square error function, R(·) is the range function, and the noise frame weights are: [0.40, 0.38, 0.38];

[0225] (4) Perform pixel weighted replacement on the noise frame according to the weights. The calculation formulas for pixel weighted replacement are:

[0226]

[0227] Among them, I w (x, y) is the weighted pixel value, I i (x,y),I j (x, y) represents the pixel value of the i-th noise image and the j-th normal frame image closest to it at the coordinate (x, y), w i is the weight of the i-th frame image; the three noise frames are weightedly replaced with the 2nd, 16th, and 19th normal frame images closest to them respectively;

[0228] Step S103, smoothing the image after receiving noise replacement using an improved adaptive weighted average filter;

[0229] In step S103, the specific steps of using the improved filtering to smooth the image are:

[0230] In this embodiment, the smoothing process of the improved adaptive weighted average filtering is as follows: Figure 2 , 3 As shown;

[0231] (1) For a given 3×3 filter window S xy For each pixel (x, y) in the min , maximum gray value Z max And the average gray value Z of all pixels mean , and set the noise judgment threshold T∈[1.7,2.1];

[0232] (2) Determine whether the mean of the filter window is a noise point, that is, if T min <(Z min +Z max ) / Z mean <T max If true, then the mean Z mean If it is a normal point, use the mean. Otherwise, expand the window length by 2 and continue to look for the normal mean.

[0233] (3) Determine Z min <I(x,y)<Z max Whether the condition is met, if so, the pixel value of the point is output, otherwise the Euclidean distance is used to calculate the difference between each pixel of the input image and the pixels in its neighborhood;

[0234] (4) Calculate the pixel weight according to the difference value. The calculation formula of the pixel weight is:

[0235]

[0236] Among them, W(x,y) represents the weight of the pixel point (x,y), Indicates the difference between the pixel and the center point (128,128);

[0237] (5) Multiply the weight by the corresponding pixel value to calculate the weighted average pixel value. The calculation method of the weighted average pixel value is:

[0238]

[0239] Among them, FP(x,y) represents the pixel value after filtering, I w (x,y) represents the pixel value in the original image;

[0240] (6) Repeat the steps until all pixels are traversed;

[0241] (7) When there are still noisy pixels after the window reaches the maximum size, use Gaussian filtering to replace them;

[0242] In this embodiment, the pixel information in the 3×3 filter window is: I(1,1)=100, I(1,2)=120, …, I(3,3)=150, the difference between each point and the central pixel is: diff(1,1)=10, diff(1,2)=5, diff(1,3)=15, …, diff(3,3)=0, the weight of each pixel is calculated according to the calculation formula of pixel weight, the standard deviation σ of the Gaussian function is 5, and the pixel weight is calculated as follows:

[0243]

[0244]

[0245] The weighted average pixel value of each pixel is calculated according to the weighted average pixel value calculation formula:

[0246] W(1,1)≈107.2

[0247] W(1,2)≈110.7

[0248] W(1,3)≈102.2

[0249] …

[0250] W(3,3)≈150

[0251] Step S104, arranging the smoothed images in chronological order, and performing temporal noise reduction processing based on the jump structure;

[0252] In step S104, the specific steps of the temporal noise reduction processing based on the jump structure are as follows:

[0253] In this embodiment, the model of timing denoising based on the jump structure is as follows: Figure 4 As shown;

[0254] (1) Seven consecutive frames of images: I1, I2, I3, ..., I7 are passed to the encoder in sequence, and three convolution kernels are used to perform convolution operations on each frame of the image to obtain the encoded features:

[0255]

[0256] In this embodiment, the number and size of convolution kernels in the encoder are:

[0257] The first convolution kernel: N1=32, size is 7×7;

[0258] The second convolution kernel: N2=64, size is 5×5;

[0259] The third convolution kernel: N3 = 128, size is 3 × 3;

[0260] (2) Encoding feature Z E The images are divided into 3 groups in chronological order, with 5 images in each group, and input into the bidirectional recurrent neural network to learn the correlation between the forward and reverse time series in the time series to obtain the time series feature Z. R ;

[0261] (3) The image temporal feature Z R The input is sent to the decoder for upsampling, and matched and spliced ​​with the corresponding positions in the encoder to obtain the final denoised images I'1, I'2, I'3, ..., I'7;

[0262] Among them, the steps of learning the bidirectional recurrent neural network combined with the residual are:

[0263] (1) Calculate the forward and reverse reset gates at time t respectively and propagation gate

[0264]

[0265]

[0266] in, The forward and reverse reset gates respectively determine to what extent the hidden state information of the previous moment is reset, thereby allowing the network to forget irrelevant information; are the forward and reverse propagation gates, respectively, which control the extent to which the new candidate hidden state information is updated to the current hidden state; chx(·) and shx(·) are the hyperbolic cosine activation function and hyperbolic sine activation function used for gating degree scaling in the reset gate and propagation gate, respectively; W o , U o , b o , W s , U s , b sis the gating parameter of the model's forward and reverse propagation, W o and W s Represents the weight matrix between the reset gate and propagation gate layers, U o and U s denote the time step matrices of the reset gate and the propagation gate, respectively, and b o and b s Represent the bias matrices of the reset gate and propagation gate respectively; W o , U o , W s , U s All matrix elements obey uniform distribution U(-0.1,0.1); b o , b s The matrix elements are 0 and 0.01 respectively;

[0267] (2) Calculate the candidate hidden states for the forward and reverse directions respectively

[0268]

[0269] in, They are the candidate hidden states for the forward and reverse directions, respectively, and contain the information of the current input and the hidden state at the previous moment; W f , W b , U f , U b , b f , b b are the model forward and reverse gating parameters, W f , W b is the weight matrix for forward and reverse learning, U f , U b is the time step matrix of forward and reverse learning, b f , b b is the bias matrix for forward and reverse learning; b f , b b The matrix element is 0.01; H is the H-Swith(·) activation function, and the H-Swith activation function formula is as follows:

[0270]

[0271] ReLU6=min(ReLU,6)

[0272] (3) According to the propagation gate and the candidate hidden state, the current forward and reverse hidden states are Concatenate into the final hidden state h t , and the input encoding feature Z E Make residual connection to get the final time series feature Z R :

[0273]

[0274] Z R =Z E +h t

[0275] Among them, h t is the final hidden state, which is the concatenation of the forward and reverse hidden states; Z R is the final time series feature, which is the input encoding feature Z E and the hidden state h t residual connection; [:] is the concatenation operation, ⊙ is the element-by-element multiplication; model learning rate lr = 0.0001, iteration number epoch = 100, batch size batch_size = 64;

[0276] The step S2 specifically includes the following steps:

[0277] Step S201: By applying the improved multi-expansion rate fused dilated convolution on the input image, three preliminary feature maps C1-C3 of the input image and three corresponding sub-feature maps D1-D3 of different depths are obtained. The specific steps of using the improved multi-expansion rate fused dilated convolution operation are as follows:

[0278] In this embodiment, the improved dilated convolution operation process of multiple dilation rate fusion is as follows: Figure 5 As shown;

[0279] First, three convolutional layers of different sizes are used to extract multi-scale features of the image, and three preliminary feature maps C1-C3 at different levels are obtained;

[0280] In this embodiment, the number and size of convolution kernels are:

[0281] The first convolution kernel: N1=64, size 3×3, stride 1;

[0282] The second convolution kernel: N2 = 128, size 5 × 5, step size 1;

[0283] The third convolution kernel: N3 = 256, size is 7 × 7, step size is 1;

[0284] After connecting the ReLU(·) activation function after the convolutional layer and performing batch normalization, the preliminary feature maps C1-C3 are obtained;

[0285] Among them, the normalization formula is:

[0286]

[0287] Where x is the input data, μ, σ 2are the mean and variance of the data respectively; γ, β, ∈ are scaling parameters, which are 0.1, 0, 10 respectively -5

[0288] Then, the feature maps C1-C3 are convolved separately using the hole convolution operation with multiple dilation rates to obtain the sub-feature maps D1-D3 with different depths corresponding to them;

[0289] Among them, the hole convolution operation with multiple dilation rates is defined as follows:

[0290] The expansion rate of the i-th convolution layer Rate i Represents the distance expansion rate between adjacent elements in the convolution kernel of this layer, Rate i It can be expressed as:

[0291] Rate i =f(i)

[0292] Where i is the level, f(i) is the level function, f(i) = 2 i-1 ;

[0293] The actual effective convolution size V of the i-th layer convolution kernel size is k i It can be expressed as:

[0294] V i =(k-1)*Rate i +1

[0295] The operation of the i-th layer image convolution can be expressed as:

[0296]

[0297] Among them, I(i,j) and K(i,j) are the values ​​of the i-th and j-th positions on the image and convolution kernel respectively;

[0298] In this embodiment, the feature maps C1-C3 are respectively subjected to dilation convolution with dilation rates of 1, 2, and 4 according to the number of levels 1, 2, and 3 to obtain their respective sub-feature maps: C1-D1, C1-D2, C1-D3, ..., C3-D1, C3-D2, and C3-D3, where the receptive field sizes of D1-D3 are respectively:

[0299] D1: (3-1)×1+1=3;

[0300] D2: (3-1)×2+1=5;

[0301] D3: (3-1)×4+1=9;

[0302] Step S202: Introduce a cross attention layer in each layer between feature maps C1-C3, and learn the correlation between its sub-feature maps through the designed AN network. The specific learning process of the AN network is:

[0303] First, calculate the attention weights of different feature maps:

[0304] W = softmax(AN(D i ,D j )))

[0305] Then, apply the weights to the category feature map to obtain the fused feature map:

[0306] D=W*D i

[0307] Finally, the fused feature map D is combined with the original category sub-feature map D i Merge to get a new feature map D * :

[0308] D * =Merge(D,D i )

[0309] In this embodiment, the 9 sub-feature graphs are divided into 27 groups: {(C1-D1, C2-D1), (C2-D1, C3-D1), (C3-D1, C1-D1), ..., (C3-D3, C1-D3)} are respectively transmitted to the layer attention network AN, and the fused feature sub-graphs are then merged with the original feature sub-graphs to obtain the new feature graph D * 1,…,D * 9 and divided into three groups according to the source characteristic diagrams C1-C3:

[0310] Feature maps generated from C1: D1, D2, D3;

[0311] Feature maps generated from C2: D4, D5, D6;

[0312] Feature maps generated from C3: D7, D8, D9;

[0313] Step S203, performing multi-scale fusion on the different sub-feature maps D in the feature maps C1-C3 to obtain the final feature pyramid P;

[0314] In this embodiment, the feature sub-graphs are fused at a single scale by average pooling according to the source to obtain feature graphs: F_C1, F_C2, F_C3; F_C1, F_C2, F_C3 are concatenated and connected with ReLU (·) activation function to obtain feature pyramid P;

[0315] Step S204: Apply a multi-scale G-LBP operator to each layer of the obtained feature pyramid P to extract features to obtain a feature map. The multi-scale feature extraction operator is specifically:

[0316] In this embodiment, the MC-LBP operator calculation process is as follows: Figure 6 As shown;

[0317] (1) Each layer of the feature pyramid P is divided into 9 2×2 grids;

[0318] (2) For each grid, calculate the MC-LBP value with r=1 and p=8 as the eigenvalue of the grid:

[0319]

[0320] (3) According to each eigenvalue M in the local grid p,r (x,y), update the global eigenvalue M of the i-th local grid i , eigenvalue M i The calculation formula is as follows:

[0321]

[0322] (4) According to the updated local grid eigenvalue M i , calculate the MC-LBP operator eigenvalue T where r is r2 and p is p2 in the whole image m , eigenvalue T m The calculation formula is as follows:

[0323]

[0324] T m =softmax(M * p,r (x,y)

[0325] (5) A Gabor filter with a frequency of 0.2, an azimuth of 0 degrees, a standard deviation of 2, a phase shift of 0, and an aspect ratio of 0.5 is applied to the image. By performing a convolution operation with the image at each pixel position, the response value at that position is obtained and normalized. The features extracted by the Gabor filter are as follows:

[0326] T g =softmax(conv(G,I))

[0327] (6) Define two actions: increase a1, decrease a2 and ∈ greedy strategy, and set the initial attention weight a = 0.5, discount factor γ = 0.9, learning rate lr = 0.1, and exploration factor ∈ = 0.1;

[0328] (7) Initialize the action value function table Q-table: initialize each state-action to 0;

[0329] (8) Based on the current attention weight ξ, explore how to increase or decrease the weight ξ with a certain probability

[0330] (9) Execute actions and update Q value:

[0331]

[0332] Where Q(s,a) is the Q value of executing action a in state s, α is the learning rate, r is the immediate reward after executing action a, γ is the discount factor, which indicates the importance of future rewards, s' is the next state transferred to after executing action a, and Q(s,a) is updated according to the current reward signal r and the expected future reward maxQ(s,a), guiding the model to choose the best action in a given state;

[0333] (8) Calculate the initial G-LBP features according to the formula:

[0334] T * =ξ*T m +(1-ξ)*T g

[0335] In this embodiment, the weight ξ is 0.65;

[0336] The step S3 specifically includes the following steps:

[0337] Step S301: randomly rotate and crop the input feature map and adjust it to a feature map of the same size to complete data augmentation;

[0338] In this embodiment, each frame of the image is randomly rotated by an angle of [-10°, 10°], and the size is cut off by 80% to 100%;

[0339] Step S302: construct a deep learning driven target detection module to accurately identify and locate road surface defects on the feature map. The specific steps of constructing the target detection algorithm are as follows:

[0340] (1) For each scale feature map, calculate the channel attention weight CA between different time frames i and spatial attention weight SA i ;

[0341] (2) For different feature maps, use the attention weights to adjust the original feature maps. The adjustment formula is as follows;

[0342]

[0343] (3) Add the adjusted feature maps. The addition formula is as follows:

[0344]

[0345] In this embodiment, ε1 and ε2 are 0.3 and 0.7 respectively;

[0346] (4) Setting multiple prior boxes with different aspect ratios in each unit as the reference for the subsequent predicted bounding boxes;

[0347] In this embodiment, a priori frame with an aspect ratio of 1:1 suitable for a square target, a priori frame with an aspect ratio of 1:2 suitable for a tall target, and a priori frame with an aspect ratio of 2:1 suitable for a horizontally long target are established respectively;

[0348] (5) Flatten the regional features corresponding to each prior box into a vector, and predict the offset of the bounding box for the generated prior box in the fully connected layer of regression prediction;

[0349] Among them, the default prior box generated at position (i, j) on the feature map can be expressed as:

[0350] B i,j =(x,y,w,h)

[0351] Among them, B i,j The road defect target (54,105) is taken as the center coordinate, w,h are the length and width of the prior box: 16, 32;

[0352] Offset prediction k It can be expressed as:

[0353] l k =(Δx, Δy, Δw, Δh)

[0354] Among them, Δx, Δy represent the horizontal and vertical coordinate offsets of the center coordinates of the prior frame, and Δw, Δh represent the offsets of the width and height scaling ratios of the prior frame respectively;

[0355] The center coordinates and the actual offsets for width and height can be expressed as:

[0356] t x =(xx a ) / w a

[0357] t y =(yy a ) / h a

[0358] t w =ln(w / w a )

[0359] t h =ln(h / h a )

[0360] Among them, t x ,t y Indicates the actual offset of the dimension center coordinates: 2, 8, t w ,t h Expressed as the actual offset of the prior box length and width: 0.08, 0.09, x a ,y a Represent the actual center coordinates: (54,105), w a ,h a Represent the length and width of the actual prior box: 16, 32;

[0361] The center coordinates and predicted offsets for width and height can be expressed as:

[0362] t' x =(x-Δx) / Δw

[0363] t' y =(y-Δy) / Δh

[0364] t' w =ln(w / Δw)

[0365] t' h =ln(h / Δh)

[0366] Among them, t' x ,t' y Expressed as predicted offset of center coordinates: 3, 8, t' w ,t' h The predicted offsets for the prior box length and width are: 0.1 and 0.09;

[0367] The position error is defined as follows:

[0368]

[0369] Where N is the number of samples and HL(·) is the Loss function, which is defined as follows:

[0370]

[0371] Wherein, δ is the threshold parameter of the loss function and is set to 0.8. In this embodiment, the HL(·) error is 1.028;

[0372] Step S303: further filtering the overlapping detection frames using a non-maximum suppression algorithm with an IoU threshold set to 0.5 to ensure that each defect is detected only once;

[0373] The step S4 specifically includes the following steps:

[0374] Step S401: construct a classifier to classify the road defects in the detection frame output by the target detection module into pothole defects and crack defects. The steps of constructing the classifier are as follows:

[0375] (1) Calculate the distance between each sample point and other sample points and their corresponding weights. The weight calculation formula is as follows:

[0376]

[0377] Where k is the number of K nearest neighbors of sample i, S(i) is the set of k nearest neighbor points of sample i, and d i,j is the distance between sample points i and j;

[0378] (2) Calculate the local density of each sample point based on its weight and the number of neighbors. The local density of the point indicates the density of the point in the feature space. Calculate the local density of the weighted neighbors The formula is as follows:

[0379]

[0380] (3) Calculate the relative distance η between each pair of sample points and other samples with higher density than it i , the relative distance calculation formula is as follows:

[0381]

[0382] (4) Calculate the sample decision value θ i , and select the cluster center set, assign cluster labels, and the sample decision value calculation formula is as follows:

[0383]

[0384] (5) Select k samples with the highest decision values ​​as the initial cluster centers;

[0385] (6) Calculate the similarity between samples and construct a similarity matrix. The similarity between samples is defined as follows:

[0386]

[0387] Among them, sim i,j and w i,j are the similarity and similarity weight between sample points i and j respectively;

[0388] (7) Traverse the samples that have been assigned cluster labels, find the unassigned samples with the greatest similarity, and assign the unassigned samples to the clusters where the assigned samples are located. Repeat this operation until all samples have been assigned, or the similarity between the assigned samples and the unassigned samples is 0;

[0389] (8) If there are samples that have not been assigned, assign the remaining samples to the cluster that contains the samples with higher density and closest distance to it;

[0390] (9) The sample decision value θ in the cluster i The highest sample is used as the new cluster center. Repeat the above steps until the cluster center no longer changes, and output the classification result R1;

[0391] Step S402: passing the classified results into the neural network and performing a Dropout operation to prevent overfitting;

[0392] Step S403: Pass the two classification results into the voting machine and set the weights The soft voting mechanism is used to obtain the final road defect classification results;

[0393] In this embodiment, k is 5. After calculation, the weights, relative distances, and decision values ​​of the first five samples are:

[0394] Sample 1: η1=0.2934, θ1=0.1264;

[0395] Sample 2: η2=0.3121, θ2=0.1208;

[0396] Sample 3: η3=0.2756, θ3=0.0983;

[0397] Sample 4: η4=0.2934, θ4=0.1232;

[0398] Sample 5: η5=0.3001, θ5=0.1045;

[0399] The neural network consists of three convolutional layers: 1×1, 3×3, and 5×5. The dropout rate is 0.1; the weight Take 0.65 and 0.35 respectively, and the classification results are R1, R2, R * They are all {1,0,1,1,0}, where 0 represents a pit defect and 1 represents a crack defect;

[0400] The step S5 specifically includes the following steps:

[0401] Step S501: Calculate the damage degree of the pothole defect and the crack defect respectively. The calculation formulas for the damage degree of the pothole and the crack are as follows:

[0402]

[0403] Among them, p a The number of pixels quantified for the pothole defect target, p s is the number of pixels quantified in the entire road pothole defect image, a is the number of pixels in the smallest circumscribed rectangle of the crack; by using the pothole damage degree D p The severity of pothole image damage is measured numerically by using D n To measure the severity of crack image damage;

[0404] Step S502: Classify the damage levels of pothole defects and crack defects as follows:

[0405] Road pothole damage level: D p ≤17% is a light pothole, 17%≤D p ≤21% is a medium pothole, 21%≤D p For heavy potholes;

[0406] Road crack defect damage level: D a ≤18% is a light seam defect, 18%≤D a For heavy duty seams;

[0407] The step S6 specifically includes the following steps:

[0408] Step S601, the obstacle type and location of potholes or cracks on the road identified by the classifier are transmitted back to the display of the vehicle in a flashing manner through different graphics and colors;

[0409] Step S602, uploading the road defect information and the driver's actions to the cloud server;

[0410] In this embodiment, the road defect image data is fully preprocessed. Figure 2 and Figure 3 The working principle of the improved adaptive weighted average filter on normal images and noisy images is demonstrated respectively; Figure 4 The temporal denoising structure in the proposed method can effectively utilize temporal correlation to perform multi-frame denoising on road defect videos. Figure 5 To obtain road defect features using dilated convolution with multiple dilation rates combined with attention mechanism; Figure 6 The feature operators extracted by the MC-LBP operator invented in this paper at multiple scales are demonstrated; Figure 7 A workflow for classifying road defects for the classification algorithm of the present invention; Figure 8 and Fig. 9 This is an example of the road defect grade classification of the present invention; the present invention improves the target recognition capability through rich image preprocessing work and respective attention mechanisms, which helps to increase the accuracy and speed of road defect recognition and contains richer data information.

[0411] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0412] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A road defect detection method based on deep learning, characterized in that: The following steps are involved: Step S1: construct an image preprocessor based on the original image data, and perform preprocessing operations of multi-frame smoothing and noise reduction based on the time sequence; Step S2: Based on the improved multi-scale feature extraction algorithm, feature value extraction is performed on the processed image to obtain different low, medium and high-level features of defects such as cracks and potholes on the current road surface; Step S3: constructing a target detection algorithm to identify the target of road defects in the feature map; Step S4: construct a road defect classifier to classify the detected targets into pothole defects and crack defects; Step S5: Calculate the damage degree according to different road defect categories and classify the damage levels; Step S6: Transmit the identified road defect information back to the vehicle display for early warning, and upload the relevant information to the cloud for analysis and statistics.

2. A road defect detection method based on deep learning according to claim 1, characterized in that: The step S1 specifically includes: Step S101: For each video stream to be processed, decompose the video into individual frames; Step S102: intercept k consecutive image frames, select the middle frame as the key frame, filter the noise frame according to the similarity between frames, and perform pixel weighted replacement on the noise frame. The specific operation steps are as follows: First, the similarity between each frame image and the key frame image is calculated, where the similarity calculation formula is: Among them, i, Represent the current frame and the middle frame image of the video, μ i , Represent the mean values ​​of the pixels of the two frames, Represent the variance of the pixels of the two frames, cov(·) is the covariance function, σ i is the standard deviation of the Gaussian function of the i-th frame image, C1 and C2 are stability constants; Then, a similarity threshold τ is set to filter out noise frames whose similarity to the key frame is less than the threshold τ; Furthermore, the pixel weight of the noise frame image is calculated, where the pixel weight calculation formula is: Among them, Rmse(·) is the root mean square error function, R(·) is the range function, and this formula and the subsequent exp(·) are exponential functions with the natural constant e as the base; Finally, the noise frame pixels are weighted replaced according to the weights, where the pixel weighted replacement formula is: Among them, I w (x, y) is the weighted pixel value, I i (x,y),I j (x, y) represents the pixel value of the i-th noise image and the j-th normal frame image closest to it at the coordinate (x, y), w i is the weight of the i-th frame image, i is the current time point; Step S103: using an improved adaptive weighted average filtering method to perform a smoothing operation on the received image. The specific steps of using the improved filtering method to smooth the image are as follows: (1) For a given filtering window S xy , calculate S xy The minimum gray value Z of each pixel (x, y) in min , maximum gray value Z max And the average gray value Z of all pixels mean , and set the noise judgment threshold T; (2) Determine whether the pixel mean of the filter window is a noise point. If T min <(Z min +Z max ) / Z mean <T max If true, then the mean Z mean is a normal point, otherwise, expand the window and continue to search for the normal mean; (3) Determine Z min w (x,y) <Z max Whether the condition is met, if so, the pixel value of the point is output, otherwise the Euclidean distance is used to calculate the difference between each pixel of the input image and the pixels in its neighborhood;​ (4) Calculate the pixel weight according to the difference value. The calculation formula of the pixel weight is: Among them, W(x,y) represents the weight of the pixel point (x,y), Indicates the pixel at this point and the center pixel (c x ,c y ) difference value; (5) Multiply the weight by the corresponding pixel value to calculate the filtered pixel value. The calculation method of the filtered pixel value is: Among them, FP(x,y) represents the pixel value after filtering, I w (x,y) represents the pixel value in the original image; (6) Repeat the steps until all pixels are traversed; (7) When there are still noisy pixels after the window reaches the maximum size, the noisy pixels are replaced by Gaussian filtering; Step S104: Arrange the smoothed images in chronological order and perform temporal denoising based on a jump structure. The specific steps of temporal denoising based on a jump structure are as follows: First, k frames of continuous images are passed into the encoder, and then downsampled through three convolution kernels of different sizes to extract feature information, and the encoded feature Z is obtained. E It is expressed as: WITH E =(From E1 ,WITH E2 ,…,WITH Ek ) Then, the encoded feature Z E Divide every n into a group in chronological order, and divide into k-n+1 groups in total, as the input of the bidirectional recurrent neural network combined with the residual in the skip structure, and learn the forward and reverse time series correlation in the time series; Finally, the k-n+1 groups of time series are input into the decoder for upsampling, and matched and spliced ​​with the corresponding positions in the encoder to obtain the final denoised image.

3. A road defect detection method based on deep learning according to claim 2, characterized in that: In step S104, the steps of learning the bidirectional recurrent neural network combined with the residual are: First, calculate the forward and reverse reset gates at time t respectively and propagation gate in, The forward and reverse reset gates respectively determine to what extent the hidden state information of the previous moment is reset, thereby allowing the network to forget irrelevant information; are the forward and reverse propagation gates, respectively, which control the extent to which the new candidate hidden state information is updated to the current hidden state; chx(·) and shx(·) are the hyperbolic cosine activation function and hyperbolic sine activation function used for gating degree scaling in the reset gate and propagation gate, respectively; W o , U o 、b o , W s , U s 、b s is the gating parameter of the model's forward and reverse propagation, W o and W s Represents the weight matrix between the reset gate and propagation gate layers, U o and U s denote the time step matrices of the reset gate and the propagation gate, respectively, and b o and b s They represent the bias matrices of the reset gate and the propagation gate respectively; Then, the candidate hidden states for the forward and reverse directions are calculated separately in, They are the candidate hidden states for the forward and reverse directions, respectively, and contain the information of the current input and the hidden state at the previous moment; W f , W b , U f , U b 、b f 、b b are the model forward and reverse gating parameters, W f , W b is the weight matrix for forward and reverse learning, U f , U b is the time step matrix of forward and reverse learning, b f 、b b is the bias matrix for forward and reverse learning; H is the H-Swith(·) activation function, and the H-Swith activation function formula is as follows: ReLU6=min(ReLU,6) This formula and the subsequent min(·) are all functions for finding the minimum value; Finally, according to the propagation gate and the candidate hidden state, the current forward and reverse hidden states are Concatenate into the final hidden state h t , and with the input encoding feature Z E Make residual connection to get the final time series feature Z R : Z R =Z E +h t Among them, h t is the final hidden state, which is the concatenation of the forward and reverse hidden states; Z R is the final time series feature, which is the input encoding feature Z E and the hidden state h t The residual connection of ; [:] is the concatenation operation, and ⊙ is the element-by-element multiplication.

4. The road defect detection method based on deep learning according to claim 1, characterized in that: The step S2 specifically includes: Step S201: Apply the improved multi-expansion rate fusion dilated convolution operation on the input image to obtain the preliminary feature map C1-C n And the corresponding sub-feature maps D1-D at different depths n , the specific steps of using the improved multi-expansion rate fused dilated convolution operation are: First, n convolutional layers of different sizes are used to extract multi-scale features of the image, and n preliminary feature maps C1-C n ; Then, in the feature map C1-C n The dilated convolution operation with multiple dilation rates is used to perform convolution respectively to obtain the sub-feature maps D1-D corresponding to different depths. n ; Among them, the hole convolution operation with multiple dilation rates is defined as follows: The expansion rate of the i-th convolution layer Rate i Represents the distance expansion rate between adjacent elements in the convolution kernel of this layer, Rate i Defined as: Rate i =f(i) Among them, f(i) is the hierarchical function, f(i) = 2 i-1 ; The actual effective convolution size V of the i-th layer convolution kernel size is k i It can be expressed as: V i =(k-1)*Rate i +1 The operation of the i-th layer image convolution can be expressed as: Among them, I(i,j), K(i,j) are the pixel values ​​at the (i,j) coordinates of the image and the convolution kernel respectively. This formula and the subsequent conv(·) are product sum functions; Step S202: In the feature graph C1-C n A cross attention layer is introduced between each layer, and the correlation between its sub-feature maps is learned through the designed AN network. The specific learning process of the AN network is as follows: First, calculate the attention weights of different feature maps: <h2 style=";text-align:left;direction:ltr">W = softmax(AN(D<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ,D<h2 style=";text-align:left;direction:ltr"> j <h2 style=";text-align:left;direction:ltr"> )) Among them, D i , D j Represents different sub-feature graphs; Then, apply the weights to the category feature map to obtain the fused feature map D: D=W*D i Finally, the fused feature map D is combined with the original category sub-feature map D i Merge to get a new feature map D * : D * =Merge(D,D i ) Among them, D * represents the final sub-feature map, Merge(·) represents the concatenation operation, and this formula and the subsequent softmax(·) are both normalized exponential functions; Step S203: Convert feature graph C1-C n Different sub-feature graphs D * Perform multi-scale fusion to obtain the final feature pyramid P; Step S204: Apply a multi-scale G-LBP operator on each layer of the obtained feature pyramid P to extract features to obtain a feature map. The multi-scale feature extraction method is specifically as follows; (1) Divide the image into n m×m grids, each of which includes a different radius r and a different sampling point p; (2) Calculate the MC-LBP operator eigenvalue M where r is r1 and p is p1 in each local grid. The calculation formula of the eigenvalue M is as follows: Among them, M p,r (x, y) represents the MC-LBP operator eigenvalue, p is the number of sampling points, r is the radius, (x, y) is the coordinate of the center point, a i is the angle information of the i-th sampling point, s(·) is the threshold function, and the threshold function formula is as follows: Among them, θ i is the angle of the i-th sampling point, g(x,y), g(x+rcos(θ i ),y+rsin(θ i )) are the grayscale values ​​of the center pixel and the sampling point respectively; (3) According to each eigenvalue M in the local grid p,r (x,y), update the global eigenvalue M of the i-th local grid i , eigenvalue M i The calculation formula is as follows: Where p represents the number of sampling points; (4) According to the updated local grid eigenvalue M i , calculate the MC-LBP operator eigenvalue T where r is r2 and p is p2 in the whole image m , eigenvalue T m The calculation formula is as follows: T m =softmax(M * p,r (x,y)) Among them, M * p,r (x, y) is the global MC-LBP operator eigenvalue, g c is the gray value of the center pixel, M i is the gray value of the local grid where the sampling point is located; T m is the normalized global MC-LBP operator eigenvalue; (5) Perform a convolution operation on the Gabor filter and each pixel position of the image to obtain the response value at that position. The specific method of extracting features by the Gabor filter is as follows: T g =softmax(conv(G,I)) Among them, T g represents the normalized Gabor feature, G represents the Gabor filter, and I represents the input image. (6) Apply the attention mechanism to extract the features T from the MC-LBP operator and Gabor filter. m and T g Fusion, get the G-LBP feature.

5. The road defect detection method based on deep learning according to claim 4, characterized in that: In step S204, the application steps of the attention mechanism are: First, initialize the action value function table Q-table and select the initial attention weight ξ; Then, based on the current attention weight ξ, explore how to increase or decrease the weight ξ with a certain probability; Furthermore, a reward signal Q is defined to guide the model to learn how to select feature weights. The calculation formula of the reward signal Q is defined as: Where Q(s,a) is the Q value of executing action a in state s, α is the learning rate, r is the immediate reward after executing action a, γ is the discount factor, which indicates the importance of future rewards, s' is the next state transferred to after executing action a, Q(s,a) is updated according to the current reward signal r and the expected future reward maxQ(s,a), guiding the model to choose the best action in a given state. This formula and the subsequent max(·) are all maximum value functions; Finally, weighted fusion is performed to obtain the final G-LBP feature. The calculation formula of G-LBP feature is: T * =ξ*T m +(1-ξ)*T g Among them, T * represents the G-LBP feature, T m , T g They are the normalized MC-LBP and Gabor features respectively.

6. The road defect detection method based on deep learning according to claim 1, characterized in that: The step S3 specifically includes: Step S301: randomly rotating and cropping the input feature map and adjusting it to a feature map of the same size to complete data augmentation; Step S302: construct a deep learning driven target detection module to accurately identify and locate road surface defects on the feature map. The specific steps of constructing the target detection module are as follows: First, through the convolutional attention module, objects of different scales on the feature map are detected; Then, multiple prior boxes with different aspect ratios are set as the basis for subsequent prediction of bounding boxes; Finally, the regional features corresponding to each priori box are flattened into a vector, and the offset of the generated priori box is predicted in the fully connected layer of regression prediction. Among them, the default prior box generated at position (i, j) on the feature map can be expressed as: B i,j =(x,y,w,h) Among them, B i,j is the coordinate centered at (x, y), w and h are the length and width of the prior box; Offset prediction k It can be expressed as: l k =(Δx,Δy,Δw,Δh) Among them, Δx, Δy represent the offset of the horizontal and vertical coordinates of the center coordinates of the prior frame, and Δw, Δh represent the offset of the width and height scaling ratio of the prior frame respectively; The center coordinates and the actual offsets for width and height can be expressed as: t x =(x-x a ) / w a t y =(y-y a ) / h a t w =ln(w / w a ) t h =ln(h / h a ) Among them, t x ,t y Indicates the actual offset of the center coordinates of the dimension, t w ,t h It is expressed as the actual offset of the length and width of the prior box, x a ,y a Represent the actual center coordinates, w a ,h a Represent the length and width of the actual prior frame respectively; The center coordinates and predicted offsets for width and height can be expressed as: t' x =(x-Δx) / Δw t' y =(y-Δy) / Δh t' w =ln(w / Δw) t' h =ln(h / Δh) Among them, t' x ,t' y Expressed as the predicted offset of the center coordinate, t' w ,t' h is the predicted offset of the prior frame length and width; The position error is defined as follows: Where N is the number of samples and HL(·) is the Loss function, which is defined as follows: Among them, δ is the threshold parameter of the loss function.

7. The road defect detection method based on deep learning according to claim 6, characterized in that: In step S302, the steps of designing the attention mechanism are: First, the channel attention weights CA between feature maps of each scale at different time frames are calculated i and spatial attention weight SA i ; Then, for different feature maps, the attention weights are used to adjust the original feature maps. The adjustment formula is as follows; in, F i Represent the adjusted feature map and the original feature map respectively; CA i ,SA i Represent channel and spatial attention weights respectively; Finally, add the adjusted feature maps together, and the addition formula is as follows; In the formula, F * is the final feature map channel; ε1 and ε2 are weights for adjusting the fusion of different feature maps, and they satisfy: ε1+ε2≤1 Step S303: further filtering the overlapping detection frames by a non-maximum suppression algorithm to ensure that each defect is detected only once.

8. The road defect detection method based on deep learning according to claim 1, characterized in that: The step S4 specifically includes: S401: Construct a road defect classifier to classify the road defects in the detection frame output by the target detection module into pothole defects and crack defects. The specific steps of constructing the road defect classifier are as follows: (1) Calculate the distance between each sample point and other sample points and their corresponding weights. The weight calculation formula is as follows: Where k is the number of K nearest neighbors of sample i, S(i) is the set of k nearest neighbor points of sample i, and d i,j is the distance between sample points i and j; (2) Calculate the local density of each sample point based on its weight and the number of neighbors. The local density of the point indicates the density of the point in the feature space. Calculate the local density of the weighted neighbors The formula is as follows: (3) Calculate the relative distance η between each pair of sample points and other samples with higher density than it i , the relative distance calculation formula is as follows: (4) Calculate the sample decision value θ i , and select the cluster center set, assign cluster labels, and the sample decision value calculation formula is as follows: (5) Select k samples with the highest decision values ​​as the initial cluster centers; (6) Calculate the similarity between samples and construct a similarity matrix. The similarity between samples is defined as follows: Among them, sim i,j and w i,j are the similarity and similarity weight between sample points i and j respectively; (7) Traverse the samples that have been assigned cluster labels, find the unassigned samples with the greatest similarity, and assign the unassigned samples to the clusters where the assigned samples are located. Repeat this operation until all samples have been assigned, or the similarity between the assigned samples and the unassigned samples is 0; (8) If there are samples that have not been assigned, assign the remaining samples to the cluster that contains the samples with higher density and closest distance to it; (9) The sample decision value θ in the cluster i The highest sample is used as the new cluster center. Repeat the above steps until the cluster center no longer changes, and output the classification result R1; Step S402: passing the classified result R1 into the neural network, performing a Dropout operation to prevent overfitting, and outputting the classified result R2; Step S403: pass the two classification results R1 and R2 into the voting machine and set the weights The final road defect classification result R is obtained by using the soft voting mechanism * ={ΠR 坑洼 ,ΠR 裂缝 }, where R 坑洼 =0, R 裂缝 =1.

9. The road defect detection method based on deep learning according to claim 1, characterized in that: The step S5 specifically includes: Step S501: According to the classification result R * The damage degree of pothole defects and crack defects is calculated respectively. The calculation formulas for the damage degree of potholes and cracks are as follows: Among them, p a The number of pixels quantified for the pothole defect target, p s is the number of pixels quantified in the entire road pothole defect image, a is the number of pixels in the smallest circumscribed rectangle of the crack; by using the pothole damage degree D p The severity of pothole image damage is measured numerically by using D n To measure the severity of crack image damage; Step S502: Classify the damage levels of pothole defects and crack defects as follows: Road pothole damage level: D p ≤C1 is a light pothole; C1≤D p ≤C2 is a medium-sized pothole; C2≤D p For heavy potholes; Road crack defect damage level: D a ≤C3 is a light seam defect; C3≤D a For heavy duty seams.

10. The road defect detection method based on deep learning according to claim 1, characterized in that: The step S6 specifically includes: Step S601: using a classifier to identify the obstacle type and location of potholes or cracks on the road, and transmitting them back to the vehicle display in a flashing manner through different graphics and colors; Step S602: Upload the road defect information and the driver's actions to the cloud server.

Citation Information

Patent Citations

  • Road surface defect identification method and system based on video deep learning

    CN112184625A

  • Pavement defect detection method and system based on deep learning

    CN114066808A

  • Road surface condition identification method based on deep learning

    CN116363613A

  • Video noise reduction method, video playing method, video noise reduction device, video playing device and electronic equipment

    CN117474786A

  • Road defect detection method and device based on improved YOLOv8 model and medium

    CN117576073A

Cited By

  • Real-time mileage stake mark identification method and system

    CN119904855A

  • Key nuclide prediction device based on multi-channel cavity convolution attention model

    CN120069169A

  • Die defect detection method and device based on deep learning

    CN120259177A

  • Road surface collapse risk monitoring method and system

    CN120339962A

  • Highway pavement defect nondestructive testing method based on unmanned aerial vehicle inspection

    CN121883491A