Road crack detection method, system, equipment and medium

Abstract

The invention discloses a road crack detection method, system and equipment and a medium, belongs to the technical field of deep learning and road maintenance, and aims to solve the technical problem of how to assist road maintenance personnel in performing road maintenance management work and correctly know that the road maintenance personnel take emergency measures for roads with quality risks. According to the technical scheme, the method comprises the steps that cameras are installed on key roads, and road video data are collected in real time; making a corresponding training set and a corresponding test set for the collected video data through a labeling tool; an ELAN feature fusion structure is optimized by fusing a part of convolution modules of a dilated convolution unit, meanwhile, an activation function is introduced to improve a YOLOv7 model, and a small target of a road crack is detected through the improved YOLOv7 model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and road maintenance technology, and specifically to a road crack detection method, system, equipment and medium. Background Art

[0002] With the continuous development of computer vision and artificial intelligence, informatization, digitization and intelligence have gradually become important trends and trends in social development, and have also led all walks of life to gradually enter the development track of intelligent production and intelligent supervision. In the era of big data and intelligence, the supervision model of road maintenance should also tend to be digital and intelligent. With the explosive growth of the number of vehicles, road maintenance has become particularly important, not only to improve the quality of road services, but also to ensure driving safety. The sharp increase in the number of vehicles has brought huge challenges to road maintenance. On the one hand, the road bears a greater load, resulting in damage to the road structure, resulting in cracks and potholes; on the other hand, the wear rate of the road increases, resulting in material aging, decreased road hardness, and easy cracks. Therefore, road maintenance has become a pain point and difficulty for road supervision departments. Traditional road quality detection mainly includes two methods: one is to rely on staff to conduct road inspections, and the other is to rely on road maintenance feedback provided by the masses. Among them, road inspections are prone to cause road congestion and pose a greater safety risk. Secondly, it consumes a lot of manpower costs, but cannot detect road problems in a timely manner. It can be seen that the traditional road maintenance supervision model is difficult to effectively ensure driving safety.

[0003] Therefore, how to assist road maintenance personnel in carrying out road maintenance management work, correctly let road maintenance personnel take emergency measures for roads with quality risks, and realize the intelligentization and informatization of road maintenance management are technical problems that need to be solved urgently. Summary of the invention

[0004] The technical task of the present invention is to provide a road crack detection method, system, equipment and medium to solve the problem of how to assist road maintenance personnel in road maintenance management, correctly let road maintenance personnel take emergency measures for roads with quality risks, and realize the intelligent and information-based road maintenance management.

[0005] The technical task of the present invention is achieved in the following manner: a road crack detection method, the method is as follows:

[0006] Install cameras on key roads to collect road video data in real time;

[0007] Use annotation tools to create corresponding training sets and test sets for the collected video data;

[0008] The ELAN feature fusion structure is optimized by fusing the partial convolution (PConv) module of the dilated convolution unit, and the HardSwish activation function is introduced to improve the YOLOv7 model. The improved YOLOv7 model is used to detect small targets in road cracks.

[0009] The improved YOLOv7 model is iteratively trained through the training set to obtain the YOLOv7 model after iterative convergence, and the YOLOv7 model after iterative convergence is verified using the test set to improve the stability and robustness of the improved YOLOv7 model;

[0010] Based on the iteratively converged YOLOv7 model, the road cracks in the video frame are identified and detected to obtain the road crack detection and recognition results;

[0011] According to the detection and recognition results, the confidence threshold is set to filter out the effective road detection anchor frames;

[0012] Count the number of effective detection anchor frames, that is, the number of cracks and potholes detected;

[0013] Calculate the effective detection anchor box area and remove the overlapping area between anchor boxes;

[0014] According to the big data analysis of road maintenance in real scenarios, the maximum crack or pothole number threshold and the maximum crack or pothole area threshold are set, and the statistical number of effective detection anchor frames and the calculated effective detection anchor frame area are used for decision-making and judgment:

[0015] When the number of cracks or potholes exceeds the maximum threshold or the area of ​​cracks or potholes, it means that the corresponding road section needs to be immediately maintained.

[0016] When the number of cracks or potholes is less than the maximum threshold and the area of ​​cracks or potholes is less than the maximum threshold, it means that the corresponding road section needs special attention.

[0017] As a preference, the design principle of the partial convolution module integrating the dilated convolution unit is as follows:

[0018] By taking advantage of the redundancy of the feature map synthesis function, the convolution algorithm is applied only to some channel features in the input feature map, and the remaining channel features are not convolved, which reduces the number of network model parameters and redundant calculations, thereby reducing the FLOPs (Floating Point Operations Per Second) indicator and effectively improving the model's computing efficiency.

[0019] In the process of selecting some channels for convolution operation, dilated convolution is used for convolution operation. The values ​​of adjacent pixels in the asphalt road image are basically the same. In the convolution process, the receptive field is expanded and the local information between adjacent pixels is ignored, which improves the performance of the model while keeping the amount of calculation unchanged.

[0020] The partial convolution module fused with dilated convolution units is embedded into the ELAN module. Specifically, the HardSwish activation function is used to improve the calculation rate of features in forward and backward propagation. The CBS convolution unit with a 1×1 convolution kernel and the partial convolution module fused with dilated convolution units are used. That is, the computational efficiency of the model is improved by filtering on the feature dimension. The receptive field of feature extraction is expanded through dilated convolution to improve the correlation between features and enhance the detection performance of the model.

[0021] Preferably, the HardSwish activation function and derivative calculation formula are as follows:

[0022] ReLU6(x)=min(6,max(0,x));

[0023]

[0024] Among them, x represents the feature variable.

[0025] Preferably, the improved YOLOv7 model is iteratively trained as follows:

[0026] During the training process, set the initial learning rate lr0, the maximum iteration period Epoch, and the training batch batch_size;

[0027] After the training is completed, the model training results are evaluated by the convergence of the loss function and the recall rate indicator to obtain the training results; among them, the recall rate refers to the ratio of the number of correctly predicted positive samples to the actual total number of positive samples, and the formula is as follows:

[0028]

[0029] Among them, TP (True Positives) represents the number of samples correctly classified as positive, that is, the number of samples whose actual category and predicted category are both positive; FN (False Negatives) represents the number of samples incorrectly classified as negative, that is, the number of samples whose actual category is positive but whose predicted category is negative.

[0030] Preferably, the anchor box includes the center point coordinates, width and height data of the detected target and the confidence of the anchor box, and the data format is (x, y, w, h, c); (x, y) represents the center point coordinates of the detected target; (w, h) represents the width and height of the anchor box; c represents the confidence of the detection result.

[0031] More preferably, the problem of calculating the area of ​​the overlapping region between anchor frames is the problem of calculating the area of ​​the overlapping region of two rectangles; specifically, as follows:

[0032] Find the coordinates of the upper left corners of the two rectangles and the lower right corners of the opposite corners;

[0033] Calculate the width and height of the overlapping area; the width is the minimum horizontal coordinate of the right rectangle minus the maximum horizontal coordinate of the left rectangle; the height is the maximum vertical coordinate of the upper rectangle minus the minimum vertical coordinate of the lower rectangle;

[0034] Calculate the area of ​​the overlapping rectangular area based on the width and height of the overlapping area.

[0035] A road crack detection system, which is used to implement the road crack detection method as described above; the system comprises:

[0036] Data acquisition module, used to install cameras on key roads to collect road video data in real time;

[0037] The streaming media module is used to transfer video data based on the RTMP protocol, process video frames in the streaming link, and package the processed video frames and push them to the video display page;

[0038] The front-end module is used to build the front-end page structure based on the Vue architecture according to business needs;

[0039] The recognition and discrimination module is used to detect and identify road cracks in the video frames of the video collection based on the improved YOLOv7 model, and to determine whether the road surface needs to be maintained immediately through the road quality decision discrimination method under the road crack factor;

[0040] The backend module is used to design the backend business logic based on the Django architecture according to business needs, integrate various functions, and provide early warning push functions to promptly feedback roads with crack hazards to road maintenance personnel;

[0041] The deployment module is used for clustered deployment of Nginx based on Nginx cluster in order to ensure the robustness of the system and effectively improve the stability of the system. The hardware equipment for building the Nginx cluster includes the master proxy server, slave proxy server, static resource server, file storage server, streaming media server and ordinary server.

[0042] Preferably, the identification and discrimination module comprises:

[0043] The data set production submodule is used to produce corresponding training sets and test sets for the collected video data through annotation tools;

[0044] The model improvement submodule is used to optimize the ELAN feature fusion structure by fusing the partial convolution (PConv) module with the dilated convolution unit, and to improve the YOLOv7 model by introducing the HardSwish activation function. The improved YOLOv7 model is used to detect small targets in road cracks.

[0045] The model training submodule is used to iteratively train the improved YOLOv7 model through the training set to obtain the YOLOv7 model after iterative convergence, and to verify the YOLOv7 model after iterative convergence using the test set to improve the stability and robustness of the improved YOLOv7 model;

[0046] The recognition and detection submodule is used to recognize and detect road cracks in the video frame based on the YOLOv7 model after iterative convergence, and obtain the road crack detection and recognition results;

[0047] The filtering submodule is used to set the confidence threshold according to the detection and recognition results, and filter out the effective road detection anchor frame; wherein the anchor frame includes the center point coordinate width and height data of the detection target and the confidence of the anchor frame, and the data format is (x, y, w, h, c); (x, y) represents the center point coordinate of the detection target; (w, h) represents the width and height of the anchor frame; c represents the confidence of the detection result;

[0048] The statistics submodule is used to count the number of effective detection anchor frames, that is, the number of cracks and potholes detected;

[0049] The area calculation submodule is used to calculate the effective detection anchor box area and remove the overlapping area between anchor boxes;

[0050] The decision-making and discrimination submodule is used to set the maximum crack or pothole number threshold and the maximum crack or pothole area threshold according to the big data analysis of road maintenance in real scenarios, and make decisions and discriminations on the statistical number of effective detection anchor frames and the calculated effective detection anchor frame area:

[0051] When the number of cracks or potholes exceeds the maximum threshold or the area of ​​cracks or potholes, it means that the corresponding road section needs to be immediately maintained.

[0052] When the number of cracks or potholes is less than the maximum threshold and the area of ​​cracks or potholes is less than the maximum threshold, it means that the corresponding road section needs special attention.

[0053] An electronic device comprising: a memory and at least one processor;

[0054] Wherein, the memory stores a computer program;

[0055] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the road crack detection method as described above.

[0056] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the road crack detection method as described above.

[0057] The road crack detection method, system, device and medium of the present invention have the following advantages:

[0058] (I) The present invention takes the maintenance and supervision mode of key roads as the entry point, and intelligently upgrades the traditional road maintenance and supervision mode based on computer vision and deep learning technology. By placing cameras on key roads, real-time acquisition and analysis of road surface images are carried out to provide road maintenance decisions, and better assist road maintenance personnel in maintenance management. In particular, in order to ensure the timeliness and accuracy of real-time video analysis, the present invention adopts an improved YOLOv7 model to detect cracks on key roads, correctly guide road maintenance personnel to take emergency measures for roads with quality risks, and realize intelligent and information-based road maintenance management.

[0059] (ii) The present invention detects and identifies cracks on key roads, introduces an improved YOLOv7 model algorithm and a road quality decision algorithm under road crack factors, performs real-time analysis and processing on video frames collected by cameras, correctly detects crack areas on key roads, analyzes road quality, provides correct road maintenance decisions, and promotes a road maintenance supervision model that is more information-based and intelligent;

[0060] (III) Based on the real scene requirements of key roads, the present invention optimizes and reconstructs the traditional YOLOv7 model in consideration of the challenges of placing a large number of cameras on the algorithm processing rate. First, the characteristics of asphalt roads are basically consistent, that is, the road pixel values ​​on each frame are similar. In view of this feature, the present invention proposes a partial convolution (PConv) module that integrates a dilated convolution unit, optimizes the extended efficient layer aggregation network (ELAN) in the traditional YOLOv7 model, improves the feature processing rate of the feature extraction network and the feature fusion network, and adopts the hard-Swish activation function to prevent the gradient from disappearing during forward propagation and improve the forward and reverse propagation rates of the model gradient.

[0061] (IV) The present invention adopts an improved YOLOv7 model to detect and identify cracks on key roads, and improves the deduction rate of the model by optimizing the network structure, thereby providing a model optimization strategy that meets the timeliness requirements in the field of real-time video analysis; at the same time, the current road maintenance supervision model is optimized, providing a scientific solution and architecture for the development of road maintenance supervision function software based on deep learning and computer vision;

[0062] (V) The present invention uses multiple cameras to collect real-time road video frames, and improves the YOLOv7 model to detect cracks on key roads, guiding road maintenance personnel to carry out road maintenance supervision, which is conducive to extending the service life of roads and improving the quality of road services. At the same time, it provides a strong guarantee for driving safety and improves the informationization and intelligence level of the road maintenance supervision system;

[0063] (VI) The present invention aims at the road features in the real environment, and effectively improves the operation efficiency of the YOLOv7 model by integrating the partial convolution module of the hole convolution unit and the hard-Swish activation function, meets the needs of real business scenarios, and is applied to accurately detect road cracks in video frames;

[0064] (VII) The present invention provides efficient data set training, based on open source related data sets, integrating data sets annotated in real application scenarios, and jointly participating in training, thereby improving the generalization ability of the training model;

[0065] (VIII) The road quality decision-making judgment under the road crack factor of the present invention is to set the tolerance value of road crack maintenance, judge the number of effective detection anchor frames and the area of ​​effective detection anchor frames, and guide the maintenance decision of road maintenance personnel;

[0066] (IX) The present invention uses dispatch detection and road quality decision discrimination to timely push road decision results to road maintenance personnel through text messages and other means, providing reliable road maintenance decisions and optimizing the road maintenance supervision model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present invention is further described below in conjunction with the accompanying drawings.

[0068] Attached Figure 1 This is a comparison chart of the HardSwish activation function image and the SiLU activation function image;

[0069] Attached Figure 2 It is a comparison diagram of HardSwish derivative image and SiLU derivative image;

[0070] Attached Figure 3 This is a schematic diagram of the structure of a partial convolution module that integrates a dilated convolution unit;

[0071] Attached Figure 4 It is a schematic diagram of the structure of a traditional ELAN module;

[0072] Attached Figure 5 It is a schematic diagram of the structure of the improved ELAN module;

[0073] Attached Figure 6 This is the training result diagram of the improved YOLOv7 model;

[0074] Attached Figure 7 Structural image for crack detection;

[0075] Attached Figure 8 This is the pothole detection result image;

[0076] Attached Fig. 9 It is a structural schematic diagram of the road crack detection system;

[0077] Attached Fig.10 This is a flow chart for decision-making on road maintenance. DETAILED DESCRIPTION

[0078] The road crack detection method, system, device and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Embodiment 1:

[0080] This embodiment provides a road crack detection method, which is specifically as follows:

[0081] S1. Install cameras on key roads to collect road video data in real time;

[0082] S2. Use annotation tools to create corresponding training and test sets for the collected video data; specifically: select the open source RDD2022 road dataset, which covers 47,420 road images in six countries and annotates them according to road damage conditions. In view of complex roads and scenes, 8,300 road images including viaducts, intersections, and highways were collected, and the labelme annotation tool was used to select and annotate road cracks and potholes. The annotated dataset was integrated into the RDD2022 road dataset, and divided into a ratio of 9:1 between the training set and the test set to generate the training set and test set for model training;

[0083] S3, by fusing the partial convolution (PConv) module of the hole convolution unit to optimize the ELAN feature fusion structure, and introducing the HardSwish activation function to improve the YOLOv7 model, the improved YOLOv7 model is used to detect small targets in road cracks;

[0084] Among them, real-time video data is collected as video frames and input into the improved YOLOv7 model for road crack detection and identification. According to the characteristics of asphalt roads and the need for high timeliness, the traditional YOLOv7 model is reconstructed from YOLOv7 model optimization, data set preparation, model training and result analysis;

[0085] The YOLOv7 model has excellent detection accuracy and high detection efficiency in the field of target detection through multi-scale feature fusion, anchor box optimization, and lightweight network structure. The model is mainly composed of four parts: input end, feature extraction network, feature fusion network, and output end. The input end uses mosaic technology to enhance data by rotating, scaling, cropping, and splicing the input image to improve the generalization ability of model training; the feature extraction network consists of multiple CBS convolution modules, ELAN feature fusion modules, and MP pooling modules. Among them, the CBS module consists of Conv convolution layer, BN normalization layer, and SiLU activation function; the ELAN feature fusion module consists of multiple CBS modules and residual modules; the MP pooling module mainly performs downsampling through maximum pooling operations, eliminating the complex convolution process; the feature fusion network follows the YOLO series Path Aggregation Feature Pyramid Network (PAFPN) architecture, which fuses the feature information of feature maps of different scales and improves the sensitivity of the network model to small targets. In summary, the YOLOv7 model has the advantages of fast deduction speed and strong feature learning ability, especially for small targets with higher detection accuracy, and is suitable for road crack detection application scenarios to a certain extent.

[0086] Considering that a large number of cameras need to be installed on key roads, higher algorithm operation efficiency is required, and the traditional YOLOv7 model is difficult to achieve high operation efficiency. Therefore, for this business scenario, it is necessary to optimize the deduction rate of the traditional YOLOv7 model to further improve the operation efficiency of the model. This embodiment introduces the hard-Swish activation function into the traditional YOLOv7 model, and proposes a partial convolution (PConv) module that integrates a dilated convolution unit to optimize the ELAN feature fusion structure, which greatly improves the operation efficiency of the model and meets the needs of real business scenarios.

[0087] S4. Iteratively train the improved YOLOv7 model through the training set to obtain the YOLOv7 model after iterative convergence, and use the test set to verify the YOLOv7 model after iterative convergence to improve the stability and robustness of the improved YOLOv7 model;

[0088] S5. Identify and detect road cracks in the video frame based on the iteratively converged YOLOv7 model to obtain a road crack detection and identification result;

[0089] S6. In order to accurately and efficiently judge the current quality of the road and save road maintenance costs, this example designs a road maintenance decision-making process based on the detection results of road cracks. The determination flow chart is shown in the attached figure. Fig.10 As shown in the figure, according to the detection and recognition results, a confidence threshold is set to filter out valid road detection anchor frames; the threshold set in this embodiment is 0.7. The YOLOv7 model is used to obtain the road crack detection anchor frame according to the confidence threshold. When c<0.7, the detection result is considered inaccurate and the anchor frame is an invalid detection; when c>0.7, the detection result is considered accurate and the anchor frame is a valid detection.

[0090] S7, counting the number of effective detection anchor frames, that is, the number of cracks and potholes detected;

[0091] S8, calculating the effective detection anchor box area and removing the overlapping area between anchor boxes;

[0092] S9. According to the big data analysis of road maintenance in real scenarios, set the maximum crack or pothole number threshold and the maximum crack or pothole area threshold, and make decisions on the statistical number of effective detection anchor frames and the calculated effective detection anchor frame area:

[0093] ① When the number of cracks or potholes exceeds the maximum threshold or the maximum crack or pothole area threshold, it means that the corresponding road section needs to be immediately maintained;

[0094] ② When it is less than the maximum crack or pothole number threshold and the maximum crack or pothole area threshold, it means that the corresponding road section needs special attention.

[0095] The design principle of the partial convolution module fused with the dilated convolution unit in step S3 of this embodiment is as follows:

[0096] S301, using the redundancy of the feature map synthesis function, only apply the convolution algorithm to some channel features in the input feature map, and do not perform convolution operations on the remaining channel features, thereby reducing the number of network model parameters and cutting down on redundant calculations, thereby reducing the FLOPs (Floating Point Operations Per Second) index and effectively improving the computational efficiency of the model;

[0097] S302, in the process of selecting some channels for convolution operation, dilated convolution is used for convolution operation; wherein, the values ​​of adjacent pixels of the asphalt road image are basically the same, and the receptive field is expanded in the convolution process, and the local information between adjacent pixels is ignored, so as to improve the performance of the model while keeping the amount of calculation unchanged;

[0098] S303, embedding the partial convolution module fused with the dilated convolution unit into the ELAN module, specifically: using the HardSwish activation function to improve the calculation rate of the features in forward and backward propagation; and using the CBS convolution unit with a 1×1 convolution kernel and the partial convolution module fused with the dilated convolution unit; that is, firstly improving the calculation efficiency of the model by filtering on the feature dimension; and then expanding the receptive field of feature extraction through dilated convolution, improving the correlation between features, and improving the detection performance of the model.

[0099] The partial convolution (PConv) module fused with the dilated convolution unit is shown in the attached figure. Figure 3 As shown in the figure. First, the input multi-dimensional feature map is effectively filtered and only part of the feature map is convolved, which effectively improves the calculation efficiency; then, the filtered feature map is subjected to feature extraction again through a dilated convolution operation. Compared with traditional convolution, the dilated convolution kernel has a larger receptive field range, and the output of each convolution contains a larger feature range, which effectively fits the characteristics of the local pixel value differences of the road image. By expanding the feature calculation range, the detection accuracy of road cracks is improved.

[0100] The traditional YOLOv7 model uses the Efficient Layer Aggregation Network (ELAN) for feature extraction and feature fusion. By controlling the shortest and longest gradient paths, the network model can learn more features and effectively superimpose and fuse the learned features. Figure 4 and5 It can also be seen that the traditional ELAN module is embedded with 7 CBS convolution units, which leads to an excessively deep structural network, greatly reducing the model's operational efficiency and not meeting the requirements for efficient operation of road crack detection. Therefore, the present invention optimizes the structure of the traditional ELAN module, embeds the partial convolution (PConv) module that integrates the hole convolution unit into the ELAN module, and replaces the traditional CBS convolution unit. Under the premise of ensuring detection accuracy, the operational efficiency of the network model is greatly improved. The structural comparison diagram of the traditional ELAN module and the improved ELAN module is shown in the attached figure. Figure 4 and 5 shown.

[0101] As attached Figure 4 and 5 In the figure, k represents the size of the convolution kernel, k=1 represents a 1×1 convolution kernel, and k=3 represents a 3×3 convolution kernel; s represents the step size of the convolution operation; Figure 4 and 5 The design ideas for improving the ELAN module network structure are:

[0102] ① The HardSwitch activation function is used to replace the SiLU activation function in the CBS basic convolution unit to improve the calculation rate of features in forward propagation and back propagation;

[0103] ② The CBS convolution unit with 1×1 convolution kernel in the traditional ELAN module remains unchanged, and the CBS convolution unit with 3×3 convolution kernel is replaced by the PConv convolution unit with fused dilated convolution. First, the computational efficiency of the model is improved by filtering in the feature dimension; second, the receptive field range of feature extraction is expanded through dilated convolution, the correlation between features is increased, and the detection performance of the model is improved.

[0104] The traditional YOLOv7 model uses the SiLU nonlinear activation function, which is smooth over the entire real number domain, helping the gradient to flow more stably during back propagation and reducing the problem of gradient vanishing or exploding. However, the calculation and derivation process of this function is too complicated, affecting the computational efficiency. The present invention introduces the hard-Swish activation function to replace the SiLU activation function, because the hard-Swish activation function has a similar nonlinear structure to the SiLU activation function, and the calculation process of the hard-Swish activation function is simpler and has higher computational efficiency. The comparison of the function image and derivative image of the hard-Swish activation function and the SiLU activation function is shown in the attached figure. Figure 1 and 2 shown.

[0105] The calculation formula of SiLU activation function and derivative is as follows, where x represents the feature variable:

[0106]

[0107] SiLU(x)=x*sigmoid(x);

[0108]

[0109] The calculation formula of the HardSwish activation function and derivative in step S3 of this embodiment is as follows:

[0110] ReLU6(x)=min(6,max(0,x));

[0111]

[0112] Among them, x represents the feature variable.

[0113] The realization of a fast network model is inseparable from the optimization of the number of floating point operations (FLOPs) during the network deduction and calculation process.

[0114] The iterative training of the improved YOLOv7 model in step S4 of this embodiment is specifically as follows:

[0115] S401, during the training process, set the initial learning rate lr0, the maximum iteration period Epoch and the training batch batch_size;

[0116] S402. The hardware environment for network training in this embodiment is to use the Ubuntu operating system, and use the NVIDIA GeForce GTX1080Ti graphics card for improved YOLOv7 model training, with a video memory of 11GB. The software environment is Python, and the deep learning part is implemented using PyTorch framework programming. During the model training process, the initialization learning rate lr0 is set to 0.01, the maximum iteration cycle Epoch is 1000, and the training batch batch_size is 16. After the training is completed, the model training results are evaluated by the convergence of the loss function and the recall rate (Recall) indicator. The training results obtained are as shown in the attached figure. Figure 6 As shown; where recall refers to the ratio of the number of correctly predicted positive samples to the actual total number of positive samples, the formula is as follows:

[0117]

[0118] Among them, TP (True Positives) represents the number of samples correctly classified as positive, that is, the number of samples whose actual category and predicted category are both positive; FN (False Negatives) represents the number of samples incorrectly classified as negative, that is, the number of samples whose actual category is positive but whose predicted category is negative.

[0119] By adjusting the model training parameters, the optimal model training weights are obtained, and the trained YOLOv7 model is applied to road crack detection in video frames. The detection results are shown in the attached figure. Figure 7 and 8 As shown in the figure, the experimental results show that the improved YOLOv7 model can correctly detect and identify cracks and potholes. Through model structure optimization, it not only improves the efficiency of road crack detection and meets the real-time requirements of road crack detection, but also improves the stability and generalization of the algorithm model to a certain extent.

[0120] The anchor frame in step S6 of this embodiment includes the center point coordinate width and height data of the detected target and the confidence of the anchor frame, and the data format is (x, y, w, h, c); (x, y) represents the center point coordinates of the detected target; (w, h) represents the width and height of the anchor frame; c represents the confidence of the detection result.

[0121] The problem of calculating the area of ​​the overlapping regions between anchor frames in step S8 of this embodiment is the problem of calculating the area of ​​the overlapping regions between two rectangles; the details are as follows:

[0122] S801, find the coordinate points of the upper left corners of two rectangles and the coordinate points of the lower right corners of the opposite corners;

[0123] S802, calculating the width and height of the overlapping area; wherein the width is the minimum horizontal coordinate of the right rectangle minus the maximum horizontal coordinate of the left rectangle; and the height is the maximum vertical coordinate of the upper rectangle minus the minimum vertical coordinate of the lower rectangle;

[0124] S803: Calculate the area of ​​the overlapping rectangular region according to the width and height of the overlapping region.

[0125] Embodiment 2:

[0126] As attached Fig. 9 As shown, this embodiment provides a road crack detection system, which is used to implement the road crack detection method in embodiment 1; the system includes:

[0127] Data acquisition module, used to install cameras on key roads to collect road video data in real time;

[0128] The streaming media module is used to transfer video data based on the RTMP protocol, process video frames in the streaming link, and package the processed video frames and push them to the video display page;

[0129] The front-end module is used to build the front-end page structure based on the Vue architecture according to business needs;

[0130] The recognition and discrimination module is used to detect and identify road cracks in the video frames of the video collection based on the improved YOLOv7 model, and to determine whether the road surface needs to be maintained immediately through the road quality decision discrimination method under the road crack factor;

[0131] The backend module is used to design the backend business logic based on the Django architecture according to business needs, integrate various functions, and provide early warning push functions to promptly feedback roads with crack hazards to road maintenance personnel;

[0132] The deployment module is used for clustered deployment of Nginx based on Nginx cluster in order to ensure the robustness of the system and effectively improve the stability of the system. The hardware equipment for building the Nginx cluster includes the master proxy server, slave proxy server, static resource server, file storage server, streaming media server and ordinary server.

[0133] The identification module in this embodiment includes:

[0134] The data set production submodule is used to produce corresponding training sets and test sets for the collected video data through annotation tools;

[0135] The model improvement submodule is used to optimize the ELAN feature fusion structure by fusing the partial convolution (PConv) module with the dilated convolution unit, and to improve the YOLOv7 model by introducing the HardSwish activation function. The improved YOLOv7 model is used to detect small targets in road cracks.

[0136] The model training submodule is used to iteratively train the improved YOLOv7 model through the training set to obtain the YOLOv7 model after iterative convergence, and to verify the YOLOv7 model after iterative convergence using the test set to improve the stability and robustness of the improved YOLOv7 model;

[0137] The recognition and detection submodule is used to recognize and detect road cracks in the video frame based on the YOLOv7 model after iterative convergence, and obtain the road crack detection and recognition results;

[0138] The filtering submodule is used to identify the results of the detection, such as the attached Fig.10As shown in the figure, a confidence threshold is set to filter out valid road detection anchor frames; wherein the anchor frame includes the center point coordinate width and height data of the detection target and the confidence of the anchor frame, and the data format is (x, y, w, h, c); (x, y) represents the center point coordinate of the detection target; (w, h) represents the width and height of the anchor frame; c represents the confidence of the detection result;

[0139] The statistics submodule is used to count the number of effective detection anchor frames, that is, the number of cracks and potholes detected;

[0140] The area calculation submodule is used to calculate the effective detection anchor box area and remove the overlapping area between anchor boxes;

[0141] The decision-making and discrimination submodule is used to set the maximum crack or pothole number threshold and the maximum crack or pothole area threshold according to the big data analysis of road maintenance in real scenarios, and make decisions and discriminations on the statistical number of effective detection anchor frames and the calculated effective detection anchor frame area:

[0142] When the number of cracks or potholes exceeds the maximum threshold or the area of ​​cracks or potholes, it means that the corresponding road section needs to be immediately maintained.

[0143] When the number of cracks or potholes is less than the maximum threshold and the area of ​​cracks or potholes is less than the maximum threshold, it means that the corresponding road section needs special attention.

[0144] The working process of the system is as follows:

[0145] Step 1: Place cameras on key roads to collect real-time images of key roads.

[0146] Step 2: Build an rtmp streaming server, repackage the video frames processed by the algorithm into a video stream in rtmp format through the Ffmpeg tool, and push it to the front-end page for display.

[0147] Step 3: Build the Vue front-end page, obtain the video stream processed by the algorithm from the rtmp streaming server, and realize the real-time video screen display after algorithm detection.

[0148] Step 4: Build the Django backend architecture to complete data interaction with the front end and database on the one hand, and coordinate and schedule to improve the road crack detection algorithm and road quality decision-making method of the YOLOv7 model on the other hand, and push the decision-making results to road maintenance personnel in a timely manner through SMS and other means.

[0149] Step 5: System testing: Test the algorithm’s execution efficiency and accuracy in real road scenarios, test the system’s execution, and fix any derivative problems that arise during the testing process.

[0150] Step 6: System deployment. In order to improve the stability and robustness of the system, the system deployment adopts a clustered deployment solution based on Nginx. The deployment architecture mainly consists of a master proxy server, a slave proxy server, a static resource server, a file storage server, a streaming media server, and a back-end ordinary server. Among them, the file storage server saves video frames with cracks or holes during the algorithm model detection process.

[0151] Embodiment 3:

[0152] This embodiment also provides an electronic device, including: a memory and a processor;

[0153] Wherein, the memory stores computer-executable instructions;

[0154] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the road crack detection method in any embodiment of the present invention.

[0155] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0156] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0157] Embodiment 4:

[0158] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the road crack detection method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0159] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0160] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0161] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0162] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A road crack detection method, characterized in that: The method is as follows: Install cameras on key roads to collect road video data in real time; Use annotation tools to create corresponding training sets and test sets for the collected video data; The ELAN feature fusion structure is optimized by fusing the partial convolution module of the dilated convolution unit, and the HardSwish activation function is introduced to improve the YOLOv7 model. The improved YOLOv7 model is used to detect small targets in road cracks. The improved YOLOv7 model is iteratively trained through the training set to obtain the YOLOv7 model after iterative convergence, and the test set is used to verify the YOLOv7 model after iterative convergence; Based on the iteratively converged YOLOv7 model, the road cracks in the video frame are identified and detected to obtain the road crack detection and recognition results; According to the detection and recognition results, the confidence threshold is set to filter out the effective road detection anchor frames; Count the number of effective detection anchor frames, that is, the number of cracks and potholes detected; Calculate the effective detection anchor box area and remove the overlapping area between anchor boxes; According to the big data analysis of road maintenance in real scenarios, the maximum crack or pothole number threshold and the maximum crack or pothole area threshold are set, and the statistical number of effective detection anchor frames and the calculated effective detection anchor frame area are used for decision-making and judgment: When the number of cracks or potholes exceeds the maximum threshold or the area of ​​cracks or potholes, it means that the corresponding road section needs to be immediately maintained. When the number of cracks or potholes is less than the maximum threshold and the area of ​​cracks or potholes is less than the maximum threshold, it means that the corresponding road section needs special attention.

2. The road crack detection method according to claim 1, characterized in that: The design principle of the partial convolution module fused with the dilated convolution unit is as follows: By taking advantage of the redundancy of the feature map synthesis function, the convolution algorithm is applied only to some channel features in the input feature map, and the remaining channel features are not convolved, which reduces the number of network model parameters and reduces redundant calculations. In the process of selecting some channels for convolution operation, dilated convolution is used for convolution operation. The values ​​of adjacent pixels in the asphalt road image are basically the same. In the convolution process, the receptive field is expanded and the local information between adjacent pixels is ignored, which improves the performance of the model while keeping the amount of calculation unchanged. The partial convolution module fused with dilated convolution units is embedded into the ELAN module. Specifically, the HardSwish activation function is used to improve the calculation rate of features in forward and backward propagation. The CBS convolution unit with a 1×1 convolution kernel and the partial convolution module fused with dilated convolution units are used. That is, the computational efficiency of the model is improved by filtering on the feature dimension. The receptive field of feature extraction is expanded through dilated convolution to improve the correlation between features and enhance the detection performance of the model.

3. The road crack detection method according to claim 1 or 2, characterized in that: The calculation formula of HardSwish activation function and derivative is as follows: ReLU6(x)=min(6,max(0,x)); Among them, x represents the feature variable.

4. The road crack detection method according to claim 3, characterized in that: The iterative training of the improved YOLOv7 model is as follows: During the training process, set the initial learning rate lr0, the maximum iteration period Epoch, and the training batch batch_size; After the training is completed, the model training results are evaluated by the convergence of the loss function and the recall rate indicator to obtain the training results; among them, the recall rate refers to the ratio of the number of correctly predicted positive samples to the actual total number of positive samples, and the formula is as follows: Among them, TP represents the number of samples correctly classified as positive, that is, the number of samples whose actual category and predicted category are both positive; FN represents the number of samples incorrectly classified as negative, that is, the number of samples whose actual category is positive but whose predicted category is negative.

5. The road crack detection method according to claim 4, characterized in that: The anchor box includes the center point coordinates, width and height data of the detected target and the confidence of the anchor box. The data format is (x, y, w, h, c); (x, y) represents the center point coordinates of the detected target; (w, h) represents the width and height of the anchor box; c represents the confidence of the detection result.

6. The road crack detection method according to claim 5, characterized in that: The problem of calculating the area of ​​the overlapping region between anchor frames is the problem of calculating the area of ​​the overlapping region between two rectangles; the details are as follows: Find the coordinates of the upper left corners of the two rectangles and the lower right corners of the opposite corners; Calculate the width and height of the overlapping area; the width is the minimum horizontal coordinate of the right rectangle minus the maximum horizontal coordinate of the left rectangle; the height is the maximum vertical coordinate of the upper rectangle minus the minimum vertical coordinate of the lower rectangle; Calculate the area of ​​the overlapping rectangular area based on the width and height of the overlapping area.

7. A road crack detection system, characterized in that: The system is used to implement the road crack detection method as claimed in any one of claims 1 to 6; the system comprises: Data acquisition module, used to install cameras on key roads to collect road video data in real time; The streaming media module is used to transfer video data based on the RTMP protocol, process video frames in the streaming link, and package the processed video frames and push them to the video display page; The front-end module is used to build the front-end page structure based on the Vue architecture according to business needs; The recognition and discrimination module is used to detect and identify road cracks in the video frames of the video collection based on the improved YOLOv7 model, and to determine whether the road surface needs to be maintained immediately through the road quality decision discrimination method under the road crack factor; The backend module is used to design the backend business logic based on the Django architecture according to business needs, integrate various functions, and provide early warning push functions to promptly feedback roads with crack hazards to road maintenance personnel; The deployment module is used to perform clustered deployment of Nginx based on the Nginx cluster; the hardware devices for building the Nginx cluster include the master proxy server, the slave proxy server, the static resource server, the file storage server, the streaming media server and the ordinary server.

8. The road crack detection system according to claim 7, characterized in that: The identification and discrimination module comprises: The data set production submodule is used to produce corresponding training sets and test sets for the collected video data through annotation tools; The model improvement submodule is used to optimize the ELAN feature fusion structure by fusing the partial convolution module of the dilated convolution unit, and introduce the HardSwish activation function to improve the YOLOv7 model. The improved YOLOv7 model is used to detect small targets in road cracks. The model training submodule is used to iteratively train the improved YOLOv7 model through the training set, obtain the YOLOv7 model after iterative convergence, and verify the YOLOv7 model after iterative convergence using the test set; The recognition and detection submodule is used to recognize and detect road cracks in the video frame based on the YOLOv7 model after iterative convergence, and obtain the road crack detection and recognition results; The filtering submodule is used to set the confidence threshold according to the detection and recognition results, and filter out the effective road detection anchor frame; wherein the anchor frame includes the center point coordinate width and height data of the detection target and the confidence of the anchor frame, and the data format is (x, y, w, h, c); (x, y) represents the center point coordinate of the detection target; (w, h) represents the width and height of the anchor frame; c represents the confidence of the detection result; The statistics submodule is used to count the number of effective detection anchor frames, that is, the number of cracks and potholes detected; The area calculation submodule is used to calculate the effective detection anchor box area and remove the overlapping area between anchor boxes; The decision-making and discrimination submodule is used to set the maximum crack or pothole number threshold and the maximum crack or pothole area threshold according to the big data analysis of road maintenance in real scenarios, and make decisions and discriminations on the statistical number of effective detection anchor frames and the calculated effective detection anchor frame area: When the number of cracks or potholes exceeds the maximum threshold or the area of ​​cracks or potholes, it means that the corresponding road section needs to be immediately maintained. When the number of cracks or potholes is less than the maximum threshold and the area of ​​cracks or potholes is less than the maximum threshold, it means that the corresponding road section needs special attention.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the road crack detection method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the road crack detection method according to any one of claims 1 to 6.

Images