Intelligent Detection and Maintenance System for Foreign Objects on Expressway Pavement

Abstract

The intelligent detection and maintenance system for foreign objects on highway pavements includes modules such as data acquisition, image processing, model training, on-site control, execution of maintenance, remote access, on-site self-organizing network, and maintenance master console. First, videos of foreign objects on highway pavements are collected to construct an image dataset. Abnormal pictures are cleaned, and the optimized dataset is labeled. The labeled data is used for model training. Then, the trained model is deployed to the on-site controller, and its detection results are received. The background server conducts remote access through the network, thereby controlling each maintenance terminal. Each terminal communicates with each other to form an on-site information interaction network to achieve collaborative detection. Finally, the detection and maintenance results are displayed on the maintenance master console. Compared with traditional highway maintenance methods, it realizes unmanned operation, can comprehensively and accurately identify foreign objects, achieves all-day real-time detection of foreign objects on highway pavements, and conducts effective maintenance, improving the maintenance efficiency and safety and solving the problem of shortage of maintenance personnel.

Description

Technical Field

[0001] The present invention belongs to the field of foreign object detection and maintenance of expressways, especially the intelligent detection and maintenance system for foreign objects on the expressway pavement. Background Art

[0002] With the rapid development of expressways in China, the safe operation of expressways has become particularly important. When vehicles are traveling at high speeds, foreign objects may suddenly fall from the vehicle in front or fly out from the roadside, or be carried up by the high-speed rotating wheels of the vehicle in front. Due to the large kinetic energy, it is very easy to cause harm to subsequent vehicles and personnel, resulting in vehicle out of control or even multi-vehicle collisions, which not only endangers the safety of the involved and passing vehicles, but also seriously affects the normal traffic on the expressway. Foreign objects on expressways mainly refer to substances such as bottles, cartons, lunch boxes, wooden boards, etc. These foreign objects not only affect the quality of expressways in China, but also pose potential safety hazards to driving. To effectively ensure the safety of citizens' travel, it is crucial to detect, prevent and maintain foreign objects on expressways.

[0003] Currently, for the detection and maintenance of foreign objects on expressways in China, most adopt manual detection and maintenance methods. Manual detection of foreign objects requires staff to enter the traffic flow for operation, which has certain safety hazards and may lead to traffic accidents. It also requires a large amount of manpower input, increasing the cost of traffic management departments. In addition, staff operating on the expressway for a long time are prone to fatigue or negligence, resulting in missed detections or misjudgments, posing potential dangers to road traffic.

[0004] To ensure the safety and effectiveness of road traffic, various methods need to be adopted to detect and maintain foreign objects on expressways. Using scientific and technological means such as video monitoring and target detection to achieve intelligent detection and reduce the risks and deficiencies of manual detection has become a preferred solution. Intelligent detection can comprehensively and accurately detect foreign objects on expressways, improving the accuracy and precision of detection. Compared with traditional manual detection and maintenance methods, intelligent detection and maintenance can achieve automated operation, reduce manual intervention, thereby improving detection efficiency, reducing management costs, reducing staff fatigue and negligence, reducing the situation of missed detections or misjudgments, reducing potential safety hazards, and ensuring the safety of staff and road traffic. Summary of the Invention

[0005] The technical problem solved by the present invention is: to overcome the deficiencies of the prior art and provide an intelligent detection and maintenance system for foreign objects on the expressway pavement, which can comprehensively and accurately identify foreign objects on expressways, realize all-day real-time and effective detection and maintenance of foreign objects on the expressway pavement, improve the safety and traffic efficiency of road traffic, and at the same time can also improve the efficiency and quality of highway maintenance and reduce maintenance costs.

[0006] The technical solution of the present invention is: an intelligent detection and maintenance system for foreign objects on highway pavements, which is characterized by including modules such as data acquisition, image processing, model training, on-site control, execution of maintenance, remote access, on-site self-organizing network, and maintenance master console. The specific steps are as follows:

[0007] 1) The data acquisition module uses a camera to capture visible light and infrared videos of foreign objects on the highway, collects interval video frame images, and constructs a foreign object image data set.

[0008] 2) The image processing module diagnoses and cleans the abnormal image data, and labels the optimized image data set.

[0009] 3) The model training module uses the labeled image data set for model training and verification.

[0010] 4) The on-site control module deploys the trained model to the on-site controller to detect foreign objects on the highway all-weather and in real time.

[0011] 5) The execution of maintenance module receives the detection results sent by the on-site controller and drives the maintenance execution mechanism to implement maintenance operations.

[0012] 6) The remote access module enables the background server to remotely access and control the on-site control module through the network, and at the same time receives the detection and maintenance results of the on-site maintenance terminal.

[0013] 7) The on-site self-organizing network module enables each maintenance terminal to communicate with each other, forms an on-site information interaction network to achieve collaborative detection, and feeds back the information fusion results of the detection and maintenance of the maintenance terminal to the background server.

[0014] 8) The maintenance master console, by connecting to the background server, displays in real time the detection and maintenance results and operating conditions of each on-site maintenance terminal, reports the information of foreign objects that cannot be processed on-site, and gives different levels of alarm reminders according to the hazard level. Subsequently, the background will conduct further manual review and processing.

[0015] The data acquisition module described in step 1) is specifically as follows: using a highway guardrail robot to capture a foreign object video data set, converting the video data set into video frame images, collecting interval video frame images, constructing a foreign object detection data set, and transmitting it to the cloud platform for preprocessing. The data set categories include water bottles, cardboard boxes, lunch boxes, potholes, mileage signs, horizontal and vertical cracks, etc.

[0016] The image processing module described in step 2) is specifically: using a data diagnosis method to check whether there are abnormal attribute pictures in the data set, such as too low resolution, insufficient clarity, etc. According to the data diagnosis results, clean the abnormal pictures, and use a labeling tool to label the optimized foreign object data set to obtain label file information.

[0017] The model training module described in step 3) is specifically as follows: The labeled image dataset and label information are fed into the network to extract image features, training parameters are set for model training. If the model is basically convergent and in a stable state, model verification can be carried out; otherwise, model training can continue through optimization methods such as modifying hyperparameters.

[0018] The on-site control module described in step 4) is specifically as follows: The trained weight file is deployed to the on-site controller. By reading the video data stream of the camera, the running effect in the actual highway application scenario is detected in real time, enabling the system to be used in other occasions and realizing the adaptive function.

[0019] The maintenance execution module described in step 5) is specifically as follows: It receives the detection results sent by the on-site controller, including the category, location, and distance information of foreign objects, and drives the maintenance execution mechanism to perform maintenance operations, such as picking up foreign objects, cleaning foreign objects, reporting the information of foreign objects that cannot be processed on-site and giving an alarm to prompt the maintenance console to handle them in a timely manner. The distance information is obtained by the monocular ranging method, and the ranging formula is as follows:

[0020]

[0021] Where P bottom represents the pixel value of the bottom of the target object on the imaging plane, P center refers to the pixel coordinate of the optical center of the lens in the y direction in the pixel coordinate system, f represents the focal length of the camera, which can be obtained through camera calibration and calibration, H is the installation height of the camera, θ is the included angle between the target object and the base plane, Dis is the distance between the target object and the camera. From this, the distance between the target object and the camera can be obtained:

[0022] Dis = (H * f) / (P bottom ― P center )

[0023] The remote access module described in step 6) is specifically as follows: The background server remotely accesses and controls the on-site control module through the network, and then controls each on-site maintenance terminal. At the same time, it also receives the detection and maintenance result situations of the on-site maintenance terminal, so as to realize the real-time and effective detection and maintenance of foreign objects on the highway pavement.

[0024] Step 7) The on-site self-organizing network module is as follows: Each on-site maintenance terminal communicates with other maintenance terminals through the network to form an on-site information interaction network for collaborative detection. The detection and maintenance results of each on-site maintenance terminal are fused on each on-site maintenance terminal. According to the fusion results, the on-site maintenance terminal continues to perform detection or maintenance and feeds back to the background server, and then controls the camera pan-tilt, and iteratively optimizes the previously trained model.

[0025] Step 8) The overall maintenance console is as follows: By connecting to the background server, it real-time displays the detection and maintenance results of each on-site maintenance terminal, as well as the operation status of each on-site maintenance terminal. According to the foreign object information reported above that cannot be processed on-site, the monitoring engineer of the overall maintenance console further makes a manual judgment on the urgency according to the hazard level generated by the system. According to the urgency and combined with the pile number information, a maintenance team is dispatched to the site for manual maintenance to eliminate potential safety hazards and ensure the fast and safe passing ability of the expressway.

[0026] The advantages of the present invention compared with the prior art are as follows: The intelligent detection and maintenance system for expressway pavement foreign objects of the present invention can comprehensively and accurately identify expressway foreign objects, realize all-day real-time and effective detection and maintenance of expressway pavement foreign objects, improve the safety and passing efficiency of road traffic, and at the same time can also improve the efficiency and quality of highway maintenance and reduce maintenance costs. Brief Description of the Drawings

[0027] Figure 1 is the overall structural schematic diagram of the method of the present invention;

[0028] Figure 2 is the flow chart of the method of the present invention;

[0029] Figure 3 is the image data set diagram of the method of the present invention;

[0030] Figure 4 is the abnormal image cleaning diagram of the method of the present invention;

[0031] Figure 5 is the image label parameter diagram of the method of the present invention;

[0032] Figure 6 is the relationship diagram of accuracy and confidence for training 500 rounds of the method of the present invention;

[0033] Figure 7 is the mean average precision diagram for training 500 rounds of the method of the present invention;

[0034] Figure 8 is the relationship diagram of recall rate and confidence for training 500 rounds of the method of the present invention;

[0035] Figure 9 It is a graph showing the box_loss, obj_loss, cls_loss, accuracy, recall, and average mAP results at different IoU thresholds after 500 rounds of training using the method of the present invention;

[0036] Figure 10 It is a detection effect diagram of the method of the present invention;

[0037] Figure 11 It is a flowchart of the web platform of the method of the present invention;

[0038] Figure 12 It is a web detection effect diagram of the method of the present invention. Detailed implementation manners

[0039] The flowchart of the present invention is shown in the appendix Figure 2 , and the specific steps are as follows:

[0040] 1) Data acquisition module: Use a camera to capture visible light and infrared videos of foreign objects on the highway, collect video frame images at intervals, and construct a foreign object image dataset;

[0041] 2) Image processing module: Diagnose and clean abnormal image data, and label the optimized image dataset;

[0042] 3) Model training module: Use the labeled image dataset to train and verify the model;

[0043] 4) On-site control module: Deploy the trained model to the on-site controller to detect foreign objects on the highway in real time all day long;

[0044] 5) Maintenance execution module: Receive the detection results sent by the on-site controller and drive the maintenance execution mechanism to implement maintenance operations;

[0045] 6) Remote access module: The background server remotely accesses and controls the on-site control module through the network, and at the same time receives the detection and maintenance results of the on-site maintenance terminal;

[0046] 7) On-site self-organizing network module: Each maintenance terminal communicates with each other to form an on-site information interaction network to achieve collaborative detection, and feedback the information fusion results of the detection and maintenance of the maintenance terminal to the background server;

[0047] 8) Maintenance console: By connecting to the background server, it displays in real time the detection and maintenance results and operation status of each on-site maintenance terminal, reports the information of foreign objects that cannot be processed on site, and gives different levels of alarm reminders according to the hazard level. Subsequently, the background will conduct further manual review and processing.

[0048] Step 1) The data acquisition module is as follows: Use the highway guardrail robot to capture a foreign object video dataset, convert the video dataset into video frame images, collect interval video frame images, construct a foreign object detection dataset, and transmit it to the pan-tilt for preprocessing. The dataset categories include water bottles, cartons, lunch boxes, potholes, mileage signs, horizontal and vertical cracks, etc.

[0049] Step 2) The image processing module is specifically: Use the data diagnosis method to check whether the dataset has abnormal attribute pictures, such as too low resolution, insufficient clarity, etc. According to the data diagnosis results, clean the abnormal pictures, and use annotation tools such as Makesense to annotate the optimized foreign object dataset to obtain label file information. It is necessary to convert json and xml format files into txt format label files.

[0050] Step 3) The model training module is specifically: Send the labeled picture dataset and txt format file label information into the YOLOv5 network, extract image features and set training parameters for model training. If the model is basically convergent and in a stable state, model verification can be carried out. Otherwise, continue model training through optimization methods such as modifying hyperparameters.

[0051] Step 4) The on-site control module is as follows: Deploy the trained weight file to the Jetson TX2 on the embedded platform, and through reading the video data stream of the camera, perform real-time detection on the operation effect in the actual highway application scenario, so that the system can also be used in other occasions to achieve the adaptive function.

[0052] Step 5) The execution and maintenance module is as follows: Receive the detection results sent by the on-site controller, including the category, location, and distance information of the foreign object, drive the maintenance execution mechanism to implement maintenance operations, such as picking up foreign objects, cleaning foreign objects, reporting the information of foreign objects that cannot be processed on-site and alarming to prompt the maintenance master console to process in time. The distance information is obtained by the monocular ranging method, and the ranging formula is as follows:

[0053]

[0054] Where P bottom represents the pixel value of the bottom of the target object on the imaging plane, P center refers to the pixel coordinate of the optical center of the lens in the y direction in the pixel coordinate system, f represents the focal length of the camera, which can be obtained through camera calibration and calibration, H is the installation height of the camera, θ is the included angle between the target object and the base plane, Dis is the distance between the target object and the camera, and thus the distance between the target object and the camera can be obtained:

[0055] Dis = (H * f) / (P bottom ―Pcenter )

[0056] The remote access module described in step 6) is specifically as follows: The background server remotely accesses and controls the on-site control module through the network. Its web platform process is as Figure 11 shown, and then controls each on-site maintenance terminal. At the same time, it also receives the detection and maintenance result situations of the on-site maintenance terminals as Figure 12 shown, so as to realize the real-time and effective detection and maintenance of foreign objects on the expressway pavement;

[0057] The on-site self-organizing network module described in step 7) is specifically as follows: Each on-site maintenance terminal communicates with other maintenance terminals through the network and forms an on-site information interaction network as Figure 1 shown. The results of detection and maintenance of each on-site maintenance terminal are fused on each on-site maintenance terminal. According to the fusion results, the on-site maintenance terminal continues to perform detection or maintenance and feeds back to the background server, and then controls the camera pan-tilt, and iteratively optimizes the aforementioned trained model;

[0058] The overall maintenance console described in step 8) is specifically as follows: By connecting to the background server, it real-time displays the detection and maintenance results of each on-site maintenance terminal, as well as the operating conditions of each on-site maintenance terminal. According to the foreign object information that cannot be processed on-site reported above, the monitoring engineer of the overall maintenance console further makes a human judgment on the emergency level according to the system hazards generated by the system. According to the emergency level and combined with the mileage information, a maintenance team is dispatched to the site for manual maintenance to eliminate potential safety hazards and ensure the fast and safe passing ability of the expressway.

[0059] Embodiment

[0060] The hyperparameter settings for model training are as follows: The initial learning rate is set to 0.01, which is the step size for updating the weights during the optimization process; The learning momentum is set to 0.937, which controls the direction of gradient descent during the optimization process, maintains the momentum and continuously adjusts the gradient direction to reach the global optimal solution faster; The weight decay coefficient is set to 0.0005, which is used to reduce the overfitting of the model. By adjusting the parameters, it prevents excessive complexity; The batch-size is set to 4, that is, the number of samples input each time during training. The initial size of the epoch is set to 100, that is, the number of rounds of iterative training. These parameters need to be adjusted according to the training data and model structure.

[0061] Optimizers are a very important part of deep learning because choosing different optimizers can have a great impact on the training effect of the model. There are three optimizer options: SGD, Adam, and AdamW. The default is the SGD (Stochastic Gradient Descent) optimization method. The label smoothing is set to 0.0, which is used to prevent overfitting. The patience value, which is the threshold number of epochs for early stopping of training, is set to 100. If the performance does not improve during this period, training will stop early to save computing resources. This can prevent the model from getting stuck in local optima during training and can improve training efficiency. Among them, the relationship graph P_curve between accuracy and confidence represents the precision of each category recognition when setting the confidence value. When the confidence is greater, the category detection is more accurate. The relationship graph PR_curve between precision and recall, mAP is the mean of APs of all categories, and AP is determined by precision and recall. And the IoU threshold and confidence threshold affect the calculation of precision and recall. The higher the precision, the lower the recall. The relationship graph R_curve between recall and confidence represents the probability of complete retrieval of each category when setting the confidence value. When the confidence is smaller, the category detection is more comprehensive.

[0062] YOLOv5 includes five model structures with different depths and widths (YOLOv5l, YOLOv5m, YOLOv5n, YOLOv5s, and YOLOv5x). Based on the YOLOv5s model structure, two different attention mechanisms (C3KTongxue and CoordAtt) were attempted. As shown in Table 1, the experimental results show that the mAP of the YOLOv5s model on the dataset reached 95.0%, making it more suitable for practical applications and engineering deployments.

[0063] Table 1 Model Metric Parameters

[0064]

[0065]

[0066] After multiple experimental verifications, the training results of the detection model tend to stabilize after about 300 training iterations. Similarly, the training cycle is 500 epochs, and the results are as Figure 6 、 Figure 7 、 Figure 8 and Figure 9 shown. The relationship graph between its accuracy and confidence, mean average precision graph, and relationship graph between recall and confidence reached 98.1%, 95.0%, and 97.0% respectively, with a significant improvement in precision. The experimental results show that, as Figure 10 shown, this model has good detection and measurement effects on both the PC and the development board.

[0067] The above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any improvements, polishing, etc. made on the premise of the principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent detection and maintenance system for foreign objects on highway pavements, characterized in that, It includes modules such as data acquisition, image processing, model training, on-site control, execution of maintenance, remote access, on-site self-organizing network, and maintenance console. The specific steps are as follows: 1) Data acquisition module: Use a camera to capture visible light and infrared videos of highway foreign objects, collect intermittent video frame images, and construct a foreign object image dataset. 2) Image processing module: Diagnose and clean the abnormal image data, and label the optimized image dataset. 3) Model training module: Use the labeled image dataset for model training and verification. 4) On-site control module: Deploy the trained model to the on-site controller to detect highway foreign objects in real time all day long. 5) Execution of maintenance module: Receive the detection results sent by the on-site controller, and drive the maintenance execution mechanism to perform maintenance operations. Specifically: Receive the detection results sent by the on-site controller, including the category, location, and distance information of the foreign object, drive the maintenance execution mechanism to perform maintenance operations, pick up the foreign object, clean the foreign object, report the information of foreign objects that cannot be processed on site and alarm to prompt the maintenance console to handle it in time. The distance information is obtained by the monocular ranging method, and the ranging formula is as follows: where P bottom represents the pixel value of the bottom of the target object on the imaging plane, and P center refers to the pixel coordinate of the optical center of the lens in the y direction in the pixel coordinate system. f represents the focal length of the camera, which can be obtained through camera calibration and calibration. H is the installation height of the camera, θ is the angle between the target object and the base plane, and Dis is the distance between the target object and the camera. Thus, the distance between the target object and the camera can be obtained as follows: Dis = (H * f) / (P bottom - P center ); 6) Remote access module: The background server remotely accesses and controls the on-site control module through the network, and at the same time receives the detection and maintenance results of the on-site maintenance terminal. 7) On-site self-organizing network module: Each maintenance terminal communicates with each other to form an on-site information interaction network to achieve collaborative detection, and feedback the information fusion result of the detection and maintenance of the maintenance terminal to the background server. Specifically: Each on-site maintenance terminal communicates with other maintenance terminals through the network to form an on-site information interaction network, fuse the detection and maintenance results of each on-site maintenance terminal on each on-site maintenance terminal, and continue to detect or maintain according to the fusion result on the on-site maintenance terminal, and feedback to the background server, and then control the camera pan-tilt, and iteratively optimize the aforementioned trained model. 8) Maintenance console: By connecting to the background server, it displays the detection and maintenance results and operation status of each on-site maintenance terminal in real time, reports the information of foreign objects that cannot be processed on site, and gives different levels of alarm reminders according to the hazard level, and then the background will conduct further manual review and processing.

2. The intelligent detection and maintenance system for foreign objects on highway pavements according to claim 1, characterized in that, The data acquisition module described in step 1) is specifically as follows: Use a highway guardrail robot to capture a foreign object video dataset, convert the video dataset into video frame images, collect intermittent video frame images, construct a foreign object detection dataset, and transmit it to the pan-tilt for preprocessing. The dataset categories include water bottles, cartons, lunch boxes, potholes, mileage signs, transverse and longitudinal crack categories.

3. The intelligent detection and maintenance system for foreign objects on highway pavements according to claim 1, wherein The image processing module described in step 2) is specifically: Use a data diagnosis method to check whether there are abnormal attribute pictures in the dataset. According to the data diagnosis result, clean the abnormal pictures, and use a labeling tool to label the optimized foreign object dataset to obtain label file information.

4. The intelligent detection and maintenance system for foreign objects on highway pavements according to claim 1, wherein The model training module described in step 3) is specifically as follows: The labeled image data set and label information are sent into the network to extract image features, and training parameters are set for model training. If the model is basically convergent and in a stable state, model verification can be carried out; otherwise, the model training can continue by modifying the hyperparameter optimization method.

5. The intelligent detection and maintenance system for foreign objects on highway pavements according to claim 1, wherein, The on-site control module described in step 4) is specifically as follows: The trained weight file is deployed to the on-site controller, and the video data stream of the camera is read to detect the running effect in the actual highway application scenario in real time, so that the system can also be used in other occasions to achieve the adaptive function.

6. The intelligent detection and maintenance system for foreign objects on highway pavements according to claim 1, characterized in that, The remote access module described in step 6) is specifically as follows: The background server remotely accesses and controls the on-site control module through the network, and then controls each on-site maintenance terminal. At the same time, it also receives the detection and maintenance result information of the on-site maintenance terminal, so as to realize the real-time and effective detection and maintenance of foreign objects on the highway pavement.

7. The intelligent detection and maintenance system for foreign objects on highway pavements according to claim 1, wherein The overall maintenance console described in step 8) is specifically as follows: By connecting to the background server, it displays in real time the detection and maintenance results of each on-site maintenance terminal, as well as the running status of each on-site maintenance terminal. According to the foreign object information reported above that cannot be processed on site, the monitoring engineer of the overall maintenance console further makes a human judgment on the urgency according to the hazard level generated by the system. According to the urgency and combined with the pile number information, a maintenance team is dispatched to the site for manual maintenance to eliminate potential safety hazards and ensure the fast and safe traffic capacity of the highway.

Images