A road condition monitoring system based on YOLOV5

Abstract

The present invention provides a road condition monitoring system based on YOLOV5, including an edge terminal, a message middleware module, and a central platform; the system combines video analysis and edge computing technology, and can monitor multiple road targets at one time, including trucks, large buses, medium-sized buses, SUVs, cars, vans, etc., with the characteristics of high recognition accuracy, strong real-time performance, wide monitoring range, and low cost.

Description

Technical Field

[0001] The present invention relates to the fields of smart transportation and road condition monitoring, and in particular to a road condition monitoring system based on YOLOV5. Background Art

[0002] With the advancement of society and the improvement of people's living standards, the number of motor vehicles has increased rapidly, and transportation demand has also increased dramatically. Traffic congestion is a common occurrence, especially in cities. Traffic congestion has become a pressing issue for major cities. Intelligent transportation systems have emerged to address this problem and are currently experiencing rapid development. Comprehensive, real-time, and accurate collection of traffic information is a fundamental requirement for intelligent transportation. Currently, there are three main technologies for collecting traffic information: permanently embedded systems based on piezoelectric circuits; suspended systems such as those using closed-circuit television, microwave, radar, infrared, or ultrasonic sensors; and traffic flow detection systems based on image processing technology. These systems use cameras driven along the road to capture real-time traffic conditions. The cameras then transmit the video data via a network to a video analysis server. The video analysis server uses image processing technology to identify and count the vehicles in the video, ultimately deriving information such as vehicle type and number.

[0003] Of the three traditional traffic information collection technologies mentioned above, the permanently embedded piezoelectric circuit-based system requires sensors to be permanently buried underground. Construction and maintenance require excavation of the road surface, which is time-consuming and labor-intensive, disrupts traffic, and significantly affects detection accuracy. Furthermore, detection accuracy is significantly affected by road deformation. While the suspension-based system, unlike the piezoelectric circuit-based technology, requires no road surface damage during installation and maintenance and is unaffected by road surface deformation, its detection is significantly affected by environmental factors. For example, its detection accuracy is significantly reduced in windy weather, becoming virtually unusable in winds exceeding level 6. Furthermore, false detections can easily occur when non-vehicle objects appear under the sensor. Finally, the third traffic flow detection system, based on image processing technology, offers significant advantages over other methods due to its simple installation and maintenance, non-destructive nature, wide detection coverage, and diverse parameter coverage. However, this method is highly dependent on the performance of video analysis servers. Large-scale real-time monitoring requires a large number of expensive, high-performance GPU servers, severely hindering its widespread application and promotion. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a road condition monitoring system based on YOLOV5, including an edge terminal, a message middleware module, and a central platform;

[0005] The edge terminal obtains the image of the current intersection through the camera and detects the image, and sends the detection result to the message middleware module;

[0006] The message middleware module obtains the messages sent by the edge terminal, forms a message queue, and transmits them to the central platform according to the first-in-first-out principle;

[0007] The central platform classifies the acquired data according to region, road, target quantity and type and then stores them.

[0008] The edge terminal includes an image acquisition module and a YOLOV5 target detection module. The image acquisition module is used to capture pictures through a camera; the YOLOV5 target detection module is used to train a target detection model and perform target detection on the pictures captured by the camera through the trained target detection model.

[0009] The YOLOV5 target detection module first obtains a certain number of images to be detected, and labels the targets in the images. The labeled images are input into the YOLOV5 network training to obtain a trained target detection model, and then the following steps are performed:

[0010] Step a1: input the camera captured image into the target detection model;

[0011] In step a2, the YOLOV5 target detection module generates two or more object boxes and the confidence scores of the object boxes for each target category on the image based on the trained target detection model;

[0012] In step a3, the prediction part of the original YOLOV5 algorithm is applied, using the GIoU loss function to obtain the coordinates, size, category, and confidence of two or more object boxes;

[0013] Let A be the predicted box and B be the true box. Regardless of whether A and B intersect, C is the smallest box containing A and B (the smallest convex closed box containing A and B).

[0014] Step a4: Sort two or more object boxes by confidence level. The first one is recorded as the first physical box BOX1, the second one is recorded as the second physical box BOX2, and so on until the nth one is recorded as the nth physical box BOXn. At the same time, remove the object boxes with a confidence level lower than 0.3.

[0015] Step a5: discard the GIoU of the original YOLOV5 algorithm and use the optimized version of IoU, DIoU, to filter the results obtained in step a4;

[0016] Step a6: Count the object frames by category based on the results obtained in step a5, count the number of each category in the picture, and generate a picture containing the object detection frames;

[0017] Step a7: encapsulate the final data into JSON data to complete target detection.

[0018] In step a3, the loss function GIoU is calculated as follows:

[0019] First calculate the intersection over union (IoU) of A and B:

[0020]

[0021] Then calculate the loss function GIoU:

[0022]

[0023] In step a3, the calculation formula of DIoU is:

[0024]

[0025] Where ρ(A, B) is the Euclidean distance between the center coordinates of object frame A and object frame B, and c is the diagonal distance of the smallest rectangle that can enclose object frame A and object frame B.

[0026] Step a5 includes:

[0027] Step a5-1: Create a new result set in the storage, store BOX1 in the result set, and name it BBOX1.

[0028] Step a5-2: Calculate the DIoU between BOX1 and BOX2. If DIoU < 0.5, store BOX2 in the result set and name it BBOX2. If DIoU ≥ 0.5, delete BOX2. Continue until BBOX1 is compared with all the results obtained in step a4.

[0029] In step a5-3, the comparison work of step a5-2 is performed on BOX22 to BOXn until all the results in step a4 are compared to form a final result set.

[0030] In step a7, the final data includes camera ID, object frame coordinates, object frame category, object frame confidence, and the number of object frames in each category.

[0031] The central platform includes an edge terminal management module, a data processing module and a data storage module;

[0032] The edge terminal management module is used to manage edge terminals and cameras associated with the edge terminals;

[0033] The data processing module monitors the message queue in real time, and whenever there is a new message in the queue, it is immediately taken out for processing.

[0034] After receiving the data, the data processing module merges the data of the same road according to the association between the camera ID and the road to form road data, and then stores the road data into the data storage module. The road includes the road name, number of vehicles, congestion level and image address.

[0035] The system of the present invention also includes an API gateway, which is used to provide a data interface for other applications.

[0036] Beneficial effects: The present invention provides a road condition monitoring system based on the YOLOV5 target detection framework. The system combines video analysis and edge computing technology, and can monitor multiple road targets at one time, including trucks, large buses, medium-sized buses, SUVs, cars, vans, etc. It has the characteristics of high recognition accuracy, strong real-time performance, wide monitoring range, and low cost. The system's image data processing function is deployed on the edge terminal, effectively avoiding the problems of centralized processing and calculation and high bandwidth pressure. At the same time, the data collection will be non-blocking, greatly improving the efficiency of collection and alleviating the pressure on the central system. Moreover, the described method fully utilizes the computing performance of the edge terminal, reduces the hardware cost of target detection, and adopts the edge node deployment method to effectively guarantee performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0038] Figure 1 It is a system architecture diagram of the present invention.

[0039] Figure 2 It is a schematic diagram of the picture to be detected.

[0040] Figure 3 It is a schematic diagram of the test results.

[0041] Figure 4 It is the YOLOV5 target detection flowchart.

[0042] Figure 5 This is the edge terminal system architecture diagram.

[0043] Figure 6 This is the edge terminal workflow diagram.

[0044] Figure 7 It is a schematic diagram of the predicted box and the true box.

[0045] Figure 8 This is a diagram of the diagonal distance of the smallest rectangle that can enclose frames A and B. DETAILED DESCRIPTION

[0046] A YOLO V5-based road condition monitoring system, characterized by including edge terminals, a message middleware module, and a central platform, uses a DIoU screening method (DIoU is an optimized version of IoU) instead of the IoU value screening method in the original YOLO V5 algorithm. This makes the target detection results more consistent with the target box regression mechanism, taking into account the distance, overlap rate, and scale between targets. This significantly improves the detection of vehicles with occlusion and overlap, making the object detection box regression more stable.

[0047] The edge terminal obtains the image of the current intersection through the camera and detects the image, and sends the detection result to the message middleware module;

[0048] The message middleware module obtains the messages sent by the edge terminal, forms a message queue, and transmits them to the central platform according to the first-in-first-out principle;

[0049] The central platform classifies the acquired data according to region, road, target quantity and type and then stores them.

[0050] The edge terminal includes an image acquisition module and a YOLOV5 target detection module. The image acquisition module is used to capture pictures through a camera; the YOLOV5 target detection module is used to train a target detection model and perform target detection on the pictures captured by the camera through the trained target detection model.

[0051] The YOLOV5 target detection module first obtains a certain number of images to be detected (for example, 100 images), labels the targets in the images, inputs the labeled images into the YOLOV5 network training, obtains the trained target detection model, and then performs the following steps:

[0052] Step a1: input the camera captured image into the target detection model;

[0053] In step a2, the YOLOV5 target detection module generates two or more object boxes and the confidence scores of the object boxes for each target category on the image based on the trained target detection model;

[0054] In step a3, the prediction part of the original YOLOV5 algorithm is applied, using the GIoU loss function to obtain the coordinates, size, category, and confidence of two or more object boxes;

[0055] Let A be the predicted box and B be the real box. Regardless of whether A and B intersect, C is the smallest box containing A and B (the smallest convex closed box containing A and B), such as Figure 7 As shown;

[0056] Step a4: sort the two or more object boxes according to their confidence level. The first one is named BOX1, the second one is named BOX2, and so on until the nth one is named BOXn. At the same time, remove the object boxes whose confidence level is lower than the threshold of 0.3.

[0057] In step a5, the GIoU of the original YOLOV5 algorithm is discarded, and DIoU is used. The DIoU non-maximum suppression method is used to filter the results obtained in step a4 to form the final detection result.

[0058] Step a6: Count the object frames by category based on the results obtained in step a5, count the number of each category in the picture, and generate a picture containing the object detection frames;

[0059] Step a7: encapsulate the final data into JSON data to complete target detection.

[0060] In step a3, the loss function GIoU is calculated as follows:

[0061] First calculate the intersection over union (IoU) of A and B:

[0062]

[0063] Then calculate the loss function GIoU:

[0064]

[0065] In step a3, the calculation formula of DIoU is:

[0066]

[0067] Where ρ(A, B) is the Euclidean distance between the center coordinates of the predicted box A and the real box B, and c is the diagonal distance of the smallest rectangle that can enclose the predicted box A and the real box B; this has been modified above as follows Figure 8 shown.

[0068] In step a7, the final data includes the camera ID, the coordinates of the upper left corner and the lower right corner of the object frame, the object frame category, the object frame confidence, and the number of object frames of each category.

[0069] The central platform includes an edge terminal management module, a data processing module and a data storage module.

[0070] The edge terminal management module is used to manage edge terminals and cameras associated with the edge terminals;

[0071] The data processing module monitors the message queue in real time, and whenever there is a new message in the queue, it is immediately taken out for processing.

[0072] After receiving the data, the data processing module merges the data of the same road according to the association between the camera ID and the road to form road data, and then stores the road data into the data storage module. The road includes the road name, number of vehicles, congestion level and image address.

[0073] The system of the present invention also includes an API gateway, which is used to provide a data interface for other applications (such as traffic command and dispatch systems, security management systems, etc.).

[0074] Example

[0075] like Figure 1 As shown, the entire system of the present invention is divided into four parts: edge terminal, message middleware module, central platform and API gateway.

[0076] The edge terminal uses a camera to capture images of the current intersection and performs object detection using YOLO V5. It detects a variety of objects within the image, including heavy trucks, large buses, medium buses, SUVs, sedans, vans, and pedestrians. The detection results are then transmitted to the data processing module of the central platform via the message middleware module. A single edge terminal can monitor multiple cameras simultaneously.

[0077] The message middleware module obtains the messages sent by the edge terminal (including terminal number, camera number, target type, target quantity and current screenshot), forms a message queue, and transmits it to the data processing module according to the first-in-first-out principle for data processing.

[0078] The data processing module is mainly used to process the data obtained from the message queue, classify the obtained data according to the area, road, target number and type, and send it to the data storage module for classified storage. At the same time, it makes judgments based on the preset thresholds and determines whether to initiate an early warning based on the thresholds.

[0079] The judgment logic is as follows:

[0080] The number of vehicles at the intersection is: n;

[0081] Calculate the number of pixels in the image by the image resolution: Pixel_Num. For example, if the image resolution is (1920*1080) Pixel_Num = 1920*1080. This data can be obtained from the image.

[0082] Get the coordinates of the upper left corner of the object frame (LEFT_X, LEFT_Y) and the coordinates of the lower right corner of the object frame (RIGHT_X, RIGHT_Y);

[0083] First calculate the area AREA of each object box in the result set n :

[0084] AREA1=(RIGHT_Y1-LEFT_Y1)*(RIGHT_X1-LEFT_X1)

[0085] AREA2=(RIGHT_Y2-LEFT_Y2)*(RIGHT_X2-LEFT_X2)

[0086] Until AREA n ;

[0087] Calculate the sum of the areas of all object boxes AREA al :

[0088] AREA all＝ AREA 1+ AREA 2+…+ AREA n ;

[0089] Calculate the ratio of the sum of the object box area and the pixel point Area_Ratio:

[0090] Area_Ratio=AREA all / Pixel_Num;

[0091] The judgment basis is shown in Table 1 below:

[0092] Table 1

[0093]

[0094] The API gateway provides a data interface for other applications, making it easier for other applications to obtain traffic data.

[0095] YOLOV5 target detection:

[0096] YOLO is an object recognition and localization algorithm based on deep neural networks. Its most notable feature is its high speed, making it suitable for real-time systems. Currently, YOLO has reached version V5. In this system, image detection is divided into two parts: training the object detection model and developing the YOLO V5 object detection module.

[0097] (1) Training the target detection model

[0098] 9,000 images containing the targets to be detected were collected manually and the targets in the images were marked, such as Figure 2 As shown;

[0099] The labeled images are fed into the YOLO network for training. After 300 training cycles, the target detection model is obtained. The model results are as follows: Figure 3 and as shown in Table 2.

[0100] Table 2

[0101]

[0102] Indicator Notes:

[0103] Precision: hit rate, the correct proportion of all predicted targets;

[0104] Recall: Recall rate, the ratio of correctly located and identified targets to the total number of targets.

[0105] IoU: The full name is Intersection over Union. IoU calculates the ratio of the intersection and union of two object frames.

[0106] PR curve: The P in the PR curve stands for Precision, and R stands for Recall. It represents the relationship between precision and recall. Generally, recall is set as the horizontal axis and precision is set as the vertical axis.

[0107] AP: Average precision, which uses integration to calculate the area enclosed by the PR curve and the coordinate axis.

[0108] mAP@.5: When IoU is set to 0.5, calculate the AP of all images in each class, and then calculate the average precision of all classes.

[0109] mAP@.5:.95: represents the average mAP at different IoU thresholds (from 0.5 to 0.95, step size 0.05) (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

[0110] Judging by the above metrics, the model training results achieved a hit rate exceeding 99% for all detection categories, a recall rate of 98%, and both mAP@.5 and mAP@.5:.95 exceeding 97%, meeting practical application requirements.

[0111] (2) Develop the YOLOV5 target detection module through the trained target detection model to realize target detection in images.

[0112] YOLOV5 target detection process is as follows Figure 4 As shown:

[0113] The image detection process is as follows:

[0114] Step a1: input the acquired image into the YOLOV5 image detection module;

[0115] In step a2, the YOLOV5 object detection model generates multiple object boxes and the confidence scores of the object boxes for each target category on the image based on the trained model, and filters these object boxes using the non-maximum suppression (NMS) algorithm.

[0116] Step a3: obtain the coordinates, size, category, and confidence of the final object box;

[0117] Step a4: Count the object frames by category based on the obtained results, count the number of each category (truck, large bus, medium bus, SUV, sedan, van) in the picture, and generate a picture containing the object detection frame;

[0118] In step a5, the above data (camera ID, object frame coordinates, object frame category, object frame confidence, and the number of object frames in each category) are encapsulated into json data to complete the detection work.

[0119] Edge terminal architecture and workflow:

[0120] The edge terminal is responsible for image data collection and graphics processing. The specific architecture is as follows: Figure 5 As shown, the edge terminal mainly consists of two modules: image acquisition module and YOLOV5 target detection module. The workflow is as follows Figure 6 Shown, including:

[0121] Step b1, obtain the camera list in the device list allocated by the central platform, including the camera's IP, port, account, password, etc.;

[0122] Step b2: Log in to the camera in a patrol mode and obtain the current image of the camera;

[0123] Step b3: pass the acquired image into the YOLOV5 target detection module, which detects the type and number of targets in the image;

[0124] Step b4, the YOLOV5 target detection module encapsulates the detected data into JSON data;

[0125] Step b5: transfer the data to the message middleware module and send it to the central platform for processing.

[0126] Central Platform:

[0127] The central platform includes three parts: edge terminal management module, data processing module and data storage module.

[0128] The edge terminal management module is primarily used to manage edge terminals and their associated cameras, including the edge terminal's ID, IP address, port number, associated cameras, each camera's latitude and longitude, edge terminal online status, and operational status (CPU, memory, and storage usage, etc.). It primarily includes two tables: the edge terminal information table and the camera information table, shown in Table 3 and Table 4, respectively.

[0129] First, manage edge terminals. You can add, modify, or delete edge terminal information, including the edge terminal's name (device_name), IP address, and associated camera ID (camera_id). The edge terminal can obtain a list of cameras to capture based on the corresponding camera ID list, thereby capturing images and performing target detection for camera patrols. Second, you can use camera information reports to maintain camera-related information, including the camera's name (camera_name), IP address, port (port), account (account), password, longitude (x), and latitude (y).

[0130] Table 3

[0131] id device_name ip camera_id 1 South Terminal 1 10.128.1.1 1,2 2 South Terminal 2 10.128.2.1 3,15

[0132] Table 4

[0133]

[0134] The data processing module is mainly used for processing detection data. In order to ensure that the data detected by the edge terminal can be processed as soon as possible, the central platform needs to notify the data processing module immediately whenever new data is received. Since the central platform and the edge terminal work independently and the data processing time is not synchronized, a message queue mechanism is adopted. That is, after each batch of data is detected by the edge terminal, the data description information such as the id, target type and target quantity of the detection data is delivered to the designated message queue of the message middleware module. The data processing module listens to the message queue in real time, and whenever there is a new message in the queue, it is immediately taken out for processing.

[0135] After receiving the data, the data processing module merges the data of the same road according to the association between the camera ID and the road to form road data, as shown in Table 5, and then stores the data into the data storage module, including the road name, number of vehicles and image address.

[0136] Table 5

[0137]

[0138]

[0139] The system was applied to the internal and surrounding roads of a 1.6 million square meter convention and exhibition park. The venue has 11 internal roads, 15 above-ground entrances, nine underground parking entrances and exits, five above-ground parking lots, and four external roads. Leveraging the existing video surveillance system, a total of 59 cameras were selected for road monitoring. These cameras provide road monitoring without compromising security. During construction, there was no need to modify existing front-end monitoring facilities or install new equipment. Instead, only edge terminals were added to the front-end weak-current room of the computer room and a central platform was deployed in the computer room.

[0140] Because traffic in the exhibition park is tidal, personnel are needed to monitor road conditions and direct traffic during the exhibition period. Before applying this method, every road and entrance and exit required personnel to monitor and direct traffic, requiring approximately 50-70 people. After implementing this method, only one platform monitoring personnel and about 20 mobile patrol personnel are required. The monitoring personnel can then dispatch mobile patrol personnel to promptly direct traffic after traffic warnings and alerts are issued.

[0141] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, it can run the invention content of a road condition monitoring system based on YOLOV5 provided by the present invention and some or all steps of each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0143] The present invention provides a road condition monitoring system based on YOLO V5. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A road condition monitoring system based on YOLOV5, characterized in that: Including edge terminals, message middleware modules, and central platforms; The edge terminal obtains the image of the current intersection through the camera and detects the image, and sends the detection result to the message middleware module; The message middleware module obtains the messages sent by the edge terminal, forms a message queue, and transmits them to the central platform according to the first-in-first-out principle; The central platform classifies and stores the acquired data according to regions, roads, number and type of targets; The edge terminal includes an image acquisition module and a YOLOV5 target detection module. The image acquisition module is used to capture pictures through a camera; the YOLOV5 target detection module is used to train a target detection model and perform target detection on the pictures captured by the camera through the trained target detection model; The YOLOV5 target detection module first obtains a certain number of images to be detected, and labels the targets in the images. The labeled images are input into the YOLOV5 network training to obtain a trained target detection model, and then the following steps are performed: Step a1: input the camera captured image into the target detection model; In step a2, the YOLOV5 target detection module generates two or more object boxes and the confidence scores of the object boxes for each target category on the image based on the trained target detection model; In step a3, the prediction part of the original YOLOV5 algorithm is applied, using the GIoU loss function to obtain the coordinates, size, category, and confidence of two or more object boxes; Let A be the predicted box and B be the true box. Regardless of whether A and B intersect, C is the smallest box containing A and B. Step a4: Sort two or more object boxes by confidence level. The first one is recorded as the first physical box BOX1, the second one is recorded as the second physical box BOX2, and so on until the nth one is recorded as the nth physical box BOXn. At the same time, remove the object boxes with a confidence level lower than 0.

3. Step a5: discard the GIoU of the original YOLOV5 algorithm and use the optimized version of IoU, DIoU, to filter the results obtained in step a4; Step a6: Count the object frames by category based on the results obtained in step a5, count the number of each category in the picture, and generate a picture containing the object detection frames; Step a7: encapsulate the final data into JSON data to complete target detection; In step a3, the loss function GIoU is calculated as follows: First calculate the intersection over union (IoU) of A and B: ， Then calculate the loss function GIoU: ； In step a3, the calculation formula of DIoU is: ， Where ρ(A, B) is the Euclidean distance between the center coordinates of object frame A and object frame B, and c is the diagonal distance of the smallest rectangle that can enclose object frame A and object frame B; Step a5 includes: Step a5-1: Create a new result set in the storage, store BOX1 in the result set, and name it BBOX1. Step a5-2: Calculate the DIoU between BOX1 and BOX2. If DIoU < 0.5, store BOX2 in the result set and name it BBOX2. If DIoU ≥ 0.5, delete BOX2. Continue until BBOX1 is compared with all the results obtained in step a4. Step a5-3: Perform the comparison of step a5-2 on BOX22 to BOXn until all the results in step a4 are compared to form a final result set. In step a7, the final data includes camera ID, object frame coordinates, object frame category, object frame confidence, and the number of object frames in each category; The central platform includes an edge terminal management module, a data processing module and a data storage module; The edge terminal management module is used to manage edge terminals and cameras associated with the edge terminals; The data processing module monitors the message queue in real time and takes out and processes new messages whenever they are added to the queue; After receiving the data, the data processing module merges the data of the same road according to the association between the camera ID and the road to form road data, and then stores the road data into the data storage module. The road data includes the road name, number of vehicles, congestion level and image address; It also includes an API gateway, which is used to provide data interfaces for other applications.

Images