A warning method, system, device and storage medium for pedestrians illegally climbing over guardrails based on a target tracking algorithm

Abstract

The present invention relates to an early warning method, system, device and storage medium for pedestrians illegally climbing over guardrails based on a target tracking algorithm, including: (1) calibrating a number of threshold lines of unequal distances parallel to the guardrail line; (2) deploying a detection rate control module and a target detection module; (3) controlling the detection rate of the target detection module through the detection rate control module; (4) recording and predicting the trajectory of pedestrians according to the pedestrian target detection result using the target tracking module; (5) using the early warning module to determine whether the guardrail has been climbed over. The present invention warns pedestrians illegally climbing over guardrails with high accuracy, and uploads the process of climbing over guardrails to the road management department. Using this information, the road management department can better control illegal behaviors; the present invention proposes a detection rate control module to reasonably control the detection rate of industrial computers, reduce energy consumption, extend the service life of equipment, and respond to the call for energy conservation and emission reduction.

Description

Technical Field

[0001] The present invention belongs to the technical field of road engineering, and particularly relates to a method, a system, a device and a storage medium for warning pedestrians' behavior of climbing over guardrails based on a target tracking algorithm. Background Art

[0002] With the rapid development of China's economy and the improvement of people's living standards, the number of private cars has increased exponentially. Behind the rapid growth of the number of cars, road traffic safety problems have become increasingly prominent, and the number of casualties in traffic accidents every year is appalling. One of the reasons for the frequent occurrence of traffic accidents is that pedestrians do not use crosswalks and randomly climb over the median guardrails of urban roads or the roadside guardrails of highways. This kind of behavior that ignores road regulations and morality seriously affects the efficient operation of the traffic system and people's lives and safety, causing serious economic losses. Therefore, how to accurately identify the illegal behavior of climbing over road guardrails is of great significance.

[0003] At present, for the inspection of guardrails on highways and urban roads at home and abroad, manual inspection or remote monitoring based on cameras is mostly used. Among them, the most common is manual inspection. By arranging staff to patrol specific sections and record the behavior of the guardrails in the illegal range, but the length of the guardrails that an individual can patrol at the same time is limited, resulting in low patrol efficiency and huge labor costs. In addition, by installing a violation detection camera at a certain distance on the road, integrating the camera images on a remote terminal, and using manual inspection of multiple camera images at the same time to monitor the behavior of climbing over guardrails, this method improves the monitoring efficiency to a certain extent, but its essence is still manual monitoring. Under the influence of the trend of artificial intelligence, the application of artificial intelligence in road violation detection has gradually emerged.

[0004] In China, work on vehicle violation detection based on deep learning has been carried out in many places, integrating related technologies such as image processing technology, image segmentation technology, target detection and tracking technology, digital signal acquisition and processing, etc., to conduct on-site detection of vehicle violations with a relatively high accident induction rate, such as illegal parking of vehicles, illegally occupying the emergency lane under non-emergency circumstances, randomly changing lanes, and drivers holding mobile phones while driving. However, at present, target detection technology rarely involves pedestrians climbing over guardrails, and the detection accuracy of the detection technologies that have been applied is relatively low. In addition, whether there are people near the guardrails or not, the detection algorithm always runs, resulting in waste of resources. Summary of the Invention

[0005] In view of the problems of low detection efficiency and low detection accuracy of pedestrians' illegal climbing over road guardrails at present, the present invention proposes a warning method for pedestrians' illegal climbing over guardrails based on a target tracking algorithm.

[0006] The present invention also provides a warning and uploading system for pedestrians' illegal climbing over guardrails based on a target tracking algorithm.

[0007] The present invention further provides a computer device and a storage medium.

[0008] Term Explanation:

[0009] 1. Deep learning: It learns the internal laws and representation levels of sample data. The information obtained during the learning process is very helpful for the interpretation of data such as text, images, and sounds. Its ultimate goal is to enable machines to have the ability of analysis and learning like humans, and be able to recognize data such as text, images, and sounds;

[0010] 2. Object detection: Its task is to find all interesting objects (objects) in an image and determine their positions and sizes, which is one of the core problems in the field of machine vision;

[0011] 3. YOLO v3: It is a network for object detection that can detect objects in images relatively quickly;

[0012] 4. OpenCV: It is an open-source computer vision library developed by Intel. It consists of a series of C functions and a small number of C++ classes, and implements many general algorithms in image processing and computer vision;

[0013] 5. Edge detection: A very important method for image feature extraction in the field of computer vision. Through edge detection, the combination of pixel points with drastic changes in pixel brightness in an image can be found. Usually, these combinations often appear as contours. Through edge detection, the contours of objects can be detected;

[0014] 6. Guardrail line: Using the edge detection algorithm, the contour of the guardrail is extracted, simplified into a curve, and drawn in each image of the video stream using OpenCV to represent the actual position of the guardrail;

[0015] 7. Detection rate control module: When there is no pedestrian target within the range of the first threshold line, this module controls the industrial computer to detect only 1 frame of image per second; when a pedestrian appears between the first threshold line and the second threshold line, this module controls the industrial computer to detect 15 frames of image per second; when a pedestrian appears between the second threshold line and the third threshold line, this module controls the industrial computer to detect 30 frames of image per second; when a pedestrian appears between the two third threshold lines, this module controls the industrial computer to detect 30 frames of image per second and take pictures at a rate of 10 frames per second to save evidence of climbing over;

[0016] 8. Canny edge detection is a popular edge detection algorithm. Its innovation lies in using double thresholds for detection and then connecting all the edges together to form an edge curve or line segment, which can very effectively detect the object contour.

[0017] 9. The target detection module mainly uses the trained YOLO v3 neural network to detect pedestrians. The detection category is only pedestrians and does not include vehicles, and the detection efficiency is high.

[0018] 10. The target tracking module mainly uses the detection box obtained by target detection, takes the geometric center of the detection box as the position of the target at the current moment, and connects the geometric centers at each moment to complete the tracking of the target.

[0019] 11. The warning module uploads warning information to the road control center when a pedestrian has the tendency to climb over the guardrail; when a pedestrian confirms climbing over the guardrail, it controls the camera to continuously shoot the target to leave evidence of the pedestrian climbing over the guardrail.

[0020] 12. The Pytorch deep learning environment, an open-source Python deep learning library, is used for applications such as target detection. Through the Pytorch deep learning environment, a convolutional neural network for target detection can be easily built.

[0021] 13. A pixel is the smallest unit in an image represented by a sequence of numbers.

[0022] The technical solution of the present invention is as follows:

[0023] A warning method for pedestrians' illegal climbing over guardrails based on a target tracking algorithm, including the following steps:

[0024] (1) Calibrate several non-equidistant threshold lines parallel to the guardrail line.

[0025] (2) Deploy a detection rate control module and a target detection module.

[0026] (3) Implement the control of the detection rate of the target detection module through the detection rate control module.

[0027] (4) According to the pedestrian target detection result, use the target tracking module to record and predict the pedestrian trajectory.

[0028] (5) Based on the pedestrian trajectory and the range of the threshold line where it is located, use the warning module to judge whether to climb over the guardrail.

[0029] Preferably according to the present invention, in step (1), a monitoring camera and an industrial computer are fixedly installed near the guardrail. Specifically, it means: installing the monitoring camera 5 m above the guardrail directly above, and installing the industrial computer beside the lamp post of the street lamp.

[0030] Preferably according to the present invention, in step (1), three threshold lines with unequal distances parallel to the guardrail line are calibrated using OpenCV, including a first threshold line, a second threshold line, and a third threshold line; the steps are as follows:

[0031] 1.1: Keep the industrial computer in an open state, with a good connection to the monitoring camera, and fix the video stream image output size at 800×600;

[0032] 1.2: Use the Canny edge detection algorithm to detect the guardrail contour line and the 6 placed anti-collision cones. Utilize the guardrail contour features to eliminate the remaining contour lines, and use OpenCV to redraw a curve representing the position of the guardrail in the center of the guardrail contour as the guardrail line;

[0033] 1.3: Place an anti-collision cone 1m to the left and right of the guardrail respectively. According to the positions of each anti-collision cone in the video, use OpenCV to mark their coordinate points, and then respectively draw parallel lines of the guardrail line passing through these two coordinate points as the first threshold line;

[0034] 1.4: Place an anti-collision cone 0.5m to the left and right of the guardrail respectively. According to the positions of each anti-collision cone in the video, use OpenCV to mark their coordinate points, and then respectively draw parallel lines of the guardrail line passing through these two coordinate points as the second threshold line;

[0035] 1.5: Place an anti-collision cone 0.2m to the left and right of the guardrail respectively. According to the positions of each anti-collision cone in the video, use OpenCV to mark their coordinate points, and then respectively draw parallel lines of the guardrail line passing through these two coordinate points as the third threshold line.

[0036] Preferably according to the present invention, in step (2), the specific working process of the detection rate control module is as follows:

[0037] Use OpenCV to intercept video data at a rate of 30 images per second;

[0038] Based on the feedback of the target detection result, different results are output, specifically: if there are no pedestrians within the range between the two first threshold lines, 1 frame of picture is output per second; if there are pedestrians within the range between the two first threshold lines, 15 images are output per second; if there are pedestrians within the range between the second threshold line and the third threshold line, 30 images are output per second.

[0039] Preferably according to the present invention, in step (2), the deployment of the target detection module is as follows:

[0040] 2.1: Create a pedestrian dataset, which includes pedestrians in various postures and pedestrians in different scenarios. Divide the pedestrian dataset into a training set and a test set according to the ratio of 8:2.

[0041] 2.2: Use the training set to train the open-source YOLO v3 object detection algorithm, and use the test set to verify the detection effect of the YOLO v3 object detection algorithm to obtain a trained YOLO v3 object detection network for pedestrian classification.

[0042] 2.3: Deploy the Pytorch deep learning environment in the industrial control computer, and run the trained YOLO v3 object detection network for pedestrian classification on the industrial control computer to process more than 30 frames of image data per second.

[0043] 2.4: After the object detection module is deployed, connect the object detection module and the detection rate control module.

[0044] 2.5: After the deployment is completed, field-test the detection results of the object detection module. When a pedestrian is detected, use a bounding box to select the target.

[0045] Further preferably, the specific implementation process of step (3) is as follows:

[0046] 3.1: Input the video data captured by the surveillance camera into the detection rate control module. The detection rate control module intercepts the video stream into several pictures and initially inputs them into the object detection module at a rate of one picture per second.

[0047] 3.2: If the object detection module does not detect a pedestrian within the range between two first threshold lines, maintain the detection rate of one frame per second.

[0048] If the object detection module detects a pedestrian within the range between the first threshold line and the second threshold line, feed the detection result back to the detection rate control module. The detection rate control module inputs the object detection module at a rate of 15 pictures per second.

[0049] If the object detection module detects a pedestrian within the range between the second threshold line and the third threshold line, feed the detection result back to the detection rate control module. The detection rate control module inputs the object detection module at a rate of 30 pictures per second.

[0050] If the object detection module detects a pedestrian within the range between the third threshold line and the guardrail line, feed the detection result back to the detection rate control module. The detection rate control module inputs the object detection module at a rate of 30 pictures per second and simultaneously controls the surveillance camera to take pictures at a rate of 10 pictures per second.

[0051] Preferably according to the present invention, in step (4), the specific implementation process of using the target tracking module to record and predict the pedestrian trajectory is as follows:

[0052] 4.1: Every time the target detection module detects an image, it outputs the detection result to the target tracking module;

[0053] 4.2: The target tracking module collects each detected target box. The target box is rectangular. Use OpenCV to take the geometric center point of the target box as the geometric position where the target is located at the current moment. The geometric positions at multiple moments are connected to obtain the trajectory line of the target in the past period of time. Use OpenCV to draw the trajectory line in each frame of the image, that is, target tracking;

[0054] 4.3: Target trajectory prediction is achieved through target tracking.

[0055] For the i-th image, the geometric center point of the detection box in the current image is P i ; For the (i + 1)-th image, the geometric center point of the detection box in the current image is P i+1 ; Connect P i and P i+1 and extend it to intersect with the guardrail line, and measure the included angle as α i ; In this way, n included angles are obtained, i = 1...n, and calculate the average value α average ;

[0056] While detecting the included angle, record the coordinate points of P i and P i+1 in the image. The coordinate system where the coordinate points are located takes the upper left vertex of the image as the origin, with the right representing the positive direction of the x-axis and the downward representing the positive direction of the y-axis; According to the time difference between the two input images and the distance between the coordinate points of the two geometric center points in the pixels, calculate the speed v i at which the geometric center point moves on the image. In this way, n speeds are obtained, and calculate the average value v average ;

[0057] 4.4: Predict the position of the target in the next m images.

[0058] Further preferably, in step 4.3, the calculation method of the average included angle is shown in formula (I):

[0059]

[0060] In formula (I), n represents n included angles, and α average represents the average value of n included angles.

[0061] Further preferably, in step 4.3, the calculation method of the speed is shown in formula (II):

[0062]

[0063] In formula (II), v i Indicates the speed at which the geometric center moves on the image at the time when the i+1th image is input, x i ,y i Indicates the position of the pixel where the geometric center point is located on the i-th image, x i+1 ,y i+1 represents the position of the pixel where the geometric center point is located on the i+1th image, and t represents the time detection of the two image inputs.

[0064] Further preferably, in step 4.3, the speed average value is calculated as shown in formula (III):

[0065]

[0066] In formula (III), n represents n speed values, v average Represents the average of n speed values.

[0067] Further preferably, the prediction process in step 4.4 is as follows:

[0068] 4.4.1: Based on the detected target trajectory at the current moment, the number of pixels of the target geometric center point in the image at the next moment is shown in formula (IV);

[0069] s=v average ×t (IV)

[0070] In formula (IV), s represents the pixel distance moved in time t;

[0071] 4.4.2: The direction of the target geometric center movement is α with the guardrail line average The direction of the angle;

[0072] 4.4.3: Based on the moving distance and moving direction, predict the location of the target in the next image;

[0073] 4.4.4: After the first prediction is successful, use this point as the data point for calculating the average speed and average angle, and repeat steps 4.4.1-4.4.3 for a total of m predictions;

[0074] 4.4.5: Connect the m predicted points with the target detected points to form a trajectory and complete the target tracking.

[0075] According to the preferred embodiment of the present invention, in step (5), based on the pedestrian trajectory and the threshold line range in which the pedestrian is located, the early warning module is used to determine whether the pedestrian has climbed over the guardrail. The prediction process is as follows:

[0076] 5.1: Introduce the first warning parameter and the second warning parameter. The first warning parameter in the area between the first threshold line and the second threshold line is 0.5 - 0.55, the first warning parameter in the area between the second threshold line and the third threshold line is 0.75 - 0.8, and the first warning parameter in the area between the third threshold line and the guardrail line is 0.95 - 1; the second warning parameter is related to the included angle α between the extension line of the target trajectory line and the guardrail line. a The calculation formula is shown in Equation (V):

[0077]

[0078] 5.2: Select the first warning parameter according to the area where the predicted geometric center of the target at the m-th time is located. α a Take the included angle between the extension line of the connection line of the first predicted point and the m-th predicted point and the guardrail line, so as to determine the second warning parameter through Equation (V);

[0079] 5.3: Calculate the judgment threshold. The calculation of the judgment threshold bar_value is shown in Equation (VI):

[0080] bar_value = the first warning parameter * the second warning parameter (VI)

[0081] 5.4: Predict the behavior of pedestrians climbing over the guardrail according to the judgment threshold. The judgment process is as follows:

[0082] If the connection line of the first predicted point and the m-th predicted point directly intersects the guardrail line, then it is determined that the guardrail has been climbed. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department;

[0083] If the connection line of the first predicted point and the m-th predicted point does not intersect the guardrail line, and the m-th predicted point is located between the third threshold line and the guardrail line, then the first warning parameter is taken as 1, the second warning parameter is calculated according to Equation (V), and bar_value is calculated using Equation (VI). If bar_value is greater than 0.8, then it is determined that the guardrail has been climbed. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department;

[0084] If the connection line of the first predicted point and the m-th predicted point does not intersect the guardrail line, and the m-th predicted point is located between the second threshold line and the third threshold line, then the first warning parameter is taken as 0.8, the second warning parameter is calculated according to Equation (V), and bar_value is calculated using Equation (VI). If bar_value is greater than 0.8, then it is determined that the guardrail has been climbed. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department;

[0085] If the line connecting the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is between the first threshold line and the second threshold line, it is determined that the guardrail is not crossed.

[0086] Further preferably, the first warning parameter of the area between the first threshold line and the second threshold line is 0.5, the first warning parameter of the area between the second threshold line and the third threshold line is 0.8, and the first warning parameter of the area between the third threshold line and the guardrail line is 1.

[0087] An early warning system for pedestrians illegally climbing over guardrails based on a target tracking algorithm comprises a detection rate control module, a target detection module, a target tracking module and an early warning module; the detection rate control module is used to control the detection rate of an industrial computer using a target detection algorithm; the target detection module mainly uses a YOLO v3 neural network to detect pedestrians; the target tracking module mainly predicts the target position in the future m frames of images based on the detection results of the target detection module; the early warning module is mainly used to determine whether pedestrians will climb over the guardrail.

[0088] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm are implemented.

[0089] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm.

[0090] The beneficial effects of the present invention are:

[0091] 1. The present invention is based on a pedestrian target detection algorithm, which can predict the trajectory of pedestrians in a short period of time. It can make real-time predictions of pedestrians who illegally climb over guardrails with an accuracy of more than 94%, and upload the process of climbing over guardrails to the road management department. With this information, the road management department can better control illegal behaviors and contribute to smart transportation projects.

[0092] 2. The present invention can effectively reduce the probability of traffic accidents caused by pedestrians illegally climbing over guardrails.

[0093] 3. The detection rate control module proposed in the present invention reasonably controls the detection rate of the industrial computer, reduces energy consumption, prolongs the service life of the equipment, and responds to the call for energy conservation and emission reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is a flow chart of an early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm of the present invention;

[0095] Figure 2 It is a distribution schematic diagram of three threshold lines and guardrail lines;

[0096] Figure 3 It is a process flow schematic diagram of the detection rate control module;

[0097] Figure 4 It is a schematic diagram of the included angle between the extension line of the target trajectory line and the guardrail line;

[0098] Figure 5 It is a process flow schematic diagram of the target tracking module;

[0099] Figure 6 It is a process flow schematic diagram of the warning module. Specific implementation mode

[0100] The present invention will be further limited below in conjunction with the accompanying drawings of the specification and embodiments, but not limited thereto.

[0101] Embodiment 1

[0102] An early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm, as Figure 1 shown, includes the following steps:

[0103] (1) Calibrate a number of threshold lines at unequal distances parallel to the guardrail line;

[0104] (2) Deploy a detection rate control module and a target detection module;

[0105] (3) Control the detection rate of the target detection module through the detection rate control module;

[0106] (4) According to the pedestrian target detection result, use the target tracking module to record and predict the pedestrian trajectory;

[0107] (5) Based on the pedestrian trajectory and the threshold line range where it is located, use the warning module to determine whether the guardrail is climbed over and upload the data.

[0108] Embodiment 2

[0109] An early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm according to Embodiment 1, which is characterized in that:

[0110] In step (1), a monitoring camera and an industrial computer are fixedly installed near the guardrail. Specifically, it means: installing the monitoring camera 5 m above the guardrail directly above, and this height can see the entire road surface condition, and installing the industrial computer beside the lamp post of the street lamp. Keep the two devices stable during use.

[0111] In step (1), three unequally spaced threshold lines parallel to the guardrail line are calibrated using OpenCV, including the first threshold line (threshold line 1), the second threshold line (threshold line 2), and the third threshold line (threshold line 3); as Figure 2 shown, the steps are as follows:

[0112] 1.1: Keep the industrial computer in an open state, with a good connection to the monitoring camera, and fix the video stream image output size at 800×600;

[0113] 1.2: Use the Canny edge detection algorithm to detect the guardrail contour line and the 6 placed anti-collision cones. Utilize the guardrail contour features to eliminate the remaining contour lines, and use OpenCV to redraw a curve representing the position of the guardrail in the center of the guardrail contour as the guardrail line;

[0114] 1.3: Place an anti-collision cone at 1m on each side of the left and right of the guardrail. According to the positions of the anti-collision cones in the video, use OpenCV to mark their coordinate points, and then respectively draw parallel lines to the guardrail line passing through these two coordinate points as the first threshold line;

[0115] 1.4: Place an anti-collision cone at 0.5m on each side of the left and right of the guardrail. According to the positions of the anti-collision cones in the video, use OpenCV to mark their coordinate points, and then respectively draw parallel lines to the guardrail line passing through these two coordinate points as the second threshold line;

[0116] 1.5: Place an anti-collision cone at 0.2m on each side of the left and right of the guardrail. According to the positions of the anti-collision cones in the video, use OpenCV to mark their coordinate points, and then respectively draw parallel lines to the guardrail line passing through these two coordinate points as the third threshold line.

[0117] In step (2), as Figure 3 shown, the specific working process of the detection rate control module is as follows:

[0118] Use OpenCV to intercept video data at a rate of 30 images per second;

[0119] Based on the feedback of the target detection result, different results are output, specifically: if there are no pedestrians within the range between the two first threshold lines, 1 frame of picture is output per second; if there are pedestrians within the range between the two first threshold lines, 15 images are output per second; if there are pedestrians within the range between the second threshold line and the third threshold line, 30 images are output per second.

[0120] The implementation pseudo-code in step 2.2 is:

[0121] if there are no pedestrians between the two threshold lines 1:

[0122] Output 1 image per second

[0123] elif there is a pedestrian between threshold line 1 and threshold line 2:

[0124] Output 15 images per second

[0125] elif there is a pedestrian between threshold line 2 and threshold line 3:

[0126] Output 30 images per second

[0127] else:

[0128] Output 30 images per second and take a photo

[0129] Encapsulate this part of the code to make it an independent module and install it in the industrial control computer to complete the deployment of the detection rate control module.

[0130] In step (2), the deployment of the target detection module is as follows:

[0131] 2.1: Make a pedestrian dataset, which includes pedestrians in various postures (standing upright, bending, squatting, walking, running, etc.) and pedestrians in different scenarios (urban roads, sidewalks), and divide the pedestrian dataset into a training set and a test set according to a ratio of 8:2;

[0132] 2.2: Use the training set to train the open-source YOLO v3 target detection algorithm, and use the test set to test the detection effect of the YOLO v3 target detection algorithm to obtain a trained YOLO v3 target detection network for pedestrian classification;

[0133] 2.3: Deploy the Pytorch deep learning environment in the industrial control computer, and run the trained YOLO v3 target detection network for pedestrian classification on the industrial control computer to process more than 30 image data frames per second;

[0134] 2.4: After the target detection module is deployed, connect the target detection module and the detection rate control module;

[0135] 2.5: After the deployment is completed, field-test the detection results of the target detection module. When a pedestrian is detected, use a target box to frame the target.

[0136] The purpose of connecting the target detection module and the detection rate control module is to control the running rate of the target detection module through the detection rate control module.

[0137] The target detection module will detect all the input video stream data, which will cause the CPU to maintain high load operation for a long time and reduce the life of the device. The present invention proposes a detection rate control module to control the detection rate of the target detection module. The specific implementation process of step (3) is as follows:

[0138] 3.1: The video data captured by the surveillance camera is input into the detection rate control module, which cuts the video stream into several pictures and initially inputs them into the target detection module at a rate of one picture per second;

[0139] 3.2: If the target detection module does not detect a pedestrian within the range between the two first threshold lines, the detection rate is maintained at 1 frame per second;

[0140] If the target detection module detects a pedestrian within the range between the first threshold line and the second threshold line, the detection result is fed back to the detection rate control module, and the detection rate control module inputs the image into the target detection module at a rate of 15 frames per second;

[0141] If the target detection module detects a pedestrian within the range between the second threshold line and the third threshold line, the detection result is fed back to the detection rate control module, and the detection rate control module inputs the image into the target detection module at a rate of 30 frames per second;

[0142] If the target detection module detects a pedestrian within the range between the third threshold line and the guardrail line, the detection result is fed back to the detection rate control module, which inputs the target detection module at a rate of 30 frames per second and controls the surveillance camera to take pictures at a rate of 10 frames per second. The evidence of climbing over is saved and the data is uploaded to the road management department through wired transmission.

[0143] In step (4), if Figure 5 As shown in the figure, the specific implementation process of using the target tracking module to record and predict pedestrian trajectories is as follows:

[0144] 4.1: Every time the target detection module detects an image, it outputs the detection result to the target tracking module;

[0145] 4.2: The target tracking module collects each detected target frame, which is a rectangle. OpenCV is used to take the geometric center point of the target frame as the geometric position of the target at the current moment. The geometric positions at multiple moments are connected to obtain the trajectory line of the target in the past period of time. OpenCV is used to draw the trajectory line in each frame of the image, which is target tracking;

[0146] 4.3: Target trajectory prediction is achieved through target tracking.

[0147] like Figure 4As shown, for the i-th image, the geometric center point of the detection box in the current image is P i ; for the (i + 1)-th image, the geometric center point of the detection box in the current image is P i+1 ; connect P i and P i+1 and extend it until it intersects with the guardrail line, and the measured included angle is α i ; for the geometric center points at other times, the above method is also adopted. In this way, n included angles are obtained, i = 1…n, and the average value α of the n included angles is calculated average ;

[0148] While detecting the included angle, record the coordinate points of P i and P i+1 in the image. The coordinate system where the coordinate points are located takes the upper left vertex of the image as the origin, with the right direction representing the positive x-axis and the downward direction representing the positive y-axis; according to the time difference between the two input images and the distance between the coordinate points of the two geometric center points in the pixels, calculate the speed v at which the geometric center point moves on the image i , and the above method is also adopted. In this way, n speeds are obtained, and the average value v of the speeds is calculated average ;

[0149] 4.4: Predict the position of the target in the next m images

[0150] In step 4.3, the calculation method of the average value of the included angle is shown in formula (I):

[0151]

[0152] In formula (I), n represents n included angles, and α average represents the average value of the n included angles

[0153] In step 4.3, the calculation method of the speed is shown in formula (II):

[0154]

[0155] In formula (II), v i represents the speed at which the geometric center point moves on the image at the input time of the (i + 1)-th image, and x i , y i represent the positions of the geometric center point in the pixels on the i-th image, and x i+1 , y i+1 represent the positions of the geometric center point in the pixels on the (i + 1)-th image, and t represents the time difference between the two input images

[0156] In step 4.3, the calculation method of the average value of the speed is shown in formula (III):

[0157]

[0158] In formula (III), n represents n speed values, v average Represents the average of n speed values.

[0159] The prediction process in step 4.4 is as follows:

[0160] 4.4.1: Based on the detected target trajectory at the current moment, the number of pixels of the target geometric center point in the image at the next moment is shown in formula (IV);

[0161] s=v average ×t (IV)

[0162] In formula (IV), s represents the pixel distance moved in time t;

[0163] 4.4.2: The direction of the target geometric center movement is α with the guardrail line average The direction of the angle;

[0164] 4.4.3: Based on the moving distance and moving direction, predict the location of the target in the next image;

[0165] 4.4.4: After the first prediction is successful, use this point as the data point for calculating the average speed and average angle, and repeat steps 4.4.1-4.4.3 for a total of m predictions;

[0166] 4.4.5: Connect the m predicted points with the target detected points to form a trajectory and complete the target tracking.

[0167] In step (5), based on the pedestrian trajectory and the threshold line range, the early warning module is used to determine whether the pedestrian has climbed over the guardrail, such as Figure 6 As shown, the prediction process is as follows:

[0168] 5.1: Introduce the first warning parameter and the second warning parameter. The first warning parameter of the area between the first threshold line and the second threshold line is 0.5-0.55, the first warning parameter of the area between the second threshold line and the third threshold line is 0.75-0.8, and the first warning parameter of the area between the third threshold line and the guardrail line is 0.95-1; the angle α between the second warning parameter and the extension line of the target trajectory line and the guardrail line a The calculation formula is shown in formula (V):

[0169]

[0170] 5.2: Select the first warning parameter according to the area where the geometric center of the mth target is predicted, α aTake the included angle between the extension line of the connection line between the first prediction point and the m-th prediction point and the guardrail line, so as to determine the second early warning parameter through formula (V);

[0171] 5.3: Calculate the judgment threshold. The calculation of the judgment threshold bar_value is shown in formula (VI):

[0172] bar_value = the first early warning parameter * the second early warning parameter (VI)

[0173] 5.4: Predict the behavior of pedestrians climbing over the guardrail according to the judgment threshold. The judgment process is as follows:

[0174] If the connection line between the first prediction point and the m-th prediction point directly intersects the guardrail line, then it is determined that the guardrail has been climbed over. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department;

[0175] If the connection line between the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is located between the third threshold line and the guardrail line, then the first early warning parameter is taken as 1, the second early warning parameter is calculated according to formula (V), bar_value is calculated using formula (VI). If bar_value is greater than 0.8, it is determined that the guardrail has been climbed over. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department;

[0176] If the connection line between the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is located between the second threshold line and the third threshold line, then the first early warning parameter is taken as 0.8, the second early warning parameter is calculated according to formula (V), bar_value is calculated using formula (VI). If bar_value is greater than 0.8, it is determined that the guardrail has been climbed over. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department;

[0177] If the connection line between the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is located between the first threshold line and the second threshold line, it is determined at this time that the guardrail has not been climbed over.

[0178] Embodiment 3

[0179] A warning method for pedestrians' illegal climbing over guardrails based on the target tracking algorithm according to Embodiment 2, characterized in that:

[0180] The first early warning parameter in the area between the first threshold line and the second threshold line is 0.5, the first early warning parameter in the area between the second threshold line and the third threshold line is 0.8, and the first early warning parameter in the area between the third threshold line and the guardrail line is 1.

[0181] The detection accuracy of each link in practical applications is shown in Table 1:

[0182] Table 1

[0183]

[0184] Example 4

[0185] An early warning system for pedestrians illegally climbing over guardrails based on a target tracking algorithm, including a detection rate control module, a target detection module, a target tracking module and an early warning module; the detection rate control module is used to control the detection rate of the industrial computer using the target detection algorithm; the target detection module mainly uses the YOLO v3 neural network to detect pedestrians; the target tracking module mainly based on the detection results of the target detection module realizes the prediction of the target position in the future m frames of images; the early warning module is mainly used to judge whether pedestrians will climb over the guardrail.

[0186] Example 5

[0187] A computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the early warning method for pedestrians illegally climbing over guardrails based on the target tracking algorithm according to any one of Examples 1-3.

[0188] Example 6

[0189] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the early warning method for pedestrians illegally climbing over guardrails based on the target tracking algorithm according to Examples 1-3.

Claims

1. An early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm, characterized in that, The steps include: (1) Mark several threshold lines at different distances parallel to the guardrail line; (2) Deploy the detection rate control module and the target detection module; (3) Controlling the detection rate of the target detection module through the detection rate control module; (4) Based on the pedestrian detection results, the target tracking module is used to record and predict pedestrian trajectories; (5) Based on the pedestrian’s trajectory and the threshold line range, the early warning module is used to determine whether the pedestrian has climbed over the guardrail; In step (5), based on the pedestrian trajectory and the threshold line range, the early warning module is used to determine whether the pedestrian has climbed over the guardrail. The prediction process is as follows: 5.1: Introduce the first warning parameter and the second warning parameter. The first warning parameter in the area between the first threshold line and the second threshold line is 0.5 - 0.55, the first warning parameter in the area between the second threshold line and the third threshold line is 0.75 - 0.8, and the first warning parameter in the area between the third threshold line and the guardrail line is 0.95 - 1; the second warning parameter is related to the included angle α between the extension line of the target trajectory line and the guardrail line, and its calculation formula is shown in Equation (Ⅴ): a It is related to, and its calculation formula is shown in Equation (Ⅴ): 5.2: Select the first warning parameter, α, according to the area where the predicted geometric center of the m-th time is located. a Take the included angle between the extension line of the connection line between the first prediction point and the m-th prediction point and the guardrail line, so as to determine the second warning parameter through formula (Ⅴ); 5.3: Calculate the judgment threshold. The calculation of the judgment threshold bar_value is shown in formula (VI): bar_value = first warning parameter * second warning parameter (VI) 5.4: Based on the judgment threshold, the pedestrian's behavior of climbing over the guardrail is predicted. The judgment process is as follows: If the line connecting the first prediction point and the mth prediction point directly intersects the guardrail line, it is determined that the guardrail has been climbed over. At this time, the detection rate control module is mobilized to take continuous photos of the target and upload the photo data to the road management department; If the line connecting the first prediction point and the mth prediction point does not intersect the guardrail line, and the mth prediction point is between the third threshold line and the guardrail line, then the first warning parameter is 1, the second warning parameter is calculated according to formula (V), and bar_value is calculated using formula (VI). If bar_value is greater than 0.8, it is determined to be climbing over the guardrail. At this time, the detection rate control module is mobilized to continuously take pictures of the target, and the picture data is uploaded to the road management department; If the line connecting the first prediction point and the mth prediction point does not intersect the guardrail line, and the mth prediction point is between the second threshold line and the third threshold line, then the first warning parameter is taken as 0.8, the second warning parameter is calculated according to formula (V), and bar_value is calculated using formula (VI). If bar_value is greater than 0.8, it is determined to be climbing over the guardrail. At this time, the detection rate control module is mobilized to continuously take pictures of the target, and the picture data is uploaded to the road management department; If the line connecting the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is between the first threshold line and the second threshold line, it is determined that the guardrail is not crossed.

2. The early warning method for pedestrians' illegal climbing over guardrails based on the target tracking algorithm according to claim 1, wherein, In step (1), three threshold lines with unequal distances parallel to the guardrail line are calibrated using OpenCV, including a first threshold line, a second threshold line, and a third threshold line; The steps include: 1.1: Keep the industrial computer turned on, connected to the surveillance camera in good condition, and set the video stream image output size to 800×600; 1.2: Use the Canny edge detection algorithm to detect the guardrail contour line and the 6 placed anti-collision cones. Use the guardrail contour features to remove the remaining contour lines. Use OpenCV to redraw a curve representing the guardrail position at the center of the guardrail contour as the guardrail line. 1.3: Place a crash cone at 1 m to the left and right of the guardrail respectively. According to the positions of the crash cones in the video, use OpenCV to mark their coordinate points, and then draw parallel lines to the guardrail line passing through these two coordinate points respectively as the first threshold lines; 1.4: Place a crash cone at 0.5 m to the left and right of the guardrail respectively. According to the positions of the crash cones in the video, use OpenCV to mark their coordinate points, and then draw parallel lines to the guardrail line passing through these two coordinate points respectively as the second threshold lines; 1.5: Place a crash cone at 0.2 m to the left and right of the guardrail respectively. According to the positions of the crash cones in the video, use OpenCV to mark their coordinate points, and then draw parallel lines to the guardrail line passing through these two coordinate points respectively as the third threshold lines.

3. The early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm according to claim 1, wherein, In step (2), the specific working process of the detection rate control module is as follows: Use OpenCV to intercept video data at a rate of 30 images per second; Based on the feedback of the object detection results, different results are output, specifically: if there are no pedestrians within the range between the two first threshold lines, 1 frame of picture is output per second; if there are pedestrians within the range between the two first threshold lines, 15 images are output per second; if there are pedestrians within the range between the second threshold line and the third threshold line, 30 images are output per second.

4. The early warning method for pedestrians illegally climbing over guardrails based on a target tracking algorithm according to claim 1, characterized in that, In step (2), the deployment of the object detection module is as follows: 2.1: Make a pedestrian dataset, which includes pedestrians in various postures and pedestrians in different scenarios, and divide the pedestrian dataset into a training set and a test set according to a ratio of 8:2; 2.2: Use the training set to train the open-source YOLO v3 object detection algorithm, and use the test set to test the detection effect of the YOLO v3 object detection algorithm to obtain a trained YOLO v3 object detection network for pedestrian classification; 2.3: Deploy the Pytorch deep learning environment in the industrial control computer, and run the trained YOLO v3 object detection network for pedestrian classification on the industrial control computer to process more than 30 frames of image data per second; 2.4: After the object detection module is deployed, connect the object detection module and the detection rate control module; 2.5: After the deployment is completed, field-test the detection results of the object detection module. When a pedestrian is detected, use a bounding box to select the target.

5. The early warning method for pedestrians' illegal climbing over guardrails based on a target tracking algorithm according to claim 1, wherein The specific implementation process of step (3) is as follows: 3.1: Input the video data captured by the surveillance camera into the detection rate control module. The detection rate control module intercepts the video stream into several pictures and initially inputs them into the object detection module at a rate of 1 picture per second; 3.2: If the object detection module does not detect a pedestrian within the range between the two first threshold lines, maintain the detection rate of 1 frame of picture per second; If the object detection module detects a pedestrian within the range between the first threshold line and the second threshold line, feedback the detection result to the detection rate control module. The detection rate control module inputs the object detection module at a rate of 15 frames of pictures per second; If the target detection module detects a pedestrian within the range between the second threshold line and the third threshold line, the detection result is fed back to the detection rate control module, and the detection rate control module inputs the image into the target detection module at a rate of 30 frames per second; If the target detection module detects a pedestrian within the range between the third threshold line and the guardrail line, the detection result is fed back to the detection rate control module. The detection rate control module inputs the target detection module at a rate of 30 frames per second, and at the same time controls the surveillance camera to take pictures at a rate of 10 frames per second.

6. The early warning method for pedestrians illegally climbing over guardrails based on target tracking algorithm according to claim 1 is characterized in that: In step (4), the specific implementation process of using the target tracking module to record and predict pedestrian trajectories is as follows: 4.1: Every time the target detection module detects an image, it outputs the detection result to the target tracking module; 4.2: The target tracking module collects each detected target frame, which is a rectangle. OpenCV is used to take the geometric center point of the target frame as the geometric position of the target at the current moment. The geometric positions at multiple moments are connected to obtain the trajectory line of the target in the past period of time. OpenCV is used to draw the trajectory line in each frame of the image, which is target tracking; 4.3: The target trajectory prediction is achieved through target tracking. For the i-th image, the geometric center point of the detection box in the current image is P i ; For the (i + 1)-th image, the geometric center point of the detection box in the current image is P i+1 ; Connect P i and P i+1 and extend it until it intersects with the guardrail line, and the measured included angle is α i ; In this way, n included angles are obtained, i = 1... n, and calculate the average value α of the n included angles average ; While detecting the included angle, record P i and P i+1 the coordinate points in the image, where the coordinate system of the coordinate points takes the top-left vertex of the image as the origin, with the right direction representing the positive x-axis and the downward direction representing the positive y-axis; calculate the speed v at which the geometric center point moves on the image based on the time difference between the two input images and the distance between the pixels where the coordinate points of the two geometric center points are located i In this way, n speeds are obtained, and calculate the average value v of the speeds average ; 4.4: Predict the location of the target in the next m images.

7. The warning method for pedestrians' illegal fence climbing based on the target tracking algorithm according to claim 6, characterized in that In step 4.3, the calculation method of the average angle is as shown in formula (I): In formula (I), n represents n included angles, and α average represents the average value of the n included angles.

8. The early warning method for pedestrians' illegal fence climbing based on the target tracking algorithm according to claim 6, characterized in that, In step 4.3, the speed is calculated as shown in formula (II): In formula (II), v i represents the velocity of the geometric center point moving on the image at the input time of the (i + 1)-th image, x i , y i represent the positions of the pixels where the geometric center point is located on the i-th image, x i+1 , y i+1 represent the positions of the pixels where the geometric center point is located on the (i + 1)-th image, and t represents the time detection of the input of two images.

9. The early warning method for pedestrians' illegal climbing over guardrails based on the target tracking algorithm according to claim 6, wherein In step 4.3, the speed average value is calculated as shown in formula (III): In formula (III), n represents n speed values, and v average represents the average value of the n speed values.

10. A warning method for pedestrians' illegal fence climbing based on a target tracking algorithm according to claim 6, characterized in that, The prediction process in step 4.4 is as follows: 4.4.1: According to the detected target trajectory at the current moment, the number of pixels of the target geometric center point in the image at the next moment is shown in formula (IV); s = v average × t (IV) In formula (IV), s represents the pixel distance moved in time t; 4.4.2: The direction of the movement of the target geometric center is the direction forming an angle α with the guardrail line; average ​ 4.4.3: Based on the moving distance and moving direction, predict the location of the target in the next image; 4.4.4: After the first prediction is successful, use this point as the data point for calculating the average speed and average angle, and repeat steps 4.4.1-4.4.3 for a total of m predictions; 4.4.5: Connect the m predicted points with the target detected points to form a trajectory and complete the target tracking.

11. The early warning method for pedestrians illegally climbing over guardrails based on target tracking algorithm according to claim 1 is characterized in that: The first warning parameter of the area between the first threshold line and the second threshold line is 0.5, the first warning parameter of the area between the second threshold line and the third threshold line is 0.8, and the first warning parameter of the area between the third threshold line and the guardrail line is 1.

12. A warning system for pedestrians illegally climbing over guardrails based on a target tracking algorithm, characterized in that: It includes detection rate control module, target detection module, target tracking module and early warning module; the detection rate control module is used to control the detection rate of the target detection algorithm used by the industrial computer; the target detection module mainly uses the YOLO v3 neural network to detect pedestrians; the target tracking module mainly predicts the target position in the future m frames of images based on the detection results of the target detection module; The early warning module is mainly used to determine whether pedestrians will climb over the guardrail; Based on the pedestrian trajectory and the threshold line range, the early warning module is used to determine whether the pedestrian has climbed over the guardrail. The prediction process is as follows: 5.1: Introduce the first warning parameter and the second warning parameter. The first warning parameter in the area between the first threshold line and the second threshold line is 0.5 - 0.55, the first warning parameter in the area between the second threshold line and the third threshold line is 0.75 - 0.8, and the first warning parameter in the area between the third threshold line and the guardrail line is 0.95 - 1; the second warning parameter is related to the included angle α between the extension line of the target trajectory line and the guardrail line, and its calculation formula is shown in Equation (Ⅴ): a is related, and its calculation formula is shown in Equation (Ⅴ): 5.2: Select the first warning parameter, α, according to the region where the predicted geometric center of the m-th target is located a Take the included angle between the extension line of the line connecting the first prediction point and the m-th prediction point and the guardrail line, so as to determine the second warning parameter through formula (V); 5.3: Calculate the judgment threshold. The calculation of the judgment threshold bar_value is shown in formula (VI): bar_value = First warning parameter * Second warning parameter (Ⅵ) 5.4: Predict the behavior of pedestrians climbing over the guardrail according to the judgment threshold, and the judgment process is as follows: If the line connecting the first prediction point and the m-th prediction point directly intersects the guardrail line, it is determined that the guardrail has been climbed. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department; If the line connecting the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is between the third threshold line and the guardrail line, then the first warning parameter is taken as 1, the second warning parameter is calculated according to formula (Ⅴ), and bar_value is calculated using formula (Ⅵ). If bar_value is greater than 0.8, it is determined that the guardrail has been climbed. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department; If the line connecting the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is between the second threshold line and the third threshold line, then the first warning parameter is taken as 0.8, the second warning parameter is calculated according to formula (Ⅴ), and bar_value is calculated using formula (Ⅵ). If bar_value is greater than 0.8, it is determined that the guardrail has been climbed. At this time, the detection rate control module is activated to continuously take pictures of the target and upload the picture data to the road management department; If the line connecting the first prediction point and the m-th prediction point does not intersect the guardrail line, and the m-th prediction point is between the first threshold line and the second threshold line, it is determined that the guardrail has not been climbed at this time.

13. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the warning method for pedestrians' illegal climbing over the guardrail based on the target tracking algorithm according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the warning method for pedestrians' illegal climbing over the guardrail based on the target tracking algorithm according to any one of claims 1-11.

Images