A traffic signal fault detection method based on image processing

Abstract

The present invention discloses a traffic signal failure detection method based on image processing. First, the real-time video of the traffic signal is acquired and the lamp group is segmented. Then, it is judged whether the type of the signal lamp is a red-yellow-green signal lamp. If it is a red-yellow-green signal lamp, a comprehensive discriminator is used to discriminate the signal lamp failure and output the signal lamp failure type. If it is a countdown sign, it is first preprocessed, and then divided into 7 segments, and the comprehensive discriminator is used to discriminate the failures of each segment respectively. Finally, the failure type of the entire countdown sign is determined by comprehensively considering the failure states of the above 7 segments. The above comprehensive discriminator calculates the failure probability P1 according to the on-off state, calculates the failure probability P2 according to the on-off fluctuation, calculates the failure probability P3 according to the SVM classifier, and finally comprehensively judges the failure state of the signal lamp based on the three probabilities. The present invention can effectively detect failure types such as long-term extinction, long-term lighting, double-lamp lighting at the same time, and countdown sign abnormality of traffic signal lamps, and has strong practicability.

Description

Technical Field

[0001] The present invention belongs to the technical field of image communication, and relates to a traffic signal fault detection method based on image processing. Background Art

[0002] Traffic signals are indispensable facilities at road intersections. When traffic signals malfunction, it is easy to lead to chaotic traffic order and frequent traffic accidents. When traffic signals malfunction, workers need to be arranged for maintenance in a timely manner. At present, the automatic detection schemes for traffic signal faults mainly include color sensor method, current-voltage measurement method, and image processing method. The color sensor method and the current-voltage measurement method require a large number of additional devices to be installed. The color sensor method is also easily interfered by external light. Since the image processing method can utilize the traffic monitoring cameras already installed at intersections, the image processing method has gradually become the mainstream scheme for automatic detection of traffic signal faults.

[0003] In outdoor scenarios, the weather conditions vary widely and the sun's irradiation angle is constantly changing. These factors lead to many false alarms and missed detections in existing image processing schemes. There are numerous intersections in a city. If there are many false alarms, maintenance personnel will be overwhelmed. To address this problem, a scheme has been proposed to detect traffic signal faults using deep learning methods, and to detect whether there are traffic signals in the designated area through deep learning detection. However, this scheme is prone to failure in night scenes or under the background of green belts. In addition, the deep learning method has very high requirements for hardware costs. If the deep learning method is adopted, a sufficient number of GPUs need to be configured to achieve accelerated analysis, and its cost is also difficult to control.

[0004] Therefore, objectively, it is necessary to improve the image processing method to improve the detection accuracy of traffic signal faults in a low-cost manner and reduce or avoid the occurrence of false alarms and missed detections. Summary of the Invention

[0005] The present invention proposes a traffic signal fault detection method based on image processing for the problems existing in the above-mentioned prior art, which can detect fault types such as long-term extinguishing, long-term lighting, double-lamp simultaneous lighting, and abnormal countdown signs of traffic signals.

[0006] To achieve the above object, the present invention designs a comprehensive discriminator. This comprehensive discriminator calculates the fault probability from different angles and finally makes a comprehensive judgment based on multiple fault probabilities to discriminate the faults of a single traffic signal. For the countdown sign, it is divided into 7 segments and the comprehensive discriminator is used for fault detection respectively. Finally, the fault states of the 7 segments are comprehensively considered to determine the fault type of the entire countdown sign.

[0007] The specific technical solution of the traffic signal fault detection method based on image processing includes the following steps:

[0008] S1: Obtain the real-time video of the traffic signal light and perform light group segmentation;

[0009] S2: Determine whether the type of the signal light is a red-yellow-green signal light;

[0010] S3: If it is a red-yellow-green signal light, use a comprehensive discriminator to discriminate the signal light failure and output the signal light failure type;

[0011] S4: If it is a countdown sign, first perform preprocessing, then divide it into 7 segments and use a comprehensive discriminator to discriminate the failures of each segment respectively. Finally, based on the failure states of the above 7 segments, determine the failure type of the entire countdown sign.

[0012] The above comprehensive discriminator calculates the probability of the signal light failure by three methods respectively. The first method is to calculate the on-off state by color segmentation, and then calculate the failure probability P1 according to the on-off states of all frames within a period of time; the second method is to first calculate the sum of the pixel values of the light area of each frame, and then calculate the failure probability P2 according to the fluctuation of the sum of the pixel values of the light area of multiple sub-segments within a period of time; the third method is to predict the on-off state of the light according to the pre-trained SVM model, and then calculate the failure probability P3 according to the on-off states of all frames within a period of time; finally, use the weighted sum of P1, P2, and P3 to determine the failure type.

[0013] Among them, in the above first method, each frame of image is subjected to adaptive binaryzation processing, and the on-off state of each frame of image is determined according to the bright point ratio and the average coordinates, and then the failure probability P1 is calculated according to the on-off states of all frames within a period of time.

[0014] In the above second method, calculating the failure probability P2 according to the fluctuation of the sum of the pixel values of the light area of multiple sub-segments within a period of time means calculating the failure probability P2 according to the maximum and minimum values of the sum of the pixel values of the light area of multiple sub-segments within a period of time.

[0015] The above third method specifically includes pre-training an SVM on-off state classifier, predicting the on-off state of each light in each frame of image during operation using the SVM model, and then calculating the failure probability P3 according to the on-off states of all frames within a period of time.

[0016] In particular, for the countdown sign, two SVM prediction models, horizontal and vertical, are trained respectively for predicting the horizontal and vertical segments of the digital lights.

[0017] Preferably, for the countdown sign, count the proportion of white dots on the straight line from the center coordinate to the edge coordinate at 1 / 3 of the height of the light area. If the proportion of white dots exceeds a certain range, it is considered that the font is too thick and morphological erosion processing is performed on it.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] 1. The present invention can effectively detect fault types such as long-term extinguishment, long-term lighting, simultaneous lighting of two lights, and abnormality of the countdown sign of traffic lights.

[0020] 2. The comprehensive discriminator provided by the present invention can solve the interference problem brought by the changes in weather and solar irradiation angle in the real scenario to the judgment of the lighting and extinguishing state of traffic lights, and eliminate false alarms and misreports.

[0021] 3. The comprehensive discriminator adopts a scheme similar to the ensemble learning in machine learning, calculates the fault probability from different angles, and finally makes a comprehensive judgment based on multiple fault probabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of the framework of the comprehensive discriminator of the present invention;

[0023] Figure 2 is a flowchart of traffic signal fault detection;

[0024] Figure 3 is an annotation diagram of the signal light;

[0025] Figure 4 is a flowchart of calculating the lighting and extinguishing of the lights in a single-frame image;

[0026] Figure 5 is a preprocessing flowchart of the countdown sign;

[0027] Figure 6 is a schematic diagram of the segmentation method of the digital lights of the countdown sign. DETAILED DESCRIPTION OF THE INVENTION

[0028] The present invention will be described in detail below with reference to the accompanying drawings.

[0029] The technical problem to be solved by the present invention is to improve the accuracy of traffic signal fault detection, reduce false alarms, and effectively control the project cost. To solve the above problems, the present invention proposes a traffic signal fault detection method based on image processing, which can detect fault types such as long-term extinguishment, long-term lighting, simultaneous lighting of two lights, and abnormality of the countdown sign of traffic lights. Since in the real scenario, various weather changes and changes in the solar irradiation angle will interfere with the judgment of the lighting and extinguishing state, and there are many false alarms and misreports in a single discriminant method, the present invention designs a comprehensive discriminator, which adopts a scheme similar to the ensemble learning in machine learning, calculates the fault probability from different angles, and finally makes a comprehensive judgment based on multiple fault probabilities to discriminate the faults of a single signal light. When calculating the fault probability of the signal light, the probability is calculated based on the data within a time period, and the length of the time period should be determined according to the signal cycle of the traffic lights, generally set to include at least 3 signal cycles.

[0030] The structure of the above comprehensive discriminator is as Figure 1 shown. The probability of the signal lamp failing is calculated by three methods respectively. The first method is to calculate the on / off state by color segmentation, and then calculate the failure probability P1 according to the on / off states of all frames within a period of time. The second method is to first calculate the sum of the pixel values of the lamp area in each frame, and then calculate the failure probability P2 according to the fluctuations of the sum of the pixel values of the lamp area in multiple sub-segments within a period of time. The third method is to predict the on / off state of the lamp according to the pre-trained SVM model, and then calculate the failure probability P3 according to the on / off states of all frames within a period of time. Finally, the weighted sum of P1, P2, and P3 is used to determine the type of failure.

[0031] The traffic signal lamp failure detection process is as Figure 2 shown. After obtaining the video frame, first perform lamp group segmentation, and then process according to the type of signal lamp.

[0032] For red, yellow, and green signal lamps, directly use the multi-condition comprehensive discriminator to determine whether the signal lamp fails and the type of failure that occurs.

[0033] For the countdown sign, first preprocess the countdown sign. The preprocessing includes two parts: normalization and thickness unification. The normalization of the digital lamp is to scale the digital lamp image to a unified size. As an optimization, it is recommended to divide the digital lamp into 7 segments for detection respectively. If the characters are too thick, there may be interference between segments, so it is necessary to detect the character thickness. If the characters are too thick, morphological erosion operations need to be performed on the character image. After preprocessing, perform 7-segment segmentation on the numbers of the countdown sign, and use an independent comprehensive discriminator for each segment to determine whether there is a failure. Finally, comprehensively obtain the failure state of a single digital lamp of the countdown sign according to the failure detection results of the 7 segments.

[0034] The following are the specific implementation steps of the present invention:

[0035] 1) Manual annotation

[0036] After the system is installed, first perform manual annotation. It is only necessary to calibrate the box where the object is located and set the number and type of lamp groups. The lamp group types include two types: red, yellow, and green signal lamps and countdown signs. The annotation method is as Figure 3 shown.

[0037] 2) Red, yellow, and green signal lamp failure detection

[0038] S1) Lamp group segmentation

[0039] The calibrated lamp group area usually contains multiple signal lamps and needs to be segmented to obtain the position of a single signal lamp.

[0040] S2) Calculate the failure probability P1

[0041] For red, yellow, and green signal lights, for each frame of image, calculate the on / off state of each light in the light group. The calculation process for the on / off state of each light is as follows Figure 4 shown: Adaptive binarization utilizes the color information of the signal lights. Generally, if the signal light is off, it usually appears gray, and when it is on, it appears red, yellow, or green, and may also appear white when the brightness is high, and is basically white at night. Assume that the pixel value of point (i, j) is Color(Rij, Gij, Bij). Then, if any one of Rij, Gij, and Bij is greater than 190, it is considered that a single pixel is on; if the difference between the maximum value and the minimum value of Rij, Gij, and Bij is greater than 60, it is also considered that a single pixel is on; otherwise, it is considered that a single pixel is off. Classify each pixel of the signal light, set the on pixels to 255 and the off pixels to 0, thereby obtaining the binarized image of the light.

[0042] Count the proportion of bright pixels in the binarized image. If the proportion is greater than 0.1, it indicates that the entire light may be on. However, to exclude some sunlight and vehicle headlight interferences, further count the average coordinates of all bright points. If the average coordinates are located in the central area of the light, it is confirmed that the light is on in the current frame; otherwise, it is off.

[0043] If there are a total of C frames of images in the time period [t1, t2], the number of images determined to be in the on state is B, and the number of images determined to be in the off state is D. For normal signal lights, generally B>0 and D>0. Let M be the smaller value of B and D. When the signal light is on for a long time or off for a long time, due to light interference or misjudgment of the state, M may be greater than 0. There are differences in the on / off cycles of signal lights at each intersection. Considering the extreme case where the on / off state ratio of a single light in the time period is 1:10 or 10:1. If M >= C / 10, it can be considered that the signal light is basically normal, and the probability of failure is 0. Take M = Min(B, D, C / 10); it can be seen that the maximum value of M is C / 10; the minimum value of M is 0. The closer M is to 0, the greater the probability of a single signal light failing. The probability P1 can be calculated according to the following formula

[0044] P1 = 1 - M / (C / 10);

[0045] S3) Calculate the failure probability P2

[0046] Assume that the width of the area of a single signal light is w and the height is h, and the pixel value of point (i, j) is Color(Rij, Gij, Bij). Define the calculation formula for the total pixel value BT of the light area as

[0047]

[0048] For a normal signal light, it generally lights up for a period of time and then goes off for a period of time, and there are significant fluctuations in BT. Obtain the maximum value Ma and the minimum value Mi of the sum of the pixel values BT in the light area within the time period [t1, t2]. Let Md = Ma - Mi. The magnitude of Md can measure the fluctuation of the signal light BT. The smaller Md is, the smaller the BT fluctuation within this time period, and the greater the probability of signal light abnormality. The maximum value of Md is Ma, and the minimum value is 0. The probability P2 can be calculated according to the following formula:

[0049] P2 = 1 - Md / Ma;

[0050] To prevent interference from instantaneous light, the time period [t1, t2] can be evenly divided into three sub - segments, and the probabilities P21, P22, and P23 are calculated respectively according to the above method. Since the time period [t1, t2] contains more than three signal cycles, for a normal signal light, P21, P22, and P23 are not too high; for an abnormal signal light, P21, P22, and P23 are generally very high. However, if there is interference from instantaneous light in the i - th sub - segment, the failure probability P2i of this sub - segment may be very low, and the failure probabilities of other sub - segments not affected by instantaneous light are still very high. Therefore, take P2 = max(P21, P22, P23). In this way, even if the failure probability P2i of a single sub - segment is very low due to interference, the overall failure probability P2 of the time period [t1, t2] is still relatively high.

[0051] S4) Calculate the failure probability P3

[0052] Collect signal light images in different time periods and different weather conditions, make them into single signal light sample images, label the on - off states of each image, and then train an SVM model that can predict the on - off states of single signal lights. Use this SVM model to predict the on - off states of each light in the real - time video frame. For each signal light, if within the time period [t1, t2], there are a total of C frames of images, B frames of images are predicted to be in the on state, and D frames of images are predicted to be in the off state. After obtaining B and D, take M = Min(B, D, C / 10). The probability P3 can be calculated according to the same principle as in step S2: P3 = 1 - M / (C / 10);

[0053] S5) Comprehensively judge whether the red, yellow, and green signal lights are abnormal

[0054] When making a comprehensive judgment, the optimal weighting parameters w1, w2, and w3 can be selected according to the experimental results. Usually, the sum of w1, w2, and w3 is required to be 1. In the time period [t1, t2], let the probability P of a single signal light being abnormal be P = w1 * P1 + w2 * P2 + w3 * P3; if P > 0.8, it is considered abnormal; otherwise, the signal light is considered normal. In the case of an abnormality, if the lit state is more, that is, B in step S2 is greater than D, it is considered a long-on fault; otherwise, it is a long-off fault. If there is no abnormality in a single light, it is detected whether there is a situation where multiple lights in the light group are lit simultaneously. If so, a double-light-on fault is reported.

[0055] 3) Fault detection of the digital lights on the countdown board

[0056] The fault detection of the digital lights on the countdown board also uses a comprehensive discriminator similar to that of the red, yellow, and green signal lights, but some additional steps need to be added for the processing of the digital lights on the countdown board. There may be a problem of incomplete display characters for the digital lights on the countdown board, and they need to be divided into 7 segments for separate detection. If the characters are too thick, there may be interference between segments, so the thickness of the characters also needs to be detected. If the characters are too thick, morphological erosion operations are performed on the character images to make the thickness of the characters basically consistent.

[0057] S1) Pretreatment of the lights on the countdown board

[0058] After the digital lights on the countdown board are segmented, preprocessing is first performed, as Figure 5 shown. The first step of the preprocessing is to normalize the image to a certain size, and then detect the font thickness of each normalized digital image. Assuming that the width of the digital area is w and the height is h, the proportion of white dots on the straight line from the coordinate (w / 2, h / 3) to the coordinate (w, h / 3) is statistically calculated, that is, the proportion of white dots on the straight line from the center coordinate to the edge coordinate at 1 / 3 of the height of the light area is statistically calculated. If the proportion of white dots exceeds a certain range, the font is considered too thick and morphological erosion processing is required. After preprocessing, the images of all digital lights are of the same size and the thickness is basically consistent.

[0059] S2) Segment the digital lights on the countdown board in each frame of the image

[0060] The segmentation of the digital lights utilizes the 7-segment display principle of the digital lights. The on and off of the 7 segments can combine to form individual numbers and letters. The segmentation method of the digital lights on the countdown board is as Figure 6 shown.

[0061] S3) Detect the faults of the segmented digital lights

[0062] After the countdown digital lights are divided into 7 segments, there are two types of segments. One is the horizontal segment and the other is the vertical segment. To achieve higher prediction accuracy, two SVM prediction models for the horizontal and vertical segments need to be trained separately, which are used for predicting the horizontal and vertical segments of the digital lights respectively. Calculate the failure probabilities P1, P2, and P3 for each segment of the digital lights respectively. The calculation method is the same as that of the red, yellow, and green signal lights. Then use the aforementioned comprehensive discriminator to determine whether it is normal, long-off, or long-on.

[0063] S4) Comprehensively determine whether the countdown lights are abnormal

[0064] During the time period [t1, t2], if there is a segment that is long-on in the countdown digital lights, it is determined that the digital lights are long-on; if there is a segment that is long-off, it is determined that the digital signal lights are incomplete. The failure status of each segment can also be reported independently.

[0065] It should be noted that the above embodiments provided by the present invention are only illustrative and do not have the effect of limiting the specific implementation scope of the present invention. The protection scope of the present invention should include those transformations or alternative solutions that are obvious to those of ordinary skill in the art.

Claims

1. A traffic signal fault detection method based on image processing, characterized in that It includes the following steps: S1: Obtain the real-time video of the traffic signal lights and perform light group segmentation; S2: Determine whether the type of the signal light is a red-yellow-green signal light; S3: If it is a red-yellow-green signal light, use a comprehensive discriminator to discriminate the signal light failure and output the signal light failure type; S4: If it is a countdown sign, first perform preprocessing, then divide it into 7 segments and use a comprehensive discriminator to discriminate the faults of each segment respectively, and finally comprehensively determine the fault type of the entire countdown sign according to the fault states of the above 7 segments; The comprehensive discriminator calculates the probability of the signal light failure by three methods respectively. The first method is to calculate the on-off state by color segmentation, and then calculate the failure probability P1 according to the on-off states of all frames within a period of time; the second method is to first calculate the sum of the pixel values of the light area of each frame, and then calculate the failure probability P2 according to the fluctuation of the sum of the pixel values of the light area of multiple sub-segments within a period of time; The third method is to predict the on-off state of the light according to the pre-trained SVM model, and then calculate the failure probability P3 according to the on-off states of all frames within a period of time; finally, the weighted sum of P1, P2, and P3 is used to determine the failure type.

2. The traffic signal failure detection method according to claim 1, characterized in that In the first method, each frame of image is subjected to adaptive binary processing, the on-off state of each frame of image is determined according to the bright point ratio and the average coordinates, and then the failure probability P1 is calculated according to the on-off states of all frames within a period of time.

3. The traffic signal failure detection method according to claim 1, characterized in that In the second method, calculating the failure probability P2 according to the fluctuation of the sum of the pixel values of the light area of multiple sub-segments within a period of time means calculating the failure probability P2 according to the maximum and minimum values of the sum of the pixel values of the light area of multiple sub-segments within a period of time.

4. The traffic signal failure detection method according to claim 1, characterized in that The third method specifically includes pre-training an SVM on-off state classifier, predicting the on-off state of each light in each frame of image during operation by using the SVM model, and then calculating the failure probability P3 according to the on-off states of all frames within a period of time.

5. The traffic signal failure detection method according to claim 4, wherein Two SVM prediction models, namely horizontal and vertical, are trained respectively for the countdown sign and are used for the prediction of the horizontal and vertical segments of the digital lights respectively.

6. The traffic signal fault detection method according to claim 1, wherein, For the countdown sign, count the proportion of white dots on the straight line from the center coordinate to the edge coordinate at 1 / 3 height of the light area. If the proportion of white dots exceeds a certain range, it is considered that the font is too thick and morphological erosion processing is performed on it.

Images