Signal lamp fault detection method and system based on video identification

Abstract

The invention discloses a signal lamp fault detection method and system based on video recognition, and the method comprises the steps: signal lamp state image data collection, signal lamp color state recognition model training, signal lamp preset position setting and matching, and various recognition scenes and fault detection strategies. A signal lamp color state identification model is trained through an artificial intelligence technology, signal lamp states are identified by using the identification model, different scenes are divided according to signal lamp types and operation states, and corresponding fault detection strategies are formulated for the different scenes, so that functions of accurately finding signal lamp faults in real time and performing early warning are realized. The traffic equipment and facility management and control scheme is optimized, the traffic fluency is improved, and the traffic safety risk is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and traffic system, and in particular to a signal light fault detection method and system based on video recognition. Background Art

[0002] Traffic congestion is a major challenge facing cities. Traffic light failures can lead to traffic congestion and traffic accidents, so timely detection of traffic light failures is crucial. Traditional traffic light failures cannot be discovered in a timely manner and need to be discovered and reported manually, and then confirmed by video, on-site confirmation, and maintenance personnel on duty. This is not smart enough, and manual discovery is time-consuming. Although there are currently solutions that use video recognition technology to detect traffic light failures, the model recognition accuracy and speed are low, and it lacks adaptability when dealing with traffic light failures in different time periods and scenarios. Summary of the invention

[0003] The object of the present invention is to provide a signal light fault detection method and system based on video recognition to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] In the first aspect, the present application provides a method for detecting signal light faults based on video recognition, which method consists of step S1 of acquiring signal light status image data, step S2 of training a signal light color status recognition model, step S3 of setting and matching signal light preset positions, and step S4 of multiple recognition scenarios and fault detection strategies.

[0006] As a further solution of the present invention: the step S1 of collecting the signal light status image data specifically includes the following steps:

[0007] Step S11: Classify video image acquisition and construct original data set;

[0008] Step S12: Screening and expanding video images to construct a training data set;

[0009] The step S2 of signal light color state recognition model training specifically includes the following steps:

[0010] Step S21: data set annotation and data set division;

[0011] Step S22: target recognition model and training;

[0012] The step S3 of setting and matching the preset positions of the signal lights specifically includes the following steps:

[0013] Step S31: signal light preset position setting;

[0014] Step S32: recognizing the distance calculation of the frame and matching it with the preset position;

[0015] The step S4 identifies multiple scenarios and fault detection strategies, uses real-time video streams and the signal light preset position data in step S31 to perform polling video detection, and uses the detection results to perform scene judgment; the fault detection strategies include: a set detection strategy for always-on lights to be inspected, a set detection strategy for digital lights to be inspected, and a set detection strategy for very bright lights to be inspected.

[0016] As a further solution of the present invention: Step S11, based on the camera of the existing traffic facility electric police checkpoint, obtains a real-time video stream, and captures images containing the operating status of traffic lights from the video stream at different frequencies at different times of the day and saves them to form an original data set.

[0017] As a further solution of the present invention: in step S12, invalid graphics in the original data set are screened out, and part of the images in the original data set are expanded using an image enhancement method of geometric transformation and noise injection to form a training data set.

[0018] As a further solution of the present invention: in step S22, the target recognition model uses YOLOv10 as a pre-training model.

[0019] As a further solution of the present invention: in step S31, a marking tool is used to mark the preset position of the signal light, and each channel of video to be inspected is marked once to obtain the preset position data of the corresponding video stream. The labels marked on the preset position of the signal light are divided into color lights, arrow lights, and digital lights.

[0020] As a further solution of the present invention: in the step S32, the distance between the recognition box in the target detection result and the preset position annotation box is calculated to obtain the matching result of the recognition box and the preset position annotation box.

[0021] In the second aspect, the present invention also provides a signal light fault detection system based on video recognition, which is characterized in that it consists of a signal light status image data acquisition unit, a signal light color status recognition model training unit, a signal light preset position setting and matching unit and a variety of recognition scenarios and fault detection strategy units.

[0022] Compared with the prior art, the beneficial effects of the present invention are: training the traffic light detection model through artificial intelligence technology, using the detection model to identify the status of the traffic light, dividing different scenes according to the status of the traffic light, and formulating corresponding fault judgment logic for different scenes, so as to realize the function of real-time and accurate detection of traffic light faults and early warning, so as to optimize the management and control plan of traffic equipment and facilities, improve traffic smoothness, and reduce traffic safety risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall flow chart of the detection method of the present invention;

[0024] Figure 2 The training flow chart of the target recognition model of the present invention;

[0025] Figure 3 This is a schematic diagram of the training results of the target recognition model of the present invention;

[0026] Figure 4 A flow chart for setting the preset position of a signal light of the present invention;

[0027] Figure 5 This is the flow chart of the signal light preset position matching algorithm;

[0028] Figure 6 This is a flow chart of the fault detection strategy under various identification scenarios of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The embodiment of the present application provides a signal light fault detection method and system based on video recognition, which realizes the function of real-time and accurate detection of signal light faults and early warning, so as to optimize the management and control plan of traffic equipment and facilities, improve traffic smoothness and reduce traffic safety risks.

[0031] In this application, signal lights are divided into color lights (displaying only colors), arrow lights (displaying arrows of different colors), and countdown lights (displaying numbers with color scrolling, also called digital lights) according to the display effects. Generally, there will be the same type of lights on the same light panel, or there may be a mixture of multiple types. This application regards an entire light panel as a signal light.

[0032] Traffic light failure defined in this application: Traffic lights have two display states: on and off. The normal on state may include three color states: red, yellow, and green. Multiple lights on the same light panel are considered to be in the on state as long as one light is on, and all lights are considered to be in the off state. If the off state lasts too long or does not match the signal control solution, it is determined to be an off failure.

[0033] The signal light that is always on as defined in this application means that the signal light is green when the vehicle is in the released state and red when the vehicle is not in the released state. As long as the vehicle is operating normally, the signal light will remain on, and the flickering at the switching interval can be ignored. If the signal light is damaged, the signal light will not be on at all. In contrast, a very bright signal light means that the signal light does not remain on all the time, and it will only be on for a few seconds when the state is about to switch. For example, the half-way countdown light that is often seen is off in the first half and only starts to countdown and flicker in the last 10 seconds or so.

[0034] Embodiment 1

[0035] This embodiment provides a signal light fault detection method based on video recognition, such as Figure 1 As shown, the method consists of step S1 of collecting signal light status image data, step S2 of training a signal light color status recognition model, step S3 of setting and matching signal light preset positions, and step S4 of multiple recognition scenarios and fault detection strategies.

[0036] Step S1: collecting image data of traffic light status, including: Step S11: collecting classified video images to construct an original data set; Step S12: screening and expanding video images to construct a training data set.

[0037] Step S11, based on the camera of the existing traffic facility electronic police checkpoint, obtain the real-time video stream, and intercept the images containing the operating status of the traffic lights from the video stream at different frequencies at different times of the day to save them, forming an original data set, wherein the collection strategy of different frequencies at different times of the day is as follows:

[0038] High-frequency time period: 05:00 to 09:00, 17:00 to 21:00, a total of 8 hours, one ticket every 20 minutes;

[0039] Medium frequency time period: 09:00 to 17:00, a total of 8 hours, one every 30 minutes;

[0040] Low-frequency period: 21:00 to 05:00, a total of 8 hours, one ticket every 60 minutes.

[0041] According to the above acquisition strategy, an image acquisition device can collect 48 images throughout the day. In order to ensure the training and recognition effect of the recognition model, the number of images in the original data set should not be less than 25,000.

[0042] Step S12: The actual collected images are of substandard quality. In order to improve the robustness of the recognition model, invalid images need to be removed and the image data needs to be enhanced and expanded. The images collected during high-frequency periods are mainly enhanced because the images collected during this period are in scenes with large changes in illumination. The image enhancement techniques used are geometric transformation and noise injection. Part of the image is slightly translated and rotated, and part of the image is enlarged. Then, random values ​​sampled from a Gaussian distribution are added to the pixels of the image to simulate the noise in the real environment. The new image data is expanded and used for model training.

[0043] Step S2: Traffic light color state recognition model training includes the following steps: Step S21: data set annotation and data set division; Step S22: target recognition model and training.

[0044] Step S21, data set annotation is a key step in machine learning and deep learning, which involves labeling and classifying the original data so that it can be used for model training. After all data sets are annotated, the label data of the image is obtained. The annotated labels are set to four types according to the operating status of the traffic light: red, yellow, green, and off. The training data set and labels are divided into training sets and test sets for model training and result testing.

[0045] The target recognition model training process in step S22 is as follows: Figure 2 As shown in the figure, the target recognition model uses the latest YOLOv10 as the pre-training model. According to the requirements of model training, put the labeled training set and test set into the corresponding directory, modify the relevant configuration files and start training.

[0046] The specific pre-training model is yolov10n, the image size is 640 pixels, the batch size is set to 16, the number of training rounds is set to 500, and the model is trained using GPU. The training results of the recognition model are as follows: Figure 3 As shown in the figure, MAP is used as a comprehensive evaluation indicator of target location and label accuracy. The verification results on the test set are shown in the following table.

[0047] Class Number of images Number of tags Average mAP mAP50 mAP50-90 red 5617 12638 0.791 0.821 0.429 yellow 813 1356 0.538 0.611 0.302 green 3738 7804 0.791 0.829 0.418 Turn off the lights 3376 5950 0.654 0.713 0.391 all 8479 27748 0.654 0.744 0.385

[0048] From the average mAP50, we can see that the comprehensive evaluation reaches 74%. The accuracy of the light-off state, which has the greatest impact on the fault, reaches 71.3%. The recognition result on the test set is good, and it can also achieve relatively ideal recognition results in the test of actual scenes, which can meet the actual use of traffic light fault detection.

[0049] The step S3 of setting and matching the preset position of the signal light comprises the following steps: step S31: setting the preset position of the signal light; step S32: calculating the distance of the recognition frame and matching the preset position.

[0050] Step S31. The preset position setting process is as follows Figure 4 shown. Use the annotation tool to annotate the preset positions of the signal lights, and annotate each video to be inspected once to obtain the preset position data corresponding to the video stream. The preset position type tags of the signal lights are divided into color lights, arrow lights, and digital lights, and different types correspond to different scenarios and different detection strategies. After the annotation is completed, export the preset position data and match it with each real-time video stream device.

[0051] In step S32, calculate the distance between the recognition frame in the target detection result and the preset position annotation frame, sort the distance values in descending order, and sequentially take the minimum value as the matched result. Finally, obtain the matching result between the recognition frame and the preset position annotation frame. The preset position matching algorithm is as follows Figure 5 shown. Suppose there are M recognition frames and N preset frames. The sizes of M and N are not related. Generally, the number of recognition frames is greater than the number of preset frames, and there are results that do not need to be recognized. The preset position matching reduces the redundancy of recognition, and after matching to a specific preset position type, subsequent scene-based detection can be performed.

[0052] The preset position matching algorithm specifically includes the following steps: (1) Calculate the distance: Calculate the distances between the M recognition frames and the N preset positions respectively. The distance is calculated using IOU. When the calculated IOU is 0, use GIOU to calculate, and obtain MxN distance values; (2) Distance sorting: Sort the distances from largest to smallest and keep the correspondence between the distance and the specific recognition frame and detection frame; (3) Screen the matching results: Based on the greedy idea, start selecting from the largest distance pairing. Mark the selected pairing to indicate that the recognition frame and the detection frame have been selected. Sequentially select down according to the distance. Skip the detection frame or recognition frame that has been selected and continue to execute downward. After traversing all the pairing distances, all the matching results are obtained, and then output the matching of the detection frame id and the preset position frame id; (4) There are two cases for the number of matching results. Suppose n detection results are matched. There are two cases

[0053] a. n = N, the number of matches is equal to the number of preset position frames, and it is exactly a complete match;

[0054] b. n < N, the number of matches is less than the number of preset position frames, and there are cases of missed recognition or no match. There may be a situation where the camera position deviation is too large. If there is no deviation, the number of image trainings needs to be increased.

[0055] Step S4 multiple recognition scenarios and fault detection strategies, using real-time video streams and the signal light preset position data in step S31, to perform polling video detection, and use the detection results to make scene judgments, specifically divided into three scenarios: always-on light detection scenario, very-bright light detection scenario, and digital light detection scenario. When it is uncertain whether the signal light is always-on, the default is always-on light detection, that is, all device video streams are classified into the always-on light to be detected set, all are always-on light detection, and then the corresponding fault detection strategy is adopted according to different scenarios. Fault detection strategy, such as Figure 6 As shown, it specifically includes: a detection strategy for a set of always-on lights to be inspected, a detection strategy for a set of digital lights to be inspected, and a detection strategy for a set of very bright lights to be inspected.

[0056] The detection strategy of the set of lights waiting to be inspected includes:

[0057] 1. Read the video stream in the always-on waiting-for-inspection set and capture a single image of the video stream;

[0058] 2. Use the recognition model to identify the target in a single image and obtain all the recognition boxes;

[0059] 3. Determine whether the preset position frame contains a digital light. Most digital lights are rectangles close to squares. By calculating the aspect ratio close to 1, it can be determined as a digital light. If the target contains a digital light, the detection is terminated and the video stream is added to the digital light to be detected set;

[0060] 4. Match the recognition frame of the target detection result with the preset position. The following situations exist: a. When all the matched detection frames are in a non-light-off state (red, yellow, green), the detection is terminated and the video stream is judged to be fault-free; b. If one or more of the matched detection frames are in an off state, the detection is terminated and the video stream is added to the very bright set to be inspected for detection.

[0061] The digital light is the countdown light. There is a flashing phenomenon in the interval between the digital light switching. A single detection cannot accurately determine the fault. In this case, the digital light is detected separately and the duration and frequency of video detection are increased. The digital light detection set detection strategy specifically includes:

[0062] 1. Read the video stream of the digital lights to be inspected, and capture a single image every 10 seconds, with a maximum of 10 minutes;

[0063] 2. Use the recognition model to identify the target in a single image and obtain all the recognition boxes;

[0064] 3. Match the recognition frame of the target detection result with the preset position. a. When all the matched detection frames are in a non-off state (red, yellow, green), the detection is terminated and the video stream is judged to be fault-free; b. If one or more of the matched detection frames are in an off state, continue to wait for the next detection;

[0065] 4. It is only necessary to detect once that all lights are not off before stopping the detection. It is not necessary to complete all the detections. If the light is still off after all the detections are completed, it is determined to be a fault and a fault alarm is issued.

[0066] Very bright lights are similar to digital lights, and are divided into short-term and long-term. The principle of short-term very bright lights is the same as that of digital lights. They are only on for a short time in a complete signal cycle, and the detection time can be increased by one cycle. Long-term very bright lights are special configurations. In order to follow the changes in the signal scheme, they are configured to be enabled and disabled. The enabled scheme will light up normally, and the disabled state is fixedly off. It is necessary to access the actual signal scheme to determine whether detection is required. Due to the existence of the preset position, as long as this light is not enabled, this preset position is removed and does not participate in the detection. The detection strategy for the very bright light to be inspected set specifically includes:

[0067] 1. Access the real-time signal control plan information of the intersection where the video is located, and dynamically update the activation status of the corresponding preset position when the plan changes;

[0068] 2. Read the video stream of the extremely bright lights to be inspected, and capture a single image every 5 seconds, with a maximum of 3 minutes;

[0069] 3. Use the recognition model to identify the target in a single image and obtain all the recognition boxes;

[0070] 4. Match with the dynamically updated preset positions. a. When all matched detection frames are in a non-off state (red, yellow, green), the detection ends and the video stream is judged to be fault-free; b. If one or more matched detection frames are in an off state, continue to wait for the next detection;

[0071] 5. It is only necessary to detect once that all lights are not off before stopping the detection. It is not necessary to complete all the detections. If the light is still off after all the detections, it is determined to be a fault and a fault alarm is issued.

[0072] In the second aspect, the present invention also provides a signal light fault detection system based on video recognition, which is characterized in that it consists of a signal light status image data acquisition unit, a signal light color status recognition model training unit, a signal light preset position setting and matching unit and a variety of recognition scenarios and fault detection strategy units.

[0073] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0074] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A signal light fault detection method based on video recognition, characterized in that: The method comprises the step S1 of collecting image data of the state of a signal light, the step S2 of training a model for recognizing the color state of a signal light, the step S3 of setting and matching preset positions of the signal light, and the step S4 of multiple recognition scenarios and fault detection strategies.

2. The signal light fault detection method based on video recognition according to claim 1, characterized in that: The step S1 of collecting the signal light status image data specifically includes the following steps: Step S11: Classify video image acquisition and construct original data set; Step S12: Screening and expanding video images to construct a training data set; The step S2 of signal light color state recognition model training specifically includes the following steps: Step S21: data set annotation and data set division; Step S22: target recognition model and training; The step S3 of setting and matching the preset positions of the signal lights specifically includes the following steps: Step S31: signal light preset position setting; Step S32: recognizing the distance calculation of the frame and matching it with the preset position; The step S4 identifies multiple scenarios and fault detection strategies, uses real-time video streams and the signal light preset position data in step S31 to perform polling video detection, and uses the detection results to perform scene judgment; the fault detection strategies include: a set detection strategy for always-on lights to be inspected, a set detection strategy for digital lights to be inspected, and a set detection strategy for very bright lights to be inspected.

3. The signal light fault detection method based on video recognition according to claim 2 is characterized in that: Step S11, based on the camera of the existing traffic facility electronic police checkpoint, obtain the real-time video stream, and intercept the images containing the operating status of the traffic lights from the video stream at different frequencies at different times of the day and save them to form an original data set.

4. The signal light fault detection method based on video recognition according to claim 2 is characterized in that: Step S12, filtering out invalid graphics in the original data set, and expanding some images in the original data set using an image enhancement method of geometric transformation and noise injection to form a training data set.

5. The signal light fault detection method based on video recognition according to claim 2 is characterized in that: In step S22, the target recognition model uses YOLOv10 as a pre-trained model.

6. The signal light fault detection method based on video recognition according to claim 2, characterized in that: Step S31, use a marking tool to mark the preset positions of the traffic lights, mark each channel of video to be inspected once, and obtain the preset position data of the corresponding video stream. The labels of the preset positions of the traffic lights are divided into color lights, arrow lights, and digital lights.

7. The signal light fault detection method based on video recognition according to claim 2 is characterized in that: Step S32, obtaining a matching result between the recognition frame and the preset position annotation frame by calculating the distance between the recognition frame and the preset position annotation frame in the target detection result.

8. A signal light fault detection system based on video recognition, characterized in that: It consists of a traffic light status image data acquisition unit, a traffic light color status recognition model training unit, a traffic light preset position setting and matching unit, and a variety of recognition scene and fault detection strategy units.

Images