An intelligent traffic signal fault identification method based on video stream

Abstract

The application discloses a kind of intelligent traffic signal lamp fault identification methods based on video stream, comprising the following steps: first, read electronic police video stream, collect various actual scene intersection electronic police video stream;Then the video stream of the four directions of "east, south, west and north" is transversely spliced and framed, and dataset is made;Signal lamp state in dataset is manually marked using LabelImg software;YOLOv5 model is improved after network structure, and model training is carried out;Finally, the identification and fault determination of traffic signal lamp are completed.The application effectively solves the problem of low efficiency and high false alarm rate of manual inspection of signal lamp fault, improves the accuracy of small-scale traffic signal lamp fault detection, and constructs a real-time traffic signal lamp fault detection system to ensure that traffic signal lamp failure can be detected and handled as soon as possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a smart traffic signal lamp fault identification method based on video stream. BACKGROUND

[0002] In urban traffic management, traffic signal lights play a crucial role in controlling and guiding traffic flow. Once an error occurs, it will have a huge impact on traffic safety and urban traffic efficiency. However, the number of urban traffic signal lights is large and widely distributed, and manual inspection is inefficient, making real-time monitoring of traffic signal light faults difficult and affecting the normal operation of traffic. How to accurately and efficiently detect the fault state of traffic signal lights is an urgent problem that needs to be solved in current urban traffic management. Therefore, constructing a real-time video stream traffic signal light fault detection system to ensure that traffic signal lights can be discovered and handled as soon as possible when they fail is the most effective strategy to improve road traffic efficiency and the travel experience of citizens.

[0003] Currently, traffic signal light fault detection mainly relies on electronic circuit detection equipment combined with manual observation to discover faults. The electronic circuit detection method requires the design of a traffic signal light fault detection circuit, which includes detecting the voltage and current of the signal light inside and the output of the signal machine. The disadvantage is that the maintenance and replacement cost of the urban traffic signal light module is very high, including remote maintenance and support work for the addition of detection modules at intersections. And the detection method through the output of the signal machine to judge the signal light state needs to adapt to different power supply types and power types of signal lights, with high complexity, and changes in the external power supply environment will also affect the accuracy of detection.

[0004] The algorithm for traffic signal light detection and recognition has developed from traditional algorithms based on handcrafted features to current widely used deep neural network-based algorithms. From the OverFeat algorithm proposed in 2013, to the R-CNN algorithm, the Fast RCNN and YOLO algorithms proposed in 2015, and the subsequent SSD algorithm, Faster RCNN, YOLOv2 and YOLOv3 algorithm, and the current YOLOv5 algorithm. In just a few years, in terms of network structure, from bottom-up to top-down, from two-stage to one-stage, a large number of algorithms with good detection performance and recognition effect have been produced. However, due to the small size of the signal light detection target, the existing research methods have low recognition accuracy, and even in the actual road scene with good light and clear vision, there are still obvious cases of missing detection and false detection of small-scale traffic signal lights. Therefore, how to improve the accuracy of small-scale traffic signal light fault detection has become an important direction of current research. SUMMARY

[0005] In order to solve the problems of low detection accuracy, poor real-time performance and high cost of manual inspection of small-scale target detection and recognition technology, the application provides a traffic signal lamp fault detection method based on video stream, which replaces the traditional electronic circuit fault detection method by combining computer vision and deep learning technology, significantly improves the work efficiency, reduces the cost of manual inspection and maintenance, and introduces the attention mechanism SE module into the YOLOv5 network and the focal loss function FocalLoss into the training process of the model, improves the accuracy of small-scale traffic signal lamp fault detection, and ensures that the traffic signal lamp fault is identified in the first time, so as to improve the road passing efficiency and the citizen travel experience.

[0006] In order to achieve the above purpose, the specific technical scheme of the application is as follows:

[0007] Step 1, read the electronic police video stream:

[0008] The electronic police video stream is read to obtain the video stream required for data set production.(Including the signal lights in the four directions of "east, south, west and north")

[0009] Step 2, data set production: according to the signal lamp video stream of the four directions of "east, south, west and north" of the multiple intersections of the electronic police obtained in step 1, using Python script, the four direction video streams are spliced, the splicing function is defined as "videopj", and then using FFmpeg to frame and intercept the spliced mp4 video, so that the data set is obtained, the data set has 2025 images, and the image resolution is 1280*720.

[0010] Step 3, improve the YOLOv5 network model, and identify the signal lamp based on the model: introduce the SE-Net attention mechanism network, embed the SE module into the Backbone, Neck and Head three network modules of the YOLOv5 network respectively, and generate three new network models SEA_YOLOv5s, SEB_YOLOv5s and SEC_YOLOv5s. In SEA_YOLOv5s, the SE module is embedded in the BottleneckCSP structure of the Backbone network, and the specific embedding position is between the output end of the Conv convolution feature fusion Concat and the input end of the BN; in SEB_YOLOv5s, the SE is embedded in the Neck, and the SE is embedded between the output end of each Contact module and the input end of BottleneckCSP2_1; in SEC_YOLOv5s, the SE is embedded in the Head, and the specific embedding position is between the output end of the BottleneckCSP2_1 of the Neck network and the input end of the YOLO detection head convolution of the Head network.

[0011] Step 4, improve the loss function of YOLOv5 network model, introduce focus loss function, including the weight control of positive and negative samples, the weight control of easy and difficult classification samples, and the combination of the two weight control methods.

[0012] Step 5, training and verification of improved YOLOv5 network model: based on YOLOv5 model and pre-training weight yolov5s.pt, create model configuration file and modify training required hyperparameters, train traffic signal training set, finally get the weight file best.pt with the highest model detection accuracy and recall rate.

[0013] Step 6, configure the weight file best.pt in YOLOv5 model for signal recognition and fault judgment.

[0014] As preferred, step 2 includes: step 2-1, after video stream key frame sampling, use LabelImg software to label and preprocess traffic signal lamps in 2025 images. In this process, the four working states of green light, red light, yellow light and extinguishing in the data set pictures need to be labeled to improve the discrimination ability of the model and ensure the accuracy of the recognition effect.

[0015] Step 2-2, divide the labeled data set into training set and validation set according to the proportion. This step is to provide more accurate initial training data for the model and use the validation set to evaluate the model. In addition, when labeling the data set, attention should be paid to the fact that the labeled area should be as close to the edge of the signal lamp panel as possible, which can avoid too much background interference.

[0016] As preferred, step 3 includes:

[0017] Step 3-1, development environment configuration: build development environment, including operating system, CUDA version, CuDNN, Python version, deep learning framework, GPU, CPU, memory and hard disk, etc., set up high-performance device deployment scheme.

[0018] Step 3-2, model improvement:

[0019] YOLOv5 adopts One-Stage structure, which is composed of four modules, namely Input input end, Backbone main network, Neck network layer and Head output end. With the deepening of network level, the extracted information becomes more and more abstract, and it is more difficult to detect small target traffic signal lamps in monitoring images, which is difficult to accurately detect.

[0020] Basic components of YOLOv5:

[0021] Focus: Follow the passthrough of YOLOv2 network.

[0022] CBL: composed of Conv + Bn + LeakyReLU activation function.

[0023] CSP1_X: based on the CSPNet network, composed of three convolutional layers and X Res Unint modules Concate.

[0024] CSP2_X: cancel the Res Unint module and use the CBL module instead.

[0025] SPP: adopts the maximum pooling method of 1x1, 5x5, 9x9 and 13x13 for multi-scale fusion

[0026] Based on the performance advantages of YOLOv5, improvements are made to improve the accuracy of signal light detection in urban road traffic, while reducing the false detection and missed detection rate.

[0027] SE-Net is a typical channel attention network, and the effectiveness of the SE module has been verified in some applications, but there is no complete theoretical explanation of which position in the network is more effective when embedding it.

[0028] The present application designs three network models with SE modules for YOLOv5 to form YOLOv5s. YOLOv5 contains four modules, the Input input end mainly performs image preprocessing, and does not have any feature extraction operation, so it does not need to introduce an attention mechanism. Therefore, the SE module is embedded in the Backbone, Neck and Head network modules, respectively, and three new network models are produced, which are SEA_YOLOv5s, SEB_YOLOv5s and SEC_YOLOv5s.

[0029] As preferred, in step 4, the loss function of the original YOLOv5 includes the target loss function and the classification loss function, and the bounding box regression loss function. The present application improves the loss function on the basis of the original loss function, and proposes a focal loss function (Focal Loss). Specifically:

[0030] YOLOv5 uses BCE With Logits as the target loss function and the classification loss function, and the calculation formula of the binary classification cross-entropy loss function is shown in formula (1). In the formula, y is the real sample label; p is the predicted output after the Sigmoid function, as shown in formula (2), and the value is between [0, 1].

[0031]

[0032]

[0033] The GIoU is used as a bounding box regression loss function, where the IoU is the intersection over union of the real box and the predicted box, and the calculation formula of the loss function is shown in equation (3).

[0034]

[0035] IoU loss If two detection boxes do not intersect, the IoU is 0, and the coincidence degree of the two boxes cannot be directly reflected, there is no gradient back propagation, and it cannot be learned and trained. The GIoU is improved based on the IoU loss function, and the calculation formula is shown in equation (4).

[0036]

[0037] In the formula, A c is the minimum area of the two boxes.

[0038] On the basis of the original loss function, the loss function is improved, and a loss calculation scheme for controlling unbalanced samples by weight is proposed, i.e. focal loss. It mainly includes three steps:

[0039] ① Control the weight of positive and negative samples.

[0040] The positive sample refers to the generated candidate box with the target, and the negative sample refers to the generated candidate box without the target. Taking a typical case of binary cross-entropy of cross-entropy as an example, see equation (2). To reduce the influence of negative samples on the basis of this function, a coefficient α t is added before the cross-entropy loss, see equation (5), which is in the range of [0, 1]. Combine α t with the BCE(p, y) formula to form equation (6).

[0041]

[0042]

[0043] By setting α, the contribution of positive and negative samples to the loss is controlled: when α is in the range of [0, 0.5], the loss weight of positive samples is reduced, and the loss weight of negative samples is increased; when α is in the range of (0.5, 1], the loss weight of positive samples is increased, and the loss weight of negative samples is reduced.

[0044] ② Control the weight of easy and difficult classification samples. If sample 1 and sample 2 belong to the target category, the probability of sample 1 and sample 2 is 0.9 and 0.6 respectively, obviously sample 2 is more difficult to classify than sample 1, therefore, sample 2 is called difficult classification sample. To control these two types of samples, multiply (1-p) γ or pγ This factor is called the modulation factor (Modulating Factor), γ, which takes values in the range [0, 5], see equation (7).

[0045]

[0046] The change of the modulation factor is achieved by adjusting the size of the index γ, and the size of the modulation factor is used to control the weight size of the two types of samples. When γ = 0, it is a standard binary cross-entropy loss function; when 0 < γ ≤ 5, it can control the weight of easy-to-classify samples on the loss, reduce the influence of such samples, and make the model pay more attention to difficult-to-classify samples.

[0047] ③ Merge the two weight control methods.

[0048] By merging the formulas obtained by the two methods, FocalLoss(p, y, α, γ) is obtained, see equation (8)

[0049]

[0050] On the basis of the binary cross-entropy loss function, a balance factor α is added to balance the number of positive and negative samples in a single-level structure; a modulation factor is added to control the difficulty of sample classification; the two parameters α and γ are coordinated to control, which can control the unbalanced samples through FocalLoss, and output a smaller loss.

[0051] As a preferred, the pre-processed data set is used for model training, and the improved YOLOv5 model is used for training. Based on the YOLOv5 model and the pre-training weight, the model configuration file is created and the hyperparameters required for training are modified, and the traffic signal training set is trained, and finally the weight file with the highest model detection accuracy and recall rate is obtained.

[0052] The pre-training weight yolov5s.pt is trained on a large-scale data set with supervised training, so that the network can effectively extract discriminative features from the input image. The pre-training weight contains rich knowledge and patterns obtained on a large-scale data set. These knowledge can be transferred to new tasks through transfer learning. In YOLOv5, by loading the pre-training weight, the network can utilize the previously learned knowledge to speed up the convergence and avoid training from scratch.

[0053] As preferred, the hyperparameters required before the start of training are modified: epoch, batch_size, initial anchor size, and a model dataset configuration file is created, which specifies the absolute path of the traffic signal picture training set and the validation set, and sets the number of classes to 2, the class names to [‘red’,‘green’,‘yellow’,‘black’], and uses the prepared dataset to train the improved YOLOv5 model. Through repeated iterative training, the model gradually improves its accuracy and generalization ability.

[0054] As preferred, the trained YOLOv5 model is verified using the validation set, and the accuracy and recall rate of the output results are comprehensively evaluated. This step is to verify the effect of the model and find out possible defects and deficiencies, so as to further optimize.

[0055] As preferred, step 6 includes:

[0056] Step 6-1, the signal light has clear shape and color characteristics, and the characteristics are fixed and obvious, wherein the shape mainly includes circle and arrow, and the color mainly includes red, green, yellow and black (extinguished).

[0057] Considering that the combination of light colors of the signal light in the normal working state is too complex, especially when the signal light is in the transition period, the light color is unstable and may appear a short "flicker", which is difficult to determine, therefore, the present application does not consider various situations in the normal working state, but only identifies the fault state.

[0058] The signal light fault types include "extinguished" fault and "conflict" fault, and the "extinguished" fault includes "red light extinguished", "green light extinguished", "single light panel extinguished" and "all extinguished". The "conflict" fault includes "green conflict", "red conflict", "yellow conflict", "red-green conflict", "red-yellow conflict", "yellow-green conflict", etc.

[0059] Since the transient state (such as light extinguished) when the signal light fails and the state when the signal light is in the normal signal transition period overlap, the result identified by the signal light needs to be observed in a long enough period to determine whether a real fault has occurred.

[0060] Step 6-2, the traffic signal light group includes red light, yellow light, green light and extinguished; the red light indicates prohibition of traffic, the green light indicates permission of traffic, the yellow light indicates warning, and the extinguished light presents black color;

[0061] Take the length of a complete signal light cycle at the current intersection as the benchmark, and count the red light, green light, yellow light and extinguished light appearing in a signal light cycle;

[0062] If the red light counted in the signal light cycle does not light up, there is a red light extinguishing failure;

[0063] If the green light counted in the signal light cycle does not light up, there is a green light extinguishing failure;

[0064] If the yellow light counted in the signal light cycle does not light up, there is a yellow light extinguishing failure;

[0065] If the red light, green light and yellow light of the single lamp plate counted in the signal light cycle do not light up, there is a single lamp plate extinguishing failure;

[0066] If all the red light, green light and yellow light counted in the signal light cycle do not light up, there is a total extinguishing failure;

[0067] If the time t of the red light counted in the signal light cycle is greater than T, T is a threshold time, then a red conflict failure is determined;

[0068] If the time t of the green light counted in the signal light cycle is greater than T, T is a threshold time, then a green conflict failure is determined;

[0069] If the time t of the yellow light counted in the signal light cycle is greater than T, T is a threshold time, then a yellow conflict failure is determined;

[0070] If the time t of the red light and yellow light counted in the signal light cycle is greater than T, then a red-yellow conflict failure is determined;

[0071] If the time t of the red light and green light counted in the signal light cycle is greater than T, then a red-green conflict failure is determined;

[0072] If the time t of the yellow light and green light counted in the signal light cycle is greater than T, then a yellow-green conflict failure is determined.

[0073] The beneficial effects of the present application are:

[0074] The present application effectively solves the problems of low efficiency and high false positive rate of manual inspection of signal light failures, improves the accuracy of small-scale traffic signal light failure detection, and constructs a real-time traffic signal light failure detection system to ensure that traffic signal light failures can be discovered and handled as soon as possible.

[0075] The application applies a YOLOv5 model to traffic signal lamp fault detection. Information is mined in existing electronic police system snapshot data, and the running state of the signal lamp is extracted, which ingeniously realizes the required function. The application analyzes and improves the feature extraction network and the loss function, so as to improve the recognition accuracy and reduce the occurrence of small-scale traffic signal lamp missed detection. The attention mechanism SE module is introduced into the Backbone, Neck and Head of the YOLOv5 network respectively, and the best position of the introduction of the SE module is obtained through comparison. In view of the imbalance problem of positive and negative samples existing in the YOLOv5 model, the focal loss function Focal Loss is introduced into the training process of the model. The test shows that after the SE module is fused with the Backbone, the average accuracy mAP of the model is improved by 0.03%, and after the introduction of the Focal Loss, the mAP is improved by 0.012%, and the accuracy Precision and the recall rate Recall are also improved.

[0076] The test on the self-built data set shows that the improved YOLOv5 model can effectively improve the recognition of the working state of the signal lamp in the traffic monitoring scene. Through modification of the detect.py program of the YOLOv5 model, the discrimination results of the four-way signal lamp at the intersection are counted and logically analyzed, which can effectively judge the fault state of the signal lamp, including lamp failure or conflict anomaly and other information. In addition to the advantages of high precision and real-time, this method also has the advantages of strong robustness, good adaptability and good scalability, which provides a beneficial reference for future traffic management and artificial intelligence technology research. BRIEF DESCRIPTION OF DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiments will be briefly introduced as follows.

[0078] Figure 1 is a flow chart of an intelligent traffic signal lamp fault identification method based on video stream according to an embodiment of the application;

[0079] Figure 2 is an example diagram of a data set according to an embodiment of the application;

[0080] Figure 3 is an operation interface diagram of LabelImg labeling software according to an embodiment of the application;

[0081] Figure 4 is an architecture diagram of YOLOv5 according to an embodiment of the application;

[0082] Figure 5 is a network structure diagram of the improved SEA_YOLOv5s according to an embodiment of the application;

[0083] Figure 6 This is a structural diagram of the SE_BottleneckCSP according to an embodiment of the present invention;

[0084] Figure 7 This is an improved SEB_YOLOv5s network structure diagram according to an embodiment of the present invention;

[0085] Figure 8 This is an improved SEC_YOLOv5s network structure diagram according to an embodiment of the present invention;

[0086] Figure 9 This is a diagram showing the detection results of a traffic signal red conflict fault according to an embodiment of the present invention. Detailed Implementation

[0087] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] like Figure 1 As shown in the flowchart, the intelligent traffic light fault identification method based on video stream includes four main steps: first, reading the video stream from the electronic police, then horizontally stitching the video stream, then creating a dataset, then training and improving the YOLOv5 model, and finally completing the identification and fault determination of the traffic light.

[0089] (1) Read the video stream of the electronic police system

[0090] (1-1) Research and analysis show that electronic police systems are now installed at major traffic intersections in cities. During the capture and evidence collection process of these systems, traffic lights at intersections are always within the system's recording range and are considered one of the relevant enforcement elements. This invention utilizes the video stream resources of existing electronic police systems, cleverly avoiding the inherent problems of traditional traffic light fault detection systems, and also enabling real-time monitoring of the operational status of traffic lights.

[0091] (1-2) Collect video streams of electronic police under various actual scenarios, including night and day, different weather, different lighting, and some distorted video streams, a total of 50 videos (including traffic lights in the four directions of "east, south, west and north"), each video is about 5 minutes long, and the key frames of each video are sampled to obtain a total of 2025 images with an image resolution of 1280×720.

[0092] (2) Dataset creation

[0093] (2-1) After the key frame sampling of the video stream, the LabelImg software is used to label and preprocess the traffic signal lights in 2025 images. During this process, the green, red, yellow, and extinguished working states in the dataset pictures need to be labeled to improve the model's discrimination ability and ensure the accuracy of the recognition effect.

[0094] Specifically, the features of the signal lights are fixed and obvious, and the shapes mainly include circular and arrow-shaped, while the color states of the signal lights mainly include red, green, yellow, and black. The xml format file obtained after labeling is converted to a txt format file through a Python script, and it is ensured that the txt file name corresponds to the sample name in the dataset. The final dataset contains 2025 pictures, of which the training set contains 1620 pictures and the validation set contains 405 pictures.

[0095] (2-2) The labeled dataset is divided into training set and validation set according to the proportion. This step is to provide more accurate initial training data for the model and use the validation set to evaluate the model. In addition, when labeling the dataset, attention should be paid to the fact that the labeled area should be as close to the edge of the signal light panel as possible, so as to avoid too much background interference.

[0096] (3) Signal light recognition

[0097] (3-1) Development environment configuration

[0098] The development environment is built, including the operating system, CUDA version, CuDNN, Python version, deep learning framework, GPU, CPU, memory, and hard disk, etc. The high-performance device deployment scheme is set up.

[0099] (3-2) Model improvement

[0100] (3-2-1) Attention mechanism

[0101] YOLOv5 adopts One-Stage structure and is composed of four modules, namely Input input end, Backbone main network, Neck network layer, and Head output end. As the network level deepens, the information extracted becomes more and more abstract, making it more difficult to detect small target traffic signal lights in the monitoring image, and it is difficult to accurately detect.

[0102] Based on the performance advantages of YOLOv5, improvements are made to improve the accuracy of signal light detection in urban road traffic and reduce the false detection and missed detection rates.

[0103] SE-Net is a typical channel attention network, and the effectiveness of the SE module has been verified in some applications, but there is no complete theoretical explanation of which position in the network is more effective when embedding it.

[0104] The present application designs three network models with SE modules for YOLOv5 to form YOLOv5s. YOLOv5 contains four modules, the Input input end mainly performs image preprocessing, and does not have any feature extraction operation, so it does not need to introduce an attention mechanism. Therefore, the SE module is embedded in the Backbone, Neck and Head network modules respectively, and three new network models produced are respectively denoted as SEA_YOLOv5s, SEB_YOLOv5s and SEC_YOLOv5s.

[0105] Among them, SEA_YOLOv5s is a network formed by embedding the SE module in Backbone. The main role of Backbone in YOLOv5 is to learn the high, medium and low level features in the data.

[0106] SEB_YOLOv5s is a network formed by embedding SE in Neck. The Neck network layer mainly includes the PANet structure for strengthening information transmission, and the SE module is embedded in the feature fusion stage, that is, after Concat ends and before BottleneckCSP2_1 starts. SEC_YOLOv5s is a network formed by embedding SE in Head. The Head output end is used for the final target detection, and the SE module is embedded before each YOLO detection head of the model.

[0107] (3-2-2) Loss function

[0108] The loss function of the original YOLOv5 includes the target loss function and the classification loss function, and the bounding box regression loss function.

[0109] YOLOv5 uses BCE With Logits as the target loss function and the classification loss function, and the calculation formula of the binary classification cross-entropy loss function is shown in formula (1). In the formula, y is the real sample label; p is the predicted output after the Sigmoid function, see formula (2), and the value is between [0, 1].

[0110]

[0111]

[0112] GIoU is used as the bounding box regression loss function, where IoU is the intersection over union of the real box and the predicted box, and the calculation formula of the loss function is shown in formula (3).

[0113]

[0114] IoU loss If two detection boxes have no intersection, IoU = 0, the degree of coincidence of the two boxes cannot be directly reflected, there is no gradient back propagation, and it cannot be learned and trained. GIoU is based on the improvement of IoU loss function, and the calculation formula is shown in equation (4).

[0115]

[0116] In the formula, A c is the minimum area of the two boxes.

[0117] On the basis of the original loss function, the loss function is improved, and the loss calculation scheme of focal loss is proposed to control the loss of unbalanced samples by weight. It mainly includes three steps:

[0118] (4) Weight control of positive and negative samples.

[0119] Positive samples refer to candidate boxes with targets, and negative samples refer to candidate boxes without targets. Taking a typical case of binary cross-entropy of cross-entropy as an example, see equation (2). To reduce the influence of negative samples based on this function, a coefficient α t is added before the cross-entropy loss, see equation (5), which ranges from [0, 1]. Combine α t with the BCE(p, y) formula to form equation (6).

[0120]

[0121]

[0122] By setting α, the contribution of positive and negative samples to the loss is controlled: when α is set in [0, 0.5], the loss weight of positive samples can be reduced, and the loss weight of negative samples can be increased; when α is set in (0.5, 1], the loss weight of positive samples can be increased, and the loss weight of negative samples can be reduced.

[0123] (5) Weight control of easy and difficult classification samples. If sample 1 and sample 2 belong to the target category, the probability of sample 1 and sample 2 is 0.9 and 0.6 respectively, obviously sample 2 is more difficult to classify than sample 1, therefore, sample 2 is called difficult classification sample. To control these two types of samples, multiply (1-p) γ or p γ by BCE(p, y), this factor is called modulating factor, γ takes value in the range of [0, 5], see equation (7).

[0124]

[0125] The change of the modulation coefficient is realized by adjusting the size of the index γ, and the weight size of the two types of samples is controlled by the size of the modulation coefficient. When γ=0, it is a standard binary cross-entropy loss function; when 0<γ≤5, the weight of easy-to-classify samples on the loss can be controlled, the influence of such samples is reduced, and the model pays more attention to difficult-to-classify samples.

[0126] (6) The two weight control methods are combined.

[0127] The formula obtained by combining the two methods is FocalLoss(p,y,α,γ), as shown in formula (8)

[0128]

[0129] On the basis of the binary cross-entropy loss function, a balance factor α is added to balance the number of positive and negative samples in a single-level structure; a modulation coefficient is added to control the difficulty of sample classification; the two parameters α and γ are coordinated to control, and the control of unbalanced samples can be realized through FocalLoss, and a smaller loss is output.

[0130] (3-3) Model training

[0131] The improved YOLOv5 model is used for training by using the pretreated data set for model training. Based on the YOLOv5 model and the pre-training weight yolov5s.pt, a model configuration file is created, the hyperparameters required for training are modified, and the traffic signal training set is trained, and finally the weight file best.pt with the highest model detection accuracy and recall rate is obtained.

[0132] In the target detection task, the pre-training weight serves as an initialization parameter, which can help the network better adapt to the new task. By loading the pre-training weight, the network can obtain the initial target detection ability from the pre-learned features in the initial stage. Then, in further training, the network will be fine-tuned according to the new task data, gradually optimizing the weights to better adapt to the target detection task.

[0133] The pre-training weight in YOLOv5 is obtained by supervised training on a large-scale dataset, which has rich feature extraction ability and pattern recognition knowledge. Loading the pre-training weight can help the network quickly obtain the target detection ability, and through fine-tuning, further optimize the network's performance on new tasks. Pre-training weight plays an important role in accelerating the training process, improving detection performance and generalization ability.

[0134] The embodiment modifies the hyperparameters required before the start of training: epoch, batch_size, initial anchor box size, and creates a model dataset configuration file that specifies the absolute paths of the traffic signal picture training set and the validation set, sets the number of classes to 2, sets the class names to [‘red’,‘green’,‘yellow’,‘black’], and uses the prepared dataset to train the improved YOLOv5 model. According to the changes in the evaluation indicators, adjust the hyperparameters, adjust the target box size, and test on the self-built dataset. Through repeated iterative training, the model gradually improves its precision and generalization ability. The final detection model with the best precision is to fuse the SE module with the Backbone, and the average accuracy mAP of the model is improved by 0.03%, and after introducing FocalLoss, mAP is improved by 0.012%, and the accuracy Precision and recall rate are also improved.

[0135] (3-4) Model verification

[0136] The trained YOLOv5 model is verified using the validation set, and the accuracy and recall rate of the output results are comprehensively evaluated. This step is to verify the effect of the model and find out possible defects and shortcomings for further optimization.

[0137] (4) Signal light fault determination

[0138] (4-1) The signal light has clear shape and color characteristics, and the characteristics are fixed and obvious, including round and arrow shapes, and red, green, yellow and black (extinguished) colors.

[0139] Considering that the combination of light colors of the signal light in the normal working state is too complex, especially when the signal light is in the transition period, the light color is unstable, and there will be a short “flicker”, the determination is difficult, therefore, the present application does not consider various situations in the normal working state, but only identifies the fault state.

[0140] The signal light fault types include “extinguished” fault and “conflict” fault. The “extinguished” fault includes “red light extinguished”, “green light extinguished”, “single light panel extinguished” and “all extinguished”. The “conflict” fault includes “green conflict”, “red conflict”, “yellow conflict”, “red-green conflict”, “red-yellow conflict”, “yellow-green conflict” and the like.

[0141] Because the transient state (such as light extinguishing) when the signal light fails and the state when the signal light is in the normal signal transition period exist overlap, the result identified by the signal light needs to be put into a period long enough to see whether a real failure occurs.

[0142] (4-2) The traffic signal light group includes red light, yellow light, green light and extinguishing; the red light indicates prohibition of passing, the green light indicates permission of passing, the yellow light indicates warning, and the extinguishing light presents black color;

[0143] According to the length of a complete signal light period at the current intersection as a reference, the red light, green light, yellow light and extinguishing light appearing in a signal light period are counted;

[0144] If the red light counted in the signal light period does not light, there is a red light extinguishing failure;

[0145] If the green light counted in the signal light period does not light, there is a green light extinguishing failure;

[0146] If the yellow light counted in the signal light period does not light, there is a yellow light extinguishing failure;

[0147] If the red light, green light and yellow light of the single light plate counted in the signal light period do not light, there is a single light plate extinguishing failure;

[0148] If all the red light, green light and yellow light counted in the signal light period do not light, there is a total extinguishing failure;

[0149] If the red light counted in the signal light period lights at the same time t>T, T being a threshold time, it is determined that there is a red conflict failure;

[0150] If the green light counted in the signal light period lights at the same time t>T, T being a threshold time, it is determined that there is a green conflict failure;

[0151] If the yellow light counted in the signal light period lights at the same time t>T, T being a threshold time, it is determined that there is a yellow conflict failure;

[0152] If the red light and yellow light counted in the signal light period light at the same time t>T, it is determined that there is a red-yellow conflict failure;

[0153] If the red light and green light counted in the signal light period light at the same time t>T, it is determined that there is a red-green conflict failure;

[0154] If the yellow light and green light counted in the signal light period light at the same time t>T, it is determined that there is a yellow-green conflict failure.

[0155] As Figure 2As shown, the example graph of the dataset contains the signal lights of the four directions of the intersection "East, South, West, North", the upper left is North, the upper right is South, the lower left is West, and the lower right is East.

[0156] The process of dataset making:

[0157] Get the video stream of the signal lights of the four directions of the intersection electronic police, set the same start and end time and intercept the video stream. Generally, the cycle time of most intersection signal lights is not more than 5 minutes, so 5 minutes are uniformly intercepted to ensure that all possible states of the signal lights are included in the video stream, and a total of 50 videos are obtained.

[0158] Use Python script to splice the video streams of the four directions, define the function as "videopj", and the input of the four video streams is videoLeftUp.read(), videoLeftUp.read(), videoRightUp.read(), and videoRightDown.read(). Then use FFmpeg to extract frames from the spliced mp4 video to obtain the dataset. The dataset contains a total of 2025 images, and the image resolution is 1280x720.

[0159] As shown in Figure 3 , use LabelImg software to label the signal light states in the dataset. The signal lights have clear shape and color characteristics, and the characteristics are fixed and obvious. The shape mainly includes circle and arrow, and the color state of the signal light mainly includes red, green, yellow, and black. Convert the xml format file obtained after labeling to txt format file through Python script, and ensure that the txt file name corresponds to the sample name in the dataset. The final dataset contains 2025 images, and the training set and the validation set are divided according to 8:2, of which the training set contains 1620 images and the validation set contains 405 images.

[0160] As shown in Figure 4 , the network structure diagram of YOLOv5 model. The whole YOLOv5 can be divided into three parts, namely Backbone, Neck and Head. YOLOv5 series is divided into s, m, l, x four models according to the different model parameters (Params) and floating point operation amount (FLOPs), and the Params and FlOPs of the models from s to x are increasing.

[0161] The traffic monitoring site is mainly embedded equipment with low computing power, so YOLOv5s with the least Params and FLOPs is used as the basic model for research.

[0162] Backbone is the backbone feature extraction network of YOLOv5. According to its structure (Focus structure + CSP structure), the input picture will first be feature-extracted in CSPDarknet. The extracted features can be called feature layers, which are a set of features of the input picture. In the main part, three feature layers are obtained for the construction of the next network. These three feature layers are called effective feature layers.

[0163] Neck is the enhanced feature extraction network of YOLOv5. According to its structure (FPN + PAN structure), the three effective feature layers obtained in the main part will be feature-fused in this part. The purpose of feature fusion is to combine feature information of different scales. In the Neck part, the effective feature layers obtained are used to continue to extract features. In YOLOv5, the structure of Panet is still used. Not only will the features be up-sampled to realize feature fusion, but also the features will be down-sampled again to realize feature fusion.

[0164] Head is the classifier and regressor of YOLOv5. Through CSPDarknet and FPN, we have obtained three enhanced effective feature layers. Each feature layer has width, height and channel number. At this time, we can regard the feature map as a set of feature points, each of which has channel number of features. The actual work of Head is to judge whether the feature points have objects corresponding to them. Like the previous version of Yolo, the decoupled head used by YOLOv5 is together, that is, classification and regression are realized in a 1x1 convolution. Therefore, the work of the entire YoloV5 network is feature extraction-feature enhancement-prediction of the object corresponding to the feature point.

[0165] Since the SE attention mechanism itself enhances important feature channels, the SE module is embedded in the Backbone, Neck, and Head network modules respectively, resulting in three new network models, SEA_YOLOv5s, SEB_YOLOv5s, and SEC_YOLOv5s, and comparative experiments are conducted.

[0166] The basic components of YOLOv5 include Focus, CBL, BottleneckCSP1_X, BottleneckCSP2_X, and SPP, where CBL includes "Conv + BN + LeakyReLU activation function"

[0167] As Figure 5As shown in the figure, the improved SEA_YOLOv5s network model. SEA_YOLOv5s is to embed the SE module into the BottleneckCSP structure of the Backbone network to improve the feature extraction ability of the Backbone network.

[0168] As shown in the figure, the improved SEA_YOLOv5s network model. SEA_YOLOv5s is to embed the SE module into the BottleneckCSP structure of the Backbone network to improve the feature extraction ability of the Backbone network. Figure 6 As shown in the figure, the improved SEA_YOLOv5s network model. SEA_YOLOv5s is to embed the SE module into the BottleneckCSP structure of the Backbone network to improve the feature extraction ability of the Backbone network. Figure 6 As shown in the figure, the improved SEA_YOLOv5s network model. SEA_YOLOv5s is to embed the SE module into the BottleneckCSP structure of the Backbone network to improve the feature extraction ability of the Backbone network.

[0169] The main role of Backbone in YOLOv5 is to learn the high, medium and low different levels of features in the data. Backbone is mainly composed of Focus, CBL, BottleneckCSP and SPP, and BottleneckCSP mainly plays the role of feature aggregation.

[0170] The role of Focus: down-sampling and reducing the amount of calculation to speed up the network.

[0171] The role of CBL: CBL is a convolutional block composed of Conv, BN (Batch Normalization) and LeakyReLU. Conv layer is a convolutional layer that uses multiple different convolutional kernels to process the input image to obtain different response feature maps. BN layer is a batch normalization layer, which is a layer of neural network placed before the activation function and after the convolutional layer. When the number of feature maps obtained is m and the size of the feature map is w*h (i.e. the number of image pixels), the data volume of BN is m*w*h. The main operation steps of BN layer are to calculate the mean and variance of all batch data, and then normalize the pixel value by subtracting the mean and dividing by the variance. LeakyReLU function is a variant of Relu function. In Relu function, when the input is negative, the learning speed of ReLU may become very slow, or even make the neuron invalid. In order to solve this defect of Relu function, a leakage (Leaky) value is introduced in the negative half of the Relu function.

[0172] BottleneckCSP function: divided into CSP1_X structure and CSP2_X structure.

[0173] Specifically, CSP1_X structure is to divide the input into two branches, one branch first passes through CBL, then passes through multiple residual structures (Bottleneck*N), and then performs convolution again; the other branch directly performs convolution; then the two branches are concatenated, then pass through BN (normal distribution), and then activate (Leaky function) again, and finally perform a CBL.

[0174] CSP1_X is applied to the Backbone main network part. The Backbone network is relatively deep, and adding residual structures can increase the gradient value of back propagation between layers, avoiding the gradient disappearance caused by deepening, so that more fine-grained features can be extracted and network degradation can be avoided. CSP2_X is different from CSP1_X in that CSP2_X replaces Resunit with 2*X CBL, mainly applied to the shallow Neck network.

[0175] SPP function: SPP structure is also known as spatial pyramid pooling, which can convert a feature map of any size into a fixed-size feature vector. First, the input is an image of any size, with a size of w*h. The output layer has 21 neurons (i.e. 21 features are expected to be extracted).

[0176] Specific process: respectively take the max value of each frame in the 1*1 block, 2*2 block and 4*4 subgraph. This step is to make maximum pooling, so that the extracted feature values (i.e. the maximum values taken out) are 1*1+2*2+4*4=21 in total. The obtained features are concatenated together. Among them, a CBL is added before and after, and the kernel size in the middle is 1*1, 5*5, 9*9 and 13*13 respectively.

[0177] As shown in Figure 7 , it is an improved SEB_YOLOv5s network model.

[0178] SEB_YOLOv5s is a network formed by embedding SE into Neck to enhance the feature fusion capability of Neck network. The Neck network layer mainly includes FPN and PAN structure to enhance information propagation. The SE module is embedded into the feature fusion stage, i.e. after the end of Concat and before the start of BottleneckCSP2_1, a total of 4 places are embedded.

[0179] FPN is a top-down feature pyramid that transmits high-level strong semantic features down, and the entire pyramid is semantically transmitted but not positioning information. In view of this, PAN adds a bottom-up pyramid behind FPN to complement FPN and transmit the up-sampled (Upsample) positioning features of the low layer up.

[0180] Specific process: first copy the bottom layer of the feature pyramid to become the bottom layer of the new feature pyramid. Then perform a down-sampling operation on the bottom layer of the new feature pyramid, and perform a 3*3 convolution on the second-to-last layer of the original feature pyramid with a stride of 2. Then perform a horizontal connection with the down-sampled bottom layer, and add them together. Finally, perform a 3*3 convolution to fuse their features to complete.

[0181] As shown in FIG. 1, it is an improved SEC_YOLOv5s network model. Figure 8

[0182] SEC_YOLOv5s is a network formed by embedding SE into Head to obtain more important feature information. The output end of Head is used for final target detection, and the SE module is embedded between the output end of BottleneckCSP2_1 of Neck network and the input end of YOLO detection head convolution of Head network, a total of 3 places.

[0183] The development environment is built, including operating system, CUDA version, CuDNN, Python version, deep learning framework, GPU, CPU, memory and hard disk, etc. In the deployment scheme of high computing power equipment, NVIDIA GeForce GTX30708GB GPU is used as the training device, and AMD Ryzen 7 5800H CPU is used as the processor.

[0184] Table 1 development environment configuration

[0185]

[0186] The improved three groups of YOLOv5 models are trained and learned in turn, and are built in the above-mentioned development environment.

[0187] Specifically, the model training is performed using epoch=100, batch_size=16, and initial anchor box size=10. During the training process, by analyzing the log file generated during the training process, the accuracy and recall rate and other evaluation indicators of the model during the training process can be obtained.

[0188] ​According to the change of evaluation index, a series of optimization measures are taken, such as adjusting the hyperparameters, adjusting the target frame size, etc., and the test is carried out on the self-built data set, the results show that the improved YOLOv5 algorithm compared with the original algorithm in the index precision, recall, average accuracy mAP are improved, which lays a good foundation for the fault detection of traffic signal lights.

[0189] The detection model with the best accuracy is the fusion of SE module and Backbone, the average accuracy mAP of the model is improved by 0.03%, after introducing Focal Loss, mAP is improved by 0.012%, and the accuracy Precision and recall rate Recall are also improved.

[0190] As shown in Figure 9 , it is the traffic signal red conflict fault detection result graph, the user interface of the system is built by using Qtdesigner and PyQt5.

[0191] After the user logs in the fault identification system, the real-time monitoring video of the four groups of signal lights of the specified intersection can be obtained.

[0192] YOLOv5 identifies the sampled video frame image data, displays and counts the red, green, yellow and off four states, when the signal light works incorrectly, the working state of the intersection will display "abnormal", prompting the staff to report early warning.

[0193] Image data and error logs can be saved to the specified path, which is convenient for users to query and process historical data.

[0194] The red conflict fault identification interface is shown in Figure 9 . When the signal lights of A1 intersection and B1 intersection have red conflict, the working state of the interface is automatically updated to "abnormal", and the fault type is determined as red conflict fault.

[0195] At this time, the user needs to confirm whether the fault determination is accurate, manually clicks the "alarm" button, and the fault log control will automatically generate a record, indicating that the fault has been reported, at this time, the system will automatically record the reporting date, time, fault type and fault description, and automatically create a "fault.csv" table, the csv file is automatically saved to the log saving path set by the user, which is convenient for query.

[0196] The above is only the preferred embodiment of the present application, it should be pointed out that: for ordinary skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should be regarded as the protection scope of the present application.

Claims

1. A method for identifying failure of an intelligent traffic signal based on a video stream, the method comprising: Comprising the following steps: Step 1: read the electronic police video stream, obtain the signal lamp video stream required for making the data set; Step 2: making the data set: according to the video stream obtained in step 1, using Python script to splice the video stream in different directions to obtain the data set; Step 3: improve the YOLOv5 network model, and identify the signal lamp based on the model: introduce SE-Net attention mechanism network, embed SE module into Backbone, Neck and Head three network modules of YOLOv5 network respectively, produce three new network models SEA_YOLOv5s, SEB_YOLOv5s and SEC_YOLOv5s, specifically: In SEA_YOLOv5s, the SE module is embedded in the BottleneckCSP structure of the Backbone network, and the specific embedding position is between the output end of the Conv convolution feature fusion Concat and the input end of the BN; In SEB_YOLOv5s, the SE module is embedded in the Neck network, and the specific embedding position is between the output end of each Contact module and the input end of BottleneckCSP2_1; In SEC_YOLOv5s, the SE module is embedded in the Head network, and the specific embedding position is between the output end of the BottleneckCSP2_1 of the Neck network and the input end of the YOLO detection head convolution of the Head network; Step 4: improve the loss function of the YOLOv5 network model, introduce the focal loss function, including the weight control of positive and negative samples, the weight control of easy and difficult classification samples, and the combination of the two weight control methods; Step 5: training and verifying the improved YOLOv5 network model: based on YOLOv5 model and pre-training weight yolov5s.pt, create model configuration file and modify training required hyperparameters, train traffic signal lamp training set, finally get the weight file best.pt with the highest model detection accuracy and recall rate; Step 6: configure the weight file best.pt in the YOLOv5 model to identify the signal lamp and determine the fault. 2.The intelligent traffic signal fault identification method based on video stream according to claim 1, characterized in that, In step 2, define the splicing function as "videopj", then use FFmpeg to frame and cut the spliced video, use LabelImg software to label the shape features and color features of the signal lamp in the video stream frame pictures, and divide the labeled data set into training set and validation set according to the proportion. 3.The intelligent traffic signal fault identification method based on video stream according to claim 1, characterized in that, The Backbone network is a backbone feature extraction network, which is a Focus+CSP structure. The output end of the Focus module is sequentially connected with four groups of CBL+Bottleneck CSP modules, and the output ends of the Bottleneck CSP modules are connected to the input end of the Neck. 4.The intelligent traffic signal fault identification method based on video stream according to claim 1, characterized in that, The weight control of the positive and negative samples is specifically: a generated candidate frame has a target as a positive sample, otherwise as a negative sample, and a coefficient is added before a loss function , ranging from ; The weight control of the easy-to-classify and difficult-to-classify samples is specifically multiplying the modulation coefficient on the basis of the loss function, and the modulation coefficient is Or , The value range is [0, 5]. The two weight control methods are combined as follows: adding a balance factor on the basis of a loss function to balance the number of positive and negative samples in a single-stage structure; and adding a modulation coefficient to control the classification of difficult and easy samples.

5. The method for intelligent traffic signal fault identification based on video stream according to claim 4, characterized in that, The loss function is a binary cross-entropy loss function, as shown in the following formula: ， For the process The predicted output of the function, The labels are the real sample labels; the weight control of the positive and negative samples is as follows: and The formulas are combined to form the following expression: ； The weight control of the easy classification and the difficult classification sample is as follows: ； The two kinds of weight control methods are combined as follows: 。 6.The intelligent traffic signal fault identification method based on video stream according to claim 1, wherein, Step 5: The improved 3 groups of YOLOv5 models are sequentially trained and learned. Before training, the hyperparameters are modified: epoch, batch_size, initial anchor box size, and a model dataset configuration file is created. The absolute path of the traffic signal picture training set and the validation set is specified, the classification number is set, the classification name is set as [‘red’, ‘green’, ‘yellow’, ‘black’], the improved YOLOv5 model is trained and learned using the prepared dataset, and the trained YOLOv5 model is repeatedly iterated and trained. 7.The intelligent traffic signal fault identification method based on video stream according to claim 1, wherein, The signal light fault types include "extinguishing" faults and "conflict" faults. The "extinguishing" faults include "red light extinguishing", "green light extinguishing", "single lamp panel extinguishing" and "all extinguishing". The "conflict" faults include "green conflict", "red conflict", "yellow conflict", "red-green conflict", "red-yellow conflict" and "yellow-green conflict". 8.The intelligent traffic signal light fault identification method based on video stream according to claim 7, characterized in that, The determination method is as follows: the traffic signal light group includes red light, yellow light, green light and extinguishing; the red light indicates prohibition of traffic, the green light indicates permission of traffic, the yellow light indicates warning, and the extinguishing light presents black color; A complete signal light cycle duration at the current intersection is taken as a reference to count the red light, green light, yellow light and extinguishing light in a signal light cycle; If the counted red light / green light / yellow light in the signal light cycle does not light, there is a red light / green light / yellow light extinguishing fault; If the counted red light, green light and yellow light of the single lamp panel in the signal light cycle do not light, there is a single lamp panel extinguishing fault; If the counted all red light, green light and yellow light in the signal light cycle do not light, there is an all extinguishing fault; If the counted red light in the signal light cycle lights at the same time for a time t>T, T being a threshold time, it is determined that there is a red conflict fault; If the counted green light in the signal light cycle lights at the same time for a time t>T, T being a threshold time, it is determined that there is a green conflict fault; If the counted yellow light in the signal light cycle lights at the same time for a time t>T, T being a threshold time, it is determined that there is a yellow conflict fault; If the counted red light and yellow light in the signal light cycle light at the same time for a time t>T, it is determined that there is a red-yellow conflict fault; If the counted red light and green light in the signal light cycle light at the same time for a time t>T, it is determined that there is a red-green conflict fault; If the time t in which the yellow and green lights are simultaneously on within the signal lamp cycle is counted and t > T, then a yellow-green conflict fault is determined.

Images