Traffic light detection method, system and device based on network flow multi-target tracking

Abstract

The application discloses a signal lamp detection method, system and device based on network flow multi-target tracking, and belongs to the field of image and video processing. First, the detection results of a YOLOv7 target detection neural network are divided into auxiliary signal lamp detection results and direction indicating signal lamp detection results, a minimum cost flow network flow graph is constructed for the auxiliary signal lamp target detection results, a plurality of auxiliary signal lamp trajectories are obtained by solving, then the shape categories of the auxiliary signal lamps in each auxiliary signal lamp trajectory are determined according to the direction indicating signal lamp detection results, and the color categories of the direction indicating signal lamps in the auxiliary signal lamps are detected by using a color detection classification network. Statistics and image color features are used to supplement detection and classification of the direction indicating signal lamps. The application can improve the detection accuracy of signal lamps by using the signal lamp detection results of a video, multi-target tracking and supplementary detection.

Description

Technical Field

[0001] This invention relates to the field of image and video processing, and in particular to a signal light detection method based on network stream multi-target tracking. Background Technology

[0002] Studies have shown that idling and acceleration while waiting for traffic lights at intersections both increase exhaust emissions. Therefore, obtaining information on traffic light transition times is crucial. By analyzing traffic light videos captured by dashcams from multiple vehicles, the red and green states of the lights at various times are identified, and the transition times (i.e., the light on / off times) are determined based on this information. Combining this with road speed limits and other conditions to guide vehicle speeds can effectively reduce the number of stops and starts, lower exhaust emissions, improve traffic flow at intersections, and alleviate traffic congestion.

[0003] Multi-object tracking involves simultaneously locating multiple targets of interest in a given video, maintaining their IDs, and recording their trajectories. Tracking methods can be categorized into online tracking and offline tracking. Object detection refers to the task of detecting the position and category of a specific object; detecting the position and status of traffic lights falls under the category of object detection tasks targeting a specific object category.

[0004] Currently, there are many methods for using object detection algorithms to identify and classify traffic lights. For example, traditional object detection methods can use the HSV color gamut and traffic light variation patterns to label red and green regions in an image, determine candidate regions, and then use a support vector machine (SVM) to classify the image features of the candidate regions. Alternatively, deep learning-based object detectors can be used to detect traffic lights, such as the classic SSD or YOLO algorithms. However, there is currently very little research on using video global information and object tracking fusion information to detect traffic lights in offline videos. Summary of the Invention

[0005] The purpose of this invention is to provide a traffic light detection method, system, and device based on network stream multi-target tracking, which improves the detection accuracy of traffic lights in videos through offline multi-target tracking.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A traffic light detection method based on network flow multi-target tracking includes:

[0008] Determine the YOLOv7 object detection neural network and color detection classification network;

[0009] Using the YOLOv7 object detection neural network, auxiliary traffic lights and each directional indicator light in the auxiliary traffic lights are identified in each frame of offline traffic light video, and the detection results of auxiliary traffic lights and directional indicator lights in each frame are obtained.

[0010] The trace slices are constructed using the auxiliary traffic light detection results, and the minimum cost flow network graph is constructed using the trace slices as nodes.

[0011] The shortest path algorithm is used to solve the minimum cost flow network graph to obtain multiple auxiliary traffic light trajectories;

[0012] Based on the direction indicator light detection results of each frame of image, determine the shape category of the auxiliary traffic light within each auxiliary traffic light trajectory;

[0013] Based on the auxiliary traffic light area image corresponding to each auxiliary traffic light trajectory, a color detection and classification network is used to detect the color category of the directional indicator lights in the auxiliary traffic lights, and combined with the shape category of the auxiliary traffic lights, the final detection result of each auxiliary traffic light trajectory is determined.

[0014] Optionally, determining the YOLOv7 object detection neural network specifically includes:

[0015] Acquire video including auxiliary traffic lights;

[0016] Frame-by-frame image extraction is performed on videos containing auxiliary traffic lights to obtain images containing the auxiliary traffic lights, which are then labeled to form a target detection dataset. The labeled content includes: the position, size, and shape of the auxiliary traffic lights, and the position, size, and category of the directional indicator lights. The shapes of the auxiliary traffic lights include horizontal and vertical traffic lights; the categories of the directional indicator lights include red round lights, green round lights, red left-turn arrows, green left-turn arrows, red straight arrows, green straight arrows, red right-turn arrows, and green right-turn arrows.

[0017] The YOLOv7 neural network was trained using the object detection dataset to obtain the YOLOv7 object detection neural network.

[0018] Optionally, a color detection and classification network is determined, specifically including:

[0019] The auxiliary traffic lights are cropped from the image containing the auxiliary traffic lights to obtain the auxiliary traffic light area image;

[0020] Cropped images of red lights, green lights, and black lights are extracted from the auxiliary traffic light area image; the red light image includes a red circular light, a red left-turn arrow, a red straight arrow, or a red right-turn arrow; the green light image includes a green circular light, a green left-turn arrow, a green straight arrow, or a green right-turn arrow; the black image is an image of the area of ​​the auxiliary traffic lights where no lights are lit.

[0021] A color classification dataset is constructed by concatenating a red image with a black image vertically to form the red class, concatenating a black image with a green image vertically to form the green class, and concatenating two black images vertically to form the black class.

[0022] An image classification network is trained using the aforementioned color classification dataset to obtain a color detection and classification network.

[0023] Optionally, the step of constructing traces using auxiliary traffic light detection results and using the traces as nodes to construct a minimum cost flow network graph further includes:

[0024] If the auxiliary traffic light detection result shows a horizontal traffic light, then delete the auxiliary traffic light target box whose height is greater than its width; if the auxiliary traffic light detection result shows a vertical traffic light, then delete the auxiliary traffic light target box whose width is greater than its height.

[0025] Statistically analyze the average aspect ratio P of all auxiliary traffic lights in the test results. m And delete those with aspect ratios less than P. m -0.5 and aspect ratio greater than P m +0.5 auxiliary signal light target frame;

[0026] Based on the positional relationship of auxiliary traffic lights between consecutive frames in the offline video of traffic lights, a frame-by-frame mapping method is used to supplement the missed detections of auxiliary traffic lights.

[0027] Optionally, the step of constructing traces using auxiliary traffic light detection results and using these traces as nodes to construct a minimum-cost flow network graph specifically includes:

[0028] The dual-threshold conservative correlation algorithm is used to associate the auxiliary signal light detection results of two adjacent frames with Gaussian kernel correlation values ​​less than the correlation value threshold as the same trace.

[0029] Define a source node s, a sink node t, and traces as intermediate nodes to establish a minimum cost flow network graph.

[0030] The cost between intermediate nodes is constructed by using the positional similarity and image similarity between traces. In the formula, c ij Represents node v i With node v jThe path cost; μ represents the overall cost decay weight; P pos (T j |T i ) indicates the tracer T i With T j Location similarity, one trace corresponds to one node. d l and d f They represent tracer T respectively i The last target detection result location coordinates and trace T j The first target detection result's location coordinates, where σ represents the Gaussian kernel correlation variance coefficient and λ represents the location similarity decay weight; P app (T j |T i ) indicates the tracer T i With T j Image similarity, P app (T j |T i )=S(a l |a f )(1-exp(-λΔt)),S(a l |a f This indicates that a similarity calculation classification network is used to calculate the similarity of the auxiliary traffic light regions in the two auxiliary traffic light detection results; This indicates the time constraint between the two traces. Δt represents the tracer T i With T j The number of frames between them, θ t This represents the maximum frame count threshold; two traces with a frame count greater than this value will not be associated.

[0031] A mask image is created by observing the location where the auxiliary traffic light exits the offline video of the traffic light and the location when the auxiliary traffic light is first detected.

[0032] Based on the mask image, the cost of constructing the connection between the source node and the sink node and the intermediate node is... and In the formula, This represents the distance from source node s to node v. i Path cost, Represents node v i The path cost to sink node t, α is the scaling factor, I() is the indicator function, A indicates that the position coordinates of the auxiliary traffic light in the first auxiliary traffic light target detection result in the trace slice are located within the mask image, B indicates that the position coordinates of the auxiliary traffic light in the last auxiliary traffic light target detection result in the trace slice are located within the mask image, and t is the total number of offline video frames of the traffic lights. s(i) Indicates the tracer T i The frame number in the video where the first object detection response is located, t e (i) Indicates the tracer T i The frame number in the offline video of the traffic light where the last auxiliary traffic light target detection result is located;

[0033] Combining the costs between the source node and sink node and intermediate nodes, as well as the costs between intermediate nodes, the optimal expression for the minimum cost flow network graph is:

[0034]

[0035] stf ij ,f i (s) ,f i (t) ∈{0,1}

[0036] and

[0037] In the formula, T represents the optimization function of the minimum cost flow network graph, T represents the set of traces, and f i (s) This represents the distance from source node s to node v. i Traffic, f ij Represents node v i to node v j Traffic, f i (t) Represents node v i The flow to sink node t.

[0038] Optionally, the step of using a similarity calculation classification network to calculate the similarity of auxiliary traffic light regions in the two auxiliary traffic light detection results specifically includes:

[0039] Extract the extended area of ​​the auxiliary traffic lights and the background area from an image containing auxiliary traffic lights;

[0040] The extended area of ​​the auxiliary traffic lights is named "light class", and the background area is named "background class", thus forming an image dataset;

[0041] The image classification network is trained using the image dataset to obtain a similarity calculation classification network;

[0042] Centered on the center point of the auxiliary traffic light area in the auxiliary traffic light detection results, an image with a height of 3h and a width of 3d is extracted and termed the multi-auxiliary traffic light area. Two multi-auxiliary traffic light areas are denoted as A. l With A fIn the auxiliary traffic light detection results, the height of the auxiliary traffic light area is h, the width is w, and the distance between the position coordinates of adjacent auxiliary traffic lights is d.

[0043] A l With A f Vertically and equally cut into three regions with a height of 3w and a width of d, denoted as: and

[0044] The multiple auxiliary traffic light regions from the two auxiliary traffic light detection results are input into a similarity calculation classification network to obtain the confidence scores of the two multiple auxiliary traffic light regions. and

[0045] Using formula Calculate the similarity of the auxiliary traffic light regions in the detection results of the two auxiliary traffic lights.

[0046] Optionally, the shortest path algorithm is used to solve the minimum cost flow network graph to obtain multiple auxiliary traffic light trajectories, specifically including:

[0047] Dijkstra's algorithm is used to find the shortest path in the minimum cost flow network graph to obtain multiple trajectories for the auxiliary traffic lights;

[0048] The CSR-DCF tracking algorithm is used to track auxiliary traffic lights belonging to the same trajectory, and missing auxiliary traffic lights are added.

[0049] Optionally, determining the shape category of the auxiliary traffic light within each auxiliary traffic light trajectory based on the direction indicator traffic light detection results of each frame image specifically includes:

[0050] The eight directional traffic lights are categorized into four types: round lights, left-turn arrows, straight arrows, and right-turn arrows.

[0051] The number of four morphological categories of directional indicator lights in the image corresponding to each auxiliary traffic light trajectory in the offline video of traffic lights is counted, and the morphological category with the most occurrences is determined as the morphological category of the auxiliary traffic light in each auxiliary traffic light trajectory.

[0052] A traffic light detection system based on network flow multi-target tracking includes:

[0053] The neural network determination module is used to determine the YOLOv7 object detection neural network and color detection classification network;

[0054] The image recognition module is used to identify auxiliary traffic lights and each direction indicator light in the auxiliary traffic lights in each frame of the offline video of traffic lights using the YOLOv7 object detection neural network, and to obtain the detection results of auxiliary traffic lights and direction indicator lights in each frame of the image.

[0055] The trace construction module is used to construct traces using the auxiliary traffic light detection results, and to construct a minimum cost flow network graph using the traces as nodes.

[0056] The trajectory solving module is used to solve the minimum cost flow network graph using the shortest path algorithm to obtain multiple auxiliary traffic light trajectories;

[0057] The morphology category determination module is used to determine the morphology category of the auxiliary traffic lights within each auxiliary traffic light trajectory based on the direction indicator traffic light detection results of each frame image.

[0058] The color category determination module is used to detect the color category of the directional indicator lights in the auxiliary traffic lights based on the auxiliary traffic light area image corresponding to each auxiliary traffic light trajectory, using a color detection and classification network, and combined with the shape category of the auxiliary traffic lights to determine the final detection result of each auxiliary traffic light trajectory.

[0059] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned signal light detection method based on network flow multi-target tracking.

[0060] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0061] This invention discloses a traffic light detection method, system, and device based on network flow multi-target tracking. First, the detection results of the YOLOv7 target detection neural network are divided into auxiliary traffic light detection results and directional indicator traffic light detection results. A minimum cost flow network graph is constructed for the auxiliary traffic light target detection results to obtain multiple auxiliary traffic light trajectories. Then, based on the directional indicator traffic light detection results, the shape category of the auxiliary traffic light within each trajectory is determined. A color detection and classification network is used to detect the color category of the directional indicator traffic lights among the auxiliary traffic lights. Statistical and image color features are used for supplementary detection and classification of the directional indicator traffic lights. This invention can improve the detection accuracy of traffic lights by utilizing video traffic light detection results through multi-target tracking and supplementary detection. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart of a traffic light detection method based on network flow multi-target tracking provided in an embodiment of the present invention;

[0064] Figure 2 A schematic diagram illustrating the types of directional indicator lights provided in embodiments of the present invention;

[0065] Figure 3 This is a flowchart illustrating the image cropping and dataset creation process provided in an embodiment of the present invention.

[0066] Figure 4 This is a schematic diagram of the minimum cost flow network flow graph construction process provided in an embodiment of the present invention;

[0067] Figure 5 This is a flowchart of the frame-by-frame mapping method during the preprocessing process provided in an embodiment of the present invention; Figure 5 (a) in the diagram is a schematic of the two auxiliary traffic lights detected in the first frame. Figure 5 (b) in the diagram shows a single auxiliary signal light detected in the second frame.

[0068] Figure 6 This is a schematic diagram illustrating the principle of the similarity calculation network provided in this embodiment of the invention.

[0069] Figure 7 This is a schematic diagram illustrating the principle of mask image fabrication provided in an embodiment of the present invention. Figure 7 (a) in the image shows a diagram of the traffic light leaving the video area from the left. Figure 7 (b) shows a schematic diagram of the traffic light leaving the video area from the top. Figure 7 (c) in the image shows a diagram of the traffic light leaving the video area from the right. Figure 7 (d) in the figure is a schematic diagram of the mask image. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] The purpose of this invention is to provide a traffic light detection method, system, and device based on network stream multi-target tracking, which can improve the detection accuracy of traffic lights by utilizing the traffic light detection results from video through multi-target tracking and supplementary detection.

[0072] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, this embodiment of the invention provides a traffic light detection method based on network flow multi-target tracking, including:

[0074] Step 1: Determine the YOLOv7 object detection neural network and color detection classification network.

[0075] For example, the process of determining the YOLOv7 object detection neural network is as follows:

[0076] Acquire video containing traffic lights; extract images frame by frame from the video to obtain images containing traffic lights, and annotate them; for each extracted image, use bounding boxes to label the position, size, and category of auxiliary traffic lights and small target traffic lights (direction indicator lights), and the categories are as follows: Figure 2 As shown, the specific indicators include: red round lights, green round lights, red left-turn arrows, green left-turn arrows, red straight arrows, green straight arrows, red right-turn arrows, and green right-turn arrows; the YOLOv7 object detection network is trained using the labeled dataset.

[0077] For example, the process of determining the color detection and classification network is as follows:

[0078] Step a: Extract the traffic light area and the black unlit area from the image containing the traffic lights to create a color classification dataset.

[0079] Image cropping and dataset creation process as follows Figure 3 As shown, the auxiliary signal light in the image is cropped out (a), and its height is h and its width is w. The resolution is scaled while keeping the aspect ratio unchanged, such as (b) making its height 100w / hpx and its width 100px. The upper 1 / 3 and lower 1 / 3 of the image are cropped respectively to obtain two images (c) with a height of 33w / hpx and a width of 100px. Then, the center is cropped by 60px to obtain two images (d) with a height of 60px and a width of 60px. Finally, the two images are stitched together vertically and zero-filled to obtain an image (e) with a height of 120px and a width of 120px for training the image classification network.

[0080] The auxiliary traffic lights with green lights are stitched together so that the upper half is green and the lower half is black, forming the "green" category; the auxiliary traffic lights with red lights are stitched together so that the upper half is black and the lower half is red, forming the "red" category; and the auxiliary traffic lights that are not lit are stitched together so that both the upper and lower halves are black, forming the "black" category. Since it is difficult to collect images of not-lit auxiliary traffic lights in reality, the lower half of the image in the "green" category is randomly stitched together with the upper half of the image in the "red" category to artificially create the "black" image.

[0081] Step b: Use the stitched red, green, and black image datasets to train the image classification network to obtain the color detection and classification network.

[0082] Step 2: Use the YOLOv7 object detection neural network to identify auxiliary traffic lights and each directional indicator light in each frame of the offline traffic light video, and obtain the auxiliary traffic light detection results and directional indicator light detection results for each frame.

[0083] The trained YOLOv7 object detection network was used to obtain the position, size, and shape of auxiliary traffic lights in the test video, as well as the position, size, and category of small target traffic lights (direction indicator lights).

[0084] Step 3: Construct traces using the auxiliary traffic light detection results, and use the traces as nodes to construct a minimum cost flow network graph.

[0085] The trajectory solving process is an offline multi-target tracking process. The specific process of step 3 is as follows: Figure 4 As shown, it specifically includes:

[0086] Step a: Preprocessing of auxiliary traffic light detection response.

[0087] Based on the two forms of auxiliary traffic lights, if the auxiliary traffic light in the video is a horizontal traffic light, then delete the target box whose height is greater than its width in the auxiliary traffic light detection response; conversely, if the auxiliary traffic light in the video is a vertical traffic light, then delete the target box whose width is greater than its height in the auxiliary traffic light detection response.

[0088] The average aspect ratio P of the auxiliary traffic light detection results in the entire video segment. m Delete the aspect ratio smaller than P m -0.5 and greater than P m +0.5 auxiliary signal light detection response.

[0089] Based on the positional relationship of auxiliary traffic lights between consecutive video frames, a frame-by-frame mapping method is used to supplement and predict missed detections in the auxiliary traffic light detection response. In two consecutive frames, if there are no missed detections of auxiliary traffic lights in the first frame, and there are missed detections but not all of them in the second frame, then let the set of auxiliary traffic light detection results in the first frame be N = {N1, N2, ..., N...}. n In the second frame, the set of auxiliary traffic light detection results is M = {M1, M2, ..., M}. p Calculate the Euclidean distance between the position coordinates of the detection results in set N and set M respectively. The two detection results with the smallest Euclidean distance are identified as the same traffic light in the two frames. Since there are no missed detections of auxiliary traffic lights in the first frame, sort the detection results in set N according to their position x-axis coordinates to obtain the auxiliary traffic lights arranged from left to right. Combining the correspondence between the auxiliary traffic lights in M ​​and N, the corresponding positions of the missed auxiliary traffic lights in the second frame can be determined.

[0090] The following is a specific embodiment of the frame-by-frame mapping method: Figure 5 For two consecutive frames of target detection response in a video with two auxiliary traffic lights, Figure 5 (a) shows that two auxiliary traffic lights were detected in the first frame. Figure 5 (b) shows that only one auxiliary traffic light was detected in the second frame. In this case, the lateral distance d between the auxiliary traffic lights in the first frame is calculated. w and longitudinal distance d h The Euclidean distances between the auxiliary traffic light position coordinates p1 detected in the second frame and the position coordinates q1 and q2 of all detected auxiliary traffic lights in the first frame are calculated. The Euclidean distance between q1 and p1 is the smallest, indicating that the auxiliary traffic light corresponding to q1 and p1 is the same traffic light in both frames, and the auxiliary traffic light corresponding to q2 is the one missed in the second frame. Using the horizontal, vertical, and size information of the auxiliary traffic lights in the first frame, the position coordinates of the missing auxiliary traffic light in the second frame can be determined as p2 = ((x1+x2) / 2+d). w ,(y1+y2) / 2+d y The coordinates of the top left corner and the bottom right corner are a2 = ((x1 + x2) / 2 + d) w -w2 / 2,(y1+y2) / 2+d y -h2 / 2), b2=((x1+x2) / 2+d w +w2 / 2,(y1+y2) / 2+d y +h2 / 2).

[0091] Step b: Construct a trace using the detection results of the auxiliary traffic lights.

[0092] The auxiliary signal light detection response is iterated through each frame of the video. The Gaussian kernel correlation value of the position coordinates of each auxiliary signal light detection result in two adjacent frames is calculated. If the correlation value is less than 0.0005, the two auxiliary signal light detection responses are considered as the same trace. The maximum length of each trace is 15.

[0093] Step c: Construct a minimum cost flow network graph using auxiliary traffic light traces as nodes.

[0094] Let the set of traces be Each trace corresponds to a node v i The path from the source node s to any node is represented as (s, v i The path cost is expressed as The flow rate is represented as f(s,v) i )=f i (s) The path from any node to the sink node t is represented as (v i The path cost is expressed as t). The flow rate is represented as f(v) i ,t)=f i (t) For each pair of association similarities P(T) j |T i For trace nodes where i ≠ 0 and i ≠ j, the path between them is represented as (v i ,v j The path cost is represented as c(v). i ,v j ) = c ij The flow rate is represented as f(v) i ,v j )=f ij The minimum cost flow network optimization problem can then be expressed in the following form:

[0095]

[0096] stf ij ,f i (s) ,f i (t) ∈{0,1}

[0097] and

[0098] The cost c is modeled as follows:

[0099]

[0100] c ij =-logP(T) j |Ti ).

[0101] Step d: Construct the minimum cost flow network by using the location similarity and image similarity between traces to calculate the cost between trace nodes.

[0102] This invention uses the following method to construct the cost between nodes:

[0103]

[0104]

[0105] P app (T j |T i )=S(a l |a f (1-exp(-λΔt))

[0106] Where, d l and d f They represent tracer T respectively i The last target detection result location coordinates and trace T j The location coordinates of the first target detection result; a l and a f They represent tracer T respectively i The last target detection result region image and trace T j The first target detection result region image; P pos (T j |T i ) indicates the tracer T i With T j Positional similarity; σ represents the Gaussian kernel correlation variance coefficient, with a value of 1; P app (T j |T i ) indicates the tracer T i With T j Image similarity; S(a l |a f ) indicates that the similarity between the two target detection result regions is calculated using a similarity calculation classification network; λ and μ represent the positional similarity attenuation weight and the overall cost attenuation weight, respectively, with values ​​of 2 and 20; The time constraint between two traces, which prevents the establishment of cost connections between traces that are too far apart or between trace nodes that overlap in the time dimension, is defined as follows:

[0107]

[0108] Δt represents the tracer T i With T jThe number of frames between them, θ t This represents the maximum frame count threshold; two traces with a frame count greater than this value will not be associated.

[0109] S(a l |a f The similarity calculation classification network is used to obtain the data, as illustrated by the following examples: Figure 6 As shown, let the height of the auxiliary traffic light region 'a' in the image be h, the width be w, and the distance between the position coordinates of adjacent auxiliary traffic lights be d. An image with a height of 3h and a width of 3d is cropped centered on the center point of the auxiliary traffic light region; this is called the multi-auxiliary traffic light region A. l With a f The multiple auxiliary traffic light areas are respectively represented as A l With A f . respectively A l With A f Vertically and equally cut into three regions with a height of 3w and a width of d, denoted as: and Each region, after being processed by a similarity calculation classification network, yields a binary classification confidence score between the light and the background. l With A f The confidence score obtained after the segmented regions are processed by a similarity calculation and classification network is expressed as follows: and Each item in the set represents a binary confidence tuple obtained for each region. Then S(a l |a f The calculation method for ) is as follows:

[0110]

[0111] Step e: Construct the minimum cost flow network graph and the cost between source nodes, sink nodes and trace nodes.

[0112] A mask image was created by observing the position of traffic lights when vehicles pass through an intersection and the position of auxiliary traffic lights when object detection first detects them near the intersection. For example... Figure 7 As shown, Figure 7 Frames (a) to (c) in the video are near the end of a portion of the test video. In the first frame, the vehicle turns right and the traffic light leaves the video area from the left. In the second frame, the vehicle goes straight and the traffic light leaves the video area from the top. In the third frame, the vehicle turns left and the traffic light leaves the video area from the right. The areas that leave the video area are represented by semi-transparent gray areas. Figure 7 In the image, (d) represents a mask image created from the locations where multiple traffic light videos exit the video area. The costs of constructing the source node and trace node, and the costs of constructing the sink node and trace node are as follows:

[0113]

[0114]

[0115] Where I(x) is the indicator function, with a value of 1 when x is true; α is the scaling factor, with a value of 25; A indicates that the position coordinates of the first auxiliary traffic light target detection result in the tracer are located within the mask image; B indicates that the position coordinates of the last auxiliary traffic light detection response in the tracer are located within the mask image; t is the total number of frames in the traffic light video; t s (i) t represents the frame number in the video containing the first object detection response in the i-th trace; e (i) This represents the frame number in the video where the last target detection response in the i-th trace is located.

[0116] Step 4: Use the shortest path algorithm to solve the minimum cost flow network graph to obtain multiple auxiliary traffic light trajectories.

[0117] Use Dijkstra's algorithm to compute the shortest paths in a network graph and find the trajectories of K auxiliary traffic lights. Let the constructed minimum-cost flow network graph be G, and the number of trajectories to be solved be K. Initialize f = 0; let the source node s be the starting node and the sink node t be the ending node. Run Dijkstra's algorithm to find the set of nodes with the shortest paths in G. Set f = f + 1; delete in network flow graph G The algorithm excludes nodes s and t; if f < K, Dijkstra's algorithm is repeated; otherwise, the algorithm terminates and outputs a set of K nodes, corresponding to the trajectories of K auxiliary traffic lights in the video.

[0118] The tracking algorithm is used to track auxiliary traffic lights that belong to the same trajectory, thus solving the problem of missing auxiliary traffic light detection in some frames.

[0119] The CSR-DCF tracking algorithm is used to track the auxiliary traffic lights.

[0120] make Let K be the set of trajectories of auxiliary traffic lights in the video, and K be the number of trajectories; iterate through each trajectory X. k T in the trace i Let two adjacent traces T i With T j The frame distance between (j=i+1) is If t d If >1, then use T. i Initialize the correlation filter tracker in the region of the last target detection result; for the t between the two traces d -1 frame for target tracking. If td -1 < 20, then perform the complete t d -1 frame tracking, if t d If -1 ≥ 20, then tracking stops after 20 frames. The coordinates and dimensions of the tracking output are associated with the trace patch and added to the trajectory. middle.

[0121] Step 5: Based on the direction indicator light detection results of each frame image, determine the shape category of the auxiliary traffic light within each auxiliary traffic light trajectory.

[0122] First, the eight small target traffic lights are divided into four categories: round lights (red and green), left-turn arrows (red and green), straight arrows (red and green), and right-turn arrows (red and green).

[0123] Secondly, the number of small target traffic lights of the four morphological categories within the auxiliary traffic light area is counted, and the category with the highest probability is determined as the morphological category of the auxiliary traffic light. For example, if the number of auxiliary traffic lights K in a video is 2, then the number of small target traffic lights of the four morphological categories within the trajectory X1 and trajectory X2 areas of the video is counted. If the left-turn arrow type has the highest probability in trajectory X1, then the morphological category of the auxiliary traffic light corresponding to trajectory X1 is determined to be a left-turn arrow; if the round light type has the highest probability in trajectory X2, then the morphological category of the auxiliary traffic light corresponding to trajectory X1 is determined to be a left-turn arrow. The corresponding auxiliary signal light is classified as a round light.

[0124] Delete small target traffic light detection results within the auxiliary traffic light detection result area that do not match the trajectory shape category of this auxiliary traffic light.

[0125] Step 6: Based on the auxiliary traffic light area image corresponding to each auxiliary traffic light trajectory, use a color detection and classification network to detect the color category of the directional indicator lights in the auxiliary traffic lights, and combine it with the shape category of the auxiliary traffic lights to determine the final detection result of each auxiliary traffic light trajectory.

[0126] If there are no small target traffic light detection results within the auxiliary traffic light detection results, then the auxiliary traffic light area image is processed according to step 1a to obtain a stitched image with a height of 120px and a width of 120px, and then the image is classified using a color detection and classification network.

[0127] By combining the obtained shape category and color category of the auxiliary traffic lights, the category of small target traffic lights within this auxiliary traffic light area can be determined, thus enabling supplementary detection.

[0128] This invention utilizes the detection results of traffic lights in videos, and concatenates these results through offline multi-target tracking to obtain multiple traffic light trajectories. Combining this with color detection improves the accuracy of traffic light detection.

[0129] This invention first obtains the position, size, and category information of traffic lights in the video to be detected. Then, it divides the YOLOv7 detection results into auxiliary traffic light category and small target traffic light category. It then performs network flow graph modeling on the target detection results of the auxiliary traffic light category and performs multi-target tracking. Finally, it uses statistical and image color features to supplement the detection and classification of small target traffic lights, which solves the problem that there are few current methods for detecting traffic lights using multi-target tracking and improves the detection accuracy of traffic lights in videos.

[0130] This invention also provides a traffic light detection system based on network flow multi-target tracking, comprising:

[0131] The neural network determination module is used to determine the YOLOv7 object detection neural network and color detection classification network;

[0132] The image recognition module is used to identify auxiliary traffic lights and each direction indicator light in the auxiliary traffic lights in each frame of the offline video of traffic lights using the YOLOv7 object detection neural network, and to obtain the detection results of auxiliary traffic lights and direction indicator lights in each frame of the image.

[0133] The trace construction module is used to construct traces using the auxiliary traffic light detection results, and to construct a minimum cost flow network graph using the traces as nodes.

[0134] The trajectory solving module is used to solve the minimum cost flow network graph using the shortest path algorithm to obtain multiple auxiliary traffic light trajectories;

[0135] The morphology category determination module is used to determine the morphology category of the auxiliary traffic lights within each auxiliary traffic light trajectory based on the direction indicator traffic light detection results of each frame image.

[0136] The color category determination module is used to detect the color category of the directional indicator lights in the auxiliary traffic lights based on the auxiliary traffic light area image corresponding to each auxiliary traffic light trajectory, using a color detection and classification network, and combined with the shape category of the auxiliary traffic lights to determine the final detection result of each auxiliary traffic light trajectory.

[0137] The traffic light detection system based on network flow multi-target tracking provided in this embodiment of the invention has a similar working principle and beneficial effects to the traffic light detection method based on network flow multi-target tracking described in the above embodiments, so it will not be described in detail here. For details, please refer to the introduction of the above method embodiments.

[0138] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned signal light detection method based on network flow multi-target tracking.

[0139] Furthermore, when the computer program in the aforementioned memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0140] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0141] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A traffic light detection method based on network flow multi-target tracking, characterized in that, include: Determine the YOLOv7 object detection neural network and color detection classification network; Using the YOLOv7 object detection neural network, auxiliary traffic lights and each directional indicator light in the auxiliary traffic lights are identified in each frame of offline traffic light video, and the detection results of auxiliary traffic lights and directional indicator lights in each frame are obtained. The trace slices are constructed using the auxiliary traffic light detection results, and the minimum cost flow network graph is constructed using the trace slices as nodes. The shortest path algorithm is used to solve the minimum cost flow network graph to obtain multiple auxiliary traffic light trajectories; Based on the direction indicator light detection results of each frame of image, determine the shape category of the auxiliary traffic light within each auxiliary traffic light trajectory; Based on the auxiliary traffic light area image corresponding to each auxiliary traffic light trajectory, a color detection and classification network is used to detect the color category of the directional indicator lights in the auxiliary traffic lights, and combined with the shape category of the auxiliary traffic lights, the final detection result of each auxiliary traffic light trajectory is determined. The process of constructing traces using auxiliary traffic light detection results and building a minimum-cost flow network graph using these traces as nodes specifically includes: The dual-threshold conservative correlation algorithm is used to associate the auxiliary signal light detection results of two adjacent frames with Gaussian kernel correlation values ​​less than the correlation value threshold as the same trace. Define a source node s, a sink node t, and traces as intermediate nodes to establish a minimum cost flow network graph. The cost between intermediate nodes is constructed by using the positional similarity and image similarity between traces. In the formula, Represents a node v i With nodes v j The path cost; Indicates the overall cost attenuation weight; Indicator film With trace Location similarity, one trace corresponds to one node. , and They represent traces respectively The last target detection result location coordinates and trace The location coordinates of the first target detection result. This represents the Gaussian kernel correlation variance coefficient. This represents the positional similarity decay weight; Indicator film With trace Image similarity, , This indicates that a similarity calculation classification network is used to calculate the similarity of the auxiliary traffic light regions in the detection results of two auxiliary traffic lights; This indicates the time constraint between the two traces. , Indicator film With trace The number of frames between them This represents the maximum frame count threshold; two traces with a frame count greater than this value will not be associated. A mask image is created by observing the location where the auxiliary traffic light exits the offline video of the traffic light and the location when the auxiliary traffic light is first detected. Based on the mask image, the cost of constructing the connection between the source node and the sink node and the intermediate node is... and In the formula, Indicates the distance from source node s to node v i Path cost, Represents a node v i The path cost to sink node t. This is the scaling factor. I () is an indicator function. A This indicates that the position coordinates of the first auxiliary signal light target in the tracer are located within the mask image. B This indicates that the position coordinates of the last auxiliary signal light in the target detection result of the tracer are located within the mask image. t This represents the total number of offline video frames for traffic lights. Indicator film The frame number in the video where the first object detection response is located. Indicator film The frame number in the offline video of the traffic light where the last auxiliary traffic light target detection result is located; Combining the costs between the source node and sink node and intermediate nodes, as well as the costs between intermediate nodes, the optimization expression for the minimum cost flow network graph is as follows: ； ； In the formula, T The optimization function represents the minimum cost flow network graph. Represents a set of traces. Indicates the distance from source node s to node v i Traffic, Represents a node v i To the node v j Traffic, Represents a node v i The flow to sink node t.

2. The traffic light detection method based on network flow multi-target tracking according to claim 1, characterized in that, The determination of the YOLOv7 object detection neural network specifically includes: Acquire video including auxiliary traffic lights; Frame-by-frame image extraction is performed on videos containing auxiliary traffic lights to obtain images containing the auxiliary traffic lights, which are then labeled to form a target detection dataset. The labeled content includes: the position, size, and shape of the auxiliary traffic lights, and the position, size, and category of the directional indicator lights. The shapes of the auxiliary traffic lights include horizontal and vertical traffic lights; the categories of the directional indicator lights include red round lights, green round lights, red left-turn arrows, green left-turn arrows, red straight arrows, green straight arrows, red right-turn arrows, and green right-turn arrows. The YOLOv7 neural network was trained using the object detection dataset to obtain the YOLOv7 object detection neural network.

3. The traffic light detection method based on network flow multi-target tracking according to claim 2, characterized in that, Determine the color detection and classification network, specifically including: The auxiliary traffic lights are cropped from the image containing the auxiliary traffic lights to obtain the auxiliary traffic light area image; Cropped images of red lights, green lights, and black lights are extracted from the auxiliary traffic light area image; the red light image includes a red circular light, a red left-turn arrow, a red straight arrow, or a red right-turn arrow; the green light image includes a green circular light, a green left-turn arrow, a green straight arrow, or a green right-turn arrow; the black image is an image of the area of ​​the auxiliary traffic lights where no lights are lit. A color classification dataset is constructed by concatenating a red image with a black image vertically to form the red class, concatenating a black image with a green image vertically to form the green class, and concatenating two black images vertically to form the black class. An image classification network is trained using the aforementioned color classification dataset to obtain a color detection and classification network.

4. The traffic light detection method based on network flow multi-target tracking according to claim 1, characterized in that, The step of constructing traces using auxiliary traffic light detection results and then using these traces as nodes to construct a minimum-cost flow network graph includes, prior to: If the auxiliary traffic light detection result shows a horizontal traffic light, then delete the auxiliary traffic light target box whose height is greater than its width; if the auxiliary traffic light detection result shows a vertical traffic light, then delete the auxiliary traffic light target box whose width is greater than its height. Statistical analysis of the average aspect ratio of all auxiliary traffic lights in the test results. P m And delete those with an aspect ratio smaller than 1. P m -0.5 and aspect ratio greater than P m +0.5 auxiliary signal light target frame; Based on the positional relationship of auxiliary traffic lights between consecutive frames in the offline video of traffic lights, a frame-by-frame mapping method is used to supplement the missed detections of auxiliary traffic lights.

5. The traffic light detection method based on network flow multi-target tracking according to claim 1, characterized in that, The calculation of the similarity between the auxiliary traffic light regions in the two auxiliary traffic light detection results using a similarity calculation classification network specifically includes: Extract the extended area of ​​the auxiliary traffic lights and the background area from an image containing auxiliary traffic lights; The extended area of ​​the auxiliary traffic lights is named "light class", and the background area is named "background class", thus forming an image dataset; The image classification network is trained using the image dataset to obtain a similarity calculation classification network; Using the center point of the auxiliary traffic light area in the auxiliary traffic light detection results as the center, a section with a height of 3 is extracted. h 3 d The image is called the multi-auxiliary traffic light area, and the two multi-auxiliary traffic light areas are represented as follows: and The height of the auxiliary signal light area in the auxiliary signal light detection results is... h , width is w The distance between the coordinates of adjacent auxiliary traffic lights is d ; Will and Vertically cut into equal sections with a height of 3 w And the width is d The three regions are represented as and ; The multiple auxiliary traffic light regions from the two auxiliary traffic light detection results are input into a similarity calculation classification network to obtain the confidence scores of the two multiple auxiliary traffic light regions. and ; Using formula Calculate the similarity of the auxiliary traffic light regions in the detection results of the two auxiliary traffic lights.

6. The traffic light detection method based on network flow multi-target tracking according to claim 1, characterized in that, The minimum cost flow network graph is solved using the shortest path algorithm to obtain multiple auxiliary traffic light trajectories, specifically including: Dijkstra's algorithm is used to find the shortest path in the minimum cost flow network graph to obtain multiple trajectories of the auxiliary traffic lights; The CSR-DCF tracking algorithm is used to track auxiliary traffic lights belonging to the same trajectory, and missing auxiliary traffic lights are added.

7. The traffic light detection method based on network flow multi-target tracking according to claim 1, characterized in that, The step of determining the shape category of the auxiliary traffic lights within each auxiliary traffic light trajectory based on the direction indicator traffic light detection results of each frame image specifically includes: The eight directional traffic lights are categorized into four types: round lights, left-turn arrows, straight arrows, and right-turn arrows. The number of four morphological categories of directional indicator lights in the image corresponding to each auxiliary traffic light trajectory in the offline video of traffic lights is counted, and the morphological category with the most occurrences is determined as the morphological category of the auxiliary traffic light in each auxiliary traffic light trajectory.

8. A traffic light detection system based on network flow multi-target tracking, characterized in that, include: The neural network determination module is used to determine the YOLOv7 object detection neural network and color detection classification network; The image recognition module is used to identify auxiliary traffic lights and each direction indicator light in the auxiliary traffic lights in each frame of the offline video of traffic lights using the YOLOv7 object detection neural network, and to obtain the detection results of auxiliary traffic lights and direction indicator lights in each frame of the image. The trace construction module is used to construct traces using the auxiliary traffic light detection results, and to construct a minimum cost flow network graph using the traces as nodes. The trajectory solving module is used to solve the minimum cost flow network graph using the shortest path algorithm to obtain multiple auxiliary traffic light trajectories; The morphology category determination module is used to determine the morphology category of the auxiliary traffic lights within each auxiliary traffic light trajectory based on the direction indicator traffic light detection results of each frame image. The color category determination module is used to detect the color category of the directional indicator lights in the auxiliary traffic lights based on the auxiliary traffic light area image corresponding to each auxiliary traffic light trajectory using a color detection and classification network, and combine it with the shape category of the auxiliary traffic lights to determine the final detection result of each auxiliary traffic light trajectory. The process of constructing traces using auxiliary traffic light detection results and building a minimum-cost flow network graph using these traces as nodes specifically includes: The dual-threshold conservative correlation algorithm is used to associate the auxiliary signal light detection results of two adjacent frames with Gaussian kernel correlation values ​​less than the correlation value threshold as the same trace. Define a source node s, a sink node t, and traces as intermediate nodes to establish a minimum cost flow network graph. The cost between intermediate nodes is constructed by using the positional similarity and image similarity between traces. In the formula, Represents a node v i With nodes v j The path cost; Indicates the overall cost attenuation weight; Indicator film With trace Location similarity, one trace corresponds to one node. , and They represent traces respectively The last target detection result location coordinates and trace The location coordinates of the first target detection result. This represents the Gaussian kernel correlation variance coefficient. This represents the positional similarity decay weight; Indicator film With trace Image similarity, , This indicates that a similarity calculation classification network is used to calculate the similarity of the auxiliary traffic light regions in the detection results of two auxiliary traffic lights; This indicates the time constraint between the two traces. , Indicator film With trace The number of frames between them This represents the maximum frame count threshold; two traces with a frame count greater than this value will not be associated. A mask image is created by observing the location where the auxiliary traffic light exits the offline video of the traffic light and the location when the auxiliary traffic light is first detected. Based on the mask image, the cost of constructing the connection between the source node and the sink node and the intermediate node is... and In the formula, Indicates the distance from source node s to node v i Path cost, Represents a node v i The path cost to sink node t. This is the scaling factor. I () is an indicator function. A This indicates that the position coordinates of the first auxiliary signal light target in the tracer are located within the mask image. B This indicates that the position coordinates of the last auxiliary signal light in the target detection result of the tracer are located within the mask image. t This represents the total number of offline video frames for traffic lights. Indicator film The frame number in the video where the first object detection response is located. Indicator film The frame number in the offline video of the traffic light where the last auxiliary traffic light target detection result is located; Combining the costs between the source node and sink node and intermediate nodes, as well as the costs between intermediate nodes, the optimization expression for the minimum cost flow network graph is as follows: ； ； In the formula, T The optimization function represents the minimum cost flow network graph. Represents a set of traces. Indicates the distance from source node s to node v i Traffic, Represents a node v i To the node v j Traffic, Represents a node v i The flow to sink node t.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the traffic light detection method based on network flow multi-target tracking as described in any one of claims 1 to 7.

Images