A method and system for multi-data fusion red-light running event detection

Abstract

This invention relates to the field of red-light violation detection technology, specifically disclosing a method and system for detecting red-light violation events through multi-data fusion. The method includes: first, installing a high-resolution camera at a designated location on the vehicle's windshield to collect real-time images of the road ahead, forming a video stream, and transmitting it to an onboard edge device with low latency; after preprocessing, initiating an intersection event when a traffic light is detected using a lightweight algorithm, and terminating the event if no detection is detected after an interval of t seconds; using an optimized SORT algorithm to track the shape and color of the traffic light and remove invalid data; then, determining the vehicle's trajectory through collaboration between GPS and a six-axis sensor; finally, filtering valid tracking sequences for the current lane, determining the traffic light type by voting, and matching the driving trajectory with the valid tracking sequences; if all traffic lights in the matched valid tracking sequences are red, then it is determined to be a red-light violation event.

Description

Technical Field

[0001] This invention relates to the field of red-light violation detection technology, specifically to a method and system for detecting red-light violation events using multi-data fusion. Background Technology

[0002] With the development of intelligent transportation systems, the automatic detection and processing of traffic violations has become crucial for improving road safety management. Among numerous traffic violations, running a red light is of particular concern due to its potential to cause serious traffic accidents.

[0003] In existing technologies, common red-light violation detection methods mainly rely on a single data source, such as video image analysis based on a fixed-position camera, which determines whether a violation has occurred by detecting whether a vehicle crosses the stop line during a red light. However, these methods are limited by factors such as camera angle, lighting changes, and occlusion, which can affect detection accuracy, especially in complex traffic scenarios or inclement weather, leading to false positives or false negatives. Furthermore, single-sensor systems often lack the ability to continuously track vehicle trajectories, making it difficult to distinguish between interfering vehicles in adjacent lanes or special driving behaviors (such as right-turning vehicles being allowed to proceed at a red light), resulting in insufficient system robustness. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for detecting red-light running incidents by fusing multiple data, and to solve the following technical problems.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for detecting red-light running events using multi-data fusion includes the following steps:

[0007] Step S1: Collect video streams in front of the vehicle in real time using the vehicle-mounted camera, and simultaneously collect the vehicle's position information and six-axis sensor data; determine the vehicle's trajectory as it passes through the intersection based on the position information and six-axis sensor data.

[0008] Step S2: Divide the video stream into several image frames, detect each image frame in sequence, and record the image frame in which the traffic light is first detected as the intersection event start frame; starting from the intersection event start frame, detect and track the traffic lights in each subsequent image frame in sequence, and assign a unique tracking identifier to the traffic lights in each subsequent image frame to obtain the tracking sequence.

[0009] Step S3: Determine the lane where the vehicle is located, and select several tracking sequences corresponding to the traffic lights and the lane from all tracking sequences. Select the longest tracking sequence from among the several tracking sequences to obtain the effective tracking sequence. Perform time correlation matching between the driving trajectory and the effective tracking sequence to obtain the matching time period. Analyze the color of the traffic lights in the effective tracking sequence within the matching time period. If all the colors are red, it is determined that the vehicle has run a red light.

[0010] As a further aspect of the present invention: the location information is based on the vehicle's GPS positioning, and the acquisition process of the six-axis sensor data includes:

[0011] The six-axis sensing data is acquired based on a six-axis sensing unit built into the vehicle. The six-axis sensing unit includes a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer acquires the linear acceleration of the vehicle in the front-back, left-rear, and up-down directions, and the three-axis gyroscope acquires the rotational angular velocity of the vehicle in the front-back, left-rear, and up-down directions.

[0012] As a further aspect of the present invention: the process of determining the travel trajectory of a vehicle passing through an intersection includes:

[0013] The vehicle's GPS signal strength is acquired in real time. When the GPS signal strength is greater than a preset threshold, the driving trajectory is calculated using the location information collected by GPS. When the GPS signal strength is less than or equal to the preset threshold, the vehicle's steering state is calculated using the six-axis sensor data to determine the driving trajectory.

[0014] As a further aspect of the present invention: the process of tracking traffic lights in subsequent image frames is based on the SORT multi-target tracking algorithm, specifically including:

[0015] For any image frame after the start frame of the intersection event, a tracking identifier is created for each traffic light detected in the image frame. The tracking identifier is specifically a state vector, which includes the width, height, and position coordinates of the traffic light in the image frame, thus obtaining the bounding box of the traffic light.

[0016] The traffic light detected in the next image frame of the image frame is obtained and denoted as the next traffic light. The bounding box of the next traffic light is obtained and denoted as the detection box. The bounding box and the detection box are associated and matched to obtain the intersection-union ratio (IU / R) between the bounding box and the detection box. If the IU / R is greater than 0, a loss matrix is ​​constructed based on the IU / R for matching. If the IU / R is 0, a loss matrix is ​​constructed based on the Euclidean distance between the center point of the detection box and the center point of the bounding box for matching.

[0017] All created tracking identifiers are recorded as existing tracking identifiers, and the bounding boxes of existing tracking identifiers are recorded as existing bounding boxes. After a new tracking identifier is created, the existing bounding box that is closest to the bounding box of the new tracking identifier is obtained and recorded as the nearest bounding box. The existing tracking identifier corresponding to the nearest bounding box is obtained and recorded as the nearest tracking identifier. The positional relationship between the new tracking identifier and the nearest tracking identifier is obtained, and the offset of the new tracking identifier relative to the nearest tracking identifier is obtained.

[0018] As a further aspect of the present invention, the process of tracking traffic lights in subsequent image frames also includes compensation for missed detections of tracking markers, the process of which includes:

[0019] For each tracking marker, record its positional relationship and offset with the most recent tracking marker. If, in a subsequent image frame, a tracking marker is missed and no detection box is updated, determine whether the most recent tracking marker recorded for the missed tracking marker has been successfully updated in the image frame. If the most recent tracking marker has been successfully updated, calculate the current position of the missed tracking marker based on the updated position of the most recent tracking marker and the positional relationship and offset, and update the missed tracking marker.

[0020] As a further aspect of the present invention, the process of obtaining an effective tracking sequence includes:

[0021] The length of each tracking sequence within a plurality of tracking sequences is obtained, and a length ratio threshold is set. For any tracking sequence within the plurality of tracking sequences, the product of the length of the tracking sequence and the length ratio threshold is obtained. If the length of the tracking sequence is less than the product, the tracking sequence is removed. Among all the remaining tracking sequences, the longest tracking sequence is selected and recorded as the valid tracking sequence.

[0022] As a further aspect of the present invention: before performing time-related matching between the driving trajectory and the valid tracking sequence, an invalid segment removal process is further included, the process comprising:

[0023] The stop line is identified in all image frames, and the timestamp corresponding to the image frame in which the stop line is first identified is obtained. Tracking markers indicating that the vehicle has not crossed the stop line are removed from the effective tracking sequence.

[0024] A system for detecting red-light violations based on multi-data fusion includes:

[0025] Data acquisition module: Acquires real-time video streams from the front of the vehicle via an onboard camera, and simultaneously collects the vehicle's position information and six-axis sensor data; based on the position information and six-axis sensor data, determines the vehicle's trajectory as it passes through the intersection.

[0026] The detection and tracking module divides the video stream into several image frames, detects each image frame sequentially, and records the first image frame in which a traffic light is detected as the intersection event start frame. Starting from the intersection event start frame, the module sequentially detects and tracks the traffic lights in each subsequent image frame, and assigns a unique tracking identifier to each traffic light in each subsequent image frame to obtain a tracking sequence.

[0027] Judgment Module: Determines the lane where the vehicle is located, and selects several tracking sequences corresponding to the traffic lights and the lane from all tracking sequences. Then, selects the longest tracking sequence from among the several tracking sequences to obtain the valid tracking sequence. Performs time correlation matching between the driving trajectory and the valid tracking sequence to obtain the matching time period. Analyzes the color of the traffic lights in the valid tracking sequence within the matching time period. If all the colors are red, it is determined that the vehicle has run a red light.

[0028] The beneficial effects of this invention are:

[0029] Compared to traditional fixed roadside monitoring, this invention deploys the sensing unit on the vehicle itself, naturally forming a monitoring network based on the vehicle's trajectory. This overcomes the limitations of location, power supply, and network connectivity, achieving low-cost, blind-spot-free full-area coverage, and providing an effective technical means, especially for areas lacking regulatory oversight. By deeply integrating video streams from vehicle-mounted cameras, GPS positioning data, and information from the vehicle's six-axis sensors, a multi-dimensional cross-verification system of vision, position, and attitude is constructed. This significantly overcomes the problem of misjudgment and missed judgments easily caused by single data sources under conditions of obstruction, inclement weather, or signal interference, greatly improving the recognition accuracy and system robustness in complex real-world environments.

[0030] Addressing the characteristics of vehicle-mounted mobile scenarios, this invention deeply optimizes the core algorithm: It employs an improved SORT tracking algorithm, utilizing the spatial relative relationship of traffic lights and a hybrid matching strategy to ensure continuous and stable target tracking; it proposes an intelligent lane-specific traffic light filtering mechanism based on the longest tracking sequence and dynamic proportional threshold, efficiently solving the multi-target interference problem from the vehicle's perspective; and it introduces a resource optimization strategy based on vehicle state, significantly reducing the computational and storage load on vehicle-mounted edge devices, ensuring the practicality and economy of the solution. Ultimately, the system outputs a structured and complete event evidence chain, providing a reliable and traceable data foundation for driving behavior monitoring, safety education, and off-site enforcement, forming a complete closed loop from accurate perception and intelligent judgment to effective application, laying the core technological foundation for building a new generation of distributed and intelligent road traffic safety governance system. Attached Figure Description

[0031] The invention will now be further described with reference to the accompanying drawings.

[0032] Figure 1This is a detailed implementation process of the method and system for detecting red-light running incidents using multi-data fusion according to the present invention.

[0033] Figure 2 This is a diagram showing the camera installation location of a method and system for detecting red-light violations using multi-data fusion, as described in this invention.

[0034] Figure 3 This is an image detection and tracking effect diagram of a method and system for detecting red light violations using multi-data fusion according to the present invention;

[0035] Figure 4 This is a flowchart of the intersection event acquisition process of a method and system for detecting red-light running events using multi-data fusion, as described in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Please see Figure 1 As shown, the multi-data fusion red-light violation event recognition process of the present invention is as follows: First, a high-resolution camera is installed at a designated position on the windshield of the vehicle to collect images of the front in real time, form a video stream, and transmit it to the vehicle edge device with low latency. After preprocessing, when a traffic light is detected by a lightweight algorithm, an intersection event is initiated (if no detection is detected after an interval of t seconds, the event is terminated). The optimized SORT algorithm is used to track the shape and color of the traffic light and remove invalid data. Then, the vehicle's driving trajectory is determined by the collaboration of GPS and a six-axis sensor. Finally, the effective tracking sequence of the lane is filtered, and the traffic light type is determined by voting. The driving trajectory is matched with the effective tracking sequence. If all traffic lights in the matched effective tracking sequence are red, it is determined to be a red-light violation event.

[0038] 1.1 Camera Installation

[0039] Hardware selection: A high-resolution camera was chosen to ensure that image details are clearly discernible.

[0040] Installation location: Fix it in the upper center area of ​​the windshield. The camera should be looking straight ahead as much as possible to avoid errors in signal light detection caused by viewing angle deviation.

[0041] Specifically, a high-resolution 720P onboard camera is installed in the upper center of the vehicle's windshield, ensuring that the camera lens axis is parallel to the horizontal direction, allowing for a level view of the road scene ahead without obstruction or distortion. This camera captures real-time images of the road scene at a frame rate of 25 frames per second (fps) and transmits them to edge computing devices via a low-latency bus.

[0042] 1.2 Real-time image acquisition and preprocessing are crucial steps in red-light violation detection, with the core objective of ensuring input data quality:

[0043] (1) Data acquisition: Real-time image monitoring data of the front of the vehicle is acquired through the camera installed in Section 1.1 to form a stable video stream;

[0044] (2) Data transmission: Transmitting the video stream to the vehicle edge device in a low-latency manner;

[0045] (3) Image preprocessing: The vehicle-mounted edge device performs standardization processing on each frame of image to eliminate noise, correct viewing angle, etc., to ensure the accuracy of subsequent analysis.

[0046] Specifically, this invention employs the NT9852x series edge computing device, which integrates a neural network processing unit (NPU). With the aid of a hardware acceleration module, it preprocesses each frame of image, efficiently performing optimization operations such as illumination equalization, dehazing enhancement, and color correction. Simultaneously, it embeds precise timestamp information into each video frame. This not only ensures the time synchronization and high quality of the data but also provides a reliable time reference and data support for subsequent traffic light detection.

[0047] 1.3 Red Light Violation Recognition

[0048] Red light violation identification consists of three core steps: intersection event acquisition, vehicle trajectory determination, and event assessment.

[0049] 1.3.1 Intersection Event Acquisition

[0050] The core objective is to accurately capture intersection scenes and achieve continuous tracking and effective recognition of traffic lights.

[0051] (1) Intersection event start / stop determination

[0052] Triggering conditions: The video stream is processed in real time using a lightweight deep learning detection algorithm. When a traffic light is detected for the first time, it is marked as an intersection event.

[0053] Termination condition: If no new traffic light frame is detected within an interval of t seconds from the timestamp of the last detected traffic light, the intersection event is considered terminated.

[0054] (2) Traffic light detection and tracking

[0055] Inspection content: The shape (round light, left turn arrow light, right turn arrow light, U-turn light) and color (red light, yellow light, green light) of the traffic lights;

[0056] Tracking Algorithm: A lightweight, real-time multi-target tracking algorithm, SORT (Simple Online and Real-time Tracking), is employed. A unique trackid label is assigned to each traffic light to ensure tracking continuity. In this invention, the target is the traffic light in the image frame, and the tracking target is the tracking identifier.

[0057] 3) SORT Algorithm Optimization

[0058] The core of the SORT algorithm consists of a Kalman filter (target position prediction) and a Hungarian matching algorithm (matching detection results with the tracker). This invention makes two key optimizations for vehicle-mounted scenarios:

[0059] Kalman filter optimization: Utilize the stability of the signal light detection frame, replace the tracker position information with the detection frame position information, and retain only the velocity parameter; at the same time, record the relative offset between the tracked target and other targets to avoid tracking failure caused by the target not being detected for a long time.

[0060] Hungarian matching algorithm optimization: A hybrid loss matrix is ​​used as the matching criterion, that is, when the intersection-union ratio (IoU) is greater than 0, matching is based on IoU; when IoU is 0, matching is based on the distance to the target center point; a maximum distance threshold τ is set to filter abnormal matching pairs; by recording the initial left-right position relationship between the tracked targets, the tracked targets are updated and adjusted after matching to improve the matching accuracy.

[0061] (4) Removal of invalid data

[0062] The system synchronously detects stop lines and, based on the stop line timestamp, removes invalid segments from the traffic light tracking sequence before the vehicle has crossed the stop line.

[0063] If the vehicle speed remains at 0 throughout the intersection event (drivers waiting for traffic lights), then the relevant data for that event will be saved, thus conserving computing and storage resources.

[0064] 1.3.2 Determination of vehicle trajectory

[0065] By employing a collaborative calculation method combining GPS and six-axis sensor data, the system accurately identifies vehicle driving behavior (straight ahead, left turn, right turn, U-turn):

[0066] When the GPS signal strength meets the preset positioning accuracy requirements, the driving trajectory is determined using the GPS sequence information within the time interval of the intersection event.

[0067] When the GPS signal is weak, the vehicle's steering behavior and steering angle are calculated using data collected by the six-axis sensor, thereby determining the driving trajectory.

[0068] 1.3.3 Event Assessment

[0069] By filtering traffic light tracking sequences, matching driving trajectories, and determining colors, the behavior of running a red light is ultimately identified.

[0070] (1) Selection of signal light tracking sequence for this lane

[0071] The lightweight YOLOv5 detection model has a longer detection range and longer tracking sequences for traffic lights in this lane; therefore, the longest tracking sequence is selected as the initial candidate sequence. A threshold σ is set to remove tracking sequences of intersection traffic lights and falsely detected targets whose length is less than σ×Lenmax. Combined with the stop line timestamp, invalid sequence segments are removed again, retaining only valid tracking data.

[0072] (2) Traffic light type determination

[0073] Based on the vehicle's approaching trend, the traffic light images captured by the camera show a gradual change from blurry to clear. Therefore, the detection results of the last n frames of the effective tracking sequence are extracted, and the voting method is used to select the mode to determine the traffic light type (round light, left turn arrow light, right turn arrow light, U-turn light).

[0074] (3) Determination of running a red light

[0075] The vehicle's trajectory is correlated and matched with the traffic light tracking sequence of the lane. The traffic light colors in the matched sequence are statistically analyzed: if all colors are red, it is determined to be a red light violation; otherwise, it is not a red light violation.

[0076] 2.1 Red Light Violation Recognition

[0077] 2.1.1 Intersection Event Acquisition

[0078] (1) Determination of the initiation and termination of intersection events

[0079] The YOLOv5 object detection algorithm is used to perform real-time object detection on the input image frames. When the algorithm first detects a traffic light, this moment is marked as the start time of the intersection event; if no traffic light frame is detected again within an interval of t seconds from the timestamp of the last detected traffic light frame, the intersection event is considered to have terminated. A single intersection event can be represented by a set of image frames: A i ={frame1,frame2,...,frame n}, where A i For the i-th intersection event, frame nThis represents the traffic light detection result in the nth image frame within the event at this intersection.

[0080] After traffic light detection is completed for each frame, the results are fed into the SORT multi-target tracking algorithm for tracking processing. A unique tracking identifier is assigned to each traffic light target, resulting in a frame data structure containing tracking information, the expression of which is: frame j ={obj1, obj2, ..., obj m}, where frame j Let j be the set of traffic light targets in the j-th frame of the image within the intersection event;

[0081] (2) Definition of traffic light target attributes

[0082] The specific attributes of a single traffic light target can be defined as: obj j，k ={trackid, color, shape, x, y, w, h, nearesid, pos, offset x offset y}, where obj j，k This represents the k-th traffic light target within the j-th frame; `trackid` is the unique tracking identifier for this target; `color` is the traffic light's color attribute, with values ​​of red, yellow, and green; `shape` is the traffic light's shape attribute, with values ​​of round light, left-turn arrow, right-turn arrow, and U-turn light; `x` and `y` are the pixel coordinates of the center point of the traffic light detection box, and `w` and `h` are the pixel width and height of the detection box, respectively; `nearestid` is the tracking identifier information of the nearest traffic light; `pos` is the left-right relative position of the traffic light and the `nearestid` traffic light; `offset`... x and offset y均是 The offset of the traffic light relative to the nearest traffic light.

[0083] (3) SORT multi-target tracking

[0084] The real-time multi-target tracking algorithm SORT mainly consists of Kalman filtering and the Hungarian algorithm. After image detection is performed using the YOLOv5 target detection algorithm, the detected targets are fed into the Kalman filter to predict and update the targets in the tracking set. Whenever a new target is generated, the nearest distance between this target and each target in the existing tracking target set is calculated, and the tracking identifier nearestid, the left-right relative position pos, and the relative offset offset of the nearest distance target are recorded. x offset y .

[0085] Definition of state vector:

[0086] This invention employs standard linear Kalman filtering as the prediction and update method for tracking targets. First, the state vector of the tracking target is defined as x. k ∈R 8 The definition is as follows, x k =[x k y k w k h k v xk v yk v hk ] T , where x k y k Let w be the center coordinate of the target detection box in the k-th frame. k h k Let v be the width and height of the target detection bounding box in the k-th frame. xk v yk Let v be the velocity of the target center in the x and y directions. yk v hk The rate of change of the width and height of the target detection box;

[0087] Prediction phase:

[0088] Using Kalman filtering to track the target set T in frame k-1 k-1 ={t k-1，1 , ..., t k-1，m Prediction is performed based on the tracking target state x in the (k-1)th frame. k-1 Predict the prior state of the k-th frame using a linear state transition model. Where F∈R 8×8 The state transition matrix is ​​defined as follows:

[0089]

[0090] in, The time interval between two adjacent detection frames;

[0091] Simultaneously, update the prior covariance matrix: , where P k-1 Let Q be the posterior covariance matrix of the (k-1)th frame, where Q∈R 8×8 Let be the process noise covariance matrix.

[0092] Observation and matching phase:

[0093] The YOLOv5 object detection model is used to detect objects in the k-th frame image, and the output detection set D is generated. k ={d k1 d k2 , ..., d km}, where for each detection box d ki =[x ki y ki w ki h ki ].

[0094] The Hungarian algorithm is used to track the set T. k-1 ={t k-1，1 , ..., t k-1，m} and the detection set D k The matching process is based on the mixture loss matrix L∈R. M×N Complete the matching calculation:

[0095] ,

[0096] Where L(i,j)) represents the i-th detection box d i With the j-th tracking box t j The matching loss matrix;

[0097] After the initial matching is completed, there may be cases where the relative offset between the detected target position and the tracked target position of the matched pair is too large, leading to a matching error. These erroneous matching pairs are recorded as abnormal matching pairs. Therefore, it is necessary to set a position deviation threshold τ to determine whether the matching pair is an abnormal match. The threshold τ is adaptively adjusted according to the actual scene, such as image resolution and motion speed. In this invention, the position deviation threshold τ is set to 100 pixels.

[0098] Let the set of matching pairs obtained after the initial matching be P = {(d i , t j )|i∈[1,M],j∈[1,N]}, for each matching pair (d i d j ), calculate the Euclidean distance between its center points. If dist(d i , t j If ) > τ, then the matching pair is determined to be an abnormal matching pair, and the detection target is removed from D. k Remove from the middle and add to the middle. middle.

[0099] After completing the above matching and outlier pair filtering, the removed D will be... k and T k-1 Repeat the above matching process for a second matching. Meanwhile, The detected target is used as the newly added tracking target.

[0100] After completing the above rematch, T is judged sequentially. kThe system tracks the left-right relative position of targets and corrects the trackid attribute of targets that are incorrectly matched.

[0101] By leveraging the rigidity of traffic signal devices, the accuracy of matching and the stability of tracking can be further improved by recording the left-right positional relationships between traffic lights. Specifically, whenever a new tracking target t is encountered... kj During generation, t is calculated. kj With T k The Euclidean distance of the target being tracked is recorded as t. kj The nearest target and its corresponding left-right position.

[0102] Update phase:

[0103] For a valid matching pair (t) k-1 d kj First, the detection box d kj Location information [x kj y kj w kj h kj As observed value z kj ∈R 4 Define the observation matrix H∈R 4 Used to extract the position dimension from the state vector:

[0104]

[0105] Then, the Kalman gain is calculated: , where R∈R 4×4 The observation noise covariance matrix (characterizing the position error of the detection model) is used.

[0106] Prior state ,at this time, [x] k y k w k h k Corrected by the detection box information, velocity information [v] xk v yk v wk v hk Updated by Kalman filtering to ensure a balance between position accuracy and motion continuity.

[0107] Finally, the updated covariance matrix , where I is the identity matrix.

[0108] Missed detection compensation mechanism:

[0109] Because the detection model has instances of missed detections, the corresponding tracked targets cannot be accurately updated, leading to mismatches between detected and tracked targets. To resolve this mismatch issue, the tracking set T is traversed... k Tracking target t in ki Determine its nearest target t. kj Whether to update; if not, then use the initially calculated offset. x offset y Update t kj Position information of the state vector:

[0110] ;

[0111] Resource optimization strategies:

[0112] During the event triggering phase at the intersection, vehicle speed needs to be monitored in real time. If the vehicle speed drops to 0 km / s during an intersection event and lasts for more than 3 seconds, it indicates that the driver is waiting for the traffic light. At this point, the relevant data for the intersection event can be cleared, thereby achieving efficient saving of computing and storage resources.

[0113] 2.2.2 Determination of vehicle trajectory

[0114] Monitor the driving status of vehicles during their passage through intersections to identify four typical driving behaviors: going straight, turning left, turning right, and making a U-turn.

[0115] Vehicle trajectory collection: The complete trajectory data of the vehicle during this intersection passage event is collected by using a collaborative fusion calculation method of GPS positioning data and six-axis sensor data.

[0116] Vehicle trajectory determination: When the GPS signal strength is greater than the threshold δ, the GPS positioning data sequence within the time interval corresponding to the current intersection passage event is extracted. By solving and fitting the continuous positioning coordinates in the sequence, the vehicle's driving trajectory within the intersection range is accurately calibrated. When the GPS signal strength is less than the threshold δ, the vehicle's motion attitude data is collected in real time through a six-axis sensor. The motion attitude data is then solved to obtain the vehicle's steering state.

[0117] 2.2.3 Event Assessment

[0118] By tracking sequence filtering, invalid segment removal, traffic light type determination, and color sequence matching, it is finally determined whether it is a red light running event.

[0119] (1) Sequence screening based on length features

[0120] Define the intersection event tracking sequence set T = [T1, T2, ..., T] m This includes the signal light sequence for this lane, the signal light sequence for intersecting lanes, and the false detection sequence.

[0121] Extract the lengths of all sequences, and denote the length of the longest sequence as Len. max This is used as the initial candidate sequence for the traffic lights in this lane. A length threshold σ is set, with a value range of [0.3, 0.5]. When Len(T) i ) < Len max ×σ,T i If a sequence is invalid, it will be removed from the tracking sequence set T.

[0122] (2) Removal of invalid segments based on stop line timestamps

[0123] First, the stop line is detected in real time at the intersection using the YOLOv5 detection algorithm, and its timestamp is extracted. Then, the timing of each tracked target in the tracking sequence set T is judged in turn, and invalid sequence segments before the vehicle crosses the stop line are removed, while valid tracking data is retained.

[0124] (3) Traffic light type determination

[0125] When a vehicle is traveling at an intersection, it moves closer to the traffic light in its lane, and the corresponding traffic light image captured by the camera shows a gradual change from blurry to clear. Therefore, the last n frames of the filtered traffic light tracking sequence for this lane are selected as valid samples (n is a preset number of frames, ranging from 5 to 10). A voting method is used to determine the mode of traffic lights corresponding to the tracking sequence (including round lights, left-turn arrow lights, right-turn arrow lights, and U-turn lights).

[0126] (4) Judgment of red light running incidents

[0127] Based on the vehicle's trajectory, a time-related matching is established between this trajectory and the traffic light tracking sequence detection results for this lane. The matched traffic light color sequence S = {s1, s2, ..., s...} is then statistically analyzed. k}, where s i The values ​​of are red, yellow, and green. If the color sequence S is entirely red, the event at this intersection is determined to be a red-light violation; otherwise, it is determined to be a non-red-light violation.

[0128] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A method for detecting red-light running events using multi-data fusion, characterized in that, Includes the following steps: Step S1: Collect video streams from the front of the vehicle in real time using the vehicle-mounted camera, and simultaneously collect the vehicle's position information and six-axis sensor data; Based on the location information and six-axis sensor data, the vehicle's trajectory through the intersection is determined; Step S2: Divide the video stream into several image frames, detect each image frame in sequence, and record the image frame in which the traffic light is first detected as the intersection event start frame; starting from the intersection event start frame, detect and track the traffic lights in each subsequent image frame in sequence, and assign a unique tracking identifier to the traffic lights in each subsequent image frame to obtain the tracking sequence. Step S3: Determine the lane where the vehicle is located, and select several tracking sequences corresponding to the traffic lights and the lane from all tracking sequences. Select the longest tracking sequence from among the several tracking sequences to obtain the effective tracking sequence. Perform time correlation matching between the driving trajectory and the effective tracking sequence to obtain the matching time period. Analyze the color of the traffic lights in the effective tracking sequence within the matching time period. If all the colors are red, it is determined that the vehicle has run a red light.

2. The method for detecting red-light violations by multi-data fusion according to claim 1, characterized in that, The location information is based on the vehicle's GPS positioning, and the acquisition process of the six-axis sensor data includes: The six-axis sensing data is acquired based on a six-axis sensing unit built into the vehicle. The six-axis sensing unit includes a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer acquires the linear acceleration of the vehicle in the front-back, left-rear, and up-down directions, and the three-axis gyroscope acquires the rotational angular velocity of the vehicle in the front-back, left-rear, and up-down directions.

3. The method for detecting red-light violations by multi-data fusion according to claim 1, characterized in that, The process of determining the trajectory of a vehicle as it passes through an intersection includes: The vehicle's GPS signal strength is acquired in real time. When the GPS signal strength is greater than a preset threshold, the driving trajectory is calculated using the location information collected by GPS. When the GPS signal strength is less than or equal to the preset threshold, the vehicle's steering state is calculated using the six-axis sensor data to determine the driving trajectory.

4. The method for detecting red-light violations by multi-data fusion according to claim 1, characterized in that, The process of tracking traffic lights in subsequent image frames is based on the SORT multi-object tracking algorithm, specifically including: For any image frame after the start frame of the intersection event, a tracking identifier is created for each traffic light detected in the image frame. The tracking identifier is specifically a state vector, which includes the width, height, and position coordinates of the traffic light in the image frame, thus obtaining the bounding box of the traffic light. The traffic light detected in the next image frame of the image frame is obtained and denoted as the next traffic light. The bounding box of the next traffic light is obtained and denoted as the detection box. The bounding box and the detection box are associated and matched to obtain the intersection-union ratio (IU / R) between the bounding box and the detection box. If the IU / R is greater than 0, a loss matrix is ​​constructed based on the IU / R for matching. If the IU / R is 0, a loss matrix is ​​constructed based on the Euclidean distance between the center point of the detection box and the center point of the bounding box for matching. All created tracking identifiers are recorded as existing tracking identifiers, and the bounding boxes of existing tracking identifiers are recorded as existing bounding boxes. After a new tracking identifier is created, the existing bounding box that is closest to the bounding box of the new tracking identifier is obtained and recorded as the nearest bounding box. The existing tracking identifier corresponding to the nearest bounding box is obtained and recorded as the nearest tracking identifier. The positional relationship between the new tracking identifier and the nearest tracking identifier is obtained, and the offset of the new tracking identifier relative to the nearest tracking identifier is obtained.

5. The method for detecting red-light violations by multi-data fusion according to claim 4, characterized in that, The process of tracking traffic lights in subsequent image frames also includes compensation for missed detections of tracking markers, which includes: For each tracking marker, record its positional relationship and offset with the most recent tracking marker. If, in a subsequent image frame, a tracking marker is missed and no detection box is updated, determine whether the most recent tracking marker recorded for the missed tracking marker has been successfully updated in the image frame. If the most recent tracking marker has been successfully updated, calculate the current position of the missed tracking marker based on the updated position of the most recent tracking marker and the positional relationship and offset, and update the missed tracking marker.

6. The method for detecting red-light violations by multi-data fusion according to claim 1, characterized in that, The process of obtaining a valid tracking sequence includes: The length of each tracking sequence within a plurality of tracking sequences is obtained, and a length ratio threshold is set. For any tracking sequence within the plurality of tracking sequences, the product of the length of the tracking sequence and the length ratio threshold is obtained. If the length of the tracking sequence is less than the product, the tracking sequence is removed. Among all the remaining tracking sequences, the longest tracking sequence is selected and recorded as the valid tracking sequence.

7. The method for detecting red-light violations by multi-data fusion according to claim 1, characterized in that, Before performing time-correlation matching between the driving trajectory and the valid tracking sequence, an invalid segment removal process is also included, which includes: The stop line is identified in all image frames, and the timestamp corresponding to the image frame in which the stop line is first identified is obtained. Tracking markers indicating that the vehicle has not crossed the stop line are removed from the effective tracking sequence.

8. A system for detecting red-light running incidents through multi-data fusion, characterized in that, include: Data acquisition module: Real-time acquisition of video stream from the front of the vehicle via onboard camera, and simultaneous acquisition of vehicle position information and six-axis sensor data; Based on the location information and six-axis sensor data, the vehicle's trajectory through the intersection is determined: The detection and tracking module divides the video stream into several image frames, detects each image frame sequentially, and records the first image frame in which a traffic light is detected as the intersection event start frame. Starting from the intersection event start frame, the module sequentially detects and tracks the traffic lights in each subsequent image frame, and assigns a unique tracking identifier to each traffic light in each subsequent image frame to obtain a tracking sequence. Judgment Module: Determines the lane where the vehicle is located, and selects several tracking sequences corresponding to the traffic lights and the lane from all tracking sequences. Then, selects the longest tracking sequence from among the several tracking sequences to obtain the valid tracking sequence. Performs time correlation matching between the driving trajectory and the valid tracking sequence to obtain the matching time period. Analyzes the color of the traffic lights in the valid tracking sequence within the matching time period. If all the colors are red, it is determined that the vehicle has run a red light.

Images