A traffic abnormal event detection method based on multi-source data fusion

Abstract

The application discloses a traffic abnormal event detection method based on multi-source data fusion, comprising: a single millimeter wave radar and multiple industrial cameras, target detection data is acquired for sensor characteristics; fusion information is obtained by associating multi-source information; a fusion target is tracked, a target trajectory is formed, the trajectory is smoothed and filtered, and a filtered target trajectory is formed; illegal event detection is realized by analyzing the target trajectory, the illegal events include intrusion, overspeed, underspeed, lane changing, illegal parking, reverse driving and line pressing; traffic state event detection is realized by analyzing the correlation between a lane level detection area and target data, the traffic state events include congestion, and event evidence is left for aligned event time, including event video and event photo; event reexamination is carried out for specific events through target trajectory anti-shake verification, event qualitative analysis and reporting are carried out. The application can effectively realize intelligent prevention and control of traffic safety risks, optimize traffic efficiency and reduce the probability of traffic accidents.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic detection technology, and in particular, it is a method for detecting traffic anomalies based on multi-source data fusion. Background Technology

[0002] In recent years, the automotive industry has experienced rapid development. Electric vehicles and hybrid vehicles, with their environmental friendliness and energy efficiency, have quickly penetrated the market. The widespread adoption of low-cost, low-noise transportation tools has accelerated the iteration of urban transportation networks, and cars have become the primary mode of daily travel for residents. However, at the same time, the surge in car ownership and the increase in travel frequency have brought unprecedented pressure to urban and highway traffic. The continuous increase in road saturation during peak hours urgently requires intelligent traffic light control to provide optimal strategies to alleviate traffic pressure. Intelligent perception of traffic conditions requires lane-level congestion information and coordinated decision-making based on traffic phases. The accumulation of traffic pressure directly increases the risk of traffic accidents. Under these circumstances, compliance with traffic rules is crucial. Illegal driving threatens road traffic safety, disrupts traffic order, endangers public safety, and consumes public resources. Accurate identification of illegal behavior helps to further restrain driver behavior, reduce the probability of accidents, and create a safe, efficient, and orderly environment for transportation.

[0003] Among existing traffic incident detection methods, induction coils can only identify vehicle stagnation and speeding incidents, and have high maintenance costs. Microwave detectors are not affected by lighting conditions, but have low accuracy in target type identification and a high false alarm rate. Camera-based incident detection is greatly affected by weather, and while it is good for speed measurement in specific sections, it is prone to false alarms in real-time speed measurement. Furthermore, the detection frame may jump during inference, resulting in large trajectory deviations and significant interference with the statistical analysis of relevant traffic parameters. Detection methods based on multi-source data fusion have become the mainstream. The challenges lie in multi-source data synchronization and coordinate calibration, as well as how to effectively utilize the fused data. Summary of the Invention

[0004] The purpose of this invention is to address the problems existing in the prior art by providing a traffic anomaly event detection method based on multi-source data fusion, thereby achieving intelligent prevention and control of traffic safety risks, optimizing traffic efficiency, and reducing the probability of traffic accidents. This invention employs both radar and camera sensors. Radar point cloud data is processed to obtain radar target data. A depth model is used to perform target recognition and attribute identification on the image information. Radar and camera target tracking are processed separately, and target fusion is performed after spatial synchronization. The fused trajectory is analyzed and processed. An event detection system and method are designed for the characteristics of various traffic events. In summary, this method provides a reliable approach for intelligent traffic event detection.

[0005] The technical solution for achieving the objective of this invention is: a method for detecting traffic anomalies based on multi-source data fusion, the method comprising the following steps:

[0006] Step 1: Collect radar target detection point status information and perform radar target detection; collect camera video image information and perform image target detection; capture target screenshots based on target position and size information, and perform attribute detection on the target screenshots;

[0007] Step 2: Track radar targets and image targets, establish target association between camera and radar, realize multi-source information target fusion, obtain fused targets, and then execute steps 3 and 4 simultaneously.

[0008] Step 3: Based on the positional relationship between the fusion target and the preset detection area, statistically analyze relevant traffic indicators to achieve traffic status event detection;

[0009] Step 4: Establish a tracker, use an unscented Kalman filter to track the fused target, and perform secondary filtering on the trajectory; then detect violation events based on the filtered trajectory.

[0010] Step 5: Verify the trajectory stabilization of the detected violations and report the violations that pass the verification.

[0011] Further, in step 1, the target screenshot is obtained by cropping the original image based on screenshot parameters calculated according to the position and size of the target detected in the image; wherein, the calculation formula for the screenshot parameters is: ; In the formula, The coordinates of the top-left corner of the image. The coordinates of the bottom right corner of the image. The coordinates of the top right corner of the screenshot. The coordinates of the bottom right corner of the screenshot. This is the screenshot coefficient.

[0012] Furthermore, in step 2, target association is established between the camera and the radar to achieve multi-source information target fusion, specifically including:

[0013] In the radar coordinate system, spatial and temporal synchronization is performed on radar targets and image targets to achieve target association between the camera and the radar, resulting in a fused target. The information of the fused target includes radar target information, associated camera target detection box information, category, probability, target attribute information, and detection serial number given by each detection.

[0014] Specifically, spatial synchronization between the radar target and the image target is achieved through the camera's intrinsic and extrinsic parameter transformation matrix; time synchronization is achieved by predicting the radar target's position based on the time difference of multi-source data; the formula for time synchronization is:

[0015] ;

[0016] In the formula, Predict the location of radar targets. For radar to detect location, and These represent the average speeds of the radar in the x and y directions over 1 second. and These are the detection times for the camera and radar, respectively.

[0017] Furthermore, the information configuration of the preset detection area in step 3 includes the area size, the direction of entry and exit of the area, the solid and dashed lines of the area, the left and right detection area numbers, and the relevant detection event thresholds.

[0018] Furthermore, the relevant traffic indicators in step 3 include: the number of vehicles in the area, the distance between vehicles, the average speed of vehicles, the static queue length, and the dynamic queue length;

[0019] The method for counting the number of vehicles within the area is as follows:

[0020]

[0021] In the formula, P is the target location, and A and B represent adjacent points within the region. , , The coordinates of A, B, and P are respectively. To determine whether a ray emanating from point P intersects the boundary of the region; m is the number of points in the region. These are the sets of coordinate points for the i-th and (i+1)-th regions, respectively. is_inregion indicates whether the target point is within the region; 1 indicates it is within the region, and 0 indicates it is not. N is the total number of detected targets, and num_cars represents the number of targets within the region. Indicates whether target point j is within the region;

[0022] The average vehicle speed is the average speed of targets within the area;

[0023] The calculation method for vehicle body spacing is as follows:

[0024]

[0025] In the formula, For vehicle body spacing, and These represent the distances from the radar sensors to the targets at the two ends of the area. For the length of the target vehicle at the near end;

[0026] The static queue length is determined by statistical analysis of the target's motion state, where the motion state is identified as follows:

[0027]

[0028] In the formula, This indicates whether the state is at rest; a value of 1 indicates that the state is at rest, and a value of 0 indicates that the state is not at rest. The target position at the current moment. The target position is given by ths, the distance threshold is given by v. ave In a short period of time and The average speed between positions over time, thv is the stationary speed threshold; when the first car in the area, i.e. the head of the queue, is stationary, it indicates that there is a static queue length; when a car after the head of the queue is stationary and the distance between it and the car behind it exceeds the safe distance, that car is the tail of the queue, and the static queue length is the distance between the head of the queue and the tail of the queue.

[0029] The dynamic queue length is the distance between the front and the back of the queue. By viewing the distance between adjacent vehicles in the area, the vehicle in front becomes the back of the queue when the distance between the vehicle in front and the vehicle behind is greater than the safe distance.

[0030] Furthermore, step 3 implements traffic state event detection, specifically including:

[0031] When the static queue length is less than the static queue threshold, or when two of the following three conditions are met: the dynamic queue length is less than the dynamic queue threshold, the number of vehicles in the interval is less than the vehicle threshold, and the vehicle spacing is less than the vehicle spacing threshold, the congestion time is refreshed; otherwise, the congestion start time is not refreshed.

[0032] When the congestion time reaches the preset time threshold, the congestion is recorded and the congestion event is reported; otherwise, the congestion end time is recorded and the congestion start time is updated.

[0033] Furthermore, in step 4, a moving average filter is used for trajectory filtering. The filtering formula is as follows:

[0034]

[0035] In the formula, Here are the x and y coordinates of the filtered trajectory point k, and N is the size of the filtering window. Let k+i be the x and y coordinates of the target trajectory point k+i within the sliding window.

[0036] Furthermore, step 4 involves detecting traffic violations based on the filtered trajectory, specifically including:

[0037] When the target's trajectory contains a target direction opposite to the direction of the preset detection area, it is marked as a reverse state. When the reverse state continues for a first preset threshold distance, it is marked as a reverse pending event.

[0038] When the target speed continuously exceeds the preset threshold speed within the preset detection area, it is marked as an overspeed pending event;

[0039] When no queue occurs, if the target is a restricted parking area type, and the target stays in the restricted parking area for more than a preset threshold time and moves less than a preset distance, it is marked as a pending parking violation event.

[0040] Based on the preset detection area and lane direction, the area in the direction of movement is intercepted. The intercepted area is the effective lane change detection area. When the lane line crossed by the target trajectory is a solid line and the continuous movement before and after crossing meets the second preset threshold distance, it is marked as a lane change pending event.

[0041] When the target detection box continuously contains a solid line within the middle threshold range of its lower edge, it indicates that the target is in a line-crossing state. When the line-crossing state continues for a third preset threshold distance, it is marked as a pending line-crossing event. When the target's trajectory crosses the solid line in both the lane area and the non-lane area, it is also marked as a pending line-crossing event.

[0042] When the target is not in a queue or the distance between vehicles is greater than the safe distance, it is marked as a pending underspeed event.

[0043] Furthermore, the formula for determining the target line is:

[0044]

[0045] In the formula, and These are the left and right x-coordinates of the area to be judged as the area under the line. and These are the x-coordinates of the left and right corner points of the bottom edge of the target detection box, respectively. The parameter used to extract the lower edge interval has a value range of 0-1. The x-coordinate of the intersection of the solid line and the lower edge of the target detection box is given, and s represents the state of being pressed against the line. When the value is less than 0, it indicates a pressure line.

[0046] Furthermore, in step 5, trajectory stabilization verification is performed on the detected violations. The verification formula is as follows:

[0047]

[0048] If any formula returns 0, the verification fails; otherwise, the verification passes.

[0049] In the formula, , , The anomaly values ​​for position, velocity, and acceleration changes are respectively calculated. , , These are the target's position, velocity, and acceleration in the next frame, respectively. , , These are the target's position, velocity, and acceleration from the previous frame, respectively. The average velocity is between two frames, and t is the time interval between the two frames of data. , , These are the threshold values ​​for position, velocity, and acceleration, respectively.

[0050] Compared with the prior art, the significant advantages of this invention are:

[0051] (1) This invention integrates radar and multi-video information, breaks through the limitations of blind spots and poor environmental adaptability of single sensor monitoring, makes full use of target attribute information for event identification, constructs a joint identification model by extracting multi-dimensional target attribute information, improves the accuracy of event judgment, and the synchronously triggered evidence retention mechanism reversely verifies the reliability of the identification results, while providing data support for subsequent tracing and optimization.

[0052] (2) The present invention designs flexibly configurable parameter items and dynamic threshold adjustment mechanism for the unique attributes of different events, so as to achieve accurate adaptation and recognition expansion for multiple scenarios and multiple types of events.

[0053] (3) This invention performs secondary filtering on the fusion trajectory of multi-source sensors and performs anti-shake verification on the trajectory of the event area after event detection, which adds a reliable guarantee for the accuracy of event recognition, reduces the false alarm rate of events, and provides a new method for traffic event detection.

[0054] (4) Based on multi-source sensors, the time synchronization processing is specifically as follows: according to the position, speed and time difference given by each sensor, the data is preprocessed forward and backward, the data timestamp is unified, and then fusion processing is performed; spatial synchronization is based on the transformation relationship between the camera and the world coordinate system, the transformation between the camera and the radar is simulated, the optimal external parameters are found by traversing each rotation angle, and the spatial synchronization matrix is ​​calculated based on the camera intrinsic parameters, breaking the limitation of local matching of the single mapping matrix in hand-eye calibration.

[0055] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating a traffic anomaly detection method based on multi-source data fusion in one embodiment.

[0057] Figure 2 This is a schematic diagram of target attribute detection in a traffic anomaly detection method based on multi-source data fusion in one embodiment.

[0058] Figure 3This is a schematic diagram of the radar-visual and multi-camera target fusion tracking process of a traffic anomaly detection method based on multi-source data fusion in one embodiment.

[0059] Figure 4 This is a schematic diagram of the traffic state event detection process of a traffic anomaly event detection method based on multi-source data fusion in one embodiment.

[0060] Figure 5 This is a schematic diagram of the violation event detection process of a traffic anomaly event detection method based on multi-source data fusion in one embodiment.

[0061] Figure 6 This is a schematic diagram of the anti-shake verification algorithm of a traffic anomaly detection method based on multi-source data fusion in one embodiment.

[0062] Figure 7 This is a schematic diagram of the evidence collection process for a traffic anomaly detection method based on multi-source data fusion in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0065] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0066] In one embodiment, combined Figure 1 This paper provides a method for detecting traffic anomalies based on multi-source data fusion, the method comprising the following steps:

[0067] Step 1: Collect radar target detection point status information and perform radar target detection; collect camera video image information and perform image target detection; capture target screenshots based on target position and size information, and perform attribute detection on the target screenshots;

[0068] Here, target attributes include target category, vehicle color, vehicle model, license plate number, license plate, and pedestrian attributes. Different attribute detection models are used to detect the target based on its category.

[0069] Step 2: Track radar targets and image targets, establish target association between camera and radar, realize multi-source information target fusion, obtain fused targets, and then execute steps 3 and 4 simultaneously.

[0070] Step 3: Based on the positional relationship between the fusion target and the preset detection area, statistically analyze relevant traffic indicators to achieve traffic status event detection;

[0071] Step 4: Establish a tracker, use an unscented Kalman filter to track the fused target, and perform secondary filtering on the trajectory; then detect violation events based on the filtered trajectory.

[0072] Step 5: Verify the trajectory stabilization of the detected violations and report the violations that pass the verification.

[0073] Furthermore, in one embodiment, in step 1, the target screenshot is obtained by cropping the original image based on screenshot parameters calculated according to the position and size of the target detected in the image; wherein, the calculation formula for the screenshot parameters is:

[0074]

[0075] In the formula, The coordinates of the top-left corner of the image. The coordinates of the bottom right corner of the image. The coordinates of the top right corner of the screenshot. The coordinates of the bottom right corner of the screenshot. This is the screenshot coefficient.

[0076] Furthermore, in one embodiment, combined with Figure 3 In step 2, radar target tracking is achieved through unscented Kalman filtering and Hungarian matching. Image target correlation is achieved using image-target intersection-over-union ratio, attribute similarity, and motion similarity. The solution formula is:

[0077]

[0078] The tracker has a total of m targets, and the current frame detects n targets. Calculate the intersection-union ratio (I) of the detection boxes of each tracker target and the current frame target.OU The value of i ranges from 1 to m, and the value of j ranges from 1 to n. The best target matcher is calculated using the Hungarian matching method. At the same time, the target is constrained by attribute probability, attribute category, motion direction and motion attribute, and the target is associated with the video target of the previous and next frames.

[0079] Furthermore, in one embodiment, step 2 involves establishing target association between the camera and radar to achieve multi-source information target fusion, specifically including:

[0080] In the radar coordinate system, spatial and temporal synchronization is performed on radar targets and image targets to achieve target association between the camera and the radar, resulting in a fused target. The information of the fused target includes radar target information, associated camera target detection box information, category, probability, target attribute information, and detection serial number given by each detection.

[0081] Specifically, spatial synchronization between the radar target and the image target is achieved through the camera's intrinsic and extrinsic parameter transformation matrix; time synchronization is achieved by predicting the radar target's position based on the time difference of multi-source data; the formula for time synchronization is:

[0082]

[0083] In the formula, Predict the location of radar targets. For radar to detect location, and These represent the average speeds of the radar in the x and y directions over 1 second. and These are the detection times for the camera and radar, respectively.

[0084] Furthermore, in one embodiment, the information configuration of the preset detection area in step 3 includes the area size, the directionality of the area's entry and exit, the area's solid and dashed lines, the left and right detection area numbers, and the relevant detection event thresholds.

[0085] Furthermore, in one embodiment, the relevant traffic indicators in step 3 include: the number of vehicles in the area, the distance between vehicles, the average speed of vehicles, the static queue length, and the dynamic queue length.

[0086] The method for counting the number of vehicles within the area is as follows:

[0087]

[0088] In the formula, P is the target location, and A and B represent adjacent points within the region. , , The coordinates of A, B, and P are respectively. To determine whether a ray emanating from point P intersects the boundary of the region; m is the number of points in the region. These are the sets of coordinate points for the i-th and (i+1)-th regions, respectively. is_inregion indicates whether the target point is within the region; 1 indicates it is within the region, and 0 indicates it is not. N is the total number of detected targets, and num_cars represents the number of targets within the region. Indicates whether target point j is within the region;

[0089] The average vehicle speed is the average speed of targets within the area;

[0090] The calculation method for vehicle body spacing is as follows:

[0091]

[0092] In the formula, For vehicle body spacing, and These represent the distances from the radar sensors to the targets at the two ends of the area. For the length of the target vehicle at the near end;

[0093] The static queue length is determined by statistical analysis of the target's motion state, where the motion state is identified as follows:

[0094]

[0095] In the formula, This indicates whether the state is at rest; a value of 1 indicates that the state is at rest, and a value of 0 indicates that the state is not at rest. The target position at the current moment. The target position is given by ths, the distance threshold is given by v. ave In a short period of time and The average speed between positions over time, thv is the stationary speed threshold; when the first car in the area, i.e. the head of the queue, is stationary, it indicates that there is a static queue length; when a car after the head of the queue is stationary and the distance between it and the car behind it exceeds the safe distance, that car is the tail of the queue, and the static queue length is the distance between the head of the queue and the tail of the queue.

[0096] The dynamic queue length is the distance between the front and the back of the queue. By viewing the distance between adjacent vehicles in the area, the vehicle in front becomes the back of the queue when the distance between the vehicle in front and the vehicle behind is greater than the safe distance.

[0097] Furthermore, in one embodiment, combined with Figure 4 Step 3 involves implementing traffic state event detection, specifically including:

[0098] When the static queue length is less than the static queue threshold, or when two of the following three conditions are met: the dynamic queue length is less than the dynamic queue threshold, the number of vehicles in the interval is less than the vehicle threshold, and the vehicle spacing is less than the vehicle spacing threshold, the congestion time is refreshed; otherwise, the congestion start time is not refreshed.

[0099] When the congestion time reaches the preset time threshold, the congestion is recorded and the congestion event is reported; otherwise, the congestion end time is recorded and the congestion start time is updated.

[0100] Furthermore, in one embodiment, step 4 utilizes moving average filtering for trajectory filtering, with the following filtering formula:

[0101]

[0102] In the formula, Here are the x and y coordinates of the filtered trajectory point k, and N is the size of the filtering window. Let k+i be the x and y coordinates of the target trajectory point k+i within the sliding window.

[0103] Furthermore, in one embodiment, combined with Figure 5 Step 4 involves detecting traffic violations based on the filtered trajectory, specifically including:

[0104] When the target's trajectory contains a target direction opposite to the direction of the preset detection area, it is marked as a reverse state. When the reverse state continues for a first preset threshold distance, it is marked as a reverse pending event.

[0105] When the target speed continuously exceeds the preset threshold speed within the preset detection area, it is marked as an overspeed pending event;

[0106] When no queue occurs, if the target is a restricted parking area type, and the target stays in the restricted parking area for more than a preset threshold time and moves less than a preset distance, it is marked as a pending parking violation event.

[0107] Based on the preset detection area and lane direction, the area in the direction of movement is intercepted. The intercepted area is the effective lane change detection area. When the lane line crossed by the target trajectory is a solid line and the continuous movement before and after crossing meets the second preset threshold distance, it is marked as a lane change pending event.

[0108] When the target detection box continuously contains a solid line within the middle threshold range of its lower edge, it indicates that the target is in a line-crossing state. When the line-crossing state continues for a third preset threshold distance, it is marked as a pending line-crossing event. When the target's trajectory crosses the solid line in both the lane area and the non-lane area, it is also marked as a pending line-crossing event.

[0109] When the target is not in a queue or the distance between vehicles is greater than the safe distance, it is marked as a pending underspeed event.

[0110] Preferably, in some embodiments, the formula for determining the target pressure line is:

[0111]

[0112] In the formula, and These are the left and right x-coordinates of the area to be judged as the area under the line. and These are the x-coordinates of the left and right corner points of the bottom edge of the target detection box, respectively. The parameter used to extract the lower edge interval has a value range of 0-1. The x-coordinate of the intersection of the solid line and the lower edge of the target detection box is given, and s represents the state of being pressed against the line. When the value is less than 0, it indicates a pressure line.

[0113] Preferably, in some embodiments, the safe distance is calculated using the following formula:

[0114]

[0115] In the formula, s is the safe distance, v is the current speed of the target vehicle, and t is the unit time. This represents the average train length in the regional bureau.

[0116] Furthermore, in one embodiment, combined with Figure 6 In step 5, trajectory stabilization verification is performed on the detected violations. The verification formula is as follows:

[0117]

[0118] If any formula returns 0, the verification fails; otherwise, the verification passes.

[0119] In the formula, , , The anomaly values ​​for position, velocity, and acceleration changes are respectively calculated. , , These are the target's position, velocity, and acceleration in the next frame, respectively. , , These are the target's position, velocity, and acceleration from the previous frame, respectively. The average velocity is between two frames, and t is the time interval between the two frames of data. , , These are the threshold values ​​for position, velocity, and acceleration, respectively.

[0120] Here, the purpose of the anti-shake verification algorithm is to verify the rationality of the trajectory. It comprehensively considers the target's position, velocity, and acceleration, calculates the velocity error through the acceleration error, and calculates the displacement error through the velocity error. When both the velocity error and the position error are outside the reasonable range, the target trajectory anti-shake verification fails; otherwise, it passes the anti-shake verification.

[0121] In one embodiment, a traffic anomaly event detection system based on multi-source data fusion is provided, the system comprising:

[0122] The first module is used to: collect radar target detection point status information and perform radar target detection; collect camera video image information and perform image target detection; and take a screenshot of the target based on the target position and size information, and perform attribute detection on the screenshot.

[0123] The second module is used to: track radar targets and image targets, establish target association between the camera and radar, realize multi-source information target fusion, obtain fused targets, and then execute the third and fourth modules simultaneously.

[0124] The third module is used to: statistically analyze relevant traffic indicators based on the positional relationship between the fusion target and the preset detection area, and realize traffic status event detection;

[0125] The fourth module is used to: establish a tracker, use an unscented Kalman filter to track the fused target, and perform secondary filtering on the trajectory; then detect violation events based on the filtered trajectory.

[0126] The fifth module is used to perform trajectory stabilization verification on detected violations and to report violations that pass verification.

[0127] It should be noted that the third and fourth modules are not limited to being executed simultaneously; they can also be executed sequentially.

[0128] Specific limitations regarding the traffic anomaly detection system based on multi-source data fusion can be found in the limitations of the traffic anomaly detection method based on multi-source data fusion described above, and will not be repeated here. Each module in the aforementioned traffic anomaly detection system based on multi-source data fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0129] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0130] Step 1: Collect radar target detection point status information and perform radar target detection; collect camera video image information and perform image target detection; capture target screenshots based on target position and size information, and perform attribute detection on the target screenshots;

[0131] Step 2: Track radar targets and image targets, establish target association between camera and radar, realize multi-source information target fusion, obtain fused targets, and then execute steps 3 and 4 simultaneously.

[0132] Step 3: Based on the positional relationship between the fusion target and the preset detection area, statistically analyze relevant traffic indicators to achieve traffic status event detection;

[0133] Step 4: Establish a tracker, use an unscented Kalman filter to track the fused target, and perform secondary filtering on the trajectory; then detect violation events based on the filtered trajectory.

[0134] Step 5: Verify the trajectory stabilization of the detected violations and report the violations that pass the verification.

[0135] For specific limitations on each step, please refer to the limitations on the traffic anomaly detection method based on multi-source data fusion mentioned above, which will not be repeated here.

[0136] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0137] Step 1: Collect radar target detection point status information and perform radar target detection; collect camera video image information and perform image target detection; capture target screenshots based on target position and size information, and perform attribute detection on the target screenshots;

[0138] Step 2: Track radar targets and image targets, establish target association between camera and radar, realize multi-source information target fusion, obtain fused targets, and then execute steps 3 and 4 simultaneously.

[0139] Step 3: Based on the positional relationship between the fusion target and the preset detection area, statistically analyze relevant traffic indicators to achieve traffic status event detection;

[0140] Step 4: Establish a tracker, use an unscented Kalman filter to track the fused target, and perform secondary filtering on the trajectory; then detect violation events based on the filtered trajectory.

[0141] Step 5: Verify the trajectory stabilization of the detected violations and report the violations that pass the verification.

[0142] For specific limitations on each step, please refer to the limitations on the traffic anomaly detection method based on multi-source data fusion mentioned above, which will not be repeated here.

[0143] As a specific example, the invention will be described in one embodiment.

[0144] In embodiments of the present invention, combined with Figure 2 The video frames are used for target detection and recognition, including motor vehicles, non-motor vehicles, and pedestrians. Screenshot parameters are calculated using target detection boxes to capture individual targets. Different detection models are used for attribute detection based on the target category: motor vehicles are detected using models for vehicle color, model, and license plate; non-motor vehicles are detected using models for non-motor vehicles and license plates; and pedestrians are detected using a pedestrian attribute model, including attributes such as gender, clothing (upper and lower garments), whether they carry a backpack, and age. For spatial synchronization of multi-source sensors, the radar coordinate system is simulated using a world coordinate system, and rotation and translation parameters are iterated in three directions to calculate the optimal extrinsic parameters. These extrinsic parameters are then used to calculate the radar-visual space using the camera's intrinsic parameters. The transformation matrix and time synchronization utilize the target's state information and the time difference between sensors to predict data from different sensors at the same time. The predicted data is then unified into the radar coordinate system, and Hungarian matching is used for coordinate association to establish a tracker. Unscented Kalman filtering is used for target tracking, and a second filtering process is performed after the trajectory is completed. For traffic event detection, the number of vehicles, vehicle spacing, average vehicle speed, static queue length, and dynamic queue length are counted in real time, and traffic status is analyzed based on traffic indicators. Traffic violation events are analyzed based on the target trajectory, targeting specific violation attributes and regional configurations. Anti-shake verification is performed on pending violation events to improve the accuracy of the events.

[0145] This invention addresses the need for intelligent prevention and control of traffic safety risks, optimization of traffic efficiency, and reduction of the probability of traffic accidents in intelligent transportation. In particular, it proposes a traffic anomaly event detection system and method based on multi-source data fusion to quickly and accurately identify abnormal traffic events in various traffic scenarios, thereby rapidly adjusting traffic control strategies and addressing safety hazards. This method utilizes spatiotemporal synchronous data fusion processing from multiple sources of sensors to achieve real-time and accurate detection of traffic state events and violations through analysis of real-time targets and trajectories.

[0146] Multi-source target fusion tracking

[0147] In this embodiment, the multi-source sensor includes one radar and two cameras, namely a near-focus camera and a far-focus camera. The radar data period is 50ms, and the camera data period is 40ms. During fusion, the radar frame is used as the starting point. The nearest near-focus and far-focus data are searched, and the camera at the radar data time point is selected and sorted according to time. The current radar data is recorded as the initial fusion data. The time difference between the fusion data and the camera data is calculated sequentially. The state prediction of the fusion data to the camera data time point is performed. The camera data is transformed to the radar coordinate system. Hungarian matching is used to associate the two data to obtain new fusion data. When tracking the fusion data, the infinite Kalman filter is used for filtering. When a complete trajectory is formed, the trajectory is filtered a second time using moving average filtering.

[0148] Traffic Status Incident Handling

[0149] In this embodiment, traffic indicators such as the number of vehicles, vehicle spacing, static queue length, and dynamic queue length are calculated for traffic state time detection. A distance threshold of 20m between the preceding and following targets is used to determine whether the following target is at the end of the queue. When the static queue is greater than the preset queue threshold, it is determined to be a static congestion state. When two or more of the following three conditions are met, the dynamic congestion state is determined, and the congestion duration is counted. The threshold for vehicle spacing is 15m, the threshold for the number of vehicles is 6, and the threshold for congestion duration is 8s. When the duration of static or dynamic congestion exceeds the threshold duration, evidence is recorded and reported.

[0150] Traffic violation incident detection

[0151] In this embodiment, intelligent identification is performed for intrusion, speeding, underspeeding, lane changing, illegal parking, driving in the wrong direction, and lane crossing violations. Intrusion events are identified based on the detection type of the detection area; if the detection type does not match, the intrusion length is statistically analyzed. Speeding refers to events where the instantaneous speed briefly exceeds the speed limit, with a short-term threshold of 3 seconds. Underspeeding takes into account congestion and the safe distance from the vehicle in front; under normal traffic conditions, if the instantaneous speed briefly exceeds the speed limit, it is considered an underspeeding state, with a short-term threshold of 3 seconds. Lane changing refers to events where the vehicle crosses a lane, with a travel distance threshold of 8 meters before and after the lane change. Illegal parking events refer to events where the target is present for an extended period in the detection area. Reverse direction detection aims to detect targets whose movement direction contradicts the direction of the area. Lane crossing is detected by the relative positional relationship between the target detection box and the solid lane line. The parameter for the lower edge detection range is 0.2. All event detections undergo anti-shake verification. After successful verification, event evidence is recorded and reported. Event evidence includes the target trajectory, event screenshots, and video evidence (such as...). Figure 7 The image shows evidence of a line-pressing incident.

[0152] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for detecting traffic anomalies based on multi-source data fusion, characterized in that, The method includes the following steps: Step 1: Collect radar target detection point status information and perform radar target detection; collect camera video image information and perform image target detection; capture target screenshots based on target position and size information, and perform attribute detection on the target screenshots; Step 2: Track radar targets and image targets, establish target association between camera and radar, realize multi-source information target fusion, obtain fused targets, and then execute steps 3 and 4 simultaneously. Step 3: Based on the positional relationship between the fusion target and the preset detection area, statistically analyze relevant traffic indicators to achieve traffic status event detection; Step 4: Establish a tracker, use an unscented Kalman filter to track the fused target, and perform secondary filtering on the trajectory; then detect violation events based on the filtered trajectory. Step 5: Verify the trajectory stabilization of the detected violations and report the violations that pass the verification. The relevant traffic indicators in step 3 include: number of vehicles in the area, vehicle spacing, average vehicle speed, static queue length, and dynamic queue length. The method for counting the number of vehicles within the area is as follows: In the formula, P is the target location, and A and B represent adjacent points within the region. , , The coordinates of A, B, and P are respectively. To determine whether a ray emanating from point P intersects the boundary of the region; m is the number of points in the region. These are the sets of coordinate points for the i-th and (i+1)-th regions, respectively. is_inregion indicates whether the target point is within the region; 1 indicates it is within the region, and 0 indicates it is not. N is the total number of detected targets, and num_cars represents the number of targets within the region. Indicates whether target point j is within the region; The average vehicle speed is the average speed of targets within the area; The calculation method for vehicle body spacing is as follows: In the formula, For vehicle body spacing, and These represent the distances from the radar sensors to the targets at the two ends of the area. For the length of the target vehicle at the near end; The static queue length is determined by statistical analysis of the target's motion state, where the motion state is determined as follows: In the formula, This indicates whether the state is at rest; a value of 1 indicates that the state is at rest, and a value of 0 indicates that the state is not at rest. The target position at the current moment. The target position is given by ths, the distance threshold is given by v. ave In a short period of time and The average speed between positions over time, thv is the stationary speed threshold; when the first car in the area, i.e. the head of the queue, is stationary, it indicates that there is a static queue length; when a car after the head of the queue is stationary and the distance between it and the car behind it exceeds the safe distance, that car is the tail of the queue, and the static queue length is the distance between the head of the queue and the tail of the queue. The dynamic queue length is the distance between the front and the back of the queue. By viewing the distance between adjacent vehicles in the area, the vehicle in front becomes the back of the queue when the distance between the vehicle in front and the vehicle behind is greater than the safe distance.

2. The traffic anomaly event detection method based on multi-source data fusion according to claim 1, characterized in that, In step 1, the target screenshot is obtained by cropping the original image based on screenshot parameters calculated according to the position and size of the target detected in the image; wherein, the calculation formula for the screenshot parameters is: In the formula, The coordinates of the top-left corner of the image. The coordinates of the bottom right corner of the image. The coordinates of the top right corner of the screenshot. The coordinates of the bottom right corner of the screenshot. This is the screenshot coefficient.

3. The traffic anomaly event detection method based on multi-source data fusion according to claim 1, characterized in that, Step 2 involves establishing target association between the camera and radar to achieve multi-source target information fusion, specifically including: In the radar coordinate system, spatial and temporal synchronization is performed on radar targets and image targets to achieve target association between the camera and the radar, resulting in a fused target. The information of the fused target includes radar target information, associated camera target detection box information, category, probability, target attribute information, and detection serial number given by each detection. Specifically, spatial synchronization between the radar target and the image target is achieved through the camera's intrinsic and extrinsic parameter transformation matrix; time synchronization is achieved by predicting the radar target's position based on the time difference of multi-source data; the formula for time synchronization is: In the formula, To predict the location of radar targets, For radar to detect location, and These represent the average speeds of the radar in the x and y directions over 1 second. and These are the detection times for the camera and radar, respectively.

4. The traffic anomaly event detection method based on multi-source data fusion according to claim 1, characterized in that, The information configuration of the preset detection area in step 3 includes the area size, the direction of entry and exit of the area, the solid and dashed lines of the area, the left and right detection area numbers, and the relevant detection event thresholds.

5. The traffic anomaly event detection method based on multi-source data fusion according to claim 1, characterized in that, Step 3 implements traffic state event detection, specifically including: When the static queue length is less than the static queue threshold, or when two of the following three conditions are met: the dynamic queue length is less than the dynamic queue threshold, the number of vehicles in the interval is less than the vehicle threshold, and the vehicle spacing is less than the vehicle spacing threshold, the congestion time is refreshed; otherwise, the congestion start time is not refreshed. When the congestion time reaches the preset time threshold, the congestion is recorded and the congestion event is reported; otherwise, the congestion end time is recorded and the congestion start time is updated.

6. The traffic anomaly event detection method based on multi-source data fusion according to claim 1, characterized in that, In step 4, a moving average filter is used for trajectory filtering. The filtering formula is as follows: In the formula, Here are the x and y coordinates of the filtered trajectory point k, and N is the size of the filtering window. Let k+i be the x and y coordinates of the target trajectory point k+i within the sliding window.

7. The traffic anomaly event detection method based on multi-source data fusion according to claim 1, characterized in that, Step 4 involves detecting traffic violations based on the filtered trajectory, specifically including: When the target's trajectory contains a target direction opposite to the direction of the preset detection area, it is marked as a reverse state. When the reverse state continues for a first preset threshold distance, it is marked as a reverse pending event. When the target speed continuously exceeds the preset threshold speed within the preset detection area, it is marked as an overspeed pending event; When no queue occurs, if the target is a restricted parking area type, and the target stays in the restricted parking area for more than a preset threshold time and moves less than a preset distance, it is marked as a pending parking violation event. Based on the preset detection area and lane direction, the area in the direction of movement is intercepted. The intercepted area is the effective lane change detection area. When the lane line crossed by the target trajectory is a solid line and the continuous movement before and after crossing meets the second preset threshold distance, it is marked as a lane change pending event. When the target detection box continuously contains a solid line within the middle threshold range of its lower edge, it indicates that the target is in a line-crossing state. When the line-crossing state continues for a third preset threshold distance, it is marked as a pending line-crossing event. When the target's trajectory crosses the solid line in both the lane area and the non-lane area, it is also marked as a pending line-crossing event. When the target is not in a queue or the distance between vehicles is greater than the safe distance, it is marked as a pending underspeed event.

8. The traffic anomaly event detection method based on multi-source data fusion according to claim 7, characterized in that, The formula for determining the target line is: In the formula, and These are the left and right x-coordinates of the area to be judged as the area under the line. and These are the x-coordinates of the left and right corner points of the bottom edge of the target detection box, respectively. The parameter used to extract the lower edge interval has a value range of 0-1. The x-coordinate of the intersection of the solid line and the lower edge of the target detection box is given, and s represents the state of being pressed against the line. When the value is less than 0, it indicates a pressure line.

9. The traffic anomaly event detection method based on multi-source data fusion according to claim 8, characterized in that, Step 5 involves verifying the trajectory stabilization of detected traffic violations. The verification formula is as follows: If any formula returns 0, the verification fails; otherwise, the verification passes. In the formula, , , The anomaly values ​​for position, velocity, and acceleration changes are respectively calculated. , , These are the target's position, velocity, and acceleration in the next frame, respectively. , , These are the target's position, velocity, and acceleration from the previous frame, respectively. The average velocity is between two frames, and t is the time interval between the two frames of data. , , These are the threshold values ​​for position, velocity, and acceleration, respectively.

Images