Highway abnormal event identification method and system based on multi-view video

Abstract

This invention discloses a method and system for identifying highway anomaly events based on multi-view video, belonging to the field of intelligent transportation technology. The method includes: acquiring data from various video sources; performing time synchronization and video quality assessment on each video source to obtain a set of valid synchronized frames and corresponding fusion weights for each video source; mapping the image coordinates in each video source to a unified road coordinate system based on the set of valid synchronized frames; performing target detection and tracking on the images from each video source to obtain target motion features; performing association matching and confidence fusion on the target motion features of targets in different video sources based on the fusion weights of each video source to obtain fused target motion features; and calculating anomaly event scores based on the fused target motion features to generate candidate anomaly events. This invention improves the accuracy and robustness of highway anomaly event identification by complementary fusion of fixed-view and moving-view videos, combined with adaptive video quality weighting and multi-source confidence fusion.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation, video image processing, drone inspection and artificial intelligence event recognition, specifically a method and system for identifying abnormal events on highways based on multi-view video. Background Technology

[0002] During highway operation, abnormal events may occur, including traffic accidents, illegally parked vehicles, pedestrians trespassing, large debris spills, traffic congestion, road construction, fires, smoke, fog, water accumulation on the road surface, snow accumulation on the road surface, road icing, and damage to guardrails or traffic safety facilities. These events are characterized by their suddenness, wide spatial distribution, uncertain impact range, and high requirements for timely response. Existing highway monitoring centers typically rely on roadside cameras and manual screen patrols to detect abnormal events. However, with the increasing number of video points, manual screen patrols are easily affected by factors such as fatigue, differences in experience, nighttime lighting, rain, snow, fog, and image obstruction, making it difficult to detect abnormal events in a timely and stable manner.

[0003] Existing video event detection methods based on fixed cameras typically employ single-view video streams for target detection or behavior analysis. While these methods are effective in open, unobstructed, and well-lit environments, they still face several challenges in real-world highway environments: First, side-view cameras are susceptible to obstruction by large vehicles, blind spots on curves, roadside facilities, and camera installation angles, making it difficult to accurately determine target locations across the entire road width from a single perspective. Second, the lack of spatiotemporal consistency fusion between cameras on both sides of the road can lead to repeated alarms for the same target or missed detections of obstructed targets. Third, while drone inspections offer advantages in mobility and top-down perspective, their videos suffer from limitations due to drone movement, pose changes, image jitter, and battery life, making it difficult to achieve continuous and stable monitoring solely based on drone video. Fourth, traditional target detection models often only output target categories such as vehicles, people, smoke, flames, and cones, failing to fully integrate emergency lanes, lane occupancy, dwell time, road areas, and event definitions for business semantic judgment.

[0004] Therefore, there is a need for an anomaly event identification method that can simultaneously utilize fixed video surveillance on both sides of the road and drone-borne video, integrating the continuous monitoring capability of fixed cameras with the maneuverable overhead view capability of drones, and improving the accuracy, timeliness, and traceability of highway anomaly event identification through spatiotemporal synchronization, coordinate unification, multi-source target association, anomaly event scoring, and multimodal semantic verification. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method and system for identifying abnormal events on highways based on multi-view video. By accessing fixed videos from both sides of the road and airborne videos from drones, and through spatiotemporal synchronization, coordinate unification, multi-source target association, and confidence fusion, combined with abnormal event scoring and multimodal semantic verification, the method achieves accurate identification and closed-loop handling of abnormal events on highways.

[0006] This invention is achieved through the following technical solution: In a first aspect, this application provides a method for identifying abnormal events on highways based on multi-view video, characterized by comprising the following steps: Step 1: Obtain data from each video source, perform time synchronization and video quality evaluation on each video source, and obtain the set of valid synchronized frames and the corresponding fusion weights for each video source. Step 2: Based on the set of synchronized valid frames, map the image coordinates in each video source to a unified road coordinate system; Step 3: Perform target detection and tracking on each video source image to obtain the target motion features of the target in each video source; Step 4: Based on the fusion weights of each video source, perform association matching and confidence fusion on the target motion features of the target in different video sources to obtain fused target motion features; Step 5: Based on the fused target motion features, calculate the abnormal event score and generate candidate abnormal events.

[0007] Preferably, each video source includes fixed-viewpoint video and moving-viewpoint video; The acquired video source data includes image frames, timestamps, camera intrinsic parameters, fixed camera extrinsic parameters, UAV positioning information, and UAV attitude information.

[0008] Preferably, the set of valid synchronization frames mentioned in step 1 includes: Using a unified reference time as an index, select image frames with the smallest timestamp difference that are less than the preset synchronization threshold from each video source to form an initial synchronization frame group; For each video source's image frame within the initial synchronization frame group, a comprehensive quality score is calculated by combining frame rate, bit rate, resolution, decoding error rate, frame drop rate, and occlusion ratio. Frames corresponding to video sources with an overall quality score lower than a preset quality threshold are removed. Based on the overall quality scores of the remaining valid video sources, the fusion weights corresponding to each video source are calculated through exponential normalization, and finally a set of valid synchronized frames is formed.

[0009] Preferably, the spatial location of the target in step 2 under a unified road coordinate system includes: For fixed-viewpoint videos, a pre-calibrated planar homography matrix is ​​used to map the image pixel coordinates to a unified road coordinate system; For mobile perspective videos, a spatial transformation matrix is ​​constructed based on the UAV's positioning information, attitude information, and camera parameters. The image pixel coordinates are then back-projected onto a unified road coordinate system to obtain the target's spatial position in the unified road coordinate system.

[0010] Preferably, step 3, which involves target detection and tracking of each video source image, includes: Perform target detection on the synchronized valid frames of each video source and output the original set of candidate targets, which includes detection boxes, categories, confidence scores, and appearance features; The non-maximum suppression algorithm is used to remove overlapping and redundant detection boxes from the original candidate target set to obtain the static features of the target in a single frame. Multi-target tracking is performed on the static features of targets in consecutive frames, and a unique identifier is assigned to the same target to generate a continuous motion trajectory. By combining the static characteristics of the target with its continuous motion trajectory, the motion characteristics of the target are obtained.

[0011] Preferably, step 4, which involves associating and matching the target motion features of targets in different video sources, includes: The association cost between targets from different video sources is calculated by weighting the spatial distance, crossover ratio of detection boxes, similarity of appearance features, category consistency, and time difference under a unified road coordinate system. The optimal correspondence is solved using a global matching algorithm based on the association cost, thus completing the association matching of the same target between different video sources.

[0012] Preferably, the confidence fusion in step 4 includes: To obtain the target-level confidence score, the detection confidence score of the same target in each video source is obtained by weighted fusion using probabilistic fusion combined with the fusion weights of each video source. The fused target motion features include unified spatial location, target category, target-level confidence level, and continuous motion trajectory.

[0013] Preferably, the calculation of the abnormal event score in step 5 includes: Based on the fusion of target motion features, target speed, dwell time, road area relationship, lane occupancy ratio and target-level confidence are extracted, and event scores are calculated through the classification scoring function. By combining target-level confidence, road area relationship score, dwell time score, motion anomaly score and event context score, a weighted score is calculated to obtain the real-time score of the corresponding type of abnormal event; When an event score exceeds the corresponding threshold and meets the minimum duration requirement, a candidate abnormal event is generated.

[0014] Preferably, after step 5, the method further includes: performing multimodal semantic verification on the candidate abnormal events, inputting the keyframes, target motion features and event definitions corresponding to the candidate abnormal events into the visual language model, performing multimodal semantic verification, and confirming the type and event description of the final abnormal events; Secondly, this application provides a highway anomaly event recognition system based on multi-view video, comprising: The video access module is used to acquire data from each video source, perform time synchronization and video quality evaluation on each video source, and obtain the set of valid synchronized frames and the fusion weights corresponding to each video source. The coordinate unification module is used to map the image coordinates in each video source to a unified road coordinate system based on the set of synchronized valid frames. The target detection and tracking module is used to detect and track targets in each video source image and obtain the target motion features of the targets in each video source. The multi-source fusion module is used to perform association matching and confidence fusion of the target motion features of the target in different video sources based on the fusion weights of each video source, so as to obtain fused target motion features; An abnormal event identification module is used to calculate an abnormal event score and generate candidate abnormal events based on the fused target motion features.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects: This application presents a method for identifying highway anomaly events based on multi-view video. First, it achieves time alignment and dynamic quality adaptation of multi-source data through time synchronization and quality-weighted evaluation, eliminating clock deviations from different devices and interference from low-quality images. Then, it uses a unified road coordinate system mapping to transform the pixel coordinates of fixed lateral views from the roadside and UAV overhead views to the same physical spatial reference, resolving the problem of target spatial misalignment across viewpoints. Subsequently, it outputs complete target motion features based on single-source detection and tracking, and then uses the unified spatial reference to complete the association matching and confidence fusion of multi-source targets, merging the multi-view detection results of the same target to fundamentally avoid duplicate alarms while improving target confidence and positioning accuracy. Finally, it scores anomaly events based on the fused target features combined with road area attributes, elevating the target-level identification results to business-level event judgment. This solution balances detection coverage, recognition accuracy, and scene robustness, effectively reducing missed and false detections caused by large vehicle obstruction, blind spots on curves, and severe weather.

[0016] This application also proposes a highway anomaly event recognition system based on multi-view video, an electronic device, and a computer storage medium, which possess all the advantages of the aforementioned highway anomaly event recognition method based on multi-view video. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a diagram illustrating the overall architecture of the highway anomaly event recognition system based on multi-view video according to the present invention. Figure 2 This is a flowchart of the highway anomaly event identification method based on multi-view video according to the present invention; Figure 3 This is a schematic diagram illustrating the spatiotemporal fusion of roadside monitoring and UAV collaboration according to the present invention; Figure 4 This is a schematic diagram of the detection algorithm processing flow of the present invention; Figure 5 This is a schematic diagram of the closed-loop handling process for abnormal events in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] This embodiment provides a method for identifying abnormal events on highways based on multi-view video. By accessing fixed videos from both sides of the road and drone-borne videos, and through spatiotemporal synchronization, coordinate unification, multi-source target association, and confidence fusion, combined with abnormal event scoring and multimodal semantic verification, it achieves accurate identification and closed-loop handling of abnormal events on highways.

[0022] Reference Figure 1 and Figure 2 Highway anomaly event identification based on multi-view video includes the following steps: Step 1: Based on multi-source video, a structured multi-source video dataset carrying unified timestamps, camera intrinsics, and spatial pose information is obtained through multi-protocol parsing, frame decoding, and metadata extraction processing.

[0023] The specific implementation method of step 1 will be described in detail below.

[0024] S1.1. Access the real-time video streams from the fixed video surveillance on the left side of the road and the fixed video surveillance on the right side of the road respectively, and synchronously obtain the timestamp, fixed camera intrinsic parameters, and fixed camera extrinsic parameter calibration data corresponding to each frame of image; S1.2 Receive the real-time image transmission data stream of the UAV's onboard video and synchronously acquire the timestamp, UAV onboard camera intrinsic parameters, UAV GNSS positioning information, UAV IMU attitude information, and gimbal angle information corresponding to each frame of image; S1.3. The raw data from the three types of video sources are standardized in format, unifying image resolution, timestamp format, and coordinate data units to form a structured multi-source video raw dataset. Heterogeneous roadside fixed monitoring and drone videos are uniformly accessed, and all metadata related to spatial positioning is extracted. This solves the problems of inconsistent protocols, inconsistent data formats, and separation of image and spatial information among different video sources, providing a standardized data foundation for subsequent cross-view spatiotemporal fusion and avoiding fusion errors caused by differences in data formats.

[0025] In one possible implementation, in addition to Real-Time Streaming Protocol (RTSP), GB / T 28181 national standard protocol, and UAV-specific image transmission protocol, video access protocols can also adopt Real-Time Message Transfer Protocol (RTMP), Open Network Video Interface Protocol (ONVIF), file download-based offline video access, etc., to adapt to different devices and deployment scenarios; metadata can be actively obtained through the device SDK interface or parsed through video stream extended fields.

[0026] Multi-view fusion identification of highway anomalies first requires acquiring multi-dimensional sensing data and corresponding spatial and temporal calibration parameters. Fixed monitoring on both sides of the road provides continuous and stable lateral monitoring, while drones offer flexible overhead views. These three data sources complement each other, covering the entire width of the road and reducing obstructions and blind spots. Through multi-protocol compatible access methods, it can be adapted to various existing highway monitoring equipment and drone devices, achieving multi-source sensing capability upgrades without large-scale hardware replacements. Standardized metadata formats provide a unified data benchmark for subsequent spatiotemporal synchronization and coordinate mapping, avoiding processing errors caused by data format differences.

[0027] Step 2: Based on the multi-source video dataset, obtain the set of effective frames for multi-source synchronization and the fusion weights corresponding to each video source through benchmark time alignment and multi-dimensional video quality evaluation.

[0028] The specific implementation method of step 2 will be described in detail below.

[0029] S2.1 Using a unified reference time as an index, select the image frames with the smallest timestamp difference from the reference time from the video surveillance on the left side of the road, the video surveillance on the right side of the road, and the drone-borne video. When the timestamp difference is less than the preset synchronization threshold, the corresponding image frame will be included in the same initial synchronization frame group.

[0030] Let the video source on the left side of the road be denoted as The video source on the right side of the road is recorded as The drone video source is recorded as Each video source at any time Image frame is , The corresponding timestamp is Then the reference time The corresponding optimal frame number is:

[0031] When satisfied At that time, image frame Selected into the synchronization frame group; in, Let k be the timestamp of the kth frame of the s-th video source. Let be the frame number in the s-th video source that best matches the reference time t. This is the preset time synchronization threshold.

[0032] S2.2 For each video source image frame within the initial synchronization frame group, calculate the video quality score. The score is obtained by weighting multiple dimensions such as frame rate, bit rate, resolution, decoding error rate, frame drop rate, and occlusion ratio. The calculation formula is as follows:

[0033] in, For frame rate, For bitrate, For image clarity, For decoding error rate, For frame drop rate, To achieve the desired occlusion ratio, The normalization constant is These are the weighting coefficients.

[0034] Image sharpness is calculated using Laplacian variance, as shown in the following formula:

[0035] In the formula, This is the result of performing the Laplacian operator operation on the image frame. This is the variance calculation function.

[0036] S2.3. Compare the overall quality score of each video source within the initial synchronized frame group with the preset quality threshold: When the quality score of a video source is lower than a preset quality threshold, the corresponding frame of that video source is removed so that it does not participate in the current object detection and event recognition; or the participation of that video source in subsequent fusion calculations is reduced. Based on the comprehensive quality score of all remaining valid video sources after screening, the fusion weight of each video source is calculated by normalization index, and finally the set of valid frames synchronized by multiple sources and the corresponding fusion weight set are obtained.

[0037] The formula for calculating the normalized exponent of the fusion weights is:

[0038] in, Let s be the fusion weights of the s-th video source at time t. This is the overall quality score for the corresponding video source.

[0039] Keyframes can be generated from the set of synchronous valid frames using one or a combination of methods, such as fixed time interval extraction, motion intensity trigger extraction, and target appearance trigger extraction.

[0040] Multi-source videos originate from different devices, exhibiting variations in clock reference, frame rate, and encoding quality. Time synchronization ensures that images within the same synchronized frame group correspond to the same physical moment, preventing target matching errors caused by time misalignment. Video quality assessment quantifies the reliability of each data source, reducing the interference of low-quality footage on recognition results through dynamic adjustment of fusion weights. Time synchronization eliminates time discrepancies in multi-source data, providing an accurate time reference for subsequent target association. Quantitative quality assessment enables dynamic weight adaptation of data sources, significantly improving the system's robustness in complex scenarios such as low bitrates, rain, snow, fog, and partial occlusion, reducing false positives and false negatives in harsh environments.

[0041] In some embodiments, in addition to software timestamp alignment, time synchronization can also be achieved by using hardware unified time synchronization (such as GPS / BeiDou second pulse time synchronization) to achieve higher precision time alignment. Furthermore, video quality assessment can also incorporate specific weather-related indicators such as fog concentration, rain spot interference, and strong light overexposure, to further adapt to complex outdoor scenes on highways; In other embodiments, in addition to fixed time intervals, keyframe extraction can also be triggered by motion intensity or target appearance, thereby reducing computational load while ensuring detection accuracy.

[0042] Step 3: Based on the set of valid frames from multiple sources, and combined with the camera intrinsic parameters extracted in Step 1, fixed camera extrinsic parameters, and UAV pose information, the image pixel coordinates of each video source are uniformly mapped to the road plane coordinate system using planar homography matrix mapping and three-dimensional pose back projection methods, respectively, to obtain the spatial position of all targets in the unified road coordinate system.

[0043] like Figure 3 As shown below, the specific implementation method of step 3 will be described in detail.

[0044] S3.1 For fixed video surveillance on the left and right sides of the road, a pre-calibrated planar homography matrix is ​​used to map the image pixel coordinates to road planar coordinates. Let the image pixel coordinates be ( , The road's plane coordinates are ( , The homography matrix is The mapping relationship is then:

[0045] in, Scale factor; target detection box center point coordinates By substituting the mapping relationship, the position of the target in the unified road coordinate system can be obtained. :

[0046]

[0047] In the formula, For the first s The first in the road video i The pixel coordinates of the center point of the detection bounding box of each target. These are the road plane coordinates after mapping to the corresponding target.

[0048] S3.2 For UAV-borne video, based on the UAV's GNSS positioning information, IMU attitude information, camera intrinsic parameters, and camera extrinsic parameters, construct a transformation matrix from the UAV camera coordinate system to the world coordinate system. The image pixel coordinates are back-projected onto the road plane.

[0049] Let the imaging ray direction vector be... The drone camera center is ,but:

[0050]

[0051] in, u For imaging rays and road plane The intersection scale parameters. Let the road plane equation be... ,but:

[0052] Substituting into the above formula, we can obtain the spatial position of the target in the unified road coordinate system from the perspective of the UAV.

[0053] S3.3 Verify the mapping results of the same calibration reference object under different viewpoints. When the coordinate deviation exceeds the preset threshold, recalibrate the extrinsic parameters of the corresponding camera or correct the UAV pose data to ensure the accuracy of coordinate mapping.

[0054] Video images from different perspectives only provide two-dimensional positions in a pixel coordinate system, making direct cross-view target matching impossible. By calibrating the camera and calculating the pose, all pixel coordinates are transformed to the same real physical coordinate system, eliminating positional deviations caused by differences in viewpoints and installation locations, and providing a unified spatial comparison benchmark for targets from different perspectives. Spatial alignment between the road lateral view and the UAV's overhead view is achieved, providing an accurate spatial benchmark for subsequent multi-source target association. The unified road coordinates can directly correspond to highway markers and lane positions, and the output event locations can be directly used for emergency response without secondary transformation.

[0055] In some embodiments, the unified road coordinate system can also adopt a three-dimensional road domain coordinate system, combined with a high-precision laser point cloud map to construct a three-dimensional spatial reference, further adapting to non-planar road scenarios such as slopes and curves; the UAV pose can also be optimized in real time through visual SLAM algorithms to reduce positioning errors in areas with weak GNSS signals.

[0056] Step 4: Based on the set of valid frames from multiple sources, and combining deep learning target detection algorithm with Kalman filter tracking algorithm, obtain the target motion features of all road targets in each video source; the target motion features include target detection box, target category, detection confidence, appearance feature vector and continuous motion trajectory.

[0057] like Figure 4 As shown below, the specific implementation method of step 4 will be described in detail.

[0058] S4.1 Input each valid synchronized frame into the pre-trained target detection model to obtain a set of candidate targets. The original set of candidate targets provides input data for subsequent deduplication of detection boxes, forming the static attribute basis of the target motion features.

[0059]

[0060] in, For the target detection bounding box, For the target category, To test confidence level The target appearance feature vector is used; the target categories cover common road targets such as vehicles, pedestrians, debris, smoke, flames, cones, and traffic safety facilities.

[0061] S4.2. Based on the original candidate target set, a non-maximum suppression algorithm is used to remove overlapping and redundant detection boxes, optimizing the accuracy of the target's static attributes. The intersection-union ratio (IUGR) is used to measure the spatial overlap between two detection boxes, and the calculation formula is as follows:

[0062] when and At that time, delete detection boxes with low confidence. By retaining the detection results with higher confidence, we obtain a set of static features of the target in a single frame after deduplication.

[0063] S4.3 Using the static features of the target in consecutive frames as the observation input, Kalman filtering is used to predict and update the motion state of the target, assign a unique identifier to the same target, generate continuous motion trajectories, and obtain the dynamic temporal part of the target motion features.

[0064] Define the target's planar motion state vector as:

[0065] This includes the target's planar position, velocity, and acceleration. A Kalman filter is used to predict and update the target's state, forming a continuous motion trajectory. State prediction process:

[0066]

[0067] State update process:

[0068]

[0069] in, Here is the state transition matrix. For the observation matrix, For process noise covariance, To observe the noise covariance, This represents the target observation position in the current frame.

[0070] By combining static attribute features with dynamic motion trajectories, a complete set of target motion features is ultimately formed.

[0071] Target detection can identify various road targets in a single frame image and output static attributes; multi-target tracking can assign a unique identifier to the same target in consecutive frames, forming a continuous motion trajectory, thereby extracting temporal features such as target speed, dwell time, and motion trend. The target motion features formed by the combination of these two methods fully cover the static attributes and dynamic behavior of the target, serving as the core foundational data for anomaly event determination. It can simultaneously identify multiple types of road targets, covering all related objects of common highway anomalies; the motion trajectory output by continuous tracking can accurately characterize target behavior features, providing reliable temporal data support for subsequent event scoring.

[0072] In some embodiments, the target detection model described above may also employ any one of the YOLO series, Faster R-CNN, or Transformer detection networks, and the selection can be made according to the computing power and accuracy requirements of the deployment environment; the multi-target tracking algorithm may also employ DeepSORT, ByteTrack, or graph-optimized tracking algorithms to further improve the tracking stability in complex traffic flow scenarios and optimize the continuity of target motion features.

[0073] Step 5: Based on the target motion features of each video source, the target spatial location under the unified road coordinate system, and the fusion weight of each video source, combined with multi-dimensional association cost calculation and global matching algorithm, perform association matching on the same target in different video sources to obtain a fused target motion feature set. The fused target motion features include unified spatial location, target category, target-level confidence level, and continuous motion trajectory, which can eliminate the problem of repeated alarms caused by repeated detection from multiple perspectives and improve the confidence and positional accuracy of target detection.

[0074] The specific implementation method of step 5 will be described in detail below.

[0075] S5.1 Extract the association dimension parameters for cross-source target matching from the target motion features of each video source, including the target spatial location in a unified road coordinate system, target detection box, target appearance feature vector, target category, corresponding image frame timestamp; and the fusion weight of each effective video source. Based on the extracted association feature parameters, for any two targets from different video sources, the association cost of the target pair is calculated by weighting five dimensions: spatial distance, detection box overlap, appearance similarity, category consistency, and temporal deviation, and a global association cost matrix is ​​generated.

[0076] The formula for calculating associated costs is:

[0077] in, To unify the target positions in the road coordinate system, For the target detection bounding box, For the target appearance feature vector, For the corresponding frame timestamp, This is a penalty for category inconsistency; the value is 0 when the categories are the same and 1 when the categories are different. These are the weighting coefficients.

[0078] The formula for calculating the cosine similarity of appearance features is:

[0079] The lower the correlation cost, the higher the probability that the two targets are the same physical target; this cost matrix provides a quantitative basis for subsequent global target matching.

[0080] S5.2 Based on the association cost matrix, a global matching algorithm is used to solve for the optimal matching relationship, thereby obtaining the correspondence between the same target in different video sources; Set the associated cost threshold When associated costs At that time, determine the target With the goal For the same physical target, a global matching algorithm is used to optimally allocate all target pairs, avoiding erroneous one-to-many and many-to-one matching, and finally obtaining a unified target association set across video sources.

[0081] S5.3. Based on the unified target association set and the fusion weight of each video source, the detection confidence of the same target in different video sources is weighted and fused to obtain the target-level confidence. At the same time, the unified spatial coordinates and continuous motion trajectory of the target are optimized based on the multi-source detection results to confirm the consistency of the target category and finally form a complete fused target motion feature set.

[0082] The target-level confidence score is calculated using a multi-source probability fusion formula:

[0083] in, To detect the target A collection of video sources This represents the original detection confidence level of the target in the corresponding video source.

[0084] After fusion, the output is a fused target motion feature set containing unified spatial location, target category, target-level confidence, and continuous motion trajectory.

[0085] Videos from different perspectives may repeatedly detect the same target, leading to duplicate alarms if not correlated. By matching multi-dimensional information such as spatial location, appearance features, and category under a unified coordinate system, the same target from different perspectives can be merged. Simultaneously, by combining video quality-weighted fusion confidence scores, targets detected consistently from multiple perspectives have higher confidence scores, while targets detected from a single perspective have lower confidence scores, thus improving the overall reliability of the detection results. This effectively solves the problem of duplicate alarms caused by multi-view monitoring, generating only one fused target motion feature record for the same target. Through multi-source confidence fusion, false alarms caused by single-view false detections are significantly reduced, while missed detections in occluded scenes are also reduced, resulting in a significant improvement in overall target recognition accuracy.

[0086] In some embodiments, the global target matching algorithm described above may also employ any one of the Hungarian matching algorithm, nearest neighbor matching algorithm, and graph matching algorithm; confidence fusion may also employ fusion strategies such as DS evidence theory and weighted average method to adapt to different business accuracy requirements.

[0087] In other embodiments, a trajectory direction consistency factor can be added to the association cost to further improve the matching accuracy and reduce the probability of mismatch between vehicles traveling in opposite directions, in conjunction with the target movement trend.

[0088] Step 6: Based on the fused target motion features, unified road spatial location, and target-level confidence, and combined with the categorized anomaly event scoring function and threshold determination method, a candidate anomaly event set is obtained.

[0089] The specific implementation method of step 6 will be described in detail below.

[0090] S6.1 Extract the associated features corresponding to various abnormal events from the continuous trajectory of the target's motion characteristics, including target speed, acceleration, dwell time, road area, lane occupancy ratio, target size, and status of surrounding targets.

[0091] S6.2. For different types of abnormal events, corresponding scoring functions are constructed based on the event association feature set and the target-level confidence level to quantify the probability of the event occurring and output the real-time anomaly score for each type of event. For the k-th type of abnormal event, the event score at time t is:

[0092] in, For target level confidence, Score the relationship between the target and the road area. Score based on stay time. For abnormal movement scores, Score the event context. 1,…, 5 is the weighting coefficient.

[0093] The specific scoring rules for each type of typical event are as follows: (1) Vehicle parking violations Let the vehicle speed be The probability that a vehicle is located in an emergency lane or a no-parking zone is The duration of the vehicle remaining stationary at low speed is The vehicle's illegal parking score will be:

[0094] when and At that time, candidate events for illegal parking of vehicles are generated.

[0095] (2) Pedestrian intrusion incidents Let the confidence level of the pedestrian category be... The probability that a pedestrian is located in the driving lane or emergency lane area is The staff excluded the score as The pedestrian trespassing score is:

[0096] (3) Spillage incident Let the target's rest time be... The probability that the target is located in the driving area is The target size affects the score. The score for spilled material is:

[0097] (4) Traffic congestion events Let the length of the road section to be tested be... The number of vehicles is The average speed is Lane occupancy rate ,in:

[0098]

[0099] The traffic congestion score is then:

[0100] (5) Fire and smoke incidents Let the confidence level of flame detection be... The confidence level for smoke detection is The time continuity score is The fire smoke event score is as follows:

[0101] (6) Fog Event The score is calculated based on image contrast, visible distance estimation, and lane line vanishing degree.

[0102] in, To estimate the visible distance, This represents the percentage of lane lines missing.

[0103] S6.3 When the abnormal event score meets the requirements and the duration of the corresponding event meets the minimum duration requirement, a candidate abnormal event of the corresponding type is generated. Events that do not meet the threshold conditions are judged as normal traffic conditions and no event alarm is triggered.

[0104] Simple target motion characteristics cannot directly correspond to abnormal events at the business level. It is necessary to combine road business rules with the temporal behavior characteristics of the target to elevate target-level data to event-level judgment. Through quantitative scoring functions and threshold judgments, automated initial screening of abnormal events can be achieved, distinguishing between normal traffic behavior and abnormal events. It covers more than ten types of common abnormal events on highways, such as traffic accidents, illegal parking, pedestrian intrusion, littering, congestion, fires, and fog, with a comprehensive recognition range. The scoring mechanism combines multi-dimensional information such as target confidence, behavioral characteristics, and regional attributes, and the accuracy of event judgment is far higher than that of single-target threshold judgment, effectively distinguishing easily confused scenarios such as normal construction and normal parking.

[0105] In some embodiments, the weight coefficients of the above-mentioned abnormal event scoring function can be optimized by machine learning training using historical event samples, replacing manual weight calibration; prior probability factors of historical events can also be added to dynamically adjust the scoring threshold of accident-prone road sections, further improving the adaptability of event recognition.

[0106] Step 7: Based on the key image frames, target motion features, and event definition rules corresponding to the candidate abnormal events, and combined with the multimodal semantic reasoning method of the visual language big model, the candidate events are verified a second time to obtain the confirmed final abnormal event type, event description, and target localization result.

[0107] The specific implementation method of step 7 will be described in detail below.

[0108] S7.1. The multi-view keyframes, target motion features, candidate event types, event definition rules, and target trajectory information corresponding to the candidate abnormal events are structurally assembled to form a multimodal verification input data packet.

[0109] S7.2 Input the multimodal verification data packet into the pre-trained visual language model, and the model outputs a structured verification result:

[0110] in, For the event types confirmed by the model, For natural language event description, The bounding box for the target associated with the event. The result of the decision to confirm or ignore the event.

[0111] S7.3 If the event type output by the visual language model belongs to the preset abnormal event set and the confidence level meets the preset constraints, then the candidate abnormal event is confirmed as the final valid event; otherwise, it is judged as a false alarm and the candidate event is ignored.

[0112] Rule-based scoring and initial screening are prone to misjudgments in certain edge cases, such as misidentifying normal construction cones as roadblocks or light reflections as flames. Visual language models, with their stronger semantic understanding and scene reasoning capabilities, can perform secondary verification of candidate events based on global context, thereby filtering false alarms. Simultaneously, they generate natural language descriptions to improve the interpretability of the results. This significantly reduces the false alarm rate for abnormal events and enhances the credibility of the system's output. The automatically generated event descriptions require no manual re-processing and can be directly used for event logs and notifications, improving emergency response efficiency.

[0113] In some embodiments, the above-mentioned multimodal semantic review can also adopt a multi-model voting mechanism, calling multiple visual language models with different architectures to reason separately, and taking the majority consistent result as the final review conclusion; it can also be combined with a rule engine for secondary verification to filter out results that obviously do not conform to business logic, thereby further improving the reliability of the review.

[0114] Step 8: Based on the final confirmed abnormal event results, corresponding image frames and target motion features, combined with evidence file packaging and structured record generation methods, obtain complete event evidence files and structured event records, and push the event information to the highway event handling platform.

[0115] The specific implementation method of step 8 will be described in detail below.

[0116] S8.1 Extract the original image frames, labeled images of the target boxes, and playback video clips before and after the event for the corresponding abnormal event. Classify and store them according to the event number to generate a traceable event evidence package.

[0117] S8.2 Generate standardized structured event records, which include the following fields:

[0118] in, Number the event. Encode the event. For event type, For the time of occurrence, For road location, For video source identification, For event status, For event description, A collection of evidence images. This is the URL for the replay video.

[0119] S8.3. Push structured event records to the highway event handling platform to support personnel in performing event retrieval, detail viewing, image verification, video playback, status updates, and statistical analysis.

[0120] The ultimate goal of anomaly event identification is to support business operations. Complete evidence preservation and standardized information dissemination are key links connecting the intelligent identification system with actual business operations. The structured record format can be directly integrated with existing handling platforms without additional data conversion. This achieves a closed-loop process for anomaly events, from automatic discovery and intelligent verification to dissemination and handling, significantly shortening the response time from event discovery to handling. Standardized evidence preservation meets the business needs of event tracing and liability determination, providing data support for highway safety management.

[0121] In some embodiments, the above-mentioned event push can also be tiered according to the severity of the event, with major events being pushed to multiple levels of management departments simultaneously; it can also be linked with variable message signs and broadcasting systems along highways to release early warning information, automatically generate event handling plan suggestions, and further improve the level of intelligence in event handling.

[0122] Correspondingly, this application provides a highway anomaly event recognition system based on multi-view video, comprising: The video access module is used to acquire data from each video source, perform time synchronization and video quality evaluation on each video source, and obtain the set of valid synchronized frames and the fusion weights corresponding to each video source. The coordinate unification module is used to map the image coordinates in each video source to a unified road coordinate system based on the set of synchronized valid frames. The target detection and tracking module is used to detect and track targets in each video source image and obtain the target motion features of the targets in each video source. The multi-source fusion module is used to perform association matching and confidence fusion of the target motion features of the target in different video sources based on the fusion weights of each video source, so as to obtain fused target motion features; An abnormal event identification module is used to calculate an abnormal event score and generate candidate abnormal events based on the fused target motion features.

[0123] The highway anomaly event recognition method based on multi-view video provided in this embodiment achieves accurate recognition and closed-loop handling of highway anomaly events through collaborative perception of fixed-view and moving-view videos, combined with spatiotemporal synchronization, coordinate unification, multi-source target association, confidence fusion, anomaly event scoring, and multimodal semantic verification. It has the following beneficial effects: First, by combining fixed video from both sides of the road with video from drones, and utilizing the continuous monitoring capabilities of fixed perspectives and the mobility of drones' overhead view, the full width of the road can be covered. This effectively reduces missed detections caused by large vehicles, roadside facilities, and blind spots on road curves, thus compensating for the spatial limitations of single-view monitoring.

[0124] Second, by aligning video streams with different frame rates and delays to a unified time reference through time synchronization, and by mapping target positions from different perspectives to the same road coordinate system through coordinate unification, and by merging the same target from different perspectives into a unified target through multi-dimensional association cost calculation and global matching algorithm, the probability of repeated alarms caused by repeated detection from multiple perspectives is reduced, and the accuracy of target position estimation is improved.

[0125] Third, the video sources are quantitatively evaluated and fusion weights are calculated through video quality scoring. When there is low bit rate, low frame rate, decoding abnormality or unclear picture, the participation of the corresponding video source in subsequent fusion is automatically reduced, avoiding interference of low quality video frames on the recognition results and improving the robustness of the system in complex environments such as rain, snow, fog, and insufficient light at night.

[0126] Fourth, by using a classification-based abnormal event scoring function, combined with multi-dimensional information such as target movement trajectory, road area relationship, dwell time, lane occupancy ratio, and target-level confidence, simple target identification is upgraded to business event identification. This can effectively distinguish between easily confused scenarios such as normal construction workers and pedestrians entering the area, normal parking and vehicles illegally parked, and normal traffic cones and roadblocks, thereby reducing the false alarm rate.

[0127] Fifth, through multimodal semantic verification, the scene understanding capabilities of the visual language model are used to perform secondary verification of candidate events, automatically generating event descriptions and target location results, which can further filter false alarms and provide interpretable judgment basis for subsequent manual verification and emergency response.

[0128] Sixth, by automatically saving original image frames, labeled images, and event playback videos, a structured and traceable event record is generated, realizing a closed loop of the entire process from automatic discovery and intelligent verification of abnormal events to evidence preservation and push for handling, thereby improving the informatization level of highway operation safety supervision.

[0129] It should be noted that, in the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another device, or some features may be ignored or not executed. The modules described as separate components may or may not be physically separated. The components shown as modules may be one or more physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0130] Furthermore, in the various embodiments of the present invention, the modules can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0131] An electronic device provided in this application includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the highway anomaly event recognition method based on multi-view video as described in any of the above embodiments.

[0132] Another electronic device provided in this application embodiment may further include: an input port connected to a processor for transmitting multimodal data collected by an external acquisition device to the processor; a display unit connected to the processor for displaying the processor's processing results to the outside world; and a communication module connected to the processor for enabling communication between the electronic device and the outside world. The display unit may be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module includes, but is not limited to, Mobile High Definition Link (HML), Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), and wireless connection (including Wi-Fi, Bluetooth, Bluetooth Low Energy, and IEEE 802.11s-based communication technology).

[0133] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the highway anomaly event recognition method based on multi-view video as described in any of the above embodiments.

[0134] For descriptions of relevant parts of the highway anomaly event recognition system, electronic device, and computer-readable storage medium based on multi-view video provided in this application, please refer to the detailed descriptions of the corresponding parts in the highway anomaly event recognition method based on multi-view video provided in this application, which will not be repeated here. Furthermore, parts of the technical solutions provided in this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0135] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for identifying abnormal events on highways based on multi-view video, characterized in that, Includes the following steps: Step 1: Obtain data from each video source, perform time synchronization and video quality evaluation on each video source, and obtain the set of valid synchronized frames and the corresponding fusion weights for each video source. Step 2: Based on the set of synchronized valid frames, map the image coordinates in each video source to a unified road coordinate system; Step 3: Perform target detection and tracking on each video source image to obtain the target motion features of the target in each video source; Step 4: Based on the fusion weights of each video source, perform association matching and confidence fusion on the target motion features of the target in different video sources to obtain fused target motion features; Step 5: Based on the fused target motion features, calculate the abnormal event score and generate candidate abnormal events.

2. The method for identifying highway anomaly events based on multi-view video according to claim 1, characterized in that, The video sources include fixed-viewpoint videos and moving-viewpoint videos; The acquired video source data includes image frames, timestamps, camera intrinsic parameters, fixed camera extrinsic parameters, UAV positioning information, and UAV attitude information.

3. The method for identifying highway anomaly events based on multi-view video according to claim 1, characterized in that, The set of valid synchronization frames mentioned in step 1 includes: Using a unified reference time as an index, select image frames with the smallest timestamp difference that are less than the preset synchronization threshold from each video source to form an initial synchronization frame group; For each video source's image frame within the initial synchronization frame group, a comprehensive quality score is calculated by combining frame rate, bit rate, resolution, decoding error rate, frame drop rate, and occlusion ratio. Frames corresponding to video sources with an overall quality score lower than a preset quality threshold are removed. Based on the overall quality scores of the remaining valid video sources, the fusion weights corresponding to each video source are calculated through exponential normalization, and finally a set of valid synchronized frames is formed.

4. The method for identifying highway anomaly events based on multi-view video according to claim 2, characterized in that, The spatial location of the target in the unified road coordinate system mentioned in step 2 includes: For fixed-viewpoint videos, a pre-calibrated planar homography matrix is ​​used to map the image pixel coordinates to a unified road coordinate system; For mobile perspective videos, a spatial transformation matrix is ​​constructed based on the UAV's positioning information, attitude information, and camera parameters. The image pixel coordinates are then back-projected onto a unified road coordinate system to obtain the target's spatial position in the unified road coordinate system.

5. The method for identifying highway anomaly events based on multi-view video according to claim 1, characterized in that, Step 3, which involves target detection and tracking of each video source image, includes: Perform target detection on the synchronized valid frames of each video source and output the original set of candidate targets, which includes detection boxes, categories, confidence scores, and appearance features; The non-maximum suppression algorithm is used to remove overlapping and redundant detection boxes from the original candidate target set to obtain the static features of the target in a single frame. Multi-target tracking is performed on the static features of targets in consecutive frames, and a unique identifier is assigned to the same target to generate a continuous motion trajectory. By combining the static characteristics of the target with its continuous motion trajectory, the motion characteristics of the target are obtained.

6. The method for identifying highway anomaly events based on multi-view video according to claim 1, characterized in that, Step 4, which involves associating and matching the target motion features of targets in different video sources, includes: The association cost between targets from different video sources is calculated by weighting the spatial distance, crossover ratio of detection boxes, similarity of appearance features, category consistency, and time difference under a unified road coordinate system. The optimal correspondence is solved using a global matching algorithm based on the association cost, thus completing the association matching of the same target between different video sources.

7. The method for identifying highway anomaly events based on multi-view video according to claim 6, characterized in that, The confidence fusion described in step 4 includes: To obtain the target-level confidence score, the detection confidence score of the same target in each video source is obtained by weighted fusion using probabilistic fusion combined with the fusion weights of each video source. The fused target motion features include unified spatial location, target category, target-level confidence level, and continuous motion trajectory.

8. The method for identifying highway anomaly events based on multi-view video according to claim 1, characterized in that, Step 5, which involves calculating the anomaly event score, includes: Based on the fusion of target motion features, target speed, dwell time, road area relationship, lane occupancy ratio and target-level confidence are extracted, and event scores are calculated through the classification scoring function. By combining target-level confidence, road area relationship score, dwell time score, motion anomaly score and event context score, a weighted score is calculated to obtain the real-time score of the corresponding type of abnormal event; When an event score exceeds the corresponding threshold and meets the minimum duration requirement, a candidate abnormal event is generated.

9. The method for identifying highway anomaly events based on multi-view video according to claim 1, characterized in that, Step 5 is followed by: performing multimodal semantic verification on the candidate abnormal events, inputting the keyframes, target motion features and event definitions corresponding to the candidate abnormal events into the visual language model, performing multimodal semantic verification, and confirming the type and description of the final abnormal events.

10. A highway anomaly event recognition system based on multi-view video as described in any one of claims 1-9, characterized in that, include: The video access module is used to acquire data from each video source, perform time synchronization and video quality evaluation on each video source, and obtain the set of valid synchronized frames and the fusion weights corresponding to each video source. The coordinate unification module is used to map the image coordinates in each video source to a unified road coordinate system based on the set of synchronized valid frames. The target detection and tracking module is used to detect and track targets in each video source image and obtain the target motion features of the targets in each video source. The multi-source fusion module is used to perform association matching and confidence fusion of the target motion features of the target in different video sources based on the fusion weights of each video source, so as to obtain fused target motion features; An abnormal event identification module is used to calculate an abnormal event score and generate candidate abnormal events based on the fused target motion features.

Images