Road abnormal event monitoring system based on video image analysis

Abstract

The application discloses a highway abnormal event monitoring system based on video image analysis and relates to the technical field of intelligent transportation.The system comprises a video stream access module, a vehicle behavior data access module, a core processing module and an output module.The video stream access module acquires real-time video stream, and the vehicle behavior data access module anonymously receives floating car data.The core processing module is arranged on a roadside edge computing unit, floating car data is mapped to an image coordinate system through space-time alignment, a behavior heat map is generated, video image feature enhancement is guided, the heat map is optimized through enhanced features, and finally abnormal events and confidence are output through collaborative determination logic.The output module uploads high-confidence abnormal event information to a cloud management platform.The application realizes deep fusion and mutual enhancement of video data and vehicle behavior data, has real-time performance and robustness, can automatically switch working modes, fully utilizes existing traffic resources and reduces deployment costs.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a highway abnormal event monitoring system based on video image analysis. Background Technology

[0002] With the construction and development of intelligent transportation systems, utilizing existing fixed highway surveillance video resources for automated anomaly monitoring has become a key technological direction for improving road safety management efficiency. This type of technology aims to replace or assist traditional manual monitoring through real-time intelligent analysis of video images, enabling early detection and rapid response to anomalies, thereby ensuring smooth road network operation and driving safety. Currently, the application of artificial intelligence algorithms such as deep learning in this field is becoming increasingly mature, significantly improving the accuracy of video analysis in areas such as target detection and behavior recognition.

[0003] Currently, highway anomaly event monitoring solutions based on video images mainly follow a technical approach of single-modal perception and simple backend fusion. One mainstream approach relies purely on deep analysis of video streams. For example, existing technologies focus on improving the detection rate of small target events by modifying digital retina frameworks and deformable convolutional models; another type of existing technology extracts image features through convolutional neural networks to achieve vehicle and pedestrian detection, as well as event identification such as congestion and parking. These solutions all use video as a single data source for event recognition. Another approach attempts to fuse multi-source data. For example, the existing patent technology (publication number CN116740952A) "Highway Abnormal Behavior Monitoring System and Method" monitors vehicle driving status through cameras and combines it with basic vehicle information obtained from toll stations to analyze and warn about abnormal behavior. While these solutions incorporate vehicle data, they typically only process or logically correlate it at the decision-making level. For example, events are first determined by video and then verified by vehicle data, or information from different sources is "data splicing and post-fusion"—the former refers to simply overlaying video features, vehicle attributes, and other data by dimension, while the latter refers to fusion occurring only in the final decision-making stage, where the two types of data are processed independently before the conclusion is integrated, without mutual guidance in front-end stages such as perception and feature extraction. This shallow fusion fails to address the decisive constraint of front-end perception quality on the overall system performance.

[0004] However, the aforementioned existing technologies face a common engineering practicality problem: their core event perception capabilities rely excessively on the quality of optical video images. In complex and challenging traffic scenarios such as rain, fog, low-light conditions at night, or obstructions, video image quality degrades significantly, leading to a sharp decline in the performance of vision-based feature extraction and event recognition algorithms, resulting in numerous false negatives and false positives. Although some solutions incorporate vehicle data, the fusion layer is shallow, and the overall robustness of the system does not substantially improve when video modalities fail. Therefore, how to build an abnormal event monitoring system that can mutually reinforce each other at the perception level and maintain high robustness and low false positive rates under complex conditions, without relying on or adding high-cost dedicated sensors, has become a pressing technical challenge in this field.

[0005] In summary, existing technologies mostly focus on algorithm optimization for a single video modality or backend correlation of multi-source data, and have not yet effectively solved the bottleneck problem of insufficient monitoring reliability caused by video quality degradation in complex environments. This invention is proposed against this background. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a highway abnormal event monitoring system based on video image analysis. This system can achieve deep fusion and closed-loop interaction of video data and vehicle behavior data at the feature level, thereby improving the accuracy and reliability of automatic monitoring of highway abnormal events in all weather conditions.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a highway abnormal event monitoring system based on video image analysis, comprising:

[0008] The video stream access module is used to acquire real-time video streams from fixed cameras along the highway.

[0009] The vehicle behavior data access module is used to anonymously receive floating car data of moving vehicles, wherein the floating car data includes at least vehicle position, speed and acceleration;

[0010] The core processing module, deployed in the roadside edge computing unit, includes a dynamically fused cross-modal mutual enhancement and collaborative verification unit. The core processing module is configured as follows:

[0011] Perform a spatiotemporal alignment operation to map the floating car data onto the image coordinate system of the real-time video stream;

[0012] Generate a behavior heatmap reflecting the spatial distribution of abnormal vehicle behavior based on the mapped floating car data;

[0013] The behavior heatmap is used as an attention-guided signal to enhance the initial image features extracted from the real-time video stream, resulting in enhanced image features.

[0014] The enhanced image features are used to verify and optimize the behavior heatmap to obtain an optimized behavior heatmap;

[0015] Based on the enhanced image features and the optimized behavior heatmap, abnormal events and their confidence levels are output through collaborative judgment logic.

[0016] The output module is used to upload high-confidence abnormal event information to the cloud management platform.

[0017] Furthermore, the core processing module includes a spatiotemporal alignment submodule, a mutual enhancement submodule, and a collaborative determination submodule connected in sequence;

[0018] The spatiotemporal alignment submodule is configured to synchronize the real-time video stream and the floating car data using a unified time, and to map the global coordinates of the floating car data to the pixel coordinates of the video image through perspective transformation based on camera calibration parameters and high-precision map data.

[0019] The mutual enhancement submodule is configured to perform the generation of the behavior heatmap, the extraction of the initial image features, the generation of the enhanced image features, and the output of the optimized behavior heatmap;

[0020] The collaborative determination submodule is configured to execute the collaborative determination logic.

[0021] Furthermore, the mutual enhancement submodule is configured to generate the behavior heatmap through the following steps:

[0022] For any pixel position in the video image Its corresponding behavioral thermal value The results were obtained using the kernel density estimation algorithm:

[0023]

[0024] in, This indicates the number of floating cars within the time synchronization window. Indicates the first The vehicle's three-dimensional position coordinates, A function representing the mapping from world coordinates to image pixel coordinates. Represents the Euclidean distance on the image pixel plane. Represents a kernel function that satisfies and , Indicates bandwidth parameter, and They represent the first The vehicle's velocity vector and acceleration vector The abnormal behavior weighting function is determined by the rate of change of the velocity vector and the magnitude of the acceleration vector.

[0025] Furthermore, the abnormal behavior weight function The calculation method is as follows:

[0026]

[0027] in, The normal driving speed vector of the monitored road segment is obtained from historical floating car data. Indicates the first Vehicle speed deviation, The magnitude of the acceleration vector is represented by the acceleration vector. and These are pre-defined non-negative weighting coefficients used to quantify the contributions of velocity mutations and acceleration anomalies, respectively.

[0028] Furthermore, the mutual enhancement submodule includes a behavior-guided video feature enhancement network, which is an encoder-decoder structure and integrates an attention gating mechanism in the skip connections;

[0029] The encoder is used to extract the features of the initial image;

[0030] The attention gating mechanism takes the behavior heatmap as input to generate an attention weight map, which is used to weight the features passed by the skip connections when the decoder performs feature upsampling.

[0031] The decoder outputs the enhanced image features.

[0032] Furthermore, the behavior-guided video feature enhancement network is trained in a self-supervised manner, and its loss function is constructed as follows: the enhanced image features are encouraged to decode and recover the target contour in the high-value region of the behavior heatmap, which conforms to the physical constraint model learned from large-scale normal traffic data. The physical constraint model includes target size distribution constraints and motion trajectory continuity constraints.

[0033] Furthermore, the collaborative determination logic executed by the collaborative determination submodule is as follows:

[0034] Potential anomalous visual targets and their regions in the image are identified from the enhanced image features, wherein the anomalous visual targets include visual feature codes extracted by a target detection algorithm;

[0035] Statistical features of abnormal behavior patterns in the corresponding regions are extracted from the optimized behavior heatmap;

[0036] Application judgment rule: A high-confidence abnormal event is judged to have occurred only when a specific abnormal visual target is identified and the statistical characteristics of the abnormal behavior pattern in its corresponding area exceed a preset threshold.

[0037] Furthermore, the judgment rules implemented by the collaborative judgment submodule are executed by a multilayer perceptron. The input of the multilayer perceptron is a concatenated feature vector, which is composed of the visual feature encoding of the abnormal visual target and the statistical features of the abnormal behavior pattern of the corresponding region. The output of the multilayer perceptron is the abnormal event type label and the comprehensive confidence score.

[0038] Furthermore, the statistical features of the abnormal behavior pattern are calculated as follows: for abnormal visual target regions identified in the image... Calculate the mean value of the heatmap of the optimized behavior within its internal and surrounding buffer areas. Standard deviation and the diffusion rate of thermal value over time Together they constitute the statistical feature vector. The diffusion rate The value is obtained by calculating the movement speed of the centroid of the thermal value of the region between consecutive frames.

[0039] Furthermore, the vehicle behavior data access module receives the floating car data through a vehicle-to-everything (V2X) roadside unit or a mobile communication network;

[0040] The floating car data comes from vehicles with network communication capabilities that are driving on highways, forming a floating car sample set.

[0041] When the floating car sample data is missing or insufficient, the system can switch to a single-modal working mode that relies solely on the real-time video stream.

[0042] Compared with existing technologies, this highway anomaly monitoring system based on video image analysis has the following advantages:

[0043] I. This invention achieves deep cross-modal fusion of real-time video streams from fixed cameras along highways with floating car data from moving vehicles. At the perception level, it constructs a dynamically fused cross-modal mutual enhancement and collaborative verification mechanism, enabling mutual guidance, verification, and optimization of video image features and vehicle behavior data. This effectively alleviates the constraints of video image quality degradation on monitoring performance in complex environments, reduces missed and false alarms of abnormal events, and improves the accuracy and stability of automatic monitoring of highway abnormal events in all-weather environments. Furthermore, it eliminates the need for additional high-cost dedicated sensors, fully utilizing existing transportation infrastructure resources and possessing strong engineering practicality.

[0044] Second, this invention deploys the core processing module on the roadside edge computing unit, avoiding network bandwidth occupation and transmission delay caused by transmitting a large amount of raw data to the cloud, ensuring the real-time monitoring and reporting of abnormal events, and meeting the immediate response requirements of traffic management. At the same time, the system has the ability to automatically switch between multimodal and single-modal working modes. When floating car data is missing or insufficient, it can switch to a working mode that relies solely on video streams to ensure that the monitoring task continues, significantly improving the system's adaptability and robustness in different traffic scenarios.

[0045] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from an examination of the following, or may be learned from the practice of the invention. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0047] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0048] Figure 2 This is an internal flowchart of the core processing module of the present invention;

[0049] Figure 3 This is a detailed flowchart of the mutual enhancement and synergy determination method of the present invention. Detailed Implementation

[0050] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0051] Example 1

[0052] like Figures 1 to 3 As shown in the figure, this embodiment discloses a highway anomaly event monitoring system based on video image analysis. It aims to address the problem of insufficient reliability in highway anomaly event monitoring under complex environments by deeply fusing video stream data and floating car data across modalities, thereby improving monitoring accuracy and robustness. The following detailed description of the system's components and workflow, combined with specific implementation details, provides a comprehensive overview.

[0053] In this embodiment, the system includes a video stream access module, a vehicle behavior data access module, a core processing module, and an output module. These modules work collaboratively to achieve real-time monitoring and reporting of abnormal events. The core processing module is deployed on a roadside edge computing unit and possesses parallel computing capabilities to meet real-time processing requirements. The overall architecture follows a logical flow of data access, deep fusion, collaborative judgment, and result output, ensuring smooth connection between each stage and efficient data processing.

[0054] In this embodiment, the specific implementation of each module is as follows:

[0055] The core function of the video stream access module is to acquire real-time video streams from fixed cameras along the highway, providing raw data support for subsequent image feature extraction.

[0056] In the specific implementation, high-definition network cameras already deployed along the highway are used as video acquisition devices. The camera resolution is set to 1920×1080 pixels, and the frame rate is 30 frames per second, which can clearly capture the movement status of traffic participants such as vehicles and pedestrians, as well as details of the road environment. The cameras support Ethernet or 5G network transmission and transmit real-time video streams to the roadside edge computing unit through the standard RTSP protocol. The H.264 encoding format is used during transmission to reduce data transmission bandwidth consumption while ensuring video quality.

[0057] To ensure the validity of the video data, the module performs preliminary preprocessing on the incoming video stream, including frame synchronization and distortion correction. Frame synchronization is achieved by extracting the timestamp from the camera's built-in clock, ensuring accurate time stamping for each frame. Distortion correction, based on the camera's intrinsic parameter data, corrects image distortion caused by lens distortion, laying the foundation for subsequent spatiotemporal alignment operations.

[0058] In this embodiment, the vehicle behavior data access module is used to anonymize the floating car data of the moving vehicles. This data is key information reflecting the vehicle's behavior status and complements the video stream data.

[0059] Specifically, the floating car data originates from vehicles equipped with connected communication capabilities on highways. Data transmission is achieved through vehicle-to-everything (V2X) roadside units (RSUs) or 4G / 5G mobile communication networks, employing the TCP / IP protocol to ensure data transmission reliability. To protect user privacy, the module anonymizes the received data, removing sensitive content such as vehicle identification numbers and owner information, retaining only behavioral data related to vehicle location, speed, and acceleration.

[0060] The floating car data is updated at a frequency of 10Hz to ensure real-time reflection of vehicle dynamics. Vehicle position data is in three-dimensional coordinates, collected by the vehicle's onboard GPS / BeiDou satellite navigation system, with a positioning accuracy of 1-5 meters. Speed ​​data is vector information, including speed magnitude and direction of travel, collected jointly by the vehicle's wheel speed and heading sensors. Acceleration data is also vector information, including acceleration magnitude and direction, collected by the vehicle's built-in acceleration sensor, accurately capturing abnormal motion states such as rapid acceleration and braking.

[0061] The module also has a data filtering function, which uses a sliding window algorithm to remove abnormal data and ensure the quality of data input to the core processing module.

[0062] In this embodiment, the core processing module is deployed in a roadside edge computing unit. This unit uses an industrial-grade server equipped with a GPU acceleration module (such as an NVIDIA Tesla T4) to support parallel computing and meet the real-time processing requirements of video streams and floating car data. The core processing module includes a spatiotemporal alignment submodule, a mutual enhancement submodule, and a collaborative decision-making submodule. These submodules are connected sequentially and work collaboratively to achieve deep data fusion and anomaly event determination.

[0063] The function of the spatiotemporal alignment submodule is to realize the time synchronization and spatial coordinate mapping between real-time video streams and floating car data, ensuring that the two types of data are fused and processed in the same spatiotemporal dimension.

[0064] For time synchronization, UTC time is used as the unified time base, and the video stream and floating car data are time-calibrated through the Network Time Protocol (NTP). The timestamp of the video stream is taken from the camera's built-in clock, and the timestamp of the floating car data is taken from the time of the vehicle's satellite navigation system. The submodule calibrates the two types of timestamps through the NTP server, controlling the synchronization error within 10ms, and ensuring that the video frames and floating car data within the same time window correspond and match.

[0065] In terms of spatial coordinate mapping, based on camera calibration parameters and high-precision map data, perspective transformation is used to map the global coordinates of the floating car data to the pixel coordinates of the video images. The camera calibration parameters include intrinsic and extrinsic parameters. The intrinsic parameters are obtained offline using the Zhang Zhengyou calibration method, while the extrinsic parameters are determined through on-site measurements combined with high-precision map calibration; both are stored in the local database of the edge computing unit. The high-precision map data is accurate to the centimeter level and includes geographic information such as road lane lines, shoulders, and traffic signs, used to assist in the accuracy of coordinate transformation.

[0066] The specific implementation process of perspective transformation is as follows: First, the latitude and longitude coordinates of the floating car are converted into Cartesian coordinates, and then combined with elevation data to obtain the three-dimensional coordinates in the world coordinate system; then, based on the camera's extrinsic parameters, the three-dimensional coordinates in the world coordinate system are converted into coordinates in the camera coordinate system; finally, using the camera's intrinsic parameters and the perspective projection matrix, the coordinates in the camera coordinate system are converted into pixel coordinates of the video image. The formula for constructing the projection matrix is ​​as follows:

[0067]

[0068] in, Image pixel coordinates, For the floating car's world coordinates, This represents the depth value in the camera coordinate system. , This is the equivalent focal length of the camera. , Principal point coordinates, image center pixel coordinates. Let the rotation matrix describe the camera pose. This is a translation vector describing the camera's position. Using this projection matrix, the position of each floating car can be accurately mapped to its corresponding video image pixel.

[0069] The mutual enhancement submodule is a key part of the core processing module. It realizes behavior heatmap generation, initial image feature extraction, enhanced image feature generation, and optimized behavior heatmap output. Through mutual guidance and verification between video modalities and floating car modalities, it improves the accuracy of feature representation.

[0070] Behavioral heatmaps are used to reflect the spatial distribution of abnormal vehicle behavior. They are generated based on a kernel density estimation algorithm, which provides attention-guided signals for image feature enhancement by quantifying the degree of abnormal behavior of floating cars.

[0071] For any pixel position in the video image Its corresponding behavioral thermal value Calculated using the following formula:

[0072]

[0073] The specific definitions and implementation methods of each parameter are as follows:

[0074] The number of floating cars within the time synchronization window. The time synchronization window size is set to 1 second, which means that the number of all floating cars participating in data transmission is counted every 1 second to ensure that the heat map can reflect the vehicle behavior status in real time during the current period.

[0075] : No. The vehicle's three-dimensional position coordinates, i.e., the three-dimensional coordinates in the world coordinate system within the floating car data. This provides the raw data for coordinate mapping.

[0076] The mapping function from world coordinates to image pixel coordinates, i.e., the perspective transformation function implemented in the spatiotemporal alignment submodule, directly reuses the calculation results of spatiotemporal alignment to avoid redundant calculations.

[0077] Euclidean distance on the image pixel plane, used to calculate pixels. Mapped floating car pixel coordinates The straight-line distance between them, measured in pixels, is used to measure the spatial relationship between a pixel's position and the vehicle's position.

[0078] : Kernel function, using the Gaussian kernel function, its expression is as follows This function satisfies the basic conditions of a kernel function, and has the characteristics of good smoothness and strong locality, which can effectively quantify the influence of vehicle position on surrounding pixels.

[0079] The bandwidth parameter controls the influence range of the kernel function. It adopts an adaptive value setting, dynamically adjusting based on the lane width and floating car density of the monitored road segment, with a value range of 5-20 pixels. For example, in a road segment where the lane width corresponds to 10 pixels in the image, if the floating car density is high, the bandwidth value is 5-8 pixels; if the floating car density is low, the bandwidth value is 12-20 pixels, ensuring that the thermal influence range of adjacent vehicles is reasonable and that there is no excessive overlap or insufficient coverage.

[0080] : No. The vehicle's velocity vector, which includes the speed magnitude and direction of travel, is directly taken from the output data of the vehicle behavior data access module.

[0081] : No. The vehicle's acceleration vector, which includes the magnitude and direction of acceleration, is also taken from the output data of the vehicle behavior data access module.

[0082] Abnormal behavior weighting function, used to quantify the first... The degree of abnormal behavior of a vehicle is calculated using the following formula:

[0083]

[0084] The definition and implementation of each parameter in this formula are as follows:

[0085] The normal driving speed vector for the monitored road segment is obtained by statistically analyzing floating car speed data for the same time period over the past 30 days. Specifically, statistical intervals are divided by lane, and the average speed and direction of travel for each lane at different times are calculated as the normal driving speed vector for that lane at that time period, ensuring... Matches with actual traffic scenarios.

[0086] : No. The vehicle's speed deviation, i.e., its velocity vector. With normal velocity vector The Euclidean distance reflects the difference between the vehicle speed and the normal driving speed on the road segment. For example, speeding or driving at low speed will cause this value to increase.

[0087] The magnitude of the acceleration vector, i.e. the magnitude of acceleration, reflects the degree of acceleration or deceleration of the vehicle. Rapid acceleration, sudden braking, and other behaviors will cause this value to increase significantly.

[0088] and : A non-negative weighting coefficient used to balance the contribution of speed deviation and acceleration magnitude to abnormal behavior, with values ​​ranging from 0.4 to 0.6. It can be adjusted according to road segment type; for example, sudden speed changes have a greater impact on highways. Values ​​0.6 The value is 0.4; the impact of acceleration anomalies is more pronounced on urban expressways. Values ​​0.4 The value is set to 0.6, and the accuracy of abnormal behavior quantification is ensured by reasonably allocating weights.

[0089] The behavioral thermal value calculated using the above formula The range is 0-10. The larger the value, the more concentrated the abnormal behavior at the corresponding pixel location, providing clear attention guidance for subsequent image feature enhancement.

[0090] Initial image feature extraction is achieved through the encoder of the video feature enhancement network. The encoder adopts the ResNet50 convolutional neural network structure, which has the characteristics of deepness and low gradient vanishing risk, and can effectively extract multi-scale image features.

[0091] The specific implementation process is as follows: The preprocessed video frames are input into the encoder and sequentially processed through convolutional layers, batch normalization layers, ReLU activation functions, and pooling layers. The convolutional layers use 3×3 kernels with a stride of 1 or 2, extracting local features of the image through a sliding window approach; the batch normalization layers standardize the convolutional output, accelerating network training convergence; the ReLU activation function introduces a non-linear transformation, enhancing the network's feature representation ability; the pooling layers use max pooling with a 2×2 window size and a stride of 2, achieving feature dimensionality reduction while preserving key information.

[0092] The encoder outputs feature maps at five levels. Shallow feature maps correspond to low-level visual features such as edges and textures of the image and have a high resolution. Deep feature maps correspond to high-level semantic features such as vehicles and pedestrians, with a lower resolution but richer semantic information. These feature maps together constitute the initial image features, providing a foundation for subsequent enhancement processing.

[0093] Enhanced image feature generation is achieved through an encoder-decoder structure combined with an attention gating mechanism. It utilizes the attention guidance of behavioral heatmaps to highlight image features in abnormal areas and suppress irrelevant features in normal areas.

[0094] The decoder employs a transposed convolutional structure to upsample the low-resolution feature map output by the encoder, restoring the feature map resolution. An attention gating mechanism is deployed in the skip connections between the encoder and decoder. Its operation is as follows: First, the behavior heatmap is processed through a 1×1 convolutional layer to adjust its channel count, making it consistent with the channel count of the feature maps at each level of the encoder. Then, the adjusted heatmap is input into a sigmoid activation function to generate an attention weight map. The weight values ​​range from 0 to 1; regions with higher behavior heatmap values ​​have weight values ​​closer to 1, and vice versa.

[0095] When the decoder performs feature upsampling, the initial image features of the corresponding layer passed from the encoder through skip connections are multiplied element-wise with the attention weight map to achieve weighted feature enhancement. That is, the initial image features of regions with concentrated abnormal behavior are amplified, while the initial image features of normal regions are suppressed. Finally, through upsampling and feature fusion at each layer of the decoder, enhanced image features with the same resolution as the original video frame are output. These features can more accurately characterize the visual information of abnormal events.

[0096] The optimized behavior heatmap is obtained by verifying and optimizing the original behavior heatmap by enhancing image features. The purpose is to correct the deviations caused by the sparse or abnormal floating car data in the original heatmap and improve the accuracy of the heatmap.

[0097] The specific implementation process is as follows: The enhanced image features are input into a 3×3 convolutional layer. The convolutional layer outputs a verification map of the same size as the original behavior heatmap. Each pixel in the verification map has a value ranging from 0 to 1, reflecting the probability of an abnormal visual target existing at that location. Then, a weighted fusion method is used to fuse the original behavior heatmap and the verification map. The fusion formula is as follows:

[0098]

[0099] in, The optimized behavior thermal value, To verify the pixels in the image The values ​​of 0.7 and 0.3 are the fusion weights, determined through experimental verification. This ensures both the fundamental role of the original behavior heatmap and the correction of deviations through the validation map. For example, if a certain area of ​​the original behavior heatmap shows an anomaly ( The value was high, but the enhanced image features did not detect the corresponding abnormal visual target. If the value is low, the thermal value of the region will decrease after fusion, avoiding misjudgment; if the enhanced image features detect abnormal visual targets ( If the value is high, but the original behavior heatmap value is low, the heat value of the region will be increased after fusion, thus avoiding missed detection.

[0100] In this embodiment, the collaborative determination submodule outputs abnormal events and their confidence levels based on enhanced image features and optimized behavior heatmaps through collaborative determination logic, ensuring the reliability of the determination results.

[0101] First, potential abnormal visual targets are identified from the enhanced image features using the YOLOv8 target detection algorithm, which features fast detection speed and high accuracy, making it suitable for real-time monitoring scenarios.

[0102] The specific implementation process is as follows: Enhanced image features are input into the YOLOv8 detection network. The network identifies abnormal visual targets in the image through anchor box matching, feature fusion, and classification regression. These targets include sudden vehicle stops, vehicles driving in the wrong direction, pedestrians trespassing, and objects being left behind. The network then outputs the bounding box (i.e., the abnormal visual target region) for each target in the image. The YOLOv8 network is pre-trained using a large-scale traffic anomaly event dataset. This dataset contains traffic scene videos under different weather conditions and at different times, and includes bounding boxes and categories for various anomaly targets. After training, the network achieves a target detection accuracy of over 90% on the validation set.

[0103] For each identified abnormal visual target region Statistical features of abnormal behavior patterns are extracted from the target's internal and surrounding buffer regions. The width of the buffer region is set to 5 pixels to ensure coverage of the associated areas surrounding the target. Statistical features include the mean. Standard deviation and diffusion rate , constitute statistical feature vector The calculation methods for each feature are as follows:

[0104] mean : Computational area The average of the optimized behavior thermal values ​​of all pixels within the buffer region reflects the overall intensity of abnormal behavior in that region, as shown in the formula:

[0105]

[0106] in For the region and the total number of pixels in the buffer area, This indicates a buffer zone.

[0107] Standard deviation : Computational area The standard deviation of the optimized behavior heatmap values ​​of all pixels within the buffer region reflects the concentration of anomalous behavior. The formula is:

[0108]

[0109] A larger standard deviation indicates that abnormal behavior is more concentrated in a localized area within the region.

[0110] diffusion rate : Calculate regions in a series of consecutive frames (set to 10 frames) of images The optimized behavior, specifically the velocity of the centroid's movement, reflects the propagation trend of abnormal behavior. This is first determined using the formula:

[0111]

[0112] Calculate the first Frame thermal centroid Then calculate the Euclidean distance between the centroids of adjacent frames. Finally passed

[0113]

[0114] Calculate the diffusion rate, where This is the video frame interval.

[0115] The collaborative decision-making logic is implemented through a multilayer perceptron (MLP). The input of the MLP is a concatenated feature vector, which consists of the visual feature encoding of the abnormal visual target and the statistical feature vector of the abnormal behavior pattern.

[0116] Specifically, the visual features of the abnormal visual target are encoded as a target feature vector output by the YOLOv8 network, with a dimension of 256, containing semantic information such as the target's shape and texture; this 256-dimensional visual feature vector is then combined with a 3-dimensional statistical feature vector. The features are concatenated to obtain a 259-dimensional input feature vector.

[0117] The network structure of the multilayer perceptron is as follows: Input layer (259 neurons) → Hidden layer 1 (128 neurons, ReLU activation function) → Hidden layer 2 (128 neurons, ReLU activation function) → Output layer (6 neurons, 5 of which correspond to abnormal event types, and 1 corresponds to the confidence score). In the output layer, the 5 event type neurons use the Softmax activation function to output the probability of each type of event (sudden stop, wrong-way driving, pedestrian intrusion, lost object, congestion); the 1 confidence neuron uses the Sigmoid activation function to output the confidence score (0-1).

[0118] The training process of the multilayer perceptron is as follows: supervised training is performed using a labeled abnormal event feature dataset. The dataset includes input feature vectors, corresponding event type labels, and true confidence scores. The loss functions are cross-entropy loss and mean squared error loss. The number of training iterations is set to 100 rounds. Training is stopped when the event classification accuracy on the validation set reaches more than 92% and the confidence prediction error is less than 0.05.

[0119] During the determination, if the multilayer perceptron outputs a certain type of event with the highest probability and the corresponding confidence score exceeds the preset threshold (set to 0.8), then it is determined that a high-confidence abnormal event of that type has occurred; if the confidence score is lower than 0.8, or there is no clear highest probability event type, then it is determined to be a low-confidence event and no further reporting is performed.

[0120] In this embodiment, the output module is used to upload high-confidence abnormal event information to the cloud management platform to support traffic management decisions.

[0121] The abnormal event information includes multi-dimensional content: event type, occurrence time, occurrence location, confidence score, event-related video clips, and optimization behavior heatmap.

[0122] Data transmission utilizes either 5G networks or fiber optic communication. 5G networks offer low latency, making them suitable for scenarios with high real-time requirements; fiber optic communication boasts high bandwidth and stability, making it ideal for long-distance transmission. SSL encryption is employed during transmission to ensure data security and prevent information leakage or tampering.

[0123] After receiving the data, the cloud management platform analyzes and displays the information. Traffic management personnel can view the details of abnormal events in real time through the platform and take corresponding measures according to the type and severity of the event.

[0124] In this embodiment, when floating car sample data is missing or insufficient (i.e., the number of floating cars within the time synchronization window) (For vehicles less than 3), the system automatically switches to a single-modal working mode that relies solely on real-time video streams to ensure continuous system operation.

[0125] In single-modal operation mode, the system disables the processing flow related to floating car data in the mutual enhancement submodule, extracting initial image features only through the encoder of the video feature enhancement network. These features are then input into the YOLOv8 target detection algorithm for abnormal visual target recognition, combined with a simplified multilayer perceptron model for event determination. The determination threshold remains set to 0.8 to ensure monitoring accuracy in single-modal mode. When the floating car data returns to normal ( When there are ≥3 vehicles, the system automatically switches back to the multimodal fusion working mode without manual intervention.

[0126] In this embodiment, the system's workflow is as follows:

[0127] 1. The video stream access module acquires the video stream from the fixed highway camera in real time, performs preprocessing, and then transmits it to the core processing module;

[0128] 2. The vehicle behavior data access module anonymizes the floating car data, filters out abnormal data, and then transmits it to the core processing module;

[0129] 3. The spatiotemporal alignment submodule realizes time synchronization and spatial coordinate mapping between the video stream and the floating car data;

[0130] 4. The mutual enhancement submodule generates a behavior heatmap based on the mapped floating car data, uses the heatmap to guide the extraction of enhanced image features, and then optimizes the behavior heatmap through the enhanced image features;

[0131] 5. The collaborative judgment submodule identifies abnormal visual targets, extracts statistical features, and outputs abnormal events and confidence levels through a multilayer perceptron;

[0132] 6. The output module uploads high-confidence anomaly event information to the cloud management platform;

[0133] 7. If the floating car data is insufficient, the system will automatically switch to single-mode operation to ensure continuous monitoring.

[0134] The beneficial effects of this embodiment are as follows:

[0135] 1. This system achieves mutual enhancement at the perception level through deep cross-modal fusion of video streams and floating car data, effectively mitigating the problem of decreased monitoring accuracy caused by video image quality degradation in complex environments, and reducing the false alarm rate and missed alarm rate of abnormal events;

[0136] 2. The core processing module is deployed on the roadside edge computing unit, which avoids the delay caused by transmitting a large amount of raw data to the cloud, ensuring real-time monitoring and reporting of abnormal events and meeting the real-time requirements of traffic management;

[0137] 3. It supports automatic switching between multimodal and single-modal working modes, which improves the robustness of the system in scenarios with insufficient floating car data and expands the applicability of the system;

[0138] 4. By making full use of existing fixed cameras and connected vehicles on highways, there is no need to add high-cost dedicated sensors, which reduces the deployment and maintenance costs of the system and has strong engineering practicality.

[0139] Example 2

[0140] like Figures 1 to 3 As shown, this embodiment further elaborates on the specific workflow of the highway anomaly monitoring system based on video image analysis, while supplementing and optimizing alternative implementation schemes, clarifying the system operation logic under different application scenarios, and improving the feasibility and robustness of the technical solution. The workflow of this system is described in detail below with specific implementation details.

[0141] In this embodiment, the system's workflow revolves around data access, spatiotemporal alignment, cross-modal mutual enhancement, collaborative judgment, result output, and mode switching. Each step is closely integrated to ensure real-time and accurate monitoring of abnormal events. The specific steps are as follows:

[0142] The first step is data access and preprocessing. After the video stream access module starts, it continuously connects to the fixed cameras deployed along the highway and acquires real-time video streams via the standard RTSP protocol. Specifically, the cameras acquire images according to preset parameters, with a fixed high-definition resolution and a stable frame rate to capture continuous traffic conditions. Mainstream encoding formats are used during video stream transmission to balance transmission efficiency and image quality. The module performs preprocessing on the received video stream, first performing frame synchronization processing, marking the time information of each frame based on the camera's built-in clock, and then correcting image distortion caused by lens distortion based on pre-calibrated camera intrinsic parameter data to ensure the accuracy of subsequent coordinate mapping.

[0143] Simultaneously, the vehicle behavior data access module receives floating car data transmitted from vehicles with connected car communication capabilities via vehicle-to-everything (V2X) roadside units or mobile communication networks. During data reception, strict anonymization is performed, removing sensitive information about the vehicle and its owner, retaining only behavior-related data such as position, speed, and acceleration. The module filters the received data to remove abnormal data caused by signal interference, ensuring the data input to the core processing module is authentic and valid, and maintaining a stable data update frequency to reflect dynamic changes in the vehicle.

[0144] The second step is spatiotemporal alignment. The spatiotemporal alignment submodule in the core processing module receives the pre-processed video stream data and floating car data, and first performs time synchronization. Using UTC time as a unified reference, the timestamps of both types of data are calibrated using a network time protocol, keeping the synchronization error within an extremely low range and ensuring accurate matching between video frames and floating car data within the same time window.

[0145] Subsequently, spatial coordinate mapping is performed. Based on pre-stored camera calibration parameters and high-precision map data, coordinate transformation is completed through perspective transformation. First, the global coordinates of the floating car are converted into Cartesian coordinates and combined with elevation data to obtain three-dimensional coordinates in the world coordinate system. Then, based on the camera extrinsic parameters, they are converted into coordinates in the camera coordinate system. Finally, through the camera intrinsic parameters and projection matrix, they are mapped to the pixel coordinates of the video image, achieving precise alignment between the floating car data and the video image in the spatial dimension, laying the foundation for subsequent cross-modal fusion.

[0146] The third step is cross-modal mutual enhancement processing. The mutual enhancement submodule receives two types of data after spatiotemporal alignment and first generates a behavior heatmap. Based on the floating car data within the time synchronization window, the spatial distribution of abnormal vehicle behavior is quantified using a kernel density estimation algorithm. According to the spatial correlation between vehicle position and image pixels, combined with the degree of anomaly reflected by vehicle speed and acceleration, a corresponding heatmap value is assigned to each pixel in the image. The higher the heatmap value, the more concentrated the abnormal behavior in that area, forming a behavior heatmap that reflects the spatial distribution of abnormal behavior.

[0147] Next, initial image features are extracted. The encoder performs multi-scale feature extraction on the preprocessed video frames, which are then processed sequentially through convolutional batch normalization activation functions and pooling layers to obtain initial image features containing both low-level visual and high-level semantic features. Subsequently, the behavior heatmap is used as an attention guidance signal, and an attention weight map is generated through an attention gating mechanism. When the decoder performs feature upsampling, the initial image features passed through skip connections are weighted to highlight features in abnormal regions and suppress irrelevant features in normal regions, resulting in enhanced image features.

[0148] Finally, the behavior heatmap is optimized by inputting enhanced image features into a convolutional layer to generate a verification map, which reflects the probability of abnormal visual targets at various locations in the image. A weighted fusion method is used to combine the original behavior heatmap with the verification map, correcting the biases caused by data sparsity or anomalies in the original heatmap, resulting in an optimized behavior heatmap and improving its accuracy.

[0149] The fourth step is collaborative decision-making. After receiving the enhanced image features and optimized behavior heatmap, the collaborative decision-making submodule first identifies potential abnormal visual targets from the enhanced image features. It then processes the enhanced image features using a target detection algorithm to identify abnormal targets such as vehicles suddenly stopping, driving in the wrong direction, pedestrians trespassing, or objects being left behind. The algorithm determines the target's location within the image and outputs relevant target category information. This target detection algorithm is pre-trained on a large-scale traffic anomaly event dataset to ensure recognition accuracy in different scenarios.

[0150] Subsequently, statistical features of abnormal behavior patterns are extracted. For each identified abnormal visual target region, the scope is expanded to its surrounding buffer zone. The mean, standard deviation, and diffusion rate of the optimized behavior heatmap within this range are calculated to form a statistical feature vector. The mean reflects the overall intensity of abnormal behavior in the region, the standard deviation reflects the concentration of abnormal behavior, and the diffusion rate reflects the spread trend of abnormal behavior.

[0151] Finally, the collaborative decision-making logic is executed, concatenating the visual feature encodings of abnormal visual targets with statistical feature vectors to form an input feature vector, which is then fed into a multilayer perceptron. The multilayer perceptron, using a pre-defined network structure and training parameters, outputs the probability of various abnormal events and their corresponding confidence scores. When a certain type of event has the highest probability and its confidence score exceeds a pre-defined threshold, it is determined to be a high-confidence abnormal event.

[0152] The fifth step involves outputting results and interacting with the cloud. The output module receives high-confidence abnormal event information from the collaborative judgment submodule, integrating multi-dimensional content such as event type, occurrence time, location, confidence score, relevant video clips, and optimization behavior heatmaps. The integrated information is uploaded to the cloud management platform via 5G network or fiber optic communication. Encryption protocols are used during transmission to ensure data security and prevent information leakage or tampering. The cloud management platform receives the data, parses and displays it, providing traffic management personnel with detailed abnormal event information to support the formulation and implementation of subsequent response measures.

[0153] Step 6: Working Mode Switching. During system operation, the number of floating cars within the time synchronization window is continuously monitored. When the number of floating cars falls below a preset threshold, it is determined that floating car data is missing or insufficient, and the system automatically switches to single-modal working mode. In this mode, processing flows related to floating car data are disabled. Only the initial image features of the video frames are extracted by the encoder. After abnormal visual targets are identified by the target detection algorithm, the data is input into a simplified multilayer perceptron for event determination, maintaining a constant determination threshold to ensure monitoring accuracy. When the number of floating cars recovers to above the preset threshold, the system automatically switches back to multimodal fusion working mode without manual intervention, ensuring continuous and stable monitoring.

[0154] In some optional implementations, the video stream access module can employ a high-definition camera with 5G communication capabilities to directly transmit the real-time video stream to the roadside edge computing unit via the 5G network, eliminating intermediate transmission nodes and further reducing transmission latency. A brightness adaptive adjustment step is added during preprocessing to automatically adjust image brightness and contrast for low-light or strong backlight scenarios at night, improving image clarity. Simultaneously, inter-frame differencing is used for preliminary screening of the video stream, transmitting only video frames containing moving targets to subsequent processing modules, reducing the amount of invalid data processed and improving system operating efficiency.

[0155] In some optional implementations, the vehicle behavior data access module can use a dedicated vehicle-to-everything (V2X) communication protocol to receive floating car data. This protocol is optimized for traffic scenarios, improving the real-time performance and stability of data transmission. During data filtering, a Kalman filter algorithm is used to smooth the floating car's position, velocity, and acceleration data, effectively eliminating abnormal data caused by random interference and further improving data quality. Simultaneously, the module can store recent historical floating car data. When real-time data is briefly interrupted, reasonable predictions are made based on historical data to ensure the system can still operate normally during periods of data loss.

[0156] In some optional implementations, the collaborative judgment submodule can employ a judgment method combining a decision tree model and a multilayer perceptron. The decision tree model first performs preliminary screening based on the visual and statistical features of abnormal visual targets, eliminating samples that are clearly not abnormal events. The filtered samples are then input into the multilayer perceptron for accurate judgment, improving judgment efficiency. Simultaneously, the preset judgment threshold can be dynamically adjusted according to the traffic flow of the monitored road segment. When traffic flow is high, the threshold is appropriately increased to reduce false alarms; when traffic flow is low, the threshold is appropriately decreased to reduce false negatives, allowing the system to adapt to the monitoring needs of different traffic scenarios.

[0157] Through the aforementioned workflow and optimized alternatives, the system can fully leverage the cross-modal fusion advantages of video streams and floating car data, effectively improving the accuracy and reliability of anomaly event monitoring in complex environments. Spatiotemporal alignment processing ensures effective fusion of the two types of data, cross-modal mutual enhancement processing improves the accuracy of feature representation, collaborative judgment logic ensures the reliability of event judgment, and mode switching functionality enhances the system's adaptability.

[0158] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A highway anomaly event monitoring system based on video image analysis, characterized in that, include: The video stream access module is used to acquire real-time video streams from fixed cameras along the highway. The vehicle behavior data access module is used to anonymously receive floating car data of moving vehicles, wherein the floating car data includes at least vehicle position, speed and acceleration; The core processing module, deployed in the roadside edge computing unit, is configured as follows: Perform a spatiotemporal alignment operation to map the floating car data onto the image coordinate system of the real-time video stream; Generate a behavior heatmap reflecting the spatial distribution of abnormal vehicle behavior based on the mapped floating car data; The behavior heatmap is used as an attention-guided signal to enhance the initial image features extracted from the real-time video stream, resulting in enhanced image features. The enhanced image features are used to verify and optimize the behavior heatmap to obtain an optimized behavior heatmap; Based on the enhanced image features and the optimized behavior heatmap, abnormal events and their confidence levels are output through collaborative judgment logic. The output module is used to upload high-confidence abnormal event information to the cloud management platform.

2. The highway abnormal event monitoring system based on video image analysis according to claim 1, characterized in that, The core processing module includes a spatiotemporal alignment submodule, a mutual enhancement submodule, and a collaborative determination submodule connected in sequence. The spatiotemporal alignment submodule is configured to synchronize the real-time video stream and the floating car data using a unified time, and to map the global coordinates of the floating car data to the pixel coordinates of the video image through perspective transformation based on camera calibration parameters and high-precision map data. The mutual enhancement submodule is configured to perform the generation of the behavior heatmap, the extraction of the initial image features, the generation of the enhanced image features, and the output of the optimized behavior heatmap; The collaborative determination submodule is configured to execute the collaborative determination logic.

3. The highway abnormal event monitoring system based on video image analysis according to claim 2, characterized in that, The mutual enhancement submodule is configured to generate the behavior heatmap through the following steps: For any pixel position in the video image Its corresponding behavioral thermal value The results were obtained using the kernel density estimation algorithm: ； in, This indicates the number of floating cars within the time synchronization window. Indicates the first The vehicle's three-dimensional position coordinates, A function representing the mapping from world coordinates to image pixel coordinates. Represents the Euclidean distance on the image pixel plane. Represents a kernel function that satisfies and , Indicates bandwidth parameter, and They represent the first The vehicle's velocity vector and acceleration vector The abnormal behavior weighting function is determined by the rate of change of the velocity vector and the magnitude of the acceleration vector.

4. The highway abnormal event monitoring system based on video image analysis according to claim 3, characterized in that, The abnormal behavior weight function The calculation method is as follows: ； in, The normal driving speed vector of the monitored road segment is obtained from historical floating car data. Indicates the first Vehicle speed deviation, The magnitude of the acceleration vector is represented by the acceleration vector. and These are pre-defined non-negative weighting coefficients used to quantify the contributions of velocity mutations and acceleration anomalies, respectively.

5. The highway abnormal event monitoring system based on video image analysis according to claim 2, characterized in that, The mutual enhancement submodule includes a behavior-guided video feature enhancement network, which is an encoder-decoder structure and integrates an attention gating mechanism in the skip connections; The encoder is used to extract the features of the initial image; The attention gating mechanism takes the behavior heatmap as input to generate an attention weight map, which is used to weight the features passed by the skip connections when the decoder performs feature upsampling. The decoder outputs the enhanced image features.

6. The highway abnormal event monitoring system based on video image analysis according to claim 5, characterized in that, The behavior-guided video feature enhancement network is trained in a self-supervised manner, and its loss function is constructed as follows: the enhanced image features are encouraged to decode and recover the target contour in the high-value region of the behavior heatmap, which conforms to the physical constraint model learned from large-scale normal traffic data. The physical constraint model includes target size distribution constraints and motion trajectory continuity constraints.

7. The highway abnormal event monitoring system based on video image analysis according to claim 2, characterized in that, The collaborative determination logic executed by the collaborative determination submodule is as follows: Identify potential anomalous visual targets and their regions in the image from the enhanced image features; Statistical features of abnormal behavior patterns in the corresponding regions are extracted from the optimized behavior heatmap; Application judgment rule: A high-confidence abnormal event is judged to have occurred only when a specific abnormal visual target is identified and the statistical characteristics of the abnormal behavior pattern in its corresponding area exceed a preset threshold.

8. The highway abnormal event monitoring system based on video image analysis according to claim 2, characterized in that, The judgment rules implemented by the collaborative judgment submodule are executed by a multilayer perceptron. The input of the multilayer perceptron is a concatenated feature vector, which is composed of the visual feature encoding of the abnormal visual target and the statistical features of the abnormal behavior pattern of the corresponding region. The output of the multilayer perceptron is the abnormal event type label and the comprehensive confidence score.

9. The highway abnormal event monitoring system based on video image analysis according to claim 7, characterized in that, The statistical features of the abnormal behavior patterns are calculated as follows: for abnormal visual target regions identified in the image... Calculate the mean value of the heatmap of the optimized behavior within its internal and surrounding buffer areas. Standard deviation and the diffusion rate of thermal value over time Together they constitute the statistical feature vector. The diffusion rate It is obtained by calculating the movement speed of the centroid of the thermal values ​​in the region between consecutive frames.

10. The highway abnormal event monitoring system based on video image analysis according to claim 1, characterized in that, The vehicle behavior data access module receives the floating car data through a vehicle-to-everything (V2X) roadside unit or a mobile communication network. The floating car data comes from vehicles with network communication capabilities that are driving on highways, forming a floating car sample set. When the floating car sample data is missing or insufficient, the system can switch to a single-modal working mode that relies solely on the real-time video stream. In the single-modal working mode, the core processing module disables the generation of the behavior heatmap and related mutual enhancement processes, and performs abnormal event determination based on the image features extracted from the real-time video stream.

Images