Key frame filtering system combining traffic abnormal events and static events

Abstract

The application discloses a kind of traffic abnormal event and static event combination's key frame filtering system, the system includes camera end and server, it is connected by wide area network between each other, server end is deployed on cloud server;The camera end is used to record traffic monitoring video, traffic monitoring video is divided into normal monitoring video and monitoring video of traffic accident, including video generation module, filtering module and adaptive control module;The server is used to process filtered video stream, and the real-time static event detection result and abnormal event detection result are obtained by training model;Specifically including receiving module, static event detection module and abnormal event detection module, model trainer, output module and model;Compared with prior art, the application can process dynamic abnormal event and static event in real time under the condition that wide area network bandwidth is limited, and guarantee the accuracy of event detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of edge computing, and particularly relates to a key frame filtering system and method for traffic abnormal event based on video stream. BACKGROUND

[0002] With the proposal of the strategy of smart city, the market share of cameras is increasing. The analysis based on machine learning of the real-time stream collected by these cameras, such as traffic monitoring, customer tracking and event detection, is expected to make a major breakthrough in efficiency and safety. In order to obtain sufficient inference accuracy, these tasks usually require high computing power and high-resolution images (or video streams). However, this is a major challenge for real-time vision-based applications. On the one hand, many video analysis tasks have been optimized for cloud environments, and it seems that we can directly send data to the cloud in a bandwidth-limited wide area network, and then the cloud model can extract sufficient information from these data. This assumption is not feasible for advanced visual analysis tasks. Although reducing the video resolution (or frame rate) can indeed save bandwidth, it will cause a decrease in inference accuracy. On the other hand, some real-time advanced visual applications, such as autonomous driving, put expensive hardware accelerators on edge devices to perform local inference. However, this requires a huge economic cost if such applications need to be deployed on a large scale in the future.

[0003] Regarding the analysis and processing of real-time video streams, in general, there are four major challenges: first, the limited bandwidth in the wide area network, second, the processing of massive data will cause a large processing delay, third, the transmission to the cloud at a long distance will cause a propagation delay that users cannot accept, and fourth, how to maintain the accuracy of event detection. In order to solve the above four problems, there have been a lot of research results in this regard. In general, it is mainly divided into four categories: deploying a compressed target object detection model on the edge, pre-processing the video, filtering out the frames containing the objects of interest, and sending them to the data center server for further processing, thereby saving the required bandwidth of network transmission. Some first conduct experiments on specific video streams to obtain a correspondence relationship between a three-tuple of bandwidth, resolution and accuracy, and then compress the picture quality according to the change of the wide area network bandwidth, thereby saving the wide area network bandwidth. Some researches are based on the former, and pre-train an offline configuration file of bandwidth, threshold and accuracy, which can dynamically adjust the threshold value through the pixel change of the video content, and then filter the key frames through the threshold value, thereby reducing the video data transmission amount. For some video segments that cannot find a corresponding relationship in the offline configuration file, the entire video segment is transmitted to the cloud for online training, thereby perfecting the offline configuration file, so that the offline configuration file is always dynamically adjusted to adapt to different video content.

[0004] However, the methods mentioned above all have limitations to some extent. While Filterforward, utilizing neural networks, can determine whether to transmit an image to a remote server based on whether it contains an object of interest, its ability to reduce the amount of data transmitted is negligible under certain extreme conditions. For example, in detecting vehicles, Filterforward transmits almost all video frames of congested roads as keyframes to the server for processing. Similarly, Reducto can adjust the resolution of images sent to the cloud for processing based on pre-set network conditions and bandwidth, which lacks flexibility to some extent. Accuracy is not linearly related to image resolution, making this configuration somewhat inflexible. Although Reducto can dynamically adjust thresholds and dynamically acquire keyframes based on video type, it ignores the dynamic attributes of video. The keyframes filtered by Reducto can only process some static events, but are helpless against dynamic events contained in the video. In summary, it is particularly important to develop an algorithm that can filter keyframes that can detect both dynamic and static events. Summary of the Invention

[0005] Based on the aforementioned existing technologies, this invention aims to propose a keyframe filtering system and method that combines traffic anomaly events and static events. It provides an algorithm for efficiently extracting video keyframes of relevant traffic anomaly events under wide area network bandwidth constraints, and the results help in the real-time identification and processing of static and dynamic anomaly events.

[0006] This invention is achieved using the following technical solution:

[0007] This invention discloses a keyframe filtering system that combines traffic anomaly events and static events. The system includes a camera end and a server end; wherein:

[0008] The camera is used to record traffic monitoring video, which is divided into normal monitoring video and monitoring video of traffic accidents. Specifically, it includes a video generation module, a filtering module, and an adaptive control module, wherein:

[0009] The video generation module generates real-time video data and sends the video data to the filtering module; the filtering module performs graying on the video data to obtain filtered video stream and sends the filtered video stream to the receiving module of the server; the adaptive control module obtains a frame in the video, performs front / back frame pixel difference on the frame, performs binaryzation on the frame by using a dynamically adjustable threshold, counts and records a proportion of the frame in which a binaryzation result is 1, compares the proportion with a best threshold selected from an offline configuration file returned by the receiving module of the server, judges whether the frame is a key frame, and records the proportion of the frame in which the binaryzation result is 1, to provide data for subsequent screening of dynamic abnormal events; and the screened key frame is transmitted to the server through a wide area network for processing.

[0010] The server is configured to process the filtered video stream and obtain real-time static event detection results and abnormal event detection results by using a trained model; and the server comprises a receiving module, a static event detection module and an abnormal event detection module, a model trainer, an output module and a model.

[0011] The receiving module receives the filtered video stream transmitted from the filtering module of the camera end; the static event detection module and the abnormal event detection module perform real-time detection on the filtered video stream, and feed back the real-time detection results and accuracy rates to the client; the model trainer performs rapid model training for each query; and the output module is configured to output the real-time static event detection results and the abnormal event detection results.

[0012] Compared with the prior art, the application can process dynamic abnormal events and static events in real time under the condition that the bandwidth of the wide area network is limited, and the accuracy rate of event detection is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 FIG. 1 is a system architecture diagram of a key frame filtering system combining traffic abnormal events and static events according to the application.

[0014] Figure 2 FIG. 2 is a real-time filtering transmission process diagram of a camera end static event according to the application.

[0015] Figure 3 FIG. 3 is a topology diagram of a key frame filtering system combining traffic abnormal events and static events under a wide area network according to the application. DETAILED DESCRIPTION

[0016] The technical scheme of the application will be further described in detail below with reference to the drawings and specific embodiments.

[0017] As Figure 1As shown, the key frame filtering system architecture combining traffic abnormal events and static events of the application is shown. The system includes a camera end 10 and a server end 20, which are connected through a wide area network, and the network changes are real-time, and the server end is deployed on a cloud server.

[0018] The camera end 10 is used to record traffic monitoring video, which is divided into normal monitoring video and monitoring video of traffic accident, including video generation module 101 and filtering module 102, and adaptive control module 103, and the specific functions of each module are described as follows:

[0019] The video generation module 101 generates real-time video data and sends it to the filtering module 102; the filtering module 102 first performs grayscale on the video data to obtain the filtered video stream, which is sent to the receiving module 201 of the server end 20, and the main purpose is to reduce the calculation delay and ensure the real-time transmission of the video; the adaptive control module 103 performs front / back frame difference processing on the frames in the video, and performs binaryzation processing with a dynamically adjustable threshold, and the proportion of the binaryzation result being 1 is counted and recorded, and compared with the best threshold selected in the offline configuration file returned by the receiving module 201 of the server end 20 to determine whether the frame is a key frame, at the same time, the proportion of the binaryzation result being 1 is recorded to provide data for subsequent screening of dynamic abnormal events; finally, the selected key frame is transmitted to the server end 20 through the wide area network for processing.

[0020] The server end 20 includes a receiving module 201, a static event detection module 2021 and an abnormal event detection module 2022, a model trainer 203, an output module 204 and a model 205; the specific description of the function of each module is as follows:

[0021] The receiving module 201 receives the filtered video stream from the filtering module of the camera end 10; the static event detection module 2021 and the abnormal event detection module 2022 (such as vehicle counting detection program SA1 and traffic accident detection program DA1, etc.) perform real-time detection on the filtered video stream received from the filtering module of the camera end 10 by the receiving module 201, and feed back the real-time detection result and accuracy to the customer. The model trainer 203 trains a model 205 for each query.

[0022] The offline configuration file performs the following processes:

[0023] The command server 20 uses each model trainer 203 to quickly train a model 205 for each query, which represents the relationship between the difference value, the filtering threshold and the query result accuracy. The model is trained by performing K-means based clustering on the raw frames sent by the camera within a short time after the query arrives. Since a simple model is used, the training usually takes a few seconds to complete. The generated model is encoded as a hash table, where each entry represents a set of difference values, its corresponding threshold is within the same neighborhood, each key is the average difference value, each value is the threshold of the cluster, and its corresponding accuracy. The entire hash table is sent to the camera 10 for each query. The camera 10 does not stream any frames before receiving the query request. When the user specifies the query and the accuracy of the goal, it continuously tracks the differences between consecutive frames for filtering. When the camera 10 receives the hash table, the filtering module looks up the best threshold for itself from the hash table, and then performs the filtering procedure.

[0024] The present application is based on the design of edge cloud cooperation, and the system architecture and algorithm of the camera and the cloud server for real-time processing of video stream. In order to cope with various video types and various contents in each video, a configuration file is trained in advance. When the video is filtered at the camera, each unit is retrieved once in the configuration file, the threshold most suitable for the video content is found, and the key frame is filtered out through the threshold. In the filtering process, the image pixel change is recorded, and the abnormal dynamic event is detected while filtering the key frame of the static event according to the recorded image pixel change. The filtered key frame of the static event and the suspicious abnormal dynamic event are sent to the cloud for processing.

[0025] The present application records the pixel difference change value while extracting the pixel difference between adjacent frames of the static event, and then uses the different characteristics of the video pixel change before and after the abnormal dynamic event to realize the detection of the segment that may have an abnormal dynamic event while extracting the key frame of the static event, and sends them to the server for processing. In the case of real-time change of wide area network bandwidth and great possibility of bandwidth shortage, the hash table of the corresponding relationship between the pixel difference, the threshold and the accuracy returned to the camera by training the model is selected to filter out the key frame of the static event, and the pixel difference is recorded. Then, the pixel difference change of the entire video segment is analyzed to determine whether one or more dynamic abnormal events are included in the video. The key frame is sent in real time, and the dynamic abnormal event segment is sent to the cloud server for processing. Through this method, more than 50% of the bandwidth can be saved, so that the real-time detection of the video in the wide area network scene is realized.

[0026] As Figure 2As shown, it is a real-time filtering transmission process flow chart of a camera end static event of the application, which describes in detail the process of video stream from the camera end to the remote server through the wide area network based on this filtering strategy, and then the remote server feeds back the relevant information to the user through the wide area network. The specific implementation is as follows:

[0027] The obtained video stream is subjected to grayscale processing;

[0028] The pixel difference between adjacent two frames is calculated;

[0029] The value of the pixel point is set to 1 if it is greater than the threshold value, and to 0 if it is less than the threshold value;

[0030] The proportion of the pixel point value being 1 is calculated and recorded, and the following operations are performed respectively:

[0031] ①Comparison with the best threshold value; if the same, send this frame as a static event key frame, if different,

[0032] Discard;

[0033] ②Detect the pixel change of the video segment, send this abnormal event video segment to the cloud for processing.

[0034] In the specific implementation process, first, the preparation of the configuration file, the video has the characteristics of dynamic change, the video may keep the target object basically unchanged or change small in a short time, but from the long time span, the characteristic object in the video is changing rapidly, relevant research shows that the best threshold value of the video is the fast change in the order of segmentation, in order to prevent the filtering algorithm of the camera from causing the query precision to be lower than the target detection precision, the camera needs a method of dynamically adjusting the related threshold value to prevent any unacceptable precision drop. This decision needs to understand the relationship between different threshold values and the accuracy of the query result. This requires the server to establish a mapping between the pixel difference value, the threshold value and the result accuracy, then the camera can use this information to quickly find the best threshold value to be used. To generate this mapping, the camera sends unfiltered frames through a short window immediately after registering the query, which are used as the initial training set, and runs the complete pipeline on the initial training set to generate complete results about each frame segment, including the corresponding relationship of pixel frame difference, threshold value and query accuracy. The pipeline includes frame difference between adjacent frames in the video, and then different threshold values are used to test the different accuracy of the video corresponding to the different threshold values.

[0035] When the camera end does not receive the query request sent by the server, the camera end will not transmit the video stream in any way. When the server transmits the query request and the accuracy, the hash table of the pixel frame difference, the threshold value and the accuracy is also transmitted. The camera end starts to perform the frame difference on the adjacent frames, and records the proportion of the pixel mutation. At the same time, the camera end finds the optimal threshold value from the hash table transmitted by the server to filter the video stream. The video is divided into segments of 3 seconds. The optimal threshold value is found from the hash table for each segment. The whole threshold value is applied to the whole segment. The filtered key frame is transmitted to the server for processing. When the frame difference is performed on the adjacent two frames, the trend of the whole pixel difference change in 15 seconds is analyzed to determine whether an abnormal event exists in the time period. If it is determined that an abnormal event exists, all the frames in the time period of 15 seconds are transmitted to the server for processing. The principle of judging the traffic abnormal event is based on the following verification. On the road surface without an abnormal event, the vehicle flow speed is stable and the pixel change is fast. When a traffic accident occurs, the vehicle flow tends to be slow or even static in a period of time, and the pixel change is slow. The trend of the pixel change of the video segment can be obviously obtained by recording the pixel change. Whether the traffic abnormal event occurs in the time period can be determined according to the above conclusion. Although the video segments obtained by the method may not all be traffic abnormal events, the pressure on the bandwidth caused by the extracted video segments can be ignored.

[0036] Generally, a network detection module is further arranged to detect the change of the available bandwidth of the wide area network in real time. The available bandwidth of the network is calculated according to the amount of data transmitted in a unit time. The probe in the network detection module feeds back the change of the bandwidth to the transmission end. When it is detected that the bandwidth pressure is large, the transmission module adjusts the resolution and the sending rate of the filtered key frame appropriately. When the accuracy of the standard is met, the propagation delay is reduced as much as possible, and the video stream is close to real-time processing. When the bandwidth of the wide area network is sufficient, the filtered key frame and the traffic abnormal event segment are transmitted at full resolution and full rate.

[0037] After the server receives the key frame and the traffic abnormal event segment, the key frame and the abnormal event are identified and processed. Finally, the identified result and the traffic abnormal event contained therein are fed back to the user.

[0038] The application realizes the algorithm for efficiently extracting the key frame of the video under the condition that the bandwidth of the wide area network is limited. The key of the algorithm is that when the bandwidth of the wide area network faces great pressure, the threshold value can be dynamically adjusted to screen out the key frame. According to the change rule of the pixel value in the video, the video segment possibly containing the abnormal event is screened out. For example, the pixel change suddenly changes from one state to another state, which indicates that the abnormal event is possible.

[0039] AsFigure 3 As shown in the figure, the topology diagram of the key frame filtering system combining traffic abnormal events and static events of the application under a wide area network. The topology structure includes a total console, a cloud data center deployed in a wide area network (WLAN), each camera end and edge server can be deployed through a cluster mode through a local area network (LAN) connection, and then connected to the cloud data center and the total console through the wide area network (WLAN).

[0040] The above only shows the preferred embodiments of the technical idea and characteristics of the application, the purpose is to enable those skilled in the art to understand the application and implement it accordingly, and does not limit the application, that is, the modification, replacement, equivalent change and the like made within the spirit and principle of the application shall be within the protection scope of the application.

Claims

1. A keyframe filtering system combining traffic anomaly events and static events, characterized in that, The system includes a camera terminal and a server terminal; among which: The camera is used to record traffic monitoring video, which is divided into normal monitoring video and monitoring video of traffic accidents. Specifically, it includes a video generation module, a filtering module, and an adaptive control module, wherein: The video generation module generates video data in real time and sends it to the filtering module. The filtering module converts the video data to grayscale to obtain a filtered video stream, which is then sent to the receiving module on the server. The adaptive control module obtains the pixel difference between frames in the video and calculates the difference between the preceding and following frames. It then performs binarization processing on the difference with a dynamically adjustable threshold, counts and records the proportion of frames with a binarization result of 1, and compares this with an optimal threshold selected from the offline configuration file returned by the receiving module on the server to determine whether the frame is a key frame. At the same time, it records the proportion of frames with a binarization result of 1 to provide data for subsequent screening of dynamic abnormal events. The selected key frames are transmitted to the server for processing via a wide area network. The server is used to process the filtered video stream and obtain real-time static event detection results and abnormal event detection results through model training; specifically, it includes a receiving module, a static event detection module, an abnormal event detection module, a model trainer, an output module, and a model; wherein: The receiving module receives the filtered video stream from the filtering module at the camera end; the static event detection module and the abnormal event detection module perform real-time detection on the filtered video stream and feed back the real-time detection results and accuracy to the client; the model trainer performs rapid model training for each query; the output module is used to output the real-time static event detection results and abnormal event detection results; wherein, the real-time detection of the static event detection module includes the following steps: performing grayscale processing on the filtered video stream; calculating the pixel difference between two adjacent frames; comparing the pixel difference with a binarization threshold, if it is greater than the threshold, setting the pixel to 1, otherwise setting it to 0; calculating the proportion of pixels set to 1 and comparing it with the optimal threshold, if they are the same, sending it as a keyframe, otherwise discarding it; simultaneously, detecting pixel changes in the video segment and sending abnormal event video segments to the cloud for processing; The abnormal event detection module includes a vehicle counting detection program or a traffic accident detection program.

2. The keyframe filtering system combining traffic anomaly events and static events as described in claim 1, characterized in that, The offline configuration file performs the following process: The command server uses each model trainer to quickly train a model for each query, which represents the relationship between the difference value, the filtering threshold, and the accuracy of the query results. The generated model is encoded as a hash table, where each entry represents a set of difference values, the corresponding thresholds are in the same neighborhood, each key is the average difference value, and each value is the threshold of the cluster and its corresponding accuracy. The entire hash table is sent to each queried camera. When the camera receives the hash table, the filtering module looks up the optimal threshold from the hash table and then executes the filtering procedure.

3. The keyframe filtering system combining traffic anomaly events and static events as described in claim 1, characterized in that, The adjustment strategy for the dynamically adjustable threshold is that the camera sends unfiltered frames through a short window immediately after registering the query. These frames serve as the initial training set. The complete pipeline is run on the initial training set to generate complete results for each frame segment, including the correspondence between pixel frame difference, threshold, and query accuracy. The optimal threshold is found using this correspondence.

4. The keyframe filtering system combining traffic anomaly events and static events as described in claim 3, characterized in that, The optimal threshold is dynamically selected for filtering based on changes in the video content.

Images