Spark-based distributed massive video analysis system

Abstract

The invention belongs to the video data processing field and proposes a Spark-based distributed massive video analysis system. The invention aims to solve the problem of massive video data tracking and pedestrian recognition. The system includes a distributed file subsystem, a resource management module, a distributed subscription subsystem, a data processing module, and a foreground display module; the distributed file subsystem is configured to provide an interaction interface for unstructured data; the resource management module provides unified resource management and scheduling services for upper layer applications; the distributed subscription subsystem is adopted as the message middleware of the system and transmits messages and data; the data processing module includes a plurality of visual processing algorithm sub-modules which cooperate with each other, decompresses acquired video data into multi-frame images, and performs pedestrian detection, tracking, pedestrian attribute recognition, pedestrian identity re-recognition and other processing tasks on the multi-frame images; and the foreground display module displays the processing results of the video data and interacts with a user. With the system of the invention adopted, efficient and accurate pedestrian trajectory tracking can be realized in a large quantity of video data, and pedestrian identity information can be recognized.

Description

technical field [0001] The invention relates to the fields of pattern recognition and computer vision, in particular to a spark-based distributed massive video analysis system. Background technique [0002] With the massive deployment of video surveillance network systems, a large amount of video data is generated. Video data is a kind of unstructured data, which faces great challenges in the storage and content processing of massive video data. [0003] At present, big data processing technologies based on Hadoop and Spark are widely used. However, when these big data processing technologies process compressed video files, there is a dependency between each frame in the compressed video files, so it is necessary to refer to The data before and after the video frame is decompressed, and the direct segmentation like ordinary files will cause the file to be unable to be decompressed, which limits the parallel processing of the video. However, in practical applications, it is...

AI Technical Summary

Problems solved by technology

[0003] At present, big data processing technologies based on Hadoop and Spark are widely used. However, when these big data processing technologies process compressed video files, there is a dependency between each frame in the compressed video files, so it is necessary to refer to The data before and after the video frame is decompressed, and the direct segmentation like ordinary files will cause the file to be unable to be decompressed, which limits the parallel processing of the video

However, in practical applications, it is usually necessary to find a target or event of interest from a large number of offline video files, and manual search is inefficient and error-prone

Images