Mass video data storage system

Abstract

The invention relates to a mass video data storage system, belongs to the data storage field and is to solve a problem of optimizing the performance of an existing massive video data distributed storage system. The mass video data storage system comprises a storage server cluster, the use condition is periodically fed back to a state manager through a heartbeat protocol, a client initiates a datastorage request to the state manager, the state manager receives a data storage request initiated by the client, allocates storage tasks to the storage server cluster according to the feedback condition, idle storage servers of the storage server cluster receive the storage tasks, and the client performs data writing operation on a video stream through the SSD and the disk bound with a virtual machine file system in the storage server, and the state manager periodically performs space recovery on the expired data of the storage server cluster, so the effect of optimizing the performance of theexisting massive video data distributed storage system is achieved.

Description

technical field [0001] The invention belongs to the field of data storage, and relates to a massive video data storage system. Background technique [0002] With the promotion and implementation of projects such as smart transportation, safe city, and digital logistics, the amount of data generated by urban monitoring has exceeded the PB level, and has a tendency to develop to the EB level or even the ZB level. Different from structured data such as files or data tables, video surveillance has the characteristics of large data volume, complex internal structure, intensive writing, constant and high concurrency of code streams, and puts forward higher performance requirements for storage systems. To ensure distributed video surveillance The system can continuously and stably provide services to the outside world, and the real-time and efficient data writing and long-term effective storage have become hot issues in current research. Existing distributed storage systems for ma...

AI Technical Summary

Problems solved by technology

The storage model based on the file system is managed by the operating system layer. Through the secondary cache of data generated by the system's own Buffer, frequent head movement and addressing in the random write mode will cause a large number of disk fragments, which will affect the system IO performance.

At the same time, the file system needs to maintain index node management information and attribute information, which is redundant data for video data with complex internal structures, which will increase the amount of system data processing and is not conducive to efficient storage of video data

The raw device-based storage model is directly controlled by the application program to read and write, avoiding the cache at the operating system layer and improving IO efficiency. However, when dealing with high-concurrency video streams, the video data storage locations are scattered, and internal disk fragments are generated accordingly. When the system is powered off There is a greater probability of failure after failure and failure, and the system reliability is not high

In order to improve the ability of the system to repair faults, the existing technology proposes a fault-tolerant redundancy strategy using multiple copies or erasure codes. Among them, triple-mode redundancy stores data copies in different nodes of the system, which solves the problem of rapid data recovery. It causes a waste of space; although the erasure code improves the space utilization rate, it uses a huge amount of data in the data recovery process for a single disk failure, and the number of faulty nodes increases with the repair time. After exceeding the system fault tolerance limit, no service will be provided

Reducing energy consumption and prolonging disk life are issues of great concern to large-scale video surveillance storage systems. The system handles high IO operations for a long time, and disk wear will greatly reduce the service life.

In this regard, Watanabe et al. designed a disk energy-saving mechanism. In the sleep plan, the disk is divided into active, idle and standby states, which reduces the loss of the disk to a certain extent, but the response delay when the sleeping disk wakes up again limits the system service quality.

Sun Zhizhuo, Zhang Xinxin and others proposed to adopt the new Ripple-RAID, which maintains the energy saving performance of local parallelism, but the energy saving effect of the system dealing with large disk arrays is not good

Images