Virtual container based big data storage and management method

Abstract

The invention provides a virtual container based big data storage and management system and method. The system comprises a metadata server, a scheduler and a slaver server, wherein the metadata server is used for maintaining tenant information and user information and performing authority authentication when a user accesses metadata and a container; the scheduler comprises two layers of scheduling; the upper-layer scheduling is container-layer scheduling; the lower-layer scheduling is task-layer scheduling; the upper-layer scheduling is used for receiving a request of the user and assigning and configuring the corresponding container for the user; the lower-layer scheduling is completed by a master node of a big data computing framework; the scheduler implements scheduling of the container; and the slave server controls start and stop of the container on a physical machine, monitors resources of the physical machine and feeds back the resources and a task operation condition to the scheduler. According to the system, a virtual container technology is adopted to isolate the resources from the environment, adopts a double-layer scheduling architecture to reduce a load of an upper-layer scheduler and a coupling degree between a distributed application and the scheduler, and provides support for real-time scheduling on a storage system.

Description

technical field [0001] The invention relates to the technical field of storage and big data, in particular to a virtual container-based big data storage and management system and method. Background technique [0002] YARN is an important component in the second generation of Hadoop, responsible for the management of computing resources and task allocation in the cluster. By separating the resource management function into a component, the coupling degree of the system is reduced, and an interface is provided to support more computing frameworks. Currently, it can support mainstream computing frameworks such as MPI, Spark, and MapReduce. The YARN scheduler adopts a single-layer scheduling architecture, and a single ResourceManager instance is responsible for scheduling and resource allocation for all tasks in the cluster. After the ApplicationMaster corresponding to each framework receives the characters submitted by the user, it applies to the ResourceManager for resources ...

AI Technical Summary

Problems solved by technology

The two-tier scheduling of Mesos reduces the load of the scheduler and can support the deployment of multiple storage systems. However, the problem with Mesos is that the access of distributed applications still needs to write special modules, and it cannot support the deployment of multiple instances of the same storage system.

[0004] YARN supports the application of the Hadoop ecosystem very well, but the underlying storage relies on HDFS, which cannot meet the needs of diversified storage; Mesos uses a two-tier scheduling architecture to solve the bottleneck problem of the single-point scheduler, but it still cannot support the deployment of the same Store multiple instances of the application

Images