Automatic deployment system for big data processing system Hadoop on cloud platform OpenStack

Abstract

The invention provides an automatic deployment system for a big data processing system Hadoop on a cloud platform OpenStack. The automatic deployment system comprises a service customization module, a Hadoop cluster control module, a coordination control module and an OpenStack platform supporting module. The OpenStack platform supporting module is installed on the OpenStack platform and used for providing the services of virtual machine creation and management and physical resource virtualization. The service customization module is installed on the OpenStack platform supporting module and used for providing a service use interface and obtaining the Hadoop cluster demand information input by a user. The Hadoop cluster control module is called by the service customization module and used for converting the user demand information sent by the service customization module to a specific Hadoop cluster configuration file, generating a specific Hadoop business operating command script, and forwarding a resource management request and job scheduling related commands sent by a resource manager in a Hadoop cluster in the Hadoop business operating process. The coordination control module triggers the coordination control module to create the corresponding Hadoop cluster.

Description

technical field [0001] The invention relates to a system for automatically deploying a big data processing system Hadoop on a cloud platform OpenStack, and is a system for automatically deploying a Hadoop cluster on OpenStack, which solves the problem of Hadoop business failure caused by physical node failure, and belongs to big data and cloud computing. field. Background technique [0002] With the rapid development of the Internet and the Internet of Things industry, the importance of data is increasingly reflected. In the boom of data nuggets, there are problems such as high threshold for big data processing, low utilization of server cluster resources, and difficulty in data sharing. For small and medium-sized enterprises and individuals with sudden data processing needs, they do not have enough funds to purchase dedicated server clusters, and do not have enough time and energy to deeply study technologies related to big data processing; for large enterprises, each depa...

Images