Automatic deployment system for big data processing system Hadoop on cloud platform OpenStack

Abstract

The invention provides an automatic deployment system for a big data processing system Hadoop on a cloud platform OpenStack. The automatic deployment system comprises a service customization module, a Hadoop cluster control module, a coordination control module and an OpenStack platform supporting module. The OpenStack platform supporting module is installed on the OpenStack platform and used for providing the services of virtual machine creation and management and physical resource virtualization. The service customization module is installed on the OpenStack platform supporting module and used for providing a service use interface and obtaining the Hadoop cluster demand information input by a user. The Hadoop cluster control module is called by the service customization module and used for converting the user demand information sent by the service customization module to a specific Hadoop cluster configuration file, generating a specific Hadoop business operating command script, and forwarding a resource management request and job scheduling related commands sent by a resource manager in a Hadoop cluster in the Hadoop business operating process. The coordination control module triggers the coordination control module to create the corresponding Hadoop cluster.

Description

technical field [0001] The invention relates to a system for automatically deploying a big data processing system Hadoop on a cloud platform OpenStack, and is a system for automatically deploying a Hadoop cluster on OpenStack, which solves the problem of Hadoop business failure caused by physical node failure, and belongs to big data and cloud computing. field. Background technique [0002] With the rapid development of the Internet and the Internet of Things industry, the importance of data is increasingly reflected. In the boom of data nuggets, there are problems such as high threshold for big data processing, low utilization of server cluster resources, and difficulty in data sharing. For small and medium-sized enterprises and individuals with sudden data processing needs, they do not have enough funds to purchase dedicated server clusters, and do not have enough time and energy to deeply study technologies related to big data processing; for large enterprises, each depa...

AI Technical Summary

Problems solved by technology

[0003] For this reason, the present invention provides an automatic deployment system of a big data processing system Hadoop on the cloud platform OpenStack, and designs a corresponding coordination control module to solve the problem of business failure caused by physical node failure; it is simple and easy to use, has a high degree of automation, and is safe Reliable, can effectively improve the utilization rate of physical server resources, reduce the cost of enterprises and the threshold for small and medium-sized enterprises to enter the field of big data processing, strong practicability, and wide application fields

Images