Customized management method and device for big data component, equipment and medium

Abstract

The embodiment of the invention discloses a customized management method and device for a big data component, equipment and a medium. The method comprises the following steps: determining a target architecture based on a target demand; establishing a target stack, and editing a stack definition file of the target stack based on the target architecture; editing the management information of each big data component based on the stack definition file to obtain self-defined management information of each big data component; and developing component installation scripts matched with the big data components according to the custom management information, and installing the big data components based on the component installation scripts. By executing the method, the types of the big data components can be flexibly selected, a customized management method of the big data component can be realized, each big data component can be quickly deployed, and then different types of big data business requirements can be met.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of big data, and in particular to a custom management method, device, equipment and medium of big data components. Background technique [0002] Big data applications are inseparable from Hadoop. In a broad sense, Hadoop refers to the entire Hadoop ecosystem (such as Hive, Hbase, Sqoop, Zookeeper, etc.), not just Hadoop. There are two vendors in the market that can manage big data components, namely Cloudera and Hortonworks, but they only support some big data components of the Hadoop3 ecosystem, and cannot support all big data components. For example, Hortonworks only supports HDFS3.x, not Impala3 and Hive3. And Cloudera and Hortonworks can only use their own fixed version of the installation package. [0003] The problems caused by this limitation are: the big data functions realized based on the specific version of the big data components are limited and incomplete; the types o...

AI Technical Summary

Problems solved by technology

There are two vendors in the market that can manage big data components, namely Cloudera and Hortonworks, but they only support some big data components of the Hadoop3 ecosystem, and cannot support all big data components

For example, Hortonworks only supports HDFS3.x, not Impala3 and Hive3

And Cloudera and Hortonworks can only use their own fixed version of the installation package

[0003] The problems caused by this limitation are: the big data functions realized based on the specific version of the big data components are limited and incomplete; the types of business requirements that can be realized based on the big data architecture composed of some big data components are also limited; The management of big data components can only depend on the requirements and regulations of manufacturers, therefore, the management of big data components and the use of big data cannot be flexibly realized

Images