A method, device and system for automatic deployment of a big data management platform CDH
Through automated deployment methods and resource coordination detection technology, the problems of cumbersome human operations and uneven component distribution during CDH deployment are solved, and rapid and accurate deployment and performance optimization are achieved, reducing costs.
Patent Information
- Application Number
- CN202311790364.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-12-25
AI Technical Summary
When deploying the CDH, the existing technology, manual manual operations are cumbersome and prone to errors, resulting in time-consuming and laborious deployment and high cost, and the uneven distribution of components in the cluster affects overall performance.
Using an automated deployment method, the CDH full package files are automatically distributed and deployed on multiple servers through node coordination processors and shell scripts. The resource coordination detection module is used to dynamically detect the node resource situation, forming a normal distribution probability density map, and optimizing the components in Distribution in the cluster to achieve automated deployment and performance optimization.
It realizes the rapid and accurate deployment of the big data management platform CDH, reduces human errors, improves the overall performance of the cluster and the balance of component distribution, and reduces deployment costs.
Smart Images

Figure CN117873497B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data application technology, and in particular to a method, device and system for automatic deployment of a big data management platform CDH. Background Art
[0002] With the development and application of cloud computing and big data, the application demand for hardware server equipment has also grown from a single or small number of units to the current large-scale demand. The traditional deployment method of deploying the CDH (full name: Cloudera's Distribution, including Apache Hadoop) big data platform on multiple hardware servers is usually carried out step by step through manual operations, including configuration modification, copy and paste, command execution, copying data between multiple servers, etc., which is often extremely cumbersome. Once a problem occurs in a certain link, it will affect other subsequent operations, resulting in the failure of the entire platform to be deployed correctly; in addition, the general manual creation of sub-modules of big data application components will cause the sub-modules to be unevenly distributed in the entire cluster, thereby affecting the overall performance of the cluster.
[0003] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention
[0004] The technical problem to be solved by the present invention is that in the application field related to big data, the solution of the Hadoop platform related components is particularly important. Generally, when installing and deploying the Hadoop platform, either an open source platform management tool or a CDH management platform is used. The CDH management platform is superior to the open source big data management platform in terms of both function and deployment. When deploying CDH, manual deployment step by step is often time-consuming and labor-intensive, prone to errors, and has high costs.
[0005] The present invention adopts the following technical solution:
[0006] In a first aspect, the present invention provides a method for automatic deployment of a big data management platform CDH, comprising:
[0007] Select one server from multiple servers as nodes as the master server, copy the node coordination program and CDH full deployment package files to the master server, run the node coordination program, and specify the IP addresses and node login information of other nodes during the run;
[0008] After the node coordination program runs, the CDH full deployment package file is distributed to the servers of other nodes according to the IP and node login information; the node coordination program distributes tasks to each node through shell remote control commands; wherein, the CDH full deployment package file includes shell scripts and big data related components;
[0009] The node coordination processing program starts to issue the CDH full deployment package file. During the issuance process, the executable attribute of the shell script in the CDH full deployment package file is modified to notify each node to start executing the shell script; wherein, the executable attribute includes the intranet IP of all nodes in the entire cluster; after the server of each node starts executing the shell script, it is in the console waiting operation;
[0010] The node coordination handler will automatically deploy ClouderaManagerServer on the master server through a shell script. The master server, as the management node of CDH, will wait for the management node ClouderaManagerServer to be successfully deployed, and then notify the shell scripts on other nodes to start executing their own ClouderaManagerAgent deployment. The script will configure the automatic restart of Cloudera ManagerServer in the master server and Cloudera ManagerAgent in the servers of other nodes.
[0011] After completing the deployment of Cloudera ManagerAgent, the server of each node deploys the big data related components in the CDH full deployment package file; wherein, the components include one or more of the Hdfs component, Hive component, Hbase component, Yarn component, Zookeeper component, Sqoop component, Hue component and Oozie component.
[0012] Preferably, after the server of each node completes the deployment of Cloudera ManagerAgent and before deploying big data related components, the method further includes:
[0013] The shell script in the server of each node returns the time consumption, resource proportion, storage area read and write performance, network bandwidth and network latency of deploying the ClouderaManagerAgent on its own server to the master server;
[0014] The master server clusters the components according to data throughput, access immediacy, network stability, and concurrent response classes, and sends instructions to start the matching quantity and type of components to the Cloudera ManagerAgent deployed on each node via the Cloudera ManagerServer based on the time consumed in deploying the Cloudera ManagerAgent on each node, resource usage, storage area read and write performance, network bandwidth, and network latency, as well as the number of components to be supported by the CDH currently to be established.
[0015] Preferably, the method comprises:
[0016] When allocating Hue components and Oozie components on each node, the time consumption of the ClouderaManagerAgent, the network bandwidth and the network delay are used as the main weights;
[0017] When allocating Sqoop components, Zookeeper components, and Yarn components on each node, the time consumption, resource proportion, and storage area read and write performance of the Cloudera ManagerAgent deployment are used as the main weights;
[0018] When allocating Hdfs components and Hive components on each node, the time consumption of the ClouderaManagerAgent, the read and write performance of the storage area, and the network bandwidth are used as the main weights;
[0019] When allocating Hbase components on each node, the time consumption of the Cloudera ManagerAgent, the resource proportion, network bandwidth and network latency are used as main weights.
[0020] Preferably, after completing the deployment of Cloudera ManagerAgent, the server of each node deploys big data related components, specifically including:
[0021] The components are deployed in a distributed manner, and the sub-modules of each component need to be dispersed to each node in the cluster. In order to maximize the performance of the sub-modules of the components and analyze the resource consumption of each sub-module, the agent component SurveyAgent tool of the resource detection algorithm of the resource coordination detection module integrates analysis instructions, and forms a probability density map of the normal distribution by simulating the peak period, off-peak period, and normal period environment. By comparing the probability density map of the sub-modules of the components on each node resource utilization, the node with the largest weight is calculated, and finally the most reasonable node environment for the module is determined to complete the deployment of the module.
[0022] Preferably, since the submodules of each component have different characteristics of CPU intensiveness, network IO intensiveness, and disk IO intensiveness, the method specifically includes:
[0023] According to the characteristics of each submodule of each component, use the SurveyAgent tool to preset the simulation environment of peak period, off-peak period and normal period for each submodule;
[0024] Traverse each submodule that needs to be deployed and obtain the node with the largest weight for each submodule, so as to complete the optimal deployment of each submodule.
[0025] Preferably, after the deployment of each component is completed, the offline business data development of the big data platform depends on Hive, and the data of Hive depends on HDFS. In the case where double partitions need to be established for Hive, the number of subdirectories under the HDFS directory will increase. Hive traverses and authenticates all subdirectories and grandchild directories by default. The method also includes:
[0026] The hive source code class RangerHiveAuthorizer is modified to only adapt to the authenticated parent directory, allowing access to all descendant subdirectories of the parent directory, thereby reducing directory traversal and improving hive performance, ensuring the optimal performance of the entire cluster.
[0027] Preferably, the pre-operation for cluster deployment includes:
[0028] The shell scripts in each node automatically modify the hosts name configuration of each node, firewall configuration, ssh login-free configuration, remote transmission of key files in each node, ntp time synchronization configuration, Linux kernel related parameter modification and configuration, jdk configuration, mysql configuration deployment, and set the software to start automatically after the relevant deployment is completed.
[0029] In the second aspect, the present invention also provides a CDH automated deployment system for a big data management platform, including a main server and multiple node servers, wherein one server is selected from multiple servers serving as nodes as the main server, and the node coordination processing program and the CDH full deployment package file are copied to the main server, specifically:
[0030] The node coordination program runs on the master server, and the IP addresses and node login information of other nodes are specified during the operation; after the node coordination program runs, the CDH full deployment package file is distributed to the servers of other nodes according to the IP address and node login information; the node coordination program distributes tasks to each node through shell remote control commands; wherein, the CDH full deployment package file includes shell scripts and big data related components; the node coordination program starts to issue the CDH full deployment package file, and during the issuance process, the executable attribute of the shell script in the CDH full deployment package file is modified to notify each node to start executing the shell script; wherein, the executable attribute includes the intranet IP address of all nodes in the entire cluster; after the shell script is executed in the server of each node, the operation in the console waiting state is performed; the node coordination program will automatically deploy Cloudera ManagerServer on the master server through the shell script. As the management node of CDH, the master server will wait for the management node Cloudera ManagerServer to be successfully deployed, and then notify the shell scripts on other nodes to start executing the deployment of their respective Cloudera ManagerAgents. The script will configure the automatic restart of ClouderaManagerServer in the master server and Cloudera ManagerAgent in the servers of other nodes;
[0031] After completing the deployment of Cloudera ManagerAgent, the servers of other nodes deploy the big data related components in the CDH full deployment package file; wherein, the components include one or more of the Hdfs component, Hive component, Hbase component, Yarn component, Zookeeper component, Sqoop component, Hue component and Oozie component.
[0032] In a third aspect, the present invention further provides a big data management platform CDH automatic deployment device, which is used to implement the big data management platform CDH automatic deployment method described in the first aspect, and the device includes:
[0033] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the CDH automated deployment method for the big data management platform described in the first aspect.
[0034] In a fourth aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the automatic deployment method of the big data management platform CDH described in the first aspect.
[0035] The present invention proposes an automated deployment method. The method uses resource coordination detection and Shell scripts. The Shell script completes a series of tedious and error-prone manual operations. The resource coordination detection dynamically detects and simulates the utilization of resources during peak, off-peak and normal periods of each node to form a probability density map of a normal distribution. By comparing the probability density map of each node, the node with the largest weight is calculated, thereby determining on which node in the cluster a certain submodule should be deployed. Finally, the submodules of all components are reasonably distributed on different nodes, thereby optimizing the overall performance of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0037] Figure 1 This is a flow chart of a method for automatic deployment of a big data management platform CDH provided by an embodiment of the present invention;
[0038] Figure 2 This is a flow chart of a method for automatic deployment of a big data management platform CDH provided by an embodiment of the present invention;
[0039] Figure 3 This is a flow chart of a method for automatic deployment of a big data management platform CDH provided by an embodiment of the present invention;
[0040] Figure 4 This is a flow chart of another method for automatic deployment of a big data management platform CDH provided by an embodiment of the present invention;
[0041] Figure 5 This is a schematic diagram of the architecture of a CDH automated deployment system for a big data management platform provided by an embodiment of the present invention;
[0042] Figure 6 It is a structural schematic diagram of a CDH automatic deployment device for a big data management platform provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0044] In the description of the present invention, the terms "inside", "outside", "longitudinal", "lateral", "upper", "lower", "top", "bottom" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on the present invention.
[0045] The terms "first", "second", etc. in the present invention are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0046] In this application, unless otherwise specified or limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. In addition, the term "coupling" can be a way of achieving electrical connection for signal transmission.
[0047] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0048] Embodiment 1:
[0049] Embodiment 1 of the present invention provides a method for automatically deploying a big data management platform CDH, such as Figure 1 As shown, including:
[0050] In step 201, a server is selected as the main server from multiple servers serving as nodes, the node coordination processing program and the CDH full deployment package file are copied to the main server, the node coordination processing program is run, and the IP addresses and node login information of other nodes are specified during the run.
[0051] In step 202, after the node coordination processing program runs, the CDH full deployment package file is distributed to the servers of other nodes according to the IP and node login information; the node coordination processing program distributes tasks to each node through shell remote control commands; wherein, the CDH full deployment package file includes shell scripts and big data related components.
[0052] In step 203, the node coordination processing program starts to send the CDH full deployment package file. During the sending process, the executable attribute of the shell script in the CDH full deployment package file is modified to notify each node to start executing the shell script; wherein, the executable attribute includes the intranet IP address of all nodes in the entire cluster; after the server of each node starts executing the shell script, it is in the console waiting operation.
[0053] In step 204, the node coordination handler automatically deploys Cloudera ManagerServer on the master server through a shell script. The master server, as the management node of CDH, waits for the management node ClouderaManagerServer to be successfully deployed, and then notifies the shell scripts on other nodes to start executing their respective ClouderaManagerAgent deployments. The script configures the automatic restart of Cloudera ManagerServer in the master server and Cloudera ManagerAgent in the servers of other nodes.
[0054] In step 205, after completing the deployment of Cloudera ManagerAgent, the server of each node deploys the big data related components in the CDH full deployment package file; wherein the components include one or more of the Hdfs component, Hive component, Hbase component, Yarn component, Zookeeper component, Sqoop component, Hue component and Oozie component.
[0055] The embodiment of the present invention proposes an automated deployment method. The method uses resource coordination detection and Shell scripts. The Shell script completes a series of tedious and error-prone manual operations. The resource coordination detection dynamically detects and simulates the resource utilization of each node during peak, off-peak and normal periods to form a probability density map of a normal distribution. By comparing the probability density map of each node, the node with the largest weight is calculated to determine which node in the cluster a submodule should be deployed on. Finally, the submodules of all components are reasonably distributed on different nodes, so that the overall performance of the cluster is optimized.
[0056] In conjunction with the embodiment of the present invention, in step 205, after the server of each node completes the deployment of Cloudera ManagerAgent, before the deployment of big data related components, Figure 2 As shown, the method also includes:
[0057] In step 2051, the shell script in the server of each node returns the time consumption, resource usage, storage area read and write performance, network bandwidth and network latency of deploying the Cloudera ManagerAgent on its own server to the master server.
[0058] In step 2052, the master server clusters the components according to data throughput class, access immediacy class, network stability class, and concurrent response class, and based on the time consumed by the server of each node in deploying the Cloudera ManagerAgent, resource usage, storage area read and write performance, network bandwidth, and network latency, as well as the number of components to be supported by the CDH currently to be established, sends instructions to the ClouderaManagerAgent deployed on each node via the Cloudera ManagerServer to start the matching number and type of components.
[0059] In the implementation process of the embodiment of the present invention, for the allocation of each component involved in step 205, the method includes:
[0060] When allocating Hue components and Oozie components on each node, the time consumption of the ClouderaManagerAgent, the network bandwidth and the network delay are used as the main weights.
[0061] Hue is an open source Apache Hadoop UI system, which was originally evolved from Cloudera Desktop and contributed by Cloudera to the open source community. It is implemented based on the Python web framework Django. By using Hue, we can interact with the Hadoop cluster on the browser-side web console to analyze and process data, such as operating data on HDFS, running MapReduce Jobs, etc. The functional feature set supported by Hue: By default, it manages session data, user authentication and authorization based on a lightweight sqlite database, which can be customized to MySQL, Postgresql, and Oracle; it accesses HDFS based on a file browser; it develops and runs Hive queries based on a Hive editor; it supports applications based on Solr for search, and provides a visual data view and a dashboard; it supports interactive queries for applications based on Impala; it supports Spark editors and dashboards; it supports the Pig editor and can submit script tasks; it supports the Oozie editor and can submit and monitor Workflow, Coordinator, and Bundle through the dashboard; it supports the HBase browser and can visualize data, query data, and modify HBase tables; it supports the Metastore browser and can access Hive metadata and HCatalog; it supports the Job browser and can access MapReduce Job (MR1 / MR2-YARN); it supports the Job designer and can create MapReduce / Streaming / Java Job; it supports Sqoop 2 Editor and Dashboard; Support ZooKeeper browser and editor; Support MySql, PostGresql, Sqlite and Oracle database query editors.
[0062] Oozie is an open source framework based on workflow engine, which was contributed by Cloudera to Apache. It can provide task scheduling and coordination for Hadoop MapReduce and Pig Jobs. Oozie needs to be deployed in Java Servlet container to run. Oozie workflow definition, like jPDL provided by JBoss jBPM, also provides a similar process definition language hPDL, which implements process definition through XML file format. For workflow system, there are generally many nodes with different functions, such as branching, concurrency, convergence, etc. Oozie also has some similar concepts, which will not be explained in detail. For more information, please refer to the relevant documents. Oozie defines control flow nodes and action nodes. Control flow nodes define the start and end of the process, as well as the execution path of the control process, such as decision, fork, join, etc. Action nodes include Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, e-mail and Oozie sub-process.
[0063] When allocating Sqoop components, Zookeeper components, and Yarn components on each node, the time consumption, resource proportion, and storage area read and write performance of the Cloudera ManagerAgent deployment are used as the main weights.
[0064] Sqoop is a tool for transferring data between Hadoop and relational databases. It can import data from a relational database (such as MySQL, Oracle, Postgres, etc.) into Hadoop's HDFS, and also import data from HDFS into a relational database. Data is imported from a relational database into HDFS through Hadoop's MapReduce. Sqoop has about 13 commands and several common parameters.
[0065] Yarn is a new JavaScript package management tool developed by Facebook, Google, Exponent and Tilde.
[0066] Zookeeper is an open source implementation of Google's Chubby. It is a highly efficient and reliable collaborative work system. Zookeeper can be used for leader election, configuration information maintenance, etc. In a distributed environment, a Master instance is required or some configuration information is stored to ensure the consistency of file writing, etc. ZooKeeper is a distributed, open source distributed application coordination service that contains a simple set of primitives and is an important component of Hadoop and Hbase. Java and C interfaces are provided.
[0067] When allocating Hdfs components and Hive components on each node, the time consumption of the ClouderaManagerAgent, the read and write performance of the storage area, and the network bandwidth are used as the main weights;
[0068] Hadoop Distributed File System (HDFS) is designed to be a distributed file system suitable for running on commodity hardware. It has a lot in common with existing distributed file systems. But at the same time, it is also very different from other distributed file systems. HDFS is a highly fault-tolerant system suitable for deployment on cheap machines. HDFS can provide high-throughput data access and is very suitable for applications on large-scale data sets. HDFS relaxes some POSIX constraints to achieve the purpose of streaming file system data. HDFS was originally developed as the infrastructure of the Apache Nutch search engine project. HDFS is part of the Apache Hadoop Core project.
[0069] Hive plays the role of a data warehouse in Hadoop. Hive superimposes structure on data in HDFS and allows data queries to be performed using SQL-like syntax. Like Pig, Hive's core functionality is extensible.
[0070] When allocating Hbase components on each node, the time consumption of the Cloudera ManagerAgent, the resource proportion, network bandwidth and network latency are used as main weights.
[0071] HBase is a distributed, column-oriented open source database. The technology is derived from the Google paper "Bigtable: A Distributed Storage System for Structured Data" written by Fay Chang. Just as Bigtable uses the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Hadoop. HBase is a subproject of the Apache Hadoop project. HBase is different from general relational databases. It is a database suitable for unstructured data storage. Another difference is that HBase is based on columns rather than rows. HBase is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, large-scale structured storage clusters can be built on cheap PC servers.
[0072] In combination with the embodiment of the present invention, after completing the deployment of Cloudera ManagerAgent, the server of each node deploys big data related components, specifically including:
[0073] The components are deployed in a distributed manner, and the sub-modules of each component need to be dispersed to each node in the cluster. In order to maximize the performance of the sub-modules of the components and analyze the resource consumption of each sub-module, the agent component SurveyAgent tool of the resource detection algorithm of the resource coordination detection module integrates analysis instructions (including vmstat, iostat, iotop, etc.), and forms a probability density graph of the normal distribution by simulating the peak period, off-peak period, and normal period environment. By comparing the probability density graph of the resource utilization of the sub-modules of the components on each node, the node with the largest weight is calculated, and finally the most reasonable node environment for the module is determined to complete the deployment of the module.
[0074] In combination with the embodiment of the present invention, since the submodules of each component have different characteristics of CPU intensiveness, network IO intensiveness, and disk IO intensiveness, such as Figure 3 As shown, the method specifically includes:
[0075] In step 301, according to the characteristics of the submodules of each component, the SurveyAgent tool is used to preset the simulation environment of the peak period, the off-peak period and the normal period for each submodule.
[0076] In step 302, each submodule to be deployed is traversed to obtain the node with the largest weight for each submodule, thereby completing the optimal deployment of each submodule.
[0077] In combination with the embodiment of the present invention, after the deployment of each component is completed, the offline business data development of the big data platform depends on Hive, and the data of Hive depends on HDFS. In the case where double partitions need to be established for Hive, the number of subdirectories under the HDFS directory will increase. Hive traverses and authenticates all subdirectories and grandchild directories by default, which is inefficient and wastes system resources. Therefore, in combination with the embodiment of the present invention, there is also a preferred implementation method, including:
[0078] The hive source code class RangerHiveAuthorizer is modified to only adapt to the authenticated parent directory, allowing access to all descendant subdirectories of the parent directory, thereby reducing directory traversal and improving hive performance, ensuring the optimal performance of the entire cluster.
[0079] In conjunction with the embodiment of the present invention, the pre-operation of cluster deployment includes:
[0080] The shell scripts in each node automatically modify the hosts name configuration of each node, firewall configuration, ssh login-free configuration, remote transmission of key files in each node, ntp time synchronization configuration, Linux kernel related parameter modification and configuration, jdk configuration, mysql configuration deployment, and set the software to start automatically after the relevant deployment is completed.
[0081] Embodiment 2:
[0082] The embodiment of the present invention is based on the CDH automatic deployment method of the big data management platform under the common invention conception of embodiment 1, and the difference is only that the embodiment of the present invention tends to highlight the solution content from the pre-configuration side, while embodiment 1 is more written from the perspective of component distribution; therefore, the contents of the two are essentially mutually supportive and referenced. Figure 4 As shown, including:
[0083] In step 401, on all node servers in the CDH cluster, run a shell script, which modifies the network card configuration and host name of each node server; turn off the firewall configuration on each node, and only after turning off the firewall configuration can the password-free remote login operation of each node be performed, configure the ntp server time synchronization service, use the master node as the synchronization server, and all other nodes synchronize time with the master node, modify the time zone operation, and ensure that the time zone and time of each node are the same;
[0084] In step 402, after the modification in step 01 is completed, according to the operation in step 401, the login-free configuration is started. First, a key file is generated on the master node, and then the coordination program remotely distributes the key file to the specified path on other nodes. After waiting is completed, the coordination program verifies the feasibility of remote password-free login of each node. If remote login is not possible, the above operation is repeated until the remote login is completed;
[0085] In step 403, after the above step 402 is executed, the shell script controls the modification and configuration of the Linux kernel related parameters of each node to ensure that the resources of the subsequent components are sufficient when running, and then installs the Java development kit, adds the driver file to the specified path, and performs JDK related configuration to ensure that the JDK is installed correctly;
[0086] In step 404, the shell script will automatically configure, install and deploy the MySQL database on the master node, and set MySQL to start automatically when it boots up;
[0087] In step 405, after the above operation 404 is completed, the coordination program will control the shell script of the main server node to start the installation of the CDH management node Server, including the decompression of the CDH Server parcel package, the movement of the dependent driver to the specified path, the creation of the mysql internal user, and then the Server service can be started, and then the server is configured to start automatically at startup; the coordination program distributes the execution command of the Agent, and other nodes will deploy the Agent service, and then start the Agent, and finally configure the Agent server to start automatically at startup;
[0088] In step 406, after waiting for all the above steps 405 to be executed, that is, after the entire CDH management platform is deployed through the Shell script, the agent component SurveyAgent tool of the detection algorithm in the detection module program will simulate the resource usage of the cluster environment during peak, off-peak and normal periods to form a probability density graph of the normal distribution, and by comparing the probability density graph of the resource utilization of each node of a certain component's sub-module, calculate the node with the largest weight, and finally determine the most reasonable node environment for the module to complete the installation of the module.
[0089] In step 407, according to the characteristics of the sub-modules of each component (CPU intensive / network IO intensive / disk IO intensive), the SurveyAgent tool is used to preset different simulation environments for each sub-module, and each sub-module that needs to be deployed is traversed to obtain the node with the largest weight for each sub-module, thereby completing the optimal deployment of each sub-module.
[0090] In step 408, for some specific application scenarios, the source code interface of some components (such as Hive) is modified, the authentication logic of the RangerHiveAuthorizer class is modified, and Hive is recompiled, packaged and released after the modification. When adding double partitions, which leads to an increase in sub-directories and grand-sub-directories, the number of directory authentications can be significantly reduced, thereby ensuring the execution and computing efficiency of Hive, and ultimately ensuring that the performance of the entire cluster is optimized.
[0091] The embodiment of the present invention proposes an automated deployment method. The method uses resource coordination detection and Shell scripts. The Shell script completes a series of tedious and error-prone manual operations. The resource coordination detection dynamically detects and simulates the resource utilization of each node during peak, off-peak and normal periods to form a probability density map of a normal distribution. By comparing the probability density map of each node, the node with the largest weight is calculated to determine which node in the cluster a submodule should be deployed on. Finally, the submodules of all components are reasonably distributed on different nodes, so that the overall performance of the cluster is optimized.
[0092] Embodiment 3:
[0093] The embodiment of the present invention provides a big data management platform CDH automatic deployment system, such as Figure 5 As shown, it includes a main server and multiple node servers (node server 1, node server 2, ..., node server n, where n is a natural number greater than or equal to 2). One server is selected from the multiple servers as nodes as the main server, and the node coordination processing program and CDH full deployment package files are copied to the main server. Specifically:
[0094] The node coordination program runs on the master server, and the IP addresses and node login information of other nodes are specified during the operation; after the node coordination program runs, the CDH full deployment package file is distributed to the servers of other nodes according to the IP address and node login information; the node coordination program distributes tasks to each node through shell remote control commands; wherein, the CDH full deployment package file includes shell scripts and big data related components; the node coordination program starts to issue the CDH full deployment package file, and during the issuance process, the executable attribute of the shell script in the CDH full deployment package file is modified to notify each node to start executing the shell script; wherein, the executable attribute includes the intranet IP address of all nodes in the entire cluster; after the shell script is executed in the server of each node, the operation in the console waiting state is performed; the node coordination program will automatically deploy Cloudera ManagerServer on the master server through the shell script. As the management node of CDH, the master server will wait for the management node Cloudera ManagerServer to be successfully deployed, and then notify the shell scripts on other nodes to start executing the deployment of their respective Cloudera ManagerAgents. The script will configure the automatic restart of ClouderaManagerServer in the master server and Cloudera ManagerAgent in the servers of other nodes;
[0095] After completing the deployment of Cloudera ManagerAgent, the servers of other nodes deploy the big data related components in the CDH full deployment package file; wherein, the components include one or more of the Hdfs component, Hive component, Hbase component, Yarn component, Zookeeper component, Sqoop component, Hue component and Oozie component.
[0096] The embodiment of the present invention proposes an automated deployment method. The method uses resource coordination detection and Shell scripts. The Shell script completes a series of tedious and error-prone manual operations. The resource coordination detection dynamically detects and simulates the resource utilization of each node during peak, off-peak and normal periods to form a probability density map of a normal distribution. By comparing the probability density map of each node, the node with the largest weight is calculated to determine which node in the cluster a submodule should be deployed on. Finally, the submodules of all components are reasonably distributed on different nodes, so that the overall performance of the cluster is optimized.
[0097] The present invention is a system-side embodiment of the common invention concept of Embodiment 1. Therefore, the extended description of the relevant technical content in Embodiment 1 is also applicable to the embodiment of the present invention. The relevant content will not be repeated here.
[0098] Embodiment 4:
[0099] like Figure 6 , is a schematic diagram of the architecture of the CDH automatic deployment device for a big data management platform according to an embodiment of the present invention. The CDH automatic deployment device for a big data management platform according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.
[0100] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.
[0101] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the big data management platform CDH automated deployment method in Example 1. The processor 21 executes the big data management platform CDH automated deployment method by running the non-volatile software programs and instructions stored in the memory 22.
[0102] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0103] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the automatic deployment method of the big data management platform CDH in the above embodiment 1 is executed, for example, the above described Figure 1-Figure 4 The steps shown.
[0104] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.
[0105] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for automatic deployment of a big data management platform CDH, characterized in that: include: Select one server from multiple servers as nodes as the master server, copy the node coordination program and CDH full deployment package files to the master server, run the node coordination program, and specify the IP addresses and node login information of other nodes during the run; After the node coordination program runs, the CDH full deployment package file is distributed to the servers of other nodes according to the IP and node login information; the node coordination program distributes tasks to each node through shell remote control commands; wherein, the CDH full deployment package file includes shell scripts and big data related components; The node coordination processing program starts to issue the CDH full deployment package file. During the issuance process, the executable attribute of the shell script in the CDH full deployment package file is modified to notify each node to start executing the shell script; wherein, the executable attribute includes the intranet IP of all nodes in the entire cluster; after the server of each node starts executing the shell script, it is in the console waiting operation; The node coordination handler will automatically deploy Cloudera ManagerServer on the master server through a shell script. The master server, as the management node of CDH, will wait for the management node Cloudera ManagerServer to be successfully deployed, and then notify the shell scripts on other nodes to start executing their respective Cloudera ManagerAgent deployments. The script will configure the automatic restart of Cloudera ManagerServer on the master server and Cloudera ManagerAgent on the servers of other nodes. After completing the deployment of Cloudera ManagerAgent, the server of each node deploys the big data related components in the CDH full deployment package file; wherein, the components include one or more of the Hdfs component, Hive component, Hbase component, Yarn component, Zookeeper component, Sqoop component, Hue component and Oozie component; After the Cloudera ManagerAgent is deployed, the servers of each node deploy big data related components, including: The components are deployed in a distributed manner, and the sub-modules of each component need to be dispersed to each node in the cluster. In order to maximize the performance of the sub-modules of the components and analyze the resource consumption of each sub-module, the agent component SurveyAgent tool of the resource detection algorithm of the resource coordination detection module integrates analysis instructions, and forms a probability density map of the normal distribution by simulating the peak period, off-peak period, and normal period environment. By comparing the probability density map of the sub-modules of the components on each node resource utilization, the node with the largest weight is calculated, and finally the most reasonable node environment for the module is determined to complete the deployment of the module.
2. The CDH automated deployment method for a big data management platform according to claim 1, characterized in that: After the server of each node completes the deployment of Cloudera ManagerAgent and before deploying the big data related components, the method further includes: The shell script in the server of each node returns the time consumption, resource proportion, storage area read and write performance, network bandwidth and network latency of deploying the Cloudera ManagerAgent on its own server to the master server; The master server clusters the components according to data throughput, access immediacy, network stability, and concurrent response classes, and sends instructions to start the matching quantity and type of components to the Cloudera ManagerAgent deployed on each node via the Cloudera ManagerServer based on the time consumed in deploying the Cloudera ManagerAgent on each node, resource usage, storage area read and write performance, network bandwidth, and network latency, as well as the number of components to be supported by the CDH currently to be established.
3. The CDH automated deployment method for a big data management platform according to claim 2, characterized in that: The method comprises: When allocating Hue components and Oozie components on each node, the time consumption of the Cloudera ManagerAgent, the network bandwidth and the network delay are used as weights; When allocating Sqoop components, Zookeeper components, and Yarn components on each node, the time consumption, resource proportion, and storage area read and write performance of the Cloudera ManagerAgent deployment are used as weights; When allocating Hdfs components and Hive components on each node, the time consumption of the Cloudera ManagerAgent, the storage area read and write performance, and the network bandwidth are used as weights; When allocating Hbase components on each node, the time consumption of the Cloudera ManagerAgent, the resource proportion, network bandwidth and network latency are used as weights.
4. The CDH automated deployment method for a big data management platform according to claim 1, characterized in that: Since the submodules of each component have different characteristics in terms of CPU intensiveness, network IO intensiveness, and disk IO intensiveness, the specific methods include: According to the characteristics of each submodule of each component, use the SurveyAgent tool to preset the simulation environment of peak period, off-peak period and normal period for each submodule; Traverse each submodule that needs to be deployed and obtain the node with the largest weight for each submodule, so as to complete the optimal deployment of each submodule.
5. The method for automatic deployment of a big data management platform CDH according to any one of claims 1 to 4, characterized in that: After the deployment of various components is completed, the offline business data development of the big data platform depends on Hive, and the data of Hive depends on HDFS. If dual partitions need to be established for Hive, the number of subdirectories under the HDFS directory will increase. By default, Hive traverses and authenticates all subdirectories and grandchild directories. The method also includes: The hive source code class RangerHiveAuthorizer is modified to only adapt to the authenticated parent directory, allowing access to all descendant subdirectories of the parent directory, thereby reducing directory traversal and improving hive performance, ensuring the optimal performance of the entire cluster.
6. The method for automatic deployment of a big data management platform CDH according to any one of claims 1 to 4, characterized in that: Pre-operations for cluster deployment include: The shell scripts in each node automatically perform host name configuration modification, firewall configuration, ssh login-free configuration, remote transmission of key files in each node, ntp time synchronization configuration, Linux kernel related parameter modification and configuration, jdk configuration and mysql configuration deployment, and set the software to start automatically after the relevant deployment is completed.
7. A big data management platform CDH automatic deployment system, characterized in that: It includes a master server and multiple node servers. Select one server as the master server from the multiple node servers, and copy the node coordination program and CDH full deployment package files to the master server. Specifically: The node coordination program runs on the master server, and the IP addresses and node login information of other nodes are specified during the operation; after the node coordination program runs, the CDH full deployment package file is distributed to the servers of other nodes according to the IP address and node login information; the node coordination program distributes tasks to each node through shell remote control commands; wherein, the CDH full deployment package file includes shell scripts and big data related components; the node coordination program starts to issue the CDH full deployment package file, and during the issuance process, the executable attribute of the shell script in the CDH full deployment package file is modified to notify each node to start executing the shell script; wherein, the executable attribute includes the intranet IP address of all nodes in the entire cluster; after the server of each node starts executing the shell script, it is in the console waiting operation; the node coordination program will automatically deploy Cloudera ManagerServer on the master server through the shell script. The master server, as the management node of CDH, waits for the management node Cloudera ManagerServer to be successfully deployed, and then notifies the shell scripts on other nodes to start executing the deployment of their respective Cloudera ManagerAgents. The script will configure the automatic restart of ClouderaManagerServer in the master server and Cloudera ManagerAgent in the servers of other nodes; After completing the deployment of Cloudera ManagerAgent, the servers of other nodes deploy the big data related components in the CDH full deployment package file; wherein, the components include one or more of the Hdfs component, Hive component, Hbase component, Yarn component, Zookeeper component, Sqoop component, Hue component and Oozie component; After the Cloudera ManagerAgent is deployed, the servers of each node deploy big data related components, including: The components are deployed in a distributed manner, and the sub-modules of each component need to be dispersed to each node in the cluster. In order to maximize the performance of the sub-modules of the components and analyze the resource consumption of each sub-module, the agent component SurveyAgent tool of the resource detection algorithm of the resource coordination detection module integrates analysis instructions, and forms a probability density map of the normal distribution by simulating the peak period, off-peak period, and normal period environment. By comparing the probability density map of the sub-modules of the components on each node resource utilization, the node with the largest weight is calculated, and finally the most reasonable node environment for the module is determined to complete the deployment of the module.
8. A CDH automated deployment device for a big data management platform, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the CDH automated deployment method for the big data management platform as described in any one of claims 1-6.
9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the big data management platform CDH automatic deployment method described in any one of claims 1-6.
