Cluster host management method, device, electronic device and storage medium

Abstract

The present application provides a cluster host management method, device, electronic device and storage medium. The method includes: determining the host to be taken offline; obtaining the technical component type of the host to be taken offline; processing the corresponding technical component according to the technical component type to stop the technical component from working; updating the configuration information of other hosts in the cluster except the host to be taken offline; synchronizing the file data in the host to be taken offline to the other hosts; shutting down the proxy server on the host to be taken offline to achieve offline management of the host to be taken offline. In an embodiment of the present application, the authorization and service of each technical component of the host to be taken offline are first released, and then the file data in the host to be taken offline is synchronized to the other hosts to achieve a smooth migration and transition of the business data of the host to be taken offline.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cluster host management method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of big data technology, the stability requirements for daily operations of big data clusters are increasing. As we all know, the current big data platforms are deployed and implemented in the form of clusters, that is, they are composed of thousands of hosts.

[0003] Elastically scaling big data cluster resources to meet the needs of business systems is crucial. When the business system's workload increases, hosts are added to the cluster, and when the workload decreases, hosts are removed from the cluster. Existing solutions, when elastically scaling a cluster, delete the business data of removed hosts, which impacts the normal use of the business system and prevents smooth migration of the removed hosts' business data. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a cluster host management method, device, electronic device and storage medium to solve the problem in the prior art that the business data of the decommissioned host cannot be smoothly migrated.

[0005] In a first aspect, an embodiment of the present application provides a node determination method, including:

[0006] Determine the host to be taken offline;

[0007] Obtaining the technical component type of the host to be taken offline;

[0008] Processing a corresponding technical component according to the type of the technical component to stop the technical component from working;

[0009] Update the configuration information of all hosts in the cluster except the host to be taken offline;

[0010] Synchronize the file data in the host to be offline to the other hosts;

[0011] Shut down the proxy server on the host to be taken offline to implement offline management of the host to be taken offline.

[0012] In an embodiment of the present application, by identifying and judging the technical component type of the host to be taken offline and processing the corresponding technical components, the files of the host to be taken offline are synchronized to other hosts. The hosts in the cluster that are online can be taken offline, and a smooth migration transition of the business data on the offline host can be achieved without affecting the normal use of the business system.

[0013] Furthermore, the processing of the corresponding technical component according to the technical component type includes:

[0014] If the technical component type includes Hbase database and / or Yarn, then the authorization of the Hbase database and / or Yarn will be released;

[0015] If the technical component type includes the Impala query system and / or the Hive data warehouse tool, shut down the service of the Impala query system and / or the Hive data warehouse tool;

[0016] Obtaining the working status of the business system corresponding to the cluster;

[0017] If the working status of the business system is normal, the authorization of the Hadoop distributed file system as a technical component type is released.

[0018] The embodiment of the present application first performs corresponding processing on different component types, including revoking authorization of the Hbase database and / or Yarn, shutting down services for the Impala query system and / or Hive data warehouse tool, and then determining whether the normal use of the business system is affected. On the basis of not affecting the normal use of the business system, the technical component is revoked for the Hadoop distributed file system, so that the host can be offline online and the processing of the offline host technical components can be guaranteed not to affect the normal use of the business system.

[0019] Furthermore, synchronizing the file data in the host to be taken offline to the other hosts includes:

[0020] Obtaining an identifier corresponding to the data stored in the host to be taken offline;

[0021] Obtaining a corresponding data copy in the other host according to the identifier;

[0022] The data copies are copied and added to other hosts that meet the storage conditions.

[0023] In an embodiment of the present application, through the identifier corresponding to the data stored in the offline host, a copy of the offline host data can be found on other hosts in the cluster, and the corresponding data copy can be copied and added to the host that can store the corresponding data copy, thereby achieving a smooth migration transition of the business data in the offline host.

[0024] Furthermore, before shutting down the proxy server on the host to be taken offline, the method further includes:

[0025] Determine whether the number of copies of the data stored by the other hosts in the cluster exceeds a preset threshold;

[0026] If the number of data copies exceeds the preset threshold, it is determined that the data of the cluster has resumed normal use.

[0027] In an embodiment of the present application, after the business data of the offline host is smoothly migrated, statistics are taken of all copies stored by other hosts in the cluster. By checking whether the copies of all data exceed a preset threshold, it is determined whether all data in the cluster has been restored to normal use, thereby ensuring that all business data stored by the offline host has been restored, so that the business system corresponding to the cluster can operate normally.

[0028] Furthermore, the method further comprises:

[0029] Determine the host to be brought online;

[0030] Configuring parameters of the host to be put online and parameters of the online hosts in the cluster;

[0031] Add the host to be put online in the cluster;

[0032] Install technical components on the host to be put online according to pre-selected requirements.

[0033] In an embodiment of the present application, the configured host to be put online is added to the cluster, and then the service is installed and deployed on the host to be put online according to actual needs. This can realize the online launch of the cluster host, avoiding the shutdown of the entire cluster host when the host is put online, causing the business system to be unable to be used normally.

[0034] Furthermore, configuring the parameters of the host to be put online and the parameters of the online hosts in the cluster includes:

[0035] Formatting the disk of the host to be put online;

[0036] Configure the network information and time synchronization service of the host to be put online;

[0037] Modify the configuration of system files of all hosts in the cluster.

[0038] In the embodiment of the present application, the parameters of the newly online host are initialized and configured, and the system files of the online hosts in the cluster are configured, so that the time of the newly online host and the online hosts in the cluster are synchronized.

[0039] Furthermore, after installing the technical components on the host to be put online according to the pre-selected requirements, the method further includes:

[0040] The data stored on each host in the cluster is balanced.

[0041] The embodiment of the present application improves the resource utilization and work efficiency of the entire cluster by balancing the data stored on each host.

[0042] In a second aspect, an embodiment of the present application provides a cluster host management device, comprising: a first host determination module, configured to determine a host to be taken offline;

[0043] An acquisition module, configured to acquire the technical component type of the host to be delisted;

[0044] a processing module, configured to process a corresponding technical component according to the type of the technical component, so as to stop the technical component from working;

[0045] An update module, configured to update configuration information of other hosts in the cluster except the host to be taken offline;

[0046] A data synchronization module, configured to synchronize file data in the host to be taken offline to the other hosts;

[0047] The host offline determination module is used to shut down the proxy server on the host to be offline, so as to implement offline management of the host to be offline.

[0048] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

[0049] An embodiment of the present application further provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is executed.

[0050] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram of a cluster host management method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a cluster host management method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of another cluster host management method provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the structure of a cluster host management device provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0058] Figure 1 A flow chart of a cluster host management method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:

[0059] Step 101: Determine the host to be taken offline.

[0060] When a cluster's resource utilization falls below a threshold within a business cycle and the corresponding business systems have no further resource requirements, hosts in the cluster are taken offline. The business cycle can be, but is not limited to, a week, a month, or a quarter, and the thresholds are, but are not limited to, 50%, 60%, or 70%.

[0061] During the specific implementation process, the total number of offline resources can be determined based on "the total number of resources required for normal operation of the cluster = the actual number of resources required by the business system / resource utilization rate" and "the total number of offline resources = the total number of resources of the cluster - the total number of resources required for normal operation of the cluster".

[0062] For example, cluster A has 50 online hosts with a total resource of 100 CPUs and a resource utilization threshold of 60%. The actual resource demand of the business system is 30 CPUs, and there will be no new business resource demand within the next month. At this time, the cluster's resource utilization is 30%, which is below the resource utilization threshold of 60%. Therefore, some hosts can be taken offline to reduce resource waste. Calculation shows that the total resources required for normal operation of cluster A are 50 CPUs, so the total number of resources to be taken offline is determined to be 50 CPUs. Multiple hosts in cluster A are selected as hosts to be taken offline, where the total storage resources of the multiple hosts are less than or equal to 50 CPUs, and each host can include multiple CPUs.

[0063] Step 102: Obtain the technical component type of the host to be taken offline.

[0064] In the specific implementation process, technical component types include: Hbase, Yarn, Hive, Impala, HDFS, etc.

[0065] Among them, Hbase is a distributed, column-oriented open source database with the advantages of high reliability, high performance, column orientation and scalability. Using Hbase database technology, a large-scale structured storage cluster can be built on a cheap PC server.

[0066] Impala is an MPP (Massively Parallel Processing) SQL query engine for processing large amounts of data stored in Hadoop clusters. It is open source software written in C++ and Java and provides high performance and low latency compared to other Hadoop SQL engines.

[0067] Yarn is a new Hadoop resource manager. It is a general resource management system that provides unified resource management and scheduling for upper-layer applications. Its introduction has brought huge benefits to the cluster in terms of utilization, unified resource management and data sharing.

[0068] Hive is a data warehouse tool based on Hadoop that extracts, transforms, and loads data. It's a mechanism for storing, querying, and analyzing large amounts of data stored in Hadoop. The Hive data warehouse tool maps structured data files into database tables and provides SQL query functionality, converting SQL statements into MapReduce tasks for execution. It's ideal for statistical analysis of data warehouses.

[0069] HDFS is a distributed file system designed to run on general-purpose hardware. It is highly fault-tolerant and suitable for deployment on inexpensive machines. It provides high-throughput data access and is well-suited for applications with large datasets. HDFS uses a master-slave architecture. An HDFS cluster consists of a NameNode and several DataNodes. The NameNode acts as the master server, managing the file system namespace and client access to files. The DataNodes in the cluster manage stored data.

[0070] Step 103: Process the corresponding technical component according to the technical component type to stop the technical component from working.

[0071] During implementation, each host's corresponding technical components may include one or more of the HBase database, YARN, the Hive data warehouse tool, the Impala query system, and the Hadoop Distributed File System. The host's technical components are handled accordingly, disconnecting them from the cluster and deactivating the data within them.

[0072] Step 104: Update the configuration information of the hosts in the cluster except the host to be taken offline.

[0073] During the specific implementation process, after the technical components of the host to be taken offline are processed, the data of the technical components of the host to be taken offline has been offline, but the configuration data of the offline technical components of the host to be taken offline are stored on other hosts in the cluster. At this time, this part of the expired configuration data is still in a valid state. In order not to affect the normal use of other hosts in the cluster, the configuration information of other hosts in the cluster needs to be updated at this time.

[0074] Step 105: Synchronize the file data in the host to be taken offline to the other hosts.

[0075] Among them, file data is business data stored by the business system in the host to be taken offline in the cluster. Synchronizing the file data in the host to be taken offline to the other hosts means restoring the business data stored in the host to be taken offline on other hosts in the cluster, that is, realizing the migration of the business data stored in the host to be taken offline.

[0076] Step 106: shut down the proxy server on the host to be taken offline to implement offline management of the host to be taken offline.

[0077] During the specific implementation process, the technical components of the host to be taken offline, including the HBase database, Yarn, Hive data warehouse tools, Impala query system, Hadoop distributed file system, etc., are all running on the proxy server. The proxy server can be, but is not limited to, the LINUX system or UINX operating system. Shut down the proxy server on the host to be taken offline to implement offline management of the host to be taken offline.

[0078] Based on the above embodiments, Figure 2 For a schematic diagram of a cluster host management method provided in this application embodiment, please refer to Figure 1 and Figure 2 , the processing of the corresponding technical component according to the technical component type includes:

[0079] If the technical component type includes Hbase database and / or Yarn, then the authorization of the Hbase database and / or Yarn will be released;

[0080] If the technical component type includes the Impala query system and / or the Hive data warehouse tool, shut down the service of the Impala query system and / or the Hive data warehouse tool;

[0081] Obtaining the working status of the business system corresponding to the cluster;

[0082] If the working status of the business system is normal, the authorization of the Hadoop distributed file system as a technical component type is released.

[0083] During implementation, the corresponding technical components may include one or more of the HBase database, Yarn, Hive data warehouse tool, and Impala query system. Authorizations between the HBase database and / or Yarn and the cluster will be revoked, and the Impala query system and / or Hive data warehouse tool services provided to the cluster will be shut down. The operating status of the business systems corresponding to the cluster will be determined to ensure that deauthorizing and shutting down the technical components of the hosts to be deployed does not affect the normal operation of the business systems.

[0084] On the basis of normal use of the business system, revoking the authorization between the technical component type HDFS and the cluster can improve the processing efficiency of the cluster and avoid further offline of the host when the business is affected, resulting in waste of resources and subsequent impact on data recovery.

[0085] Based on the above examples, please refer to Figure 1 and Figure 2 , synchronizing the file data in the host to be offline to the other hosts includes:

[0086] Obtaining an identifier corresponding to the data stored in the host to be taken offline;

[0087] Obtaining a corresponding data copy in the other host according to the identifier;

[0088] The data copies are copied and added to other hosts that meet the storage conditions.

[0089] In an embodiment of the present application, the host in the cluster is a distributed file system. When storing data, a piece of data is divided into multiple data blocks. Each data block generates multiple data copies and is stored on different hosts. The multiple data copies here are at least 3.

[0090] For example, one of the data copies A stored in the host to be taken offline is identified as A1. Based on the identifier A1, the corresponding other data copies are found in other hosts, and any data copy a is selected as the target data copy. The data copy a is copied and added to other storage hosts. Through this method, all data copies stored in the host to be taken offline are restored to achieve synchronization of file data in the host to be taken offline.

[0091] Based on the above examples, please refer to Figure 1 and Figure 2 Before shutting down the proxy server on the host to be taken offline, the method further includes:

[0092] Determine whether the number of copies of the data stored by the other hosts in the cluster exceeds a preset threshold;

[0093] If the number of data copies exceeds the preset threshold, it is determined that the data of the cluster has resumed normal use.

[0094] The preset threshold is that the number of replicas of each data block is greater than or equal to 3. In the embodiment of the present application, due to network bandwidth limitations, the number of operations on hosts in the cluster is large, and the bandwidth occupied is high, which may result in data replicas not being replicated to other hosts capable of storage, resulting in data loss, and the need to check the synchronized data. Statistics are collected on the number of data replicas stored on all hosts in the cluster to determine whether the number of replicas of each data is greater than or equal to 3. If all data replicas meet the requirements, it is determined that the data of the host to be offline has been fully restored.

[0095] Based on the above embodiments, Figure 3 Another cluster host management method provided in this embodiment is shown in the diagram. Figure 3 , the method further comprises:

[0096] Determine the host to be brought online;

[0097] Configuring parameters of the host to be put online and parameters of the online hosts in the cluster;

[0098] Add the host to be put online in the cluster;

[0099] Install technical components on the host to be put online according to pre-selected requirements.

[0100] In the embodiments of the present application, the number of hosts online in the cluster should generally be increased according to actual demand, and the demand of the business system determines the number of resources to be added. At the same time, the host to be online has an initial configuration and operating system, and the parameters of the host to be online need to be configured first to meet the synchronization requirements of the cluster. Because a new host is to be added to the cluster, the parameters of the online hosts in the cluster need to be configured accordingly. Adding a host to be online in the cluster means adding the information of the host to be online in the cluster and giving the cluster the authority to manage the host to be online.

[0101] Host resources can be primarily categorized as compute resources and storage resources. Compute resources include CPU and memory, while storage resources include disks. In practice, to effectively utilize resources and improve host resource utilization, both types of resources are typically utilized. This means that installed technical components include both compute and storage types. Typically, the default installation for a live host includes the Hadoop distributed file system, HBase database, Hive data warehouse tool, and the Yarn and Impala query systems.

[0102] Based on the above examples, please refer to Figure 3 , configuring the parameters of the host to be put online and the parameters of the online hosts in the cluster, including:

[0103] Formatting the disk of the host to be put online;

[0104] Configure the network information and time synchronization service of the host to be put online;

[0105] Modify the configuration of system files of all hosts in the cluster.

[0106] In this embodiment of the present application, to prevent the disk of the host to be online from storing previous data or formatting errors on the disk, the hard disk of the host to be online needs to be initialized. Configuring the network information of the host to be online specifically refers to configuring the IP address of the host to be online and modifying the Hosts file of the host to be online. Modifying the configuration of system files of all hosts in the cluster specifically refers to modifying the Hosts files of the online hosts in the cluster.

[0107] Among them, the Hosts file refers to a plain text file that maps the host name to the IP address, that is, a system file without an extension. Its basic function is to establish an association "database" between some commonly used URL domain names and their corresponding IP addresses.

[0108] Configuring the Network Time Protocol (NTP) time synchronization service for the host to be brought online involves modifying the NTP service configuration file for the host. Because the cluster is a distributed file system with strict time requirements for each host, it is necessary to configure the Hosts file on the host to be brought online to achieve cluster-wide time synchronization. NTP allows computers to synchronize with their servers or clock sources, providing accurate and robust time services in the chaotic Internet environment. It provides highly accurate time correction and utilizes cryptographic authentication to prevent malicious protocol attacks.

[0109] Based on the above examples, please refer to Figure 3 After installing the technical components on the host to be put online according to the pre-selected requirements, the method further includes:

[0110] The data stored on each host in the cluster is balanced.

[0111] Here, a cluster refers to a new cluster after the host to be put online is put online in the cluster. In the embodiment of the present application, after the host to be put online is added to the cluster, it is necessary to balance the data copies stored by all hosts in the cluster to ensure that the amount of data stored by each host is basically within a certain range, that is, the amount of data stored by each host is basically balanced. At the same time, the data stored by each host can be counted to check whether there is any imbalance in data storage.

[0112] Figure 4 This is a schematic diagram of the structure of the cluster host management device provided in the embodiment of the present application. The device can be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 1 The method embodiment corresponds to the embodiment that can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here. The device 200 includes: a first host determination module 201, an acquisition module 202, a processing module 203, an update module 204, a data synchronization module 205, and a host offline determination module 206, wherein:

[0113] The first host determination module 201 is used to determine the host to be taken offline;

[0114] An acquisition module 202 is configured to acquire a technical component type of the host to be delisted;

[0115] The processing module 203 is used to process the corresponding technical component according to the type of the technical component to stop the technical component from working;

[0116] An updating module 204 is configured to update configuration information of other hosts in the cluster except the host to be taken offline;

[0117] A data synchronization module 205 is used to synchronize the file data in the host to be taken offline to the other hosts;

[0118] The host offline determination module 206 is configured to shut down the proxy server on the host to be offline, so as to implement offline management of the host to be offline.

[0119] Based on the above embodiment, the processing module 203 is specifically configured to:

[0120] If the technical component type includes Hbase database and / or Yarn, then the authorization of the Hbase database and / or Yarn will be released;

[0121] If the technical component type includes the Impala query system and / or the Hive data warehouse tool, shut down the service of the Impala query system and / or the Hive data warehouse tool;

[0122] Obtaining the working status of the business system corresponding to the cluster;

[0123] If the working status of the business system is normal, the authorization of the Hadoop distributed file system as a technical component type is released.

[0124] Based on the above embodiment, the data synchronization module 205 is specifically used to:

[0125] Obtaining an identifier corresponding to the data stored in the host to be taken offline;

[0126] Obtaining a corresponding data copy in the other host according to the identifier;

[0127] The data copies are copied and added to other hosts that meet the storage conditions.

[0128] Based on the above embodiment, the device further includes a judgment module, which is used to:

[0129] Determine whether the number of copies of the data stored by the other hosts in the cluster exceeds a preset threshold;

[0130] If the number of data copies exceeds the preset threshold, it is determined that the data of the cluster has resumed normal use.

[0131] Based on the above embodiment, the device further includes a host online module, which is used to:

[0132] Determine the host to be brought online;

[0133] Configuring parameters of the host to be put online and parameters of the online hosts in the cluster;

[0134] Add the host to be put online in the cluster;

[0135] Install technical components on the host to be put online according to pre-selected requirements.

[0136] Based on the above embodiment, the device further includes a configuration module, which is used to:

[0137] Formatting the disk of the host to be put online;

[0138] Configure the network information and time synchronization service of the host to be put online;

[0139] Modify the configuration of system files of all hosts in the cluster.

[0140] Based on the above embodiment, the device further includes a data balancing module, which is used to:

[0141] The data stored on each host in the cluster is balanced.

[0142] To sum up, in the embodiment of the present application, by identifying and judging the type of technical components of the host to be taken offline and processing the corresponding technical components, the files of the host to be taken offline are synchronized to other hosts. The hosts in the cluster can be taken offline while they are online, and a smooth migration transition of the business data on the offline host can be achieved without affecting the normal use of the business system.

[0143] Figure 5 The physical structure diagram of the electronic device provided in the embodiment of the present application is as follows: Figure 5 As shown, the electronic device includes: a processor (processor) 301, a memory (memorY) 302 and a bus 303; wherein:

[0144] The processor 301 and the memory 302 communicate with each other via the bus 303;

[0145] The processor 301 is used to call the program instructions in the memory 302 to execute the methods provided by the above-mentioned method embodiments, for example, including: determining the host to be offline; obtaining the technical component type of the host to be offline; processing the corresponding technical component according to the technical component type to stop the technical component from working; updating the configuration information of other hosts in the cluster except the host to be offline; synchronizing the file data in the host to be offline to the other hosts; and shutting down the proxy server on the host to be offline to implement offline management of the host to be offline.

[0146] The processor 301 can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 301 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0147] The memory 302 may include but is not limited to random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0148] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: determining a host to be taken offline; obtaining the technical component type of the host to be taken offline; processing the corresponding technical component according to the technical component type to stop the technical component from working; updating the configuration information of other hosts in the cluster except the host to be taken offline; synchronizing file data in the host to be taken offline to the other hosts; and shutting down the proxy server on the host to be taken offline to implement offline management of the host to be taken offline.

[0149] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: determining a host to be taken offline; obtaining the technical component type of the host to be taken offline; processing the corresponding technical component according to the technical component type to stop the technical component from working; updating the configuration information of other hosts in the cluster except the host to be taken offline; synchronizing file data in the host to be taken offline to the other hosts; and shutting down the proxy server on the host to be taken offline to implement offline management of the host to be taken offline.

[0150] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0153] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0154] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A cluster host management method, characterized in that: include: Determine the host to be taken offline; Obtaining the technical component type of the host to be taken offline; Processing a corresponding technical component according to the type of the technical component to stop the technical component from working; Update the configuration information of all hosts in the cluster except the host to be taken offline; Synchronize the file data in the host to be offline to the other hosts; Shut down the proxy server on the host to be taken offline to achieve offline management of the host to be taken offline; The processing of the corresponding technical component according to the technical component type includes: If the technical component type includes Hbase database and / or Yarn, then the authorization of the Hbase database and / or Yarn will be released; If the technical component type includes the Impala query system and / or the Hive data warehouse tool, shut down the service of the Impala query system and / or the Hive data warehouse tool; Obtaining the working status of the business system corresponding to the cluster; If the working status of the business system is normal, the authorization of the Hadoop distributed file system as a technical component type is released.

2. The method according to claim 1, characterized in that The step of synchronizing the file data in the host to be taken offline to the other hosts includes: Obtaining an identifier corresponding to the data stored in the host to be taken offline; Obtaining a corresponding data copy in the other host according to the identifier; The data copies are copied and added to other hosts that meet the storage conditions.

3. The method according to claim 2, characterized in that Before shutting down the proxy server on the host to be taken offline, the method further includes: Determine whether the number of copies of the data stored by the other hosts in the cluster exceeds a preset threshold; If the number of data copies exceeds the preset threshold, it is determined that the data of the cluster has resumed normal use.

4. The method according to claim 1, wherein The method further comprises: Determine the host to be brought online; Configuring parameters of the host to be put online and parameters of the online hosts in the cluster; Add the host to be put online in the cluster; Install technical components on the host to be put online according to pre-selected requirements.

5. The method according to claim 4, characterized in that Configuring the parameters of the host to be put online and the parameters of the online hosts in the cluster includes: Formatting the disk of the host to be put online; Configure the network information and time synchronization service of the host to be put online; Modify the configuration of system files of all hosts in the cluster.

6. The method according to claim 4, characterized in that After installing the technical components on the host to be put online according to the pre-selected requirements, the method further includes: The data stored on each host in the cluster is balanced.

7. A cluster host management device, characterized in that: include: A first host determination module is used to determine the host to be taken offline; An acquisition module, configured to acquire a technical component type of the host to be taken offline; a processing module, configured to process a corresponding technical component according to the type of the technical component, so as to stop the technical component from working; An update module, configured to update configuration information of other hosts in the cluster except the host to be taken offline; A data synchronization module, configured to synchronize file data in the host to be taken offline to the other hosts; A host offline determination module is used to shut down the proxy server on the host to be offline, so as to implement offline management of the host to be offline; The processing module is specifically used for: If the technical component type includes Hbase database and / or Yarn, then the authorization of the Hbase database and / or Yarn will be released; If the technical component type includes the Impala query system and / or the Hive data warehouse tool, shut down the service of the Impala query system and / or the Hive data warehouse tool; Obtaining the working status of the business system corresponding to the cluster; If the working status of the business system is normal, the authorization of the Hadoop distributed file system as a technical component type is released.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is performed.

9. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the method according to any one of claims 1 to 6.

Images