Big data cluster deployment method and device, and service system

Abstract

The present application discloses a big data cluster deployment method and device, and a service system. The method comprises: determining a plurality of target components from among a plurality of components; querying an installation package of the current target component to be deployed from a plurality of pre-stored installation packages; when first-type installation packages of said current target component have been queried, dragging the first-type installation packages of said current target component to the local of a target host to which said current target component is to be deployed, and deploying said current target component to the target host on the basis of the first-type installation packages of said current target component; and when the first-type installation packages of said current target component have not been queried, determining the installation package having the highest compatibility level from among second-type installation packages, dragging the installation package having the highest compatibility level to the local of the target host, and deploying said current target component to the target host on the basis of the installation package having the highest compatibility level.

Description

Big data cluster deployment method, device, and service system

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023113607223, entitled “Deployment method, device and service system for big data clusters”, filed on October 19, 2023, the entire text of which is incorporated herein by reference. Technical Field

[0003] The present application relates to the field of big data technology, and more specifically, to a deployment method and device for a big data cluster, and a service system. Background Art

[0004] Currently, the majority of existing servers in various enterprises use non-domestic architectures and operating systems, while the majority of newly purchased servers use domestic architectures and operating systems. Furthermore, with the development of the information technology application innovation industry, the requirements for the compatibility of components and the hosts on which they are deployed have also increased accordingly. Consequently, there is a need for mixed deployments of different architectures and operating systems. The installation services provided by related big data cluster deployment software are strongly architecture-dependent. Installation on heterogeneous servers often requires downloading the corresponding software version separately or recompiling the entire set of components based on the new architecture. Furthermore, deployment can only be performed within a single-architecture cluster, resulting in the lack of support for mixed deployments of heterogeneous clusters.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a big data cluster deployment method, device, and service system.

[0007] According to one aspect of an embodiment of the present application, a deployment method for a big data cluster is provided, comprising: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-class installation package and a second-class installation package, the first-class installation package being an installation package dedicated to each host among the multiple hosts, and the second-class installation package being a compatibility installation package; in case the first-class installation package of the target component to be currently deployed is queried, the first-class installation package of the target component to be currently deployed is pulled to the local of the target host to which the target component to be currently deployed is to be deployed, and the target component to be currently deployed is deployed on the target host based on the first-class installation package of the target component to be currently deployed; in case the first-class installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level among the second-class installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.

[0008] Optionally, the compatibility level of each installation package in the second category of installation packages is determined by the following method: obtaining the central processing unit architecture and operating system information of each installation package, wherein the operating system information includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs; determining the compatibility level of each installation package in the second category of installation packages based on the central processing unit architecture and operating system information of each installation package.

[0009] Optionally, the compatibility level of each installation package in the second category of installation packages is determined based on the central processing unit architecture and operating system information of each installation package, including: obtaining the central processing unit architecture and operating system information of the target host, the operating system information of the target host including: the version number of the operating system of the target host and the upstream operating system to which the operating system of the target host belongs; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose operating system version number has the smallest difference with the operating system version number of the target host as the first installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose upstream operating system is the same as the upstream operating system of the target host as the second installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose core component of the operating system is the Linux kernel as the third installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host as the fourth installation package; and determining the installation package with the highest compatibility level based on the type of installation packages in the second category of installation packages.

[0010] Optionally, the compatibility levels of the first installation package, the second installation package, the third installation package, and the fourth installation package decrease in sequence.

[0011] Optionally, determining the installation package with the highest compatibility level based on the type of installation packages in the second category of installation packages includes: when the first installation package, the second installation package, the third installation package, and the fourth installation package exist in the second category of installation packages at the same time, determining the first installation package as the installation package with the highest compatibility level; when the second installation package, the third installation package, and the fourth installation package exist in the second category of installation packages at the same time, determining the second installation package as the installation package with the highest compatibility level; when the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the third installation package as the installation package with the highest compatibility level; when only the fourth installation package exists in the second category of installation packages, determining the fourth installation package as the installation package with the highest compatibility level.

[0012] Optionally, the target component to be deployed currently is deployed on the target host based on the installation package with the highest compatibility level, including: obtaining the configuration information of the target host, wherein the configuration information of the target host includes: the central processing unit model, memory capacity, hard disk type and network card speed of the target host; generating a component configuration file template of the target host based on the configuration information, wherein the configuration file template includes at least: the occupancy rate of the central processing unit, the occupancy rate of the memory capacity, the operating parameters of the hard disk and the operating speed of the network card; modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template, so as to deploy the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host.

[0013] Optionally, before obtaining the configuration information of the target host, the method also includes: after pulling the installation package with the highest compatibility level to the local target host, running the compatibility detection script to check whether the content of the installation package with the highest compatibility level can be executed locally; in response to the content of the installation package with the highest compatibility level being able to be executed locally, obtaining the configuration information of the target host and continuing to deploy the target component to be deployed on the target host.

[0014] Optionally, the configuration parameters of the installation package with the highest compatibility level are modified to the parameters recorded in the component configuration file template so that the target component to be deployed currently corresponding to the installation package with the highest compatibility level is deployed on the target host, including: modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template so as to generate a local configuration file of the target component to be deployed currently corresponding to the installation package with the highest compatibility level; and deploying the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host according to the local configuration file of the target component to be deployed currently.

[0015] Optionally, the deployment method of the big data cluster also includes: when the first type installation package and the second type installation package of the target component to be deployed are not queried, obtaining the source code package of the target component to be deployed; compiling the source code package to obtain a compilation result; when the compilation result is a successful compilation, installing the successfully compiled file on the target host to deploy the target component to be deployed on the target host; when the compilation result is a failed compilation, recording the target information and continuing to deploy the next target component of the target component to be deployed on the target host, wherein the target information is that the installation of the target component to be deployed failed.

[0016] Optionally, the deployment method of the big data cluster also includes: determining the execution progress of each thread in multiple threads, wherein the multiple threads are used to deploy multiple target components on multiple hosts, each of the multiple hosts corresponds to a different thread, and the multiple threads are executed in parallel; when the execution progress of each thread in the multiple threads is completed, the deployment results of the multiple target components are stored in the database.

[0017] Optionally, the target component to be deployed is deployed on the target host based on the first type installation package of the target component to be deployed, including: running the first type installation package of the target component to be deployed on the target host corresponding to the target component to be deployed, so as to install the target component to be deployed on the target host corresponding to the device.

[0018] According to another aspect of the embodiment of the present application, a service system is further provided, which is used to execute the above-mentioned big data cluster deployment method, including: a cluster management service module, which is used to determine multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts, and the architectures of the multiple target components are different; a package management service module, which is used to query the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, wherein the first type of installation package is an installation package dedicated to each host in the multiple hosts, and the second type of installation package is a compatibility installation package; The management service module is also used to directly pull the first-class installation package of the target component to be deployed to the local target host where the target component to be deployed is to be deployed when the first-class installation package of the target component to be deployed is queried; if the first-class installation package of the target component to be deployed is not queried, pull the installation package with the highest compatibility level among the second-class installation packages to the local target host; the cluster management service module is used to deploy the target component to be deployed on the target host based on the first-class installation package of the target component to be deployed, or deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.

[0019] Optionally, the package management service module is further used to store the source code package of each target component in the multiple target components.

[0020] According to another aspect of the embodiment of the present application, a deployment device for a big data cluster is also provided, including: a determination module for determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; a query module for querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, the first type of installation package is an installation package dedicated to each host in the multiple hosts, and the second type of installation package is a compatibility installation package; a first deployment module for querying the current The first-class installation package of the target component to be deployed is found, the first-class installation package of the target component to be deployed is pulled to the local of the target host where the target component to be deployed is to be deployed, and the target component to be deployed is deployed on the target host based on the first-class installation package of the target component to be deployed; the second deployment module is used to determine the installation package with the highest compatibility level in the second-class installation packages if the first-class installation package of the target component to be deployed is not found, pull the installation package with the highest compatibility level to the local of the target host, and deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.

[0021] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program is stored. The device where the non-volatile storage medium is located executes the above-mentioned method for deploying a big data cluster by running the computer program.

[0022] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned big data cluster deployment method through the computer program.

[0023] According to another aspect of an embodiment of the present application, a computer program product is further provided, including a computer program, wherein the computer program implements the above-mentioned big data cluster deployment method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] FIG1 is a hardware structure block diagram of a computer terminal for implementing a method for deploying a big data cluster according to an embodiment of the present application;

[0026] FIG2 is a flowchart of a method for deploying a big data cluster according to an embodiment of the present application;

[0027] FIG3 is a schematic diagram of a service system according to an embodiment of the present application;

[0028] FIG4 is a structural diagram of a deployment device for a big data cluster according to an embodiment of the present application;

[0029] FIG5 is a workflow of a big data cluster deployment device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0034] Big data cluster: a cluster system consisting of multiple computers.

[0035] Heterogeneous architecture refers to the use of multiple different types of hardware, software, or technologies in a system to implement different functions or tasks. These different types of components can include different processors, operating systems, databases, network protocols, etc.

[0036] Hybrid deployment: refers to the use of servers with heterogeneous architectures to deploy big data clusters. Hybrid deployment can involve hardware, operating systems, networks, and other aspects.

[0037] Deployment software Apache Ambari: a cluster management tool based on the global wide area network (web).

[0038] In related technologies, big data clusters are deployed using the deployment software Apache Ambari, or using the merged software Cloudera and the open source data platform HortonWorks. Since the installation services provided by these two deployment methods are strongly architecture-dependent, i.e., they only support the deployment of big data clusters when the CPU architecture applicable to the component and the central processing unit architecture of the host on which the component is to be deployed are identical, deployment can only be performed in clusters with a single architecture. Therefore, there is an issue of not supporting mixed deployment of heterogeneous clusters. To address this issue, the present application provides a relevant solution in the embodiments, which is described in detail below.

[0039] According to an embodiment of the present application, a method embodiment of a deployment method for a big data cluster is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal for implementing a deployment method of a big data cluster. As shown in Figure 1, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used in the figure to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that the structure shown in Figure 1 is only illustrative and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0041] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the deployment method of the big data cluster in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned deployment method of the big data cluster. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0044] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0045] In the above operating environment, an embodiment of the present application provides a method for deploying a big data cluster. FIG2 is a flowchart of the steps of the method for deploying a big data cluster provided by an embodiment of the present application. As shown in FIG2 , the method includes the following steps:

[0046] In step S202 , a plurality of target components are determined from the plurality of components, wherein the plurality of target components are components to be deployed in a plurality of hosts in a big data cluster, and the central processing unit architectures corresponding to the plurality of target components are different.

[0047] The method provided in this embodiment supports hybrid deployment of heterogeneous clusters, wherein the big data cluster includes multiple computers (hosts) with different central processing unit architectures. When step S202 executes the big data cluster deployment, multiple (target) components to be deployed are determined from multiple optional components based on the multiple hosts. Since these multiple (target) components are used to be deployed on multiple computers (hosts) with different central processing unit architectures, that is, these multiple (target) components are adapted to different hosts, therefore, these multiple (target) components are adapted to different central processing unit architectures.

[0048] Step S204, query the installation package of the target component to be deployed from multiple pre-stored installation packages, where the multiple installation packages include: a first-class installation package and a second-class installation package, the first-class installation package is an installation package dedicated to each host in multiple hosts, and the second-class installation package is a compatibility installation package.

[0049] When executing the big data cluster deployment, step S204 is used to query the installation package of the (target) component to be deployed currently in the installation packages of multiple (target) components pre-stored in the system for executing the big data cluster deployment. For each (target) host to be installed on which the (target) component is to be installed, the installation packages of multiple (target) components pre-stored in the system for executing the big data cluster deployment are divided into two categories: one is a first category of installation packages (i.e., a dedicated installation package for each host) whose operating system and hardware configuration are exactly the same as those of the (target) host and are specifically used to install the (target) component to be deployed currently on the (target) host; the other is a second category of installation packages (i.e., a compatibility installation package) that is not only adapted to the (target) host, but also adapted to other hosts whose operating systems or hardware configurations are not exactly the same as those of the (target) host. The compatibility installation package mentioned in the embodiments of the present application refers to a software installation package that is compatible with the differences between different operating systems and hardware platforms and can run on different operating systems and different hardware platforms; when the installation package corresponding to the (target) component is a second-type installation package (i.e., a compatibility installation package), when the (target) component is installed / deployed on its corresponding (target) host, there is no need to change the operating system of the (target) host or replace the hardware in order to run the (target) component.

[0050] Step S206, when the first type of installation package of the target component to be deployed is queried, the first type of installation package of the target component to be deployed is pulled to the local of the target host where the target component to be deployed is to be deployed, and the target component to be deployed is deployed on the target host based on the first type of installation package of the target component to be deployed.

[0051] In step S206, if there is a dedicated installation package (i.e., a first-class installation package) for the (target) host on which the (target) component to be deployed is to be installed among the multiple pre-stored installation packages, no further operation is required. The installation package corresponding to the (target) component to be deployed is directly pulled to the local location of its corresponding (target) host, and the first-class installation package is run on the (target) host corresponding to the (target) component to be deployed to install the (target) component on its corresponding (target) host.

[0052] Step S208: If the first type of installation package for the target component to be deployed is not found, determine the installation package with the highest compatibility level in the second type of installation package, pull the installation package with the highest compatibility level to the local location of the target host, and deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.

[0053] In step S208, if the dedicated installation package (i.e., the first type of installation package) for the (target) host where the (target) component to be deployed is to be installed does not exist in the pre-stored multiple installation packages, then the compatibility installation package with the highest compatibility level among the multiple compatibility installation packages that support the (target) component is queried, the compatibility installation package with the highest compatibility level is pulled to the local (target) host, and the (target) component is installed on the (target) host through the installation package with the highest compatibility level.

[0054] Optionally, the compatibility level of each installation package in the second category of installation packages is determined by the following method: obtaining the central processing unit architecture and operating system information of each installation package, wherein the operating system information includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs; determining the compatibility level of each installation package in the second category of installation packages based on the central processing unit architecture and operating system information of each installation package.

[0055] In this embodiment, the compatibility level of each installation package is determined based on the central processing unit architecture and operating system information adapted by each installation package, wherein the operating system information of each installation package includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs. Based on the relevant information of the installation package obtained (central processing unit architecture and operating system information), the compatibility level of each installation package relative to the host where the (target) component to be deployed is to be installed is judged.

[0056] According to an optional embodiment of the present application, the compatibility level of each installation package in the second category of installation packages is determined based on the central processing unit architecture and operating system information of each installation package, including: obtaining the central processing unit architecture and operating system information of the target host, the operating system information of the target host including: the version number of the operating system of the target host and the upstream operating system to which the operating system of the target host belongs; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose operating system version number has the smallest difference with the operating system version number of the target host as the first installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose upstream operating system is the same as the upstream operating system of the target host as the second installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose core component of the operating system is the Linux kernel as the third installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host as the fourth installation package; and determining the installation package with the same central processing unit architecture as the central processing unit architecture of the target host as the fourth installation package; and determining the installation package with the highest compatibility level based on the type of installation package in the second category of installation packages.

[0057] In this embodiment, when determining the compatibility of each installation package with respect to the (target) host to which it is to be deployed, it is necessary to obtain the central processing unit architecture of the target host and the operating system information of the target host, wherein the operating system information of the target host includes: the version number of the operating system of the target host and the upstream operating system to which the operating system of the target host belongs. By comparing the operating system information and central processing unit architecture of the installation package with the central processing unit architecture and operating system information of the (target) host, the installation package that is most compatible with the (target) host is selected; wherein, for the (target) host, the pre-stored installation packages are divided into the following types: one is the first installation package that has the same central processing unit architecture as the target host and the operating system version closest to the target host among all the installation packages; In the process, whether the operating system version of the installation package is the same as the operating system version of the target host is determined based on the version and version number of the operating system. For the same type of operating system, the operating system with a larger version number has a greater degree of update than the operating system with a smaller version number, that is, a newer version. For example, when the operating systems are all version A, the operating system with version number V1.0.1 is a newer version than the operating system with version number V1.0.0. Therefore, when the operating systems are all version A, the operating system version of the installation package that is closest to the version of the operating system of the target host can also be determined by calculating the difference between the version numbers. The smaller the difference in version number with the operating system of the target host, the closer it is to the operating system of the second host. The installation package also includes a second installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose upstream operating system is the same as the operating system of the target host; the installation package also includes a third installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose upstream operating system is different from the operating system of the target host but whose core component is the Linux kernel; and a fourth installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host. The installation package types in the second type of installation package are, for example, the first installation package, the second installation package, the third installation package or the fourth installation package. The target host selects the installation package with the highest compatibility level for installing the target component from the above four types of installation packages (the first installation package, the second installation package, the third installation package and the fourth installation package).

[0058] Optionally, determining the installation package with the highest compatibility level based on the type of installation packages in the second category of installation packages includes: when the first installation package, the second installation package, the third installation package, and the fourth installation package exist in the second category of installation packages at the same time, determining the first installation package as the installation package with the highest compatibility level; when the second installation package, the third installation package, and the fourth installation package exist in the second category of installation packages at the same time, determining the second installation package as the installation package with the highest compatibility level; when the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the third installation package as the installation package with the highest compatibility level; when only the fourth installation package exists in the second category of installation packages, determining the fourth installation package as the installation package with the highest compatibility level.

[0059] According to the rules preset in the embodiments of the present application, in this embodiment, the first installation package whose CPU architecture is closest to the CPU architecture of the target host and whose operating system version number is closest to the target host's operating system version number (i.e., the version number difference is the smallest) is determined to be at the first compatibility level; the installation package whose CPU architecture is the same as the target host's CPU architecture and whose operating system belongs to the same upstream operating system as the target host's operating system is determined to be at the second compatibility level; the installation package whose CPU architecture is the same as the target host's CPU architecture and whose core component of the operating system is the Linux kernel is determined to be at the third compatibility level; the installation package whose only CPU architecture is the same as the target host's CPU architecture is determined to be at the fourth compatibility level. That is, the compatibility levels of the first, second, third, and fourth installation packages decrease in sequence. When screening the installation package with the highest compatibility level, first determine the type of installation package in the second category of installation packages. When the first installation package, the second installation package, the third installation package, and the fourth installation package exist at the same time, the installation package with the highest compatibility level is the first installation package with the first compatibility level; when the second installation package, the third installation package, and the fourth installation package exist at the same time, the installation package with the highest compatibility level is the second installation package with the second compatibility level; when the third installation package and the fourth installation package exist at the same time, the installation package with the highest compatibility level is the third installation package with the third compatibility level; when only one type of installation package exists, the installation package with the highest compatibility level is the existing type of installation package.

[0060] According to an optional embodiment of the present application, the target component to be deployed currently is deployed on the target host based on the installation package with the highest compatibility level, including: obtaining the configuration information of the target host, wherein the configuration information of the target host includes: the central processing unit model, memory capacity, hard disk type and network card speed of the target host; generating a component configuration file template of the target host based on the configuration information, wherein the configuration file template includes at least: the occupancy rate of the central processing unit, the occupancy rate of the memory capacity, the operating parameters of the hard disk and the operating speed of the network card; modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template, so as to deploy the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host.

[0061] In this embodiment, when the installation package corresponding to the (target) component to be installed on the target host is a compatibility installation package (i.e., a second-class installation package), after obtaining the installation package with the highest compatibility level, a compatibility detection script is run to check whether the content of the installation package (the obtained installation package with the highest compatibility level) can be executed locally. If so, the next step of installation is performed. When it is determined that the installation package with the highest compatibility level can be executed locally, a configuration management service is requested to obtain the configuration information of the target host, wherein the configuration information of the target host includes: the central processing unit architecture of the target host and the configuration template corresponding to the operating system, wherein the information / parameters recorded in the configuration template include: the model of the central processing unit of the target host, the memory capacity of the target host, the hard disk type of the target host, the speed of the network card of the target host, and other relevant configuration information of the target host; the configuration of the installation package with the highest compatibility level is modified to the information recorded in the above configuration template to complete the configuration of the target component on the target host.

[0062] According to an optional embodiment of the present application, the configuration parameters of the installation package with the highest compatibility level are modified to the parameters recorded in the component configuration file template so that the target component to be deployed currently corresponding to the installation package with the highest compatibility level is deployed on the target host, including: modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template so as to generate a local configuration file of the target component to be deployed currently corresponding to the installation package with the highest compatibility level; and deploying the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host according to the local configuration file of the target component to be deployed currently.

[0063] In this embodiment, the configuration of the installation package with the highest compatibility level is modified to the information recorded in the above-mentioned configuration template to complete the personalized rendering of the component configuration file, and is returned to the target host after rendering as the local configuration file of the target component. The configuration of the target component on the target host is completed according to the local configuration file of the target component.

[0064] According to another optional embodiment of the present application, the deployment method of the big data cluster also includes: when the first type installation package and the second type installation package of the target component to be deployed are not queried, obtaining the source code package of the target component to be deployed; compiling the source code package to obtain a compilation result; when the compilation result is a successful compilation, installing the successfully compiled file on the target host to deploy the target component to be deployed on the target host; when the compilation result is a failed compilation, recording the target information and continuing to deploy the next target component of the target component to be deployed on the target host, wherein the target information is that the installation of the target component to be deployed failed.

[0065] In this embodiment, if it is detected that the installation package with the highest compatibility level cannot be executed locally on the target host, or the component installation package with the same processor architecture as the target host cannot be found, the source code package of the component is directly pulled from the package management service to the local target host for local compilation; if the compilation is successful, the compilation result is uploaded to the package management service to reduce the subsequent compilation time, and the next component installation / deployment is carried out; if the compilation fails, the installation failure of this component (that is, the target component currently to be deployed) is recorded, and the next (target) component is installed.

[0066] According to some optional embodiments of the present application, the deployment method of a big data cluster also includes: determining the execution progress of each thread in a plurality of threads, wherein the plurality of threads are used to deploy a plurality of target components on a plurality of hosts, each of the plurality of hosts corresponds to a different thread, and the plurality of threads are executed in parallel; when the execution progress of each thread in the plurality of threads is completed, the deployment results of the plurality of target components are stored in a database.

[0067] According to some optional embodiments, the method provided in the embodiments of the present application supports hybrid deployment of heterogeneous architectures, and can therefore be applied to the simultaneous deployment of components on multiple hosts with different central processing unit architectures; when components are deployed on multiple hosts with different central processing unit architectures at the same time, the component deployment on different hosts is executed by different threads; as mentioned in the previous embodiment, for each thread, when deploying components on each target host, when a corresponding installation package is not compatible with the target host to be deployed, the component deployment failure is recorded and the next component is deployed; and for the component process of multiple hosts, multiple threads execute the component deployment on multiple hosts in parallel, and only after it is detected that multiple threads have completed execution, the deployment results of each component in this component deployment on multiple hosts are uniformly stored in the database.

[0068] Through the above steps, a method for obtaining the installation packages of the three components can be provided: during the deployment process, the installation packages corresponding to the CPU architecture and operating system of the big data component are dynamically pulled; when no installation package is available, the installation package with the highest compatibility is pulled in real time and an executable test is performed; if no compatible installation package is available, the source code is pulled in real time and an attempt is made to compile and install it locally; at the same time, for different CPU architectures and operating systems, the host system configuration, kernel parameters and the configuration files of the big data component are configured in a targeted manner, and the configuration parameters are adjusted in a targeted manner according to the performance of the host CPU, memory, hard disk, network card, etc., so as to improve the overall performance of the cluster. This solves the problem that the common cluster deployment software in related technologies only focuses on cluster deployment of a single architecture and a single operating system and does not support heterogeneous hybrid deployment; it reduces the deployment time and the labor cost required for hybrid deployment.

[0069] FIG3 is a schematic diagram of a service system provided according to an embodiment of the present application, and the service system is used to execute the above-mentioned method for deploying a big data cluster. In some embodiments, the service system is, for example, a server. As shown in FIG3 , the service system includes: a cluster management service module 30, for determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts, and the architectures of the multiple target components are different; a package management service module 32, for querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, the first type of installation package is a dedicated installation package for each of the multiple hosts, and the second type of installation package is a compatibility installation package; the package management service module 32 is also used to query the current target component to be deployed. In the case of the first-class installation package of the target component to be deployed, the first-class installation package of the target component to be deployed is directly pulled to the local target host where the target component to be deployed is to be deployed; in the case of no first-class installation package of the target component to be deployed, the installation package with the highest compatibility level among the second-class installation packages is pulled to the local target host; the cluster management service module 34 is used to deploy the target component to be deployed on the target host based on the first-class installation package of the target component to be deployed, or to deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.

[0070] Optionally, the package management service module 32 is further configured to store the source code package of each target component in the plurality of target components.

[0071] In the above-mentioned service system, the package management service module 32 is used to store the installation packages of each component of the big data cluster under different operating systems and different CPU architectures, as well as the source code packages of each component of the big data cluster. It is used to dynamically obtain the installation package or source code package during deployment. The cluster management service module 34 is used to provide operations such as user management, host management, permission management, cluster orchestration, and cluster deployment script execution. The cluster management service module includes: a module run by the deployment framework (Ansible), which is used to use script deployment (ansible-playbook) technology to encapsulate the deployment scripts of each component of the big data cluster, and implement functions such as host environment initialization, installation package download, configuration file rendering, big data cluster component initialization, and big data cluster component start and stop script encapsulation. The configuration management service module is used to store the personalized configuration of each component of the big data cluster under different operating systems, different CPUs, and different hardware configurations (number of CPU threads, memory size, number of hard disks, hard disk size, etc.), such as whether to enable non-uniform memory access architecture (NUMA) awareness, service startup memory parameters, service hard disk space usage configuration, etc. It is used to configure the personalized rendering big data cluster software for heterogeneous servers during deployment, and optimize the CPU architecture, hardware configuration, and operating system-related configurations to achieve the highest possible overall cluster performance.

[0072] FIG4 is a structural diagram of a deployment device for a big data cluster provided according to an embodiment of the present application. As shown in FIG4 , the deployment device for a big data cluster includes: a determination module 40 for determining a plurality of target components among a plurality of components, wherein the plurality of target components are components to be deployed in a plurality of hosts in a big data cluster, and the central processing unit architectures corresponding to the plurality of target components are different; a query module 42 for querying the installation package of the target component to be currently deployed from a plurality of pre-stored installation packages, wherein the plurality of installation packages include: a first type of installation package and a second type of installation package, wherein the first type of installation package is an installation package dedicated to each host in a plurality of hosts, and the second type of installation package is a compatibility installation package; a first deployment module 44, used for, when the first type of installation package of the target component to be deployed is queried, pulling the first type of installation package of the target component to be deployed to the local of the target host to which the target component to be deployed is to be deployed, and deploying the target component to be deployed on the target host based on the first type of installation package of the target component to be deployed; the second deployment module 46, used for, when the first type of installation package of the target component to be deployed is not queried, determining the installation package with the highest compatibility level in the second type of installation package, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be deployed on the target host based on the installation package with the highest compatibility level.

[0073] FIG5 is a workflow diagram of a deployment device for a big data cluster. As shown in FIG5 , when the deployment device for a big data cluster is working, the determination module 40 is first used to select a deployment component, such as selecting a distributed file system (Hadoop Distributed File System, HDFS), and then a list of deployment hosts generated by multiple footprints to be deployed is selected. After the selection is completed, the determination module configures component distribution for each host and starts automatic deployment. During automatic deployment, the query module traverses all hosts to be deployed with components (i.e., to be deployed) and performs deployment operations on each host. When deploying the target component to the current host (i.e., the target host), the CPU architecture, operating system, and hardware configuration information of the current host (i.e., the target host) are first confirmed to confirm whether the local library needs to be supplemented. If necessary, the specific architecture / operating system dependency library is installed through the package management service. If not, Traverse each big data component that needs to be deployed in this host and perform deployment operations. When performing deployment operations, the query module 42 first determines whether the target host supports the architecture / operating system of the target component. If so, pull the installation package of the architecture / operating system corresponding to the target component through the package management service (i.e., the first type of installation package); if not, the query module 42 queries whether there is an installation package of a compatible architecture / operating system (i.e., the second type of installation package). If so, pull or compatible architecture / operating system installation package; if not, pull the source code package of the architecture / operating system corresponding to the target component adapted to the target host through the package management service for local compilation. If the first or second type of installation package is pulled, the configuration file for the corresponding architecture / operating system, or the default configuration of the target host, is pulled through the configuration management service. If the source code package is pulled, if the source code package compilation is determined to be successful, the configuration file for the corresponding architecture / operating system, or the default configuration of the target host, is pulled through the configuration management service. If the source code package compilation fails, the current component deployment failure is recorded, and the "traversal of each big data component that needs to be deployed on this host and deployment operation" is re-executed to execute the deployment of the next component. After pulling the host's configuration file, the first deployment module 44 executes the deployment of the components corresponding to the first type of installation package and the source code package, and the second deployment module executes the deployment of the components corresponding to the second type of installation package. The deployment process is as follows: the component roles corresponding to the host are deployed according to the configuration, and after the components of all hosts are deployed, the deployment result status of all hosts is collected and the component deployment status data is persistently stored in the database.

[0074] It should be noted that the preferred implementation of the embodiment of the deployment device for a big data cluster shown in FIG4 can be found in the relevant description in the aforementioned embodiment of the deployment method for a big data cluster, which will not be repeated here.

[0075] An embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored. The above-mentioned big data cluster deployment method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0076] The above-mentioned non-volatile storage medium is used to store a program that performs the following functions: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of a big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-class installation package and a second-class installation package, the first-class installation package is an installation package dedicated to each host in the multiple hosts, and the second-class installation package is a compatibility installation package; when the first-class installation package of the target component to be currently deployed is queried, the first-class installation package of the target component to be currently deployed is pulled to the local of the target host to which the target component to be currently deployed is to be deployed, and the target component to be currently deployed is deployed on the target host based on the first-class installation package of the target component to be currently deployed; when the first-class installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level among the second-class installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.

[0077] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned big data cluster deployment method through the computer program.

[0078] The processor in the above-mentioned electronic device is used to run a program that performs the following functions: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of a big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-class installation package and a second-class installation package, the first-class installation package is an installation package dedicated to each host among the multiple hosts, and the second-class installation package is a compatibility installation package; when the first-class installation package of the target component to be currently deployed is queried, the first-class installation package of the target component to be currently deployed is pulled to the local of the target host to which the target component to be currently deployed is to be deployed, and based on the first-class installation package of the target component to be currently deployed, the target component to be currently deployed is deployed on the target host; when the first-class installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level among the second-class installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.

[0079] The features described in the aforementioned embodiment of the big data cluster deployment method are applicable to the big data cluster deployment method executed by the processor of the electronic device through the computer program, and will not be repeated here.

[0080] In one embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement a deployment method for a big data cluster according to the above-mentioned embodiment: determining multiple target components from multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-category installation package and a second-category installation package, wherein the first-category installation package is an installation package dedicated to each of the multiple hosts, and the second-category installation package is a compatibility installation package; when the first-category installation package of the target component to be currently deployed is queried, the first-category installation package of the target component to be currently deployed is pulled to the local host of the target host to which the target component to be currently deployed is to be deployed, and the target component to be currently deployed is deployed on the target host based on the first-category installation package of the target component to be currently deployed; when the first-category installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level from the second-category installation package, pulling the installation package with the highest compatibility level to the local host of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.

[0081] The features described in the aforementioned embodiment of the big data cluster deployment method are applicable to the big data cluster deployment method executed by the processor of the electronic device through the computer program, and will not be repeated here.

[0082] It should be noted that the various modules in the deployment device of the above-mentioned big data cluster can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0083] According to the various embodiments of the present application, by dynamically pulling the installation packages corresponding to the CPU architecture and operating system of the big data component during the deployment process; when no installation package is available, pulling the compatibility installation package in real time and performing an executable test; when no compatibility installation package is available, pulling the source code in real time and trying to compile and install it locally, the host system configuration, kernel parameters and configuration files of the big data component are configured in a targeted manner for different CPU architectures and operating systems, and the configuration parameters are adjusted in a targeted manner according to the performance of the host CPU, memory, hard disk, network card, etc.; the purpose of optimizing the overall performance of the big data cluster is achieved, thereby realizing the technical effect of hybrid deployment of heterogeneous big data clusters, and further solving the technical problem of not supporting hybrid deployment of heterogeneous big data clusters due to the strong correlation between the big data cluster deployment method and architecture in the related technology.

[0084] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0086] As used in this application, the terms "component," "module," and "system" are intended to refer to a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable code, a thread of execution, a program, and / or a computer. As an illustration, an application running on a server and a server can both be components. One or more components can reside in a process and / or a thread of execution. And a component can be located within a computer and / or distributed between two or more computers.

[0087] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0090] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0091] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for deploying a big data cluster, comprising: Determine multiple target components from the multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; Querying the installation package of the target component to be deployed currently from a plurality of pre-stored installation packages, wherein the plurality of installation packages include: a first-category installation package and a second-category installation package, the first-category installation package being an installation package dedicated to each of the plurality of hosts, and the second-category installation package being a compatibility installation package; In the case where the first type of installation package of the target component to be deployed is queried, the first type of installation package of the target component to be deployed is pulled to the local of the target host to which the target component to be deployed is to be deployed, and the target component to be deployed is deployed on the target host based on the first type of installation package of the target component to be deployed; If the first type of installation package of the target component to be deployed is not found, the installation package with the highest compatibility level is determined from the second type of installation packages, the installation package with the highest compatibility level is pulled to the local of the target host, and the target component to be deployed is deployed on the target host based on the installation package with the highest compatibility level.

2. The method according to claim 1, wherein: The compatibility level of each installation package in the second category of installation packages is determined by the following method: Obtaining the central processing unit architecture and operating system information of each installation package, wherein the operating system information includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs; The compatibility level of each installation package in the second type of installation packages is determined according to the central processing unit architecture and operating system information of each installation package.

3. The method according to claim 2, wherein: Determining the compatibility level of each installation package in the second category of installation packages according to the central processing unit architecture and operating system information of each installation package includes: Acquire the central processing unit architecture and operating system information of the target host, wherein the operating system information of the target host includes: the version number of the operating system of the target host and the upstream operating system to which the operating system of the target host belongs; Determine the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose operating system version number has the smallest difference with the operating system version number of the target host as the first installation package; Determine an installation package whose central processing unit architecture is the same as that of the central processing unit of the target host and whose upstream operating system is the same as that of the upstream operating system of the target host as the second installation package; Determine an installation package whose central processing unit architecture is the same as that of the central processing unit of the target host and whose core component of the operating system is a Linux kernel as the third installation package; Determine an installation package having the same CPU architecture as that of the target host as a fourth installation package; The installation package with the highest compatibility level is determined according to the type of installation packages in the second type of installation packages.

4. The method according to claim 3, wherein: The compatibility levels of the first installation package, the second installation package, the third installation package, and the fourth installation package decrease in sequence.

5. The method according to claim 3 or 4, wherein: Determining the installation package with the highest compatibility level according to the type of the installation package in the second type of installation packages includes: When the first installation package, the second installation package, the third installation package, and the fourth installation package exist in the second type of installation packages at the same time, determining the first installation package as the installation package with the highest compatibility level; When the second installation package, the third installation package and the fourth installation package exist in the second type of installation packages at the same time, determining the second installation package as the installation package with the highest compatibility level; When the third installation package and the fourth installation package exist in the second type of installation packages at the same time, determining the third installation package as the installation package with the highest compatibility level; When only the fourth installation package exists in the second type of installation packages, the fourth installation package is determined as the installation package with the highest compatibility level.

6. The method according to claim 3 or 4, wherein: Deploying the target component to be deployed on the target host based on the installation package with the highest compatibility level includes: Acquire the configuration information of the target host, wherein the configuration information of the target host includes: the CPU model, memory capacity, hard disk type and network card speed of the target host; Generate a component configuration file template of the target host according to the configuration information, wherein the configuration file template at least includes: the occupancy rate of the central processing unit, the occupancy rate of the memory capacity, the operating parameters of the hard disk and the operating speed of the network card; The configuration parameters of the installation package with the highest compatibility level are modified to the parameters recorded in the component configuration file template, so as to deploy the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host.

7. The method according to claim 6, wherein: Before obtaining the configuration information of the target host, the method further includes: After the installation package with the highest compatibility level is pulled to the local of the target host, a compatibility detection script is run to check whether the content of the installation package with the highest compatibility level can be executed locally; In response to the content of the installation package with the highest compatibility level being able to be executed locally, the configuration information of the target host is obtained and the target component to be deployed currently is deployed on the target host.

8. The method according to claim 6, wherein: Modifying the configuration parameters of the installation package with the highest compatibility level into the parameters recorded in the component configuration file template, so as to deploy the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host, includes: Modifying the configuration parameters of the installation package with the highest compatibility level into the parameters recorded in the component configuration file template, so as to generate a local configuration file of the target component to be deployed currently corresponding to the installation package with the highest compatibility level; According to the local configuration file of the target component to be currently deployed, the target component to be currently deployed corresponding to the installation package with the highest compatibility level is deployed on the target host.

9. The method according to claim 1, further comprising: If neither the first installation package nor the second installation package of the target component to be deployed is found, obtaining a source code package of the target component to be deployed; Compiling the source code package to obtain a compilation result; If the compilation result is successful, installing the successfully compiled file on the target host, so as to deploy the target component to be deployed on the target host; If the compilation result is a compilation failure, record the target information and continue to deploy the target to be deployed. The next target component of the component is deployed on the target host, wherein the target information indicates that the installation of the current target component to be deployed has failed.

10. The method according to claim 1, further comprising: Determining the execution progress of each of a plurality of threads, wherein the plurality of threads are used to deploy the plurality of target components on the plurality of hosts, each of the plurality of hosts corresponds to a different thread, and the plurality of threads are executed in parallel; When the execution progress of each of the multiple threads is completed, the deployment results of the multiple target components are stored in a database.

11. The method according to claim 1, wherein: Deploying the target component to be deployed currently on the target host based on the first type of installation package of the target component to be deployed currently includes: The first type installation package of the target component to be deployed is run on the target host corresponding to the target component to be deployed, so as to install the target component to be deployed on the target host corresponding to the device.

12. A service system, the service system being used to execute the big data cluster deployment method according to any one of claims 1 to 11, comprising: A cluster management service module, used to determine a plurality of target components among the plurality of components, wherein the plurality of target components are components to be deployed in a plurality of hosts, and the architectures of the plurality of target components are different; A package management service module, configured to retrieve an installation package of a target component to be deployed from a plurality of pre-stored installation packages, wherein the plurality of installation packages include: a first-category installation package and a second-category installation package, wherein the first-category installation package is a dedicated installation package for each of the plurality of hosts, and the second-category installation package is a compatibility installation package; The package management service module is further configured to directly pull the first-category installation package of the target component to be deployed to the local of the target host to which the target component to be deployed is to be deployed, if the first-category installation package of the target component to be deployed is found; and to pull the installation package with the highest compatibility level among the second-category installation packages to the local of the target host, if the first-category installation package of the target component to be deployed is not found. The cluster management service module is used to deploy the target component to be deployed on the target host based on the first type installation package of the target component to be deployed, or to deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.

13. The service system according to claim 12, wherein: The package management service module is also used to store the source code package of each target component in the multiple target components.

14. A deployment device for a big data cluster, comprising: A determination module, used to determine a plurality of target components from a plurality of components, wherein the plurality of target components are components to be deployed in a plurality of hosts of a big data cluster, and the central processing unit architectures corresponding to the plurality of target components are different; A query module, used to query the installation package of the target component to be deployed currently from a plurality of pre-stored installation packages, wherein the plurality of installation packages include: a first-category installation package and a second-category installation package, wherein the first-category installation package is an installation package dedicated to each of the plurality of hosts, and the second-category installation package is a compatibility installation package; A first deployment module is used for, when the first type of installation package of the target component to be deployed is queried, pulling the first type of installation package of the target component to be deployed to the local of the target host to which the target component to be deployed is to be deployed, and deploying the target component to be deployed on the target host based on the first type of installation package of the target component to be deployed; The second deployment module is used for, when the first type of installation package of the target component to be deployed is not found, Determine the installation package with the highest compatibility level in the second type of installation packages, pull the installation package with the highest compatibility level to the local of the target host, and deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.

15. A non-volatile storage medium, wherein: The non-volatile storage medium stores a computer program, wherein the method for deploying a big data cluster according to any one of claims 1 to 11 is executed by running the computer program on the device where the non-volatile storage medium is located.

16. An electronic device comprising a memory and a processor, wherein: The memory stores a computer program, and the processor is configured to execute the big data cluster deployment method according to any one of claims 1 to 11 through the computer program.

17. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images