A method, device, medium and electronic equipment for assembling a big data cluster

Abstract

The application provides a method and device for assembling a big data cluster, a medium and an electronic device, and relates to the technical field of computer application. In the method, during the deployment of the big data cluster, when a service deployment request sent by a physical node is received, target configuration information of a to-be-deployed service control is determined based on an identifier of the to-be-deployed service control included in the service deployment request. The service deployment request is sent by the physical node to a control node when it is determined that the deployed service control has a subordinate service control. The target configuration information is sent to the physical node that needs to deploy the to-be-deployed service control, so that the physical node receiving the target configuration information deploys the to-be-deployed service control. The method can realize the automatic construction of the service control without pre-arranging a deployment scheme, makes the deployment of the big data cluster more convenient, thereby saving the labor cost of the deployment of the big data cluster and improving the efficiency of the deployment of the big data cluster.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to a method, apparatus, medium and electronic device for building a big data cluster. Background Technology

[0002] Currently, big data platforms are mostly in the form of clusters. The entire platform contains multiple components, such as Hadoop and Zookeeper, and each component usually contains a number of services.

[0003] Traditional deployment of big data clusters typically requires planning the network and deployment scheme, then installing the operating system, services, and configuring parameters on physical devices—a process that is extremely cumbersome and complex. Even with cluster management tools, users still need to plan the components to be deployed according to their needs and then deploy each component following a wizard.

[0004] When deploying big data clusters, related technologies typically require pre-arranged deployment plans and the construction of the big data cluster is completed in a bottom-up order based on service dependencies. The deployment plan is complex and the workload is large, resulting in low efficiency in deploying big data clusters. Summary of the Invention

[0005] To address the existing technical problems, embodiments of this application provide a method, apparatus, medium, and electronic device for building a big data cluster, which can improve the efficiency of big data cluster deployment.

[0006] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a method for constructing a big data cluster, applied to a control node of the big data cluster, wherein the big data cluster further includes at least one physical node communicating with the control node; the method includes:

[0008] Upon receiving a service deployment request from a physical node, the target configuration information of the service control to be deployed is determined based on the identifier of the service control to be deployed included in the service deployment request; wherein, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has a direct child layer of service controls;

[0009] The target configuration information is sent to the physical node that needs to deploy the service control to be deployed, so that the physical node that receives the target configuration information can deploy the service control to be deployed.

[0010] The big data cluster construction method provided in this application embodiment, during the deployment of the big data cluster, upon receiving a service deployment request from a physical node, determines the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request. The service deployment request is sent by the physical node to the control node when it determines that a deployed service control has directly subordinate service controls. The target configuration information is then sent to the physical node that needs to deploy the service control, enabling the physical node receiving the target configuration information to deploy the service control. This method, by determining the target configuration information of the service control to be deployed upon receiving a service deployment request from a physical node and sending it to the physical node that needs to deploy the service control, allows the physical node receiving the target configuration information to deploy the service control without the need for pre-arranged deployment plans, achieving automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs, and improving the efficiency of big data cluster deployment.

[0011] In an optional embodiment, the method further includes:

[0012] Before receiving a service deployment request sent by a physical node, respond to an initial service deployment request and determine the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request; the target service control is the service control selected by the user to be used, and has at least one service control of the directly subordinate sub-layer.

[0013] The configuration information of the target service control is sent to the physical node that needs to deploy the target service control, so that the physical node that receives the configuration information of the target service control can deploy the target service control.

[0014] In this embodiment, by first responding to the initial service deployment request, and based on the service controls selected by the user in the initial service deployment request, configuration information is determined and sent to the physical nodes that need to deploy the user-selected service controls. The physical nodes receiving the configuration information then deploy the user-selected service controls. This method eliminates the need for pre-arranged deployment plans; users simply select the service controls they need for deployment, thus achieving automatic construction of the service controls. This makes big data cluster deployment more convenient, saves manpower costs, and improves the efficiency of big data cluster deployment.

[0015] In an optional embodiment, the method further includes:

[0016] When the deployment completion notification of the target service control is received, a deployment completion prompt for the big data cluster is generated.

[0017] In this embodiment, upon receiving a deployment completion notification from the target service control, a deployment completion notification for the big data cluster is generated. This method eliminates the need for manual identification, further reducing the manpower costs of big data cluster deployment and improving its efficiency.

[0018] In an optional embodiment, determining the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request includes:

[0019] Based on the identifier of the service control to be deployed included in the service deployment request, the deployment parameters corresponding to the identifier of the service control to be deployed are obtained from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters.

[0020] Based on the deployment parameters corresponding to the identifier of the service control to be deployed, the physical node where the service control to be deployed needs to be deployed and the target configuration information of the service control to be deployed are determined. The target configuration information includes a service image identifier and container startup boot parameters. The service image identifier is used to enable the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier. The container startup boot parameters are used to boot and run the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

[0021] In this embodiment, based on the identifier of the service control to be deployed included in the service deployment request, deployment parameters corresponding to the identifier of the service control to be deployed are obtained from preset deployment settings information. Then, based on the deployment parameters corresponding to the identifier of the service control to be deployed, the physical node to which the service control to be deployed and the target configuration information of the service control to be deployed are determined. This method can automatically obtain deployment parameters and determine the physical node to which the service control to be deployed and the target configuration information of the service control to be deployed, without the need for pre-arranged deployment plans. It achieves automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs and improving the efficiency of big data cluster deployment.

[0022] In an optional embodiment, the method further includes:

[0023] If a deployment completion notification is received for a service control in a direct sublayer of any service control, the runtime interface information of the service control in the direct sublayer of any service control is sent to the physical node that deployed the service control, so that the physical node starts the service control based on the runtime interface information and sends a deployment completion notification of the service control to the control node after starting.

[0024] In this embodiment, if a deployment completion notification is received for a service control in a direct sub-layer of any service control, the runtime interface information of that service control is sent to the physical node that deployed the service control. This allows the physical node to start the service control based on the runtime interface information and, after starting, send a deployment completion notification to the control node. By providing the runtime interface information of the service control in a direct sub-layer to the service control upon learning that its deployment is complete, automatic transmission of runtime interface information for the service control in the direct sub-layer is achieved. This eliminates the need for pre-arranged deployment plans, enabling automatic construction of service controls and making big data cluster deployment more convenient. It also saves manpower costs and improves the efficiency of big data cluster deployment.

[0025] Secondly, embodiments of this application also provide a method for constructing a big data cluster, applied to the physical nodes of the big data cluster, wherein the big data cluster further includes a control node that communicates with the physical nodes; the method includes:

[0026] Upon receiving target configuration information for a service control to be deployed from a control node, the service control to be deployed is deployed according to the received target configuration information.

[0027] If it is determined that the service control to be deployed has a direct sub-layer of service controls, then the identifier of the direct sub-layer of service controls is placed in the service deployment request, and the service deployment request is sent to the control node, so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request.

[0028] The big data cluster construction method provided in this embodiment deploys the big data cluster by deploying the service control according to the target configuration information of the service control to be deployed sent by the control node each time the target configuration information of the service control to be deployed is received. If it is determined that the service control to be deployed has directly subordinate service controls, the identifier of the directly subordinate service controls is placed in the service deployment request, and the service deployment request is sent to the control node so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request. This method deploys the service control to be deployed according to the target configuration information of the service control to be deployed sent by the control node each time the target configuration information of the service control to be deployed is received. Furthermore, when it is determined that the service control to be deployed has directly subordinate service controls, the method further requests the control node to deploy the directly subordinate service controls of the service control to be deployed by placing the identifier of the directly subordinate service controls in the service deployment request and sending the service deployment request to the control node. Therefore, there is no need to pre-arrange a deployment plan, achieving automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs and improving the efficiency of big data cluster deployment.

[0029] In an optional embodiment, the method further includes:

[0030] Before receiving the target configuration information of the service control to be deployed sent by the control node, the physical node that needs to deploy the target service control needs to receive the configuration information of the target service control sent by the control node; the target service control is the service control that the user selects to use, and has at least one service control of the direct sub-layer.

[0031] Deploy the target service control based on the received configuration information of the target service control;

[0032] If it is determined that the target service control has directly subordinate service controls, then the identifier of the directly subordinate service control of the target service control is placed in the service deployment request and sent to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate service control of the target service control.

[0033] The method provided in this embodiment does not require pre-arranged deployment plans. When a user selects the target service control to be deployed, the physical node deploys the target service control according to the configuration information of the target service control received. At the same time, it also requests the deployment of the service controls of the direct sub-layer of the target service control from the control node, realizing the automatic construction of service controls, making the deployment of big data clusters more convenient, saving the manpower cost of big data cluster deployment, and improving the efficiency of big data cluster deployment.

[0034] In an optional embodiment, after sending the service deployment request containing the identifier of the service control of the direct sublayer of the target service control to the control node, the method further includes:

[0035] If the runtime interface information of the service controls of each of the direct sub-layers of the target service control is received, the target service control is started, and a deployment completion notification of the target service control is sent to the control node.

[0036] In this embodiment, if the runtime interface information of the service controls of each of the direct sub-layers of the target service control is received, the target service control is started, and a deployment completion notification of the target service control is sent to the control node. This method, upon receiving the runtime interface information of the service controls of each of the direct sub-layers of the target service control, directly starts the target service control and sends a deployment completion notification to the control node, without manual intervention. This achieves automatic construction of the service control, making big data cluster deployment more convenient, saving manpower costs, and improving the efficiency of big data cluster deployment.

[0037] In an optional embodiment, deploying the service control to be deployed according to the received target configuration information includes:

[0038] Based on the received target configuration information, the installation program file of the service control to be deployed is obtained; the target configuration information includes a service image identifier and container startup boot parameters; the service image identifier is used by the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier included in the received target configuration information after receiving the target configuration information; the container startup boot parameters are used to guide the running of the service control deployed according to the obtained installation program file on the physical node where the service control to be deployed needs to be deployed;

[0039] Deploy the service control to be deployed based on the obtained installation program file.

[0040] The method provided in this embodiment obtains the installation program file of the service control to be deployed based on the received target configuration information, and deploys the service control to be deployed based on the obtained installation program file. It does not require pre-arrangement of deployment schemes, realizes automatic construction of service controls, makes big data cluster deployment more convenient, saves manpower costs for big data cluster deployment, and improves the efficiency of big data cluster deployment.

[0041] In an optional embodiment, the service image identifier is further used to obtain preset dependency information of the service control to be deployed; the dependency information is the correspondence between the service control to be deployed and the service controls of the direct sub-layer of the service control to be deployed;

[0042] The service controls that are determined to have direct child layers in the deployed service controls include:

[0043] Based on the service image identifier included in the received target configuration information, obtain the dependency information of the service control to be deployed;

[0044] Based on the dependency information of the service control to be deployed, if it is determined that there is a service control in the direct sub-layer corresponding to the service control to be deployed, then it is determined that the deployed service control to be deployed has a service control in the direct sub-layer.

[0045] The method provided in this embodiment determines that when a deployed service control to be deployed has a directly subordinate service control, specifically, it obtains the dependency information of the service control to be deployed based on the service image identifier included in the received target configuration information. Based on the dependency information of the service control to be deployed, if it is determined that there is a directly subordinate service control corresponding to the service control to be deployed, then it is determined that the deployed service control to be deployed has a directly subordinate service control. Therefore, physical nodes can automatically identify the directly subordinate service controls corresponding to the service control to be deployed, and can further automatically request the deployment of the directly subordinate service controls corresponding to the service control to be deployed from the control node. This eliminates the need for pre-arranged deployment schemes, enabling automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs in big data cluster deployment, and improving the efficiency of big data cluster deployment.

[0046] In an optional embodiment, the method further includes:

[0047] After deploying any service control, the runtime interface information of the service control is sent to the control node so that the control node stores the runtime interface information of the service control.

[0048] If any of the service controls is detected to meet the preset conditions, then the service control is started and a deployment completion notification of the service control is sent to the control node; the preset conditions include any of the following: the service control does not have any directly subordinate service controls, or the service control has received the running interface information of the service controls of each of the directly subordinate service controls.

[0049] The method provided in this embodiment sends the runtime interface information of any service control to the control node when deploying any service control. When any service control has no directly subordinate service controls or when any service control has already received the runtime interface information of all its directly subordinate service controls, the method starts the service control and sends a deployment completion notification to the control node. This achieves automatic uploading of runtime interface information to the control node when deploying any service control, and starts the service control and sends a deployment completion notification to the control node when there are no directly subordinate service controls or when all directly subordinate service controls have been deployed. Therefore, there is no need to pre-arrange a deployment plan, enabling automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs, and improving the efficiency of big data cluster deployment.

[0050] Thirdly, embodiments of this application also provide a device for building a big data cluster, deployed at the control node of the big data cluster, wherein the big data cluster further includes at least one physical node that communicates with the control node; the device includes:

[0051] The information acquisition module is used to determine the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request, whenever it receives a service deployment request sent by a physical node; wherein, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has a direct child layer of service controls;

[0052] The node allocation module is used to send the target configuration information to the physical nodes that need to deploy the service control to be deployed, so that the physical nodes that receive the target configuration information can deploy the service control to be deployed.

[0053] In an optional embodiment, the device further includes:

[0054] The initial information acquisition module is used to respond to the initial service deployment request before receiving the service deployment request sent by the physical node, and determine the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request; the target service control is the service control selected by the user to be used, and has at least one service control of the direct sub-layer.

[0055] The initial node allocation module is used to send the configuration information of the target service control to the physical node that needs to deploy the target service control, so that the physical node that receives the configuration information of the target service control can deploy the target service control.

[0056] In an optional embodiment, the device further includes:

[0057] The initial prompt generation module is used to generate a deployment completion prompt for the big data cluster when the deployment completion notification of the target service control is received.

[0058] In one optional embodiment, the information acquisition module is specifically used for:

[0059] Based on the identifier of the service control to be deployed included in the service deployment request, the deployment parameters corresponding to the identifier of the service control to be deployed are obtained from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters.

[0060] Based on the deployment parameters corresponding to the identifier of the service control to be deployed, the physical node where the service control to be deployed needs to be deployed and the target configuration information of the service control to be deployed are determined. The target configuration information includes a service image identifier and container startup boot parameters. The service image identifier is used to enable the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier. The container startup boot parameters are used to boot and run the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

[0061] In an optional embodiment, the device further includes:

[0062] The interface information transmission module is used to send the running interface information of the service control of the direct sub-layer of any service control to the physical node that deployed the service control if it receives a deployment completion notification of the service control of the direct sub-layer of any service control, so that the physical node starts the service control based on the running interface information and sends the deployment completion notification of the service control to the control node after starting.

[0063] Fourthly, embodiments of this application also provide a device for building a big data cluster, deployed on the physical nodes of the big data cluster, wherein the big data cluster further includes a control node that communicates with the physical nodes; the device includes:

[0064] The control deployment module is used to deploy the service control to be deployed according to the target configuration information received from the control node each time it receives the target configuration information of the service control to be deployed.

[0065] The sub-layer deployment request module is used to, if it is determined that the service control to be deployed has a service control with a direct sub-layer, place the identifier of the service control with the direct sub-layer in the service deployment request, and send the service deployment request to the control node, so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request.

[0066] In an optional embodiment, the device further includes:

[0067] The initial control deployment module is used to ensure that, before receiving the target configuration information of the service control to be deployed from the control node, the physical node that needs to deploy the target service control receives the configuration information of the target service control sent by the control node; the target service control is a service control selected by the user and has at least one service control of the directly subordinate sub-layer; and the target service control is deployed according to the received configuration information of the target service control.

[0068] The initial sub-layer deployment request module is used to, if it is determined that the target service control has a directly subordinate sub-layer service control, place the identifier of the directly subordinate sub-layer service control in the service deployment request and send it to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate sub-layer service control of the target service control.

[0069] In an optional embodiment, the initial sub-layer deployment request module is further configured to:

[0070] If the runtime interface information of the service controls of each of the direct sub-layers of the target service control is received, the target service control is started, and a deployment completion notification of the target service control is sent to the control node.

[0071] In an optional embodiment, the control deployment module is specifically used for:

[0072] Based on the received target configuration information, the installation program file of the service control to be deployed is obtained; the target configuration information includes a service image identifier and container startup boot parameters; the service image identifier is used by the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier included in the received target configuration information after receiving the target configuration information; the container startup boot parameters are used to guide the running of the service control deployed according to the obtained installation program file on the physical node where the service control to be deployed needs to be deployed;

[0073] Deploy the service control to be deployed based on the obtained installation program file.

[0074] In an optional embodiment, the service image identifier is further used to obtain preset dependency information of the service control to be deployed; the dependency information is the correspondence between the service control to be deployed and the service controls of its direct sub-layers; the sub-layer deployment request module is further used to:

[0075] Based on the service image identifier included in the received target configuration information, obtain the dependency information of the service control to be deployed;

[0076] Based on the dependency information of the service control to be deployed, if it is determined that there is a service control in the direct sub-layer corresponding to the service control to be deployed, then it is determined that the deployed service control to be deployed has a service control in the direct sub-layer.

[0077] In an optional embodiment, the sub-layer deployment request module is further configured to:

[0078] After deploying any service control, the runtime interface information of the service control is sent to the control node so that the control node stores the runtime interface information of the service control.

[0079] If any of the service controls is detected to meet the preset conditions, then the service control is started and a deployment completion notification of the service control is sent to the control node; the preset conditions include any of the following: the service control does not have any directly subordinate service controls, or the service control has received the running interface information of the service controls of each of the directly subordinate service controls.

[0080] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for constructing a big data cluster according to the first aspect; or implements the method for constructing a big data cluster according to the second aspect.

[0081] Sixthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it enables the implementation of the method for building a big data cluster of the first aspect; or the implementation of the method for building a big data cluster of the second aspect.

[0082] The technical effects of any of the implementation methods in aspects three through six can be found in the technical effects of the corresponding implementation methods in aspects one or two, and will not be repeated here. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1This application provides a schematic diagram of the structure of a big data cluster.

[0085] Figure 2 An interactive diagram illustrating a method for constructing a big data cluster, as provided in an embodiment of this application;

[0086] Figure 3 A flowchart illustrating a method for constructing a big data cluster applied to a control node, as provided in an embodiment of this application;

[0087] Figure 4 A flowchart illustrating a method for building a big data cluster applied to physical nodes, provided in an embodiment of this application;

[0088] Figure 5 A schematic diagram of a device for building a big data cluster deployed at a control node, provided in an embodiment of this application;

[0089] Figure 6 A schematic diagram of the structure of a device for building a big data cluster that can deploy user-selected services and is located on a control node, provided in an embodiment of this application;

[0090] Figure 7 A schematic diagram of the structure of a big data cluster building device that can generate a cluster deployment completion notification and is deployed on a control node, provided in an embodiment of this application;

[0091] Figure 8 A schematic diagram of the structure of a big data cluster building device that can automatically transmit interface information and is deployed on a control node, provided in an embodiment of this application;

[0092] Figure 9 A schematic diagram of a device for building a big data cluster deployed on a physical node, provided in an embodiment of this application;

[0093] Figure 10 A schematic diagram of the structure of a device for building a big data cluster deployed on a physical node, which can be used to deploy user-selected services, as provided in an embodiment of this application;

[0094] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0096] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0097] The following are explanations of some of the words that appear in the text:

[0098] (1) Container: A container is a lightweight, operating system-level virtualization technology that allows applications and their dependencies to run in a resource-isolated environment. All the necessary components required to run an application are packaged into a single, reusable image. When the image runs, it operates in an isolated environment and does not share the host operating system's memory, CPU, or disk with other applications. This ensures that processes inside the container do not affect any processes outside the container.

[0099] (2) Image: An image refers to an image file. Similar to a ZIP archive, an image file is a single file created by combining a series of specific files in a certain format for easy download and use by users. Examples include a test version of an operating system or game. Image files not only have the "combination" function of ZIP archives, but their most important feature is that they can be recognized by specific software and directly burned to a CD. The concept of an image file can be expanded further; it can contain more information, such as system files, boot files, and partition table information. Thus, an image file can contain all the information of a partition or even a hard drive. A classic software for using this type of image file is Ghost, which also has burning capabilities. However, its burning function only saves the image file itself to the CD, while conventional burning software can directly burn the contents of a supported image file to a CD.

[0100] (3) Agent program: An agent program is a type of autonomous computing entity or program that can perceive the environment under a specific environment and can run flexibly and autonomously to achieve a series of design goals.

[0101] (4) Dependency Relationship: A dependency relationship refers to the requirement for a service control to read the runtime interface information of other service controls when it starts up. Runtime interface information is typically the network information of the container where the service control resides, such as an Internet Protocol (IP) address and domain name. In some embodiments, the runtime interface information also includes the container's namespace, which is the identifier information representing the big data cluster to which the container belongs. In the embodiments of this application, a service control has a dependency relationship with other service controls, indicating that when the service control starts up, it needs to read the runtime interface information of the other service controls. For example, a service control of a business cluster contains a dependency relationship "H: I, J, K", indicating that when service control H starts up, it needs to sequentially read the runtime interface information of service controls I, J, and K.

[0102] Currently, big data platforms are mostly clustered, with the entire platform containing multiple components, such as Hadoop and Zookeeper, and each component typically contains a varying number of services.

[0103] Traditional deployment of big data clusters typically requires planning the network and deployment scheme, then installing the operating system, services, and configuring parameters on physical devices—a process that is extremely cumbersome and complex. Even with cluster management tools, users still need to plan the components to be deployed according to their needs and then deploy each component following a wizard.

[0104] When deploying big data clusters, related technologies typically require pre-arranged deployment plans, with the cluster built from the bottom up based on service dependencies. This deployment process is complex and labor-intensive, resulting in low efficiency for big data cluster deployment.

[0105] To address existing technical problems, this application provides a method for building a big data cluster. During the deployment of the big data cluster, upon receiving a service deployment request from a physical node, the target configuration information of the service control to be deployed is determined based on the identifier of the service control to be deployed included in the service deployment request. The service deployment request is sent by the physical node to the control node when it determines that a deployed service control has directly subordinate service controls. The target configuration information is then sent to the physical node that needs to deploy the service control, enabling the physical node receiving the target configuration information to deploy the service control. This method determines the target configuration information of the service control to be deployed upon receiving a service deployment request from a physical node and sends it to the physical node that needs to deploy the service control, enabling the physical node receiving the target configuration information to deploy the service control. This eliminates the need for pre-arranged deployment plans, achieving automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs, and improving the efficiency of big data cluster deployment.

[0106] The technical solutions provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0107] This application provides an overall structure for a big data cluster, such as... Figure 1 As shown, it includes a control node 100 and at least one physical node 200. All component services run in containers, each container residing on a physical node. In this embodiment, the component service may also be referred to as a service control or a big data component service. Each container runs a component service and an agent program, where the agent program is responsible for interacting with the control node and requesting the deployment of dependent service functions from the control node. The control node manages the deployment templates of each component service and is responsible for service scheduling and deployment. Understandably, the number of containers on a physical node can be one or more, and this application does not specifically limit this.

[0108] In one embodiment, the method for building a big data cluster provided in this application can be by... Figure 1 The control node 100 and the physical nodes work together to execute the commands. Figure 2 This application provides an embodiment of an interaction diagram between a control node and a physical node, as shown in the diagram. Figure 2 As shown, the method may include the following steps:

[0109] Step S201: The control node responds to the initial service deployment request and determines the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request.

[0110] The target service control is the service control selected by the user that needs to be used, and has at least one direct child service control.

[0111] For example, a big data cluster needs to use the Hive service control, and the dependencies of the relevant service controls are as follows:

[0112] Hive: Spark, Hadoop, Kafka, Zookeeper;

[0113] Spark: Hadoop;

[0114] Hadoop: No dependencies;

[0115] Kafka: No dependencies;

[0116] Zookeeper: No dependencies.

[0117] The user selects the Hive service control for deployment on control node 100, generating an initial service deployment request Re1. Control node 100 responds to the initial service deployment request Re1, determining the configuration information of the target service control based on the identifier "Hive" included in the initial service deployment request Re1. The target service control is the Hive service control selected by the user. The Hive service control has multiple directly subordinate service controls, including: Spark, Hadoop, Kafka, and Zookeeper.

[0118] In one optional embodiment, the configuration information of the target service control is determined based on the identifier of the target service control included in the initial service deployment request, through the following steps:

[0119] Step F1: Based on the identifier of the target service control included in the initial service deployment request, obtain the deployment parameters corresponding to the identifier of the target service control from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters.

[0120] In the embodiments of this application, deployment configuration information, image configuration files, and image files of service controls can be pre-created. The image files contain service component programs and agent programs; the image configuration files declare the dependency information of the service controls; the deployment configuration information includes, but is not limited to, the following: service image identifier, container startup boot parameters, persistent storage configuration of the big data cluster, network configuration, and service deployment strategy. The service deployment strategy can declare that when the control node receives a deployment request for a service control, it first checks whether the big data cluster has already deployed the service control. If the service control has been deployed, it directly returns the runtime interface information of the deployed service control to the physical node that issued the deployment request. The deployment configuration information is uploaded to the control node after creation; the image configuration files and image files of the service controls can be stored on the control node or in a separate database.

[0121] In some embodiments, the control node has multiple caches. For example, one cache may be used to store deployment settings information, and another cache may be used to store runtime interface information uploaded by physical nodes.

[0122] Step F2: Based on the deployment parameters corresponding to the identifier of the target service control, determine the physical node where the target service control needs to be deployed and the configuration information of the target service control. The configuration information includes the service image identifier and container startup boot parameters. The service image identifier is used to enable the physical node to obtain the installation program file of the target service control based on the service image identifier. The container startup boot parameters are used to boot and run the target service control deployed according to the obtained installation program file on the physical node where the target service control needs to be deployed.

[0123] The installation program file for the service control can be a mirror file of the service control.

[0124] In step S202, the control node sends the configuration information of the target service control to the physical node where the target service control needs to be deployed.

[0125] For example, control node 100 sends the configuration information of the service control Hive to the physical node that needs to deploy the service control Hive, such as physical node 1.

[0126] Step S203: The physical node receives the configuration information of the target service control sent by the control node.

[0127] The target service control is the service control selected by the user that needs to be used, and has at least one direct sub-layer service control.

[0128] In practice, the physical node that deploys the target service control needs to receive the configuration information of the target service control sent by the control node; the target service control is the service control selected by the user and has at least one direct sub-layer service control.

[0129] For example, physical node 1 receives configuration information for the service control Hive sent by control node 100.

[0130] Step S204: The physical node deploys the target service control according to the configuration information of the received target service control.

[0131] Specifically, the physical node that needs to deploy the target service control can deploy the target service control based on the configuration information received from the target service control.

[0132] For example, physical node 1 receives the configuration information of the received service control Hive, deploys the service control Hive, and obtains container 1. Container 1 includes not only the service control Hive but also an Agent program. The Agent program is responsible for interacting with the control node and requesting the deployment of service controls in the direct sub-layer corresponding to the service controls in the container where the Agent program resides.

[0133] In step S205, if the physical node determines that the target service control has directly subordinate service controls, it places the identifier of the directly subordinate service control of the target service control in the service deployment request and sends it to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate service control of the target service control.

[0134] In practice, the physical node determines whether the target service control has a direct sub-service control. If it is determined that the target service control has a direct sub-service control, the identifier of the direct sub-service control of the target service control is placed in the service deployment request and sent to the control node, so that the control node notifies the corresponding physical node to deploy the direct sub-service control of the target service control.

[0135] For example, if physical node 1 determines that the service control Hive has directly subordinate service controls, namely service control Spark, service control Hadoop, service control Kafka, and service control Zookeeper, then it will place the identifiers of the directly subordinate service controls of service control Hive in the service deployment request and send it to control node 100, so that control node 100 will notify the corresponding physical node to deploy the directly subordinate service controls of service control Hive: service control Spark, service control Hadoop, service control Kafka, and service control Zookeeper.

[0136] In an optional embodiment, if a physical node determines that a target service control has directly subordinate service controls, it places the identifiers of the directly subordinate service controls of the target service control in the service deployment request one by one in a preset order and sends them to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate service controls of the target service control one by one, thereby realizing the deployment of the directly subordinate service controls of the target service control one by one in a preset order.

[0137] Optionally, the default order is the order of service controls in the direct child layer of the target service control in the dependency relationship.

[0138] For example, because the service control Hive has the dependency "Hive: Spark, Hadoop, Kafka, Zookeeper;", the default order is to deploy the service control Spark, the service control Hadoop, the service control Kafka, and the service control Zookeeper in sequence.

[0139] Step S206: Each time the control node receives a service deployment request sent by a physical node, it determines the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request.

[0140] Among them, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has a direct child service control.

[0141] For example, after physical node 1 determines that the direct sub-service controls of service control Hive are service control Spark, service control Hadoop, service control Kafka, and service control Zookeeper, it places the identifiers of the aforementioned direct sub-service controls of service control Hive in a service deployment request and sends it to control node 100. For instance, it places the identifier of service control Spark, a direct sub-service control of service control Hive, in service deployment request Re2 and sends it to control node 100. Understandably, the control node can receive service deployment requests for service control Spark, service control Hadoop, service control Kafka, and service control Zookeeper sequentially. Each time the control node receives a service deployment request from a physical node, it determines the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request. That is, the control node determines the corresponding target configuration information according to the order of service control Spark, service control Hadoop, service control Kafka, and service control Zookeeper. The following example illustrates the process of deploying service control Spark after the control node determines the target configuration information of service control Spark.

[0142] In an optional embodiment, determining the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request can be achieved through the following steps:

[0143] Step A1: Based on the identifier of the service control to be deployed included in the service deployment request, obtain the deployment parameters corresponding to the identifier of the service control to be deployed from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters.

[0144] Step A2: Based on the deployment parameters corresponding to the identifier of the service control to be deployed, determine the physical node where the service control to be deployed needs to be deployed and the target configuration information of the service control to be deployed. The target configuration information includes the service image identifier and container startup boot parameters. The service image identifier is used to enable the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier. The container startup boot parameters are used to boot and run the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

[0145] In step S207, the control node sends the target configuration information to the physical node that needs to deploy the service control to be deployed, so that the physical node that receives the target configuration information can deploy the service control to be deployed.

[0146] For example, the control node sends the target configuration information of the service control Spark to the physical node that needs to deploy the service control Spark, assuming it is physical node 2, so that physical node 2 that receives the target configuration information of the service control Spark can deploy the service control Spark.

[0147] In step S208, each time a physical node receives target configuration information for a service control to be deployed from a control node, it deploys the service control to be deployed according to the received target configuration information.

[0148] For example, physical node 2 receives the target configuration information of the service control Spark sent by the control node, and deploys the service control Spark according to the target configuration information of the service control Spark.

[0149] In one optional embodiment, the service control to be deployed is deployed according to the received target configuration information through the following steps:

[0150] Step B1: Obtain the installation program file of the service control to be deployed based on the received target configuration information.

[0151] The target configuration information includes a service image identifier and container startup parameters. The service image identifier is used by the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier included in the received target configuration information. The container startup parameters are used to guide the execution of the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

[0152] Step B2: Deploy the service control to be deployed based on the obtained installation program file.

[0153] In one alternative embodiment, after deploying any service control, the physical node sends the runtime interface information of any service control to the control node, so that the control node stores the runtime interface information of any service control.

[0154] For example, after deploying the service control Spark, physical node 2 sends the service control Spark runtime interface information Run_Spark_1 to control node 100 so that control node 100 can store the service control Spark runtime interface information Run_Spark_1.

[0155] In step S209, if a physical node determines that the service control to be deployed has a directly subordinate service control, it places the identifier of the directly subordinate service control in the service deployment request and sends the service deployment request to the control node, so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request.

[0156] For example, after deploying the service control Spark, physical node 2 determines through the Agent program in the container where the service control Spark resides that the service control Spark has a direct sub-layer service control Hadoop. Then, it places the identifier "Hadoop" of the service control Hadoop in the service deployment request Re3 and sends the service deployment request Re3 to control node 100, so that control node 100 can notify the corresponding physical node to deploy the service control Hadoop corresponding to the identifier "Hadoop" included in the service deployment request Re3 by executing the steps described in S206 to S207. Assume that the physical node is physical node i.

[0157] After receiving the service deployment request Re3, physical node i executes the steps as described in step S208 to deploy the service control Hadoop corresponding to the identifier "Hadoop" included in the service deployment request Re3, and sends the runtime interface information Run_Hadoop_1 of the service control Hadoop to the control node through the Agent program of the container where the service control Hadoop resides. Physical node i also determines, according to step S209, whether the service control Hadoop has any directly subordinate service controls. Because the service control Hadoop does not have any directly subordinate service controls, physical node i does not execute the action of instructing the control node to notify the corresponding physical node to deploy the directly subordinate service controls of the service control Hadoop.

[0158] In an optional embodiment, the service image identifier is further used to obtain preset dependency information of the service control to be deployed; the dependency information is the correspondence between the service control to be deployed and the service controls of its direct child layers; determining that the deployed service control has direct child layer service controls is achieved by performing the following steps:

[0159] Step C1: Based on the service image identifier included in the received target configuration information, obtain the dependency information of the service control to be deployed.

[0160] For example, physical node 2 can determine that the service control Spark has a direct sub-layer service control Hadoop by obtaining the service control Spark's dependency information "Spark: Hadoop" based on the service image identifier "Spark" included in the received target configuration information.

[0161] Step C2: Based on the dependency information of the service control to be deployed, if it is determined that there is a service control with a direct sub-layer corresponding to the service control to be deployed, then it is determined that the deployed service control to be deployed has a service control with a direct sub-layer.

[0162] For example, physical node 2 determines that there is a service control Hadoop that is a direct sub-layer of the service control Spark based on the service control Spark's dependency information "Spark:Hadoop". Therefore, it determines that the deployed service control Spark has a direct sub-layer of service controls.

[0163] Step S210: If a physical node detects that any service control meets the preset conditions, then the service control is started.

[0164] The preset conditions include any of the following: any service control does not have any directly subordinate service controls, or any service control has received the runtime interface information of each of the directly subordinate service controls of any service control.

[0165] For example, physical node i detects that the service control Hadoop does not have any directly subordinate service controls, which meets the preset conditions, and therefore starts the service control Hadoop.

[0166] Step S211: After starting any service control, the physical node sends a deployment completion notification for any service control to the control node.

[0167] In practice, after any physical node starts any service control, it sends a notification to the control node that the deployment of any service control is complete.

[0168] For example, after physical node i starts the Hadoop service control, it sends a Hadoop deployment completion notification Mes.1 to the control node.

[0169] In step S212, if the control node receives a deployment completion notification from a service control in a direct sub-layer of any service control, it sends the runtime interface information of the service control in the direct sub-layer of any service control to the physical node that deployed the service control, so that the physical node starts the service control based on the runtime interface information and sends a deployment completion notification to the control node after starting the service control.

[0170] For example, when control node 100 receives the deployment completion notification Mes.1 of service control Hadoop, which is a direct sub-layer of service control Spark, it sends the runtime interface information Run_Hadoop_1 of service control Hadoop to the physical node 2 that deploys service control Spark, so that physical node 2 starts service control Spark based on the runtime interface information and sends the deployment completion notification of service control Spark to control node after service control Spark starts.

[0171] Step S213: If the physical node receives the running interface information of the service controls of each direct sub-layer of the target service control, then the target service control is started.

[0172] For example, when physical node 2 receives the runtime interface information Run_Hadoop_1 of the service control Hadoop, it also receives the runtime interface information of the service controls of each direct sub-layer of the service control Spark. Therefore, physical node 2 starts the service control Spark based on the runtime interface information of the service controls of each direct sub-layer of the service control Spark, and sends a deployment completion notification Mes.2 of the service control Spark to the control node. Subsequently, control node 100 receives the deployment completion notification Mes.2 of the service control Spark, a direct sublayer of the service control Hive. It then sends the runtime interface information Run_Spark_1 of the service control Spark to the physical node 1 that deployed the service control Hive. Similarly, physical node 1 will also request the deployment of the following service controls in sequence according to the dependencies of the service control Hive: service control Hadoop, service control Kafka, and service control Zookeeper. It will also receive the runtime interface information of each deployed service control. When physical node 1 receives the runtime interface information of service controls Spark, Hadoop, Kafka, and Zookeeper, it has received the runtime interface information of each direct sublayer of the service control Hive, and can therefore start the service control Hive based on the runtime interface information.

[0173] Step S214: After the physical node starts the target service control, the physical node sends a deployment completion notification of the target service control to the control node.

[0174] For example, after physical node 1 starts the service control Hive, physical node 1 sends a notification to the control node that the deployment of the service control Hive is complete.

[0175] In an optional embodiment, after receiving the runtime interface information Run_Spark_1 of the service control Spark, physical node 1 places the identifier of the service control Hadoop, the direct sub-layer of service control Hive, in the service deployment request according to the dependency relationship of service control Hive, and sends it to control node 100. At this time, service control Hadoop has already been deployed based on the dependency relationship of service control Spark after the deployment of service control Spark. Then, control node 100 directly sends the runtime interface information Run_Hadoop_1 of service control Hadoop to physical node 1.

[0176] Similarly, after receiving the runtime interface information Run_Hadoop_1 from the service control Hadoop, physical node 1, based on the dependency relationship of the service control Hive, includes the identifier of the service control Kafka, a direct sub-layer of the service control Hive, in the service deployment request and sends it to control node 100. At this time, the service control Kafka has not yet been deployed, and the process of the control node causing the physical node to deploy the service control Kafka is executed. Subsequently, after receiving the runtime interface information of the service control Kafka, physical node 1, based on the dependency relationship of the service control Hive, includes the identifier of the service control Zookeeper, a direct sub-layer of the service control Hive, in the service deployment request and sends it to control node 100. At this time, the service control Zookeeper has not yet been deployed, and the process of the control node causing the physical node to deploy the service control Zookeeper is executed.

[0177] Step S215: When the control node receives the deployment completion notification of the target service control, it generates a deployment completion prompt for the big data cluster.

[0178] For example, when the control node receives a deployment completion notification from the service control Hive, it generates a deployment completion prompt for the big data cluster.

[0179] The big data cluster construction method provided in this application involves the control node, upon receiving a service deployment request from a physical node during the deployment process, determining the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request. The service deployment request is sent by the physical node to the control node when it determines that a deployed service control has directly subordinate service controls. The target configuration information is then sent to the physical node that needs to deploy the service control, enabling the physical node receiving the target configuration information to deploy the service control. This method, by having the control node determine the target configuration information of the service control to be deployed upon receiving a service deployment request from a physical node and sending it to the physical node that needs to deploy the service control, allows the physical node receiving the target configuration information to deploy the service control. This eliminates the need for pre-arranged deployment plans, achieving automatic construction of service controls, making big data cluster deployment more convenient, saving manpower costs, and improving the efficiency of big data cluster deployment.

[0180] While this application provides operational steps for the methods shown in the above embodiments or accompanying drawings, more or fewer operational steps may be included in the above methods based on conventional methods or without inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the above methods, they may be executed sequentially or in parallel as shown in the embodiments or accompanying drawings.

[0181] In the above embodiments, the big data cluster includes a control node and at least one physical node that communicates with the control node. The control node of the big data cluster executes a method for building the big data cluster, such as... Figure 3 As shown, the following steps may be included:

[0182] Step S301: Upon receiving a service deployment request from a physical node, determine the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request.

[0183] Among them, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has a direct child service control.

[0184] Step S302: Send the target configuration information to the physical node that needs to deploy the service control to be deployed, so that the physical node that receives the target configuration information can deploy the service control to be deployed.

[0185] In other embodiments, the method further includes:

[0186] Before receiving a service deployment request from a physical node, respond to an initial service deployment request and determine the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request; the target service control is the service control selected by the user that needs to be used and has at least one direct sub-layer service control;

[0187] The configuration information of the target service control is sent to the physical node where the target service control needs to be deployed, so that the physical node that receives the configuration information of the target service control can deploy the target service control.

[0188] In some embodiments, during the deployment of a big data cluster according to the above method, when a deployment completion notification of the target service control is received, a deployment completion prompt for the big data cluster is generated.

[0189] In other embodiments, based on the identifier of the service control to be deployed included in the service deployment request, the target configuration information of the service control to be deployed is determined, including:

[0190] Based on the identifier of the service control to be deployed included in the service deployment request, the deployment parameters corresponding to the identifier of the service control to be deployed are obtained from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters.

[0191] Based on the deployment parameters corresponding to the identifier of the service control to be deployed, determine the physical node where the service control to be deployed needs to be deployed and the target configuration information of the service control to be deployed. The target configuration information includes the service image identifier and container startup boot parameters. The service image identifier is used to enable the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier. The container startup boot parameters are used to boot and run the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

[0192] In other embodiments, the method further includes:

[0193] If a deployment completion notification is received for a service control in a direct sublayer of any service control, the runtime interface information of the service control in the direct sublayer of any service control is sent to the physical node that deployed the service control, so that the physical node starts the service control based on the runtime interface information and sends a deployment completion notification for the service control to the control node after starting.

[0194] In the above embodiments, the big data cluster includes a control node and at least one physical node that communicates with the control node. The physical node of the big data cluster executes a method for building the big data cluster, such as... Figure 4 As shown, the following steps may be included:

[0195] Step S401: Upon receiving target configuration information for the service control to be deployed from the control node, deploy the service control to be deployed according to the received target configuration information.

[0196] Step S402: If it is determined that the service control to be deployed has a directly subordinate service control, then the identifier of the directly subordinate service control is placed in the service deployment request, and the service deployment request is sent to the control node so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request.

[0197] In some embodiments, the above method further includes:

[0198] Before receiving the target configuration information of the service control to be deployed from the control node, the physical node that needs to deploy the target service control needs to receive the configuration information of the target service control from the control node; the target service control is the service control selected by the user and has at least one direct child layer service control.

[0199] Deploy the target service control based on the received configuration information of the target service control;

[0200] If it is determined that the target service control has directly subordinate service controls, then the identifier of the directly subordinate service control of the target service control is placed in the service deployment request and sent to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate service control of the target service control.

[0201] Optionally, after sending the service deployment request containing the identifier of the service control of the direct sub-layer of the target service control to the control node, the above method further includes:

[0202] If the runtime interface information of the service controls of each direct sub-layer of the target service control is received, the target service control is started, and a deployment completion notification of the target service control is sent to the control node.

[0203] In some embodiments, deploying the service control to be deployed based on the received target configuration information includes:

[0204] Based on the received target configuration information, obtain the installer file of the service control to be deployed; the target configuration information includes the service image identifier and container startup boot parameters; the service image identifier is used by the physical node to obtain the installer file of the service control to be deployed based on the service image identifier included in the received target configuration information; the container startup boot parameters are used to boot and run the service control deployed based on the obtained installer file on the physical node where the service control to be deployed needs to be deployed.

[0205] Deploy the service controls to be deployed based on the obtained installer file.

[0206] Optionally, the service image identifier is also used to obtain the preset dependency information of the service control to be deployed; the dependency information is the correspondence between the service control to be deployed and the service controls of the direct sub-layer of the service control to be deployed.

[0207] The deployed service controls to be deployed have direct child controls, including:

[0208] Based on the service image identifier included in the received target configuration information, obtain the dependency information of the service control to be deployed;

[0209] Based on the dependency information of the service control to be deployed, if it is determined that there is a service control with a direct sub-layer corresponding to the service control to be deployed, then it is determined that the deployed service control to be deployed has a service control with a direct sub-layer.

[0210] In some embodiments, the above method further includes:

[0211] After deploying any service control, send the runtime interface information of any service control to the control node so that the control node can store the runtime interface information of any service control.

[0212] If any service control is detected to meet the preset conditions, then the service control is started and a deployment completion notification for the service control is sent to the control node. The preset conditions include any of the following: the service control does not have any directly subordinate service controls, or the service control has received the running interface information of the service controls of each of its directly subordinate service controls.

[0213] The method provided in the above embodiments can automatically build service controls without pre-arranged deployment plans, making big data cluster deployment more convenient, thereby saving manpower costs and improving the efficiency of big data cluster deployment.

[0214] Based on the same inventive concept, this application also provides a device for building a big data cluster, which can be deployed in a control node. Since this device corresponds to the method for building a big data cluster applied to a control node in this application, and the principle by which this device solves the problem is similar to that of the method, the implementation of this device can refer to the implementation process of the above method embodiments, and repeated details will not be elaborated further.

[0215] Figure 5 This illustration shows a schematic diagram of a big data cluster building apparatus according to an embodiment of this application. The big data cluster building apparatus is deployed at the control node of the big data cluster. The big data cluster also includes at least one physical node that communicates with the control node, such as... Figure 5 As shown, it includes: an information acquisition module 501 and a node allocation module 502; wherein,

[0216] The information acquisition module 501 is used to determine the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request when it receives a service deployment request sent by a physical node; wherein, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has a direct child layer of service controls;

[0217] The node allocation module 502 is used to send target configuration information to the physical nodes that need to deploy the service control to be deployed, so that the physical nodes that receive the target configuration information can deploy the service control to be deployed.

[0218] In one alternative embodiment, such as Figure 6 As shown, the device also includes:

[0219] The initial information acquisition module 601 is used to respond to the initial service deployment request before receiving the service deployment request sent by the physical node, and determine the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request; the target service control is the service control selected by the user to be used, and has at least one direct sub-layer service control;

[0220] The initial node allocation module 602 is used to send the configuration information of the target service control to the physical node that needs to deploy the target service control, so that the physical node that receives the configuration information of the target service control can deploy the target service control.

[0221] In one alternative embodiment, such as Figure 7 As shown, the device also includes:

[0222] The initial prompt generation module 701 is used to generate a deployment completion prompt for the big data cluster when a deployment completion notification is received from the target service control.

[0223] In an optional embodiment, the information acquisition module 501 is specifically used for:

[0224] Based on the identifier of the service control to be deployed included in the service deployment request, the deployment parameters corresponding to the identifier of the service control to be deployed are obtained from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters.

[0225] Based on the deployment parameters corresponding to the identifier of the service control to be deployed, determine the physical node where the service control to be deployed needs to be deployed and the target configuration information of the service control to be deployed. The target configuration information includes the service image identifier and container startup boot parameters. The service image identifier is used to enable the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier. The container startup boot parameters are used to boot and run the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

[0226] In one alternative embodiment, such as Figure 8 As shown, the device also includes:

[0227] The interface information transmission module 801 is used to send the running interface information of the service control of the direct sub-layer of any service control to the physical node that deployed the service control if it receives a deployment completion notification of the service control of the direct sub-layer of any service control, so that the physical node starts the service control based on the running interface information and sends a deployment completion notification of the service control to the control node after starting.

[0228] Based on the same inventive concept, this application also provides a device for building a big data cluster, which can be deployed in physical nodes. Since this device corresponds to the method for building a big data cluster on physical nodes in this application, and the principle by which this device solves the problem is similar to that of the method, the implementation of this device can refer to the implementation process of the above method embodiments, and repeated details will not be elaborated further.

[0229] Figure 9 This illustration shows a structural diagram of a big data cluster building device according to an embodiment of this application. The big data cluster building device is deployed at the physical nodes of the big data cluster, which also includes a control node that communicates with the physical nodes. The device includes:

[0230] The control deployment module 901 is used to deploy the service control to be deployed according to the target configuration information received from the control node each time it receives the target configuration information of the service control to be deployed.

[0231] The sub-layer deployment request module 902 is used to, if it is determined that the service control to be deployed has a service control with a direct sub-layer, place the identifier of the service control with the direct sub-layer in the service deployment request and send the service deployment request to the control node, so that the control node notifies the corresponding physical node to deploy the service control with the identifier included in the service deployment request.

[0232] In one alternative embodiment, such as Figure 10 As shown, the device also includes:

[0233] The initial control deployment module 1001 is used to receive the target configuration information of the service control to be deployed sent by the control node. Before receiving the target configuration information of the service control to be deployed sent by the control node, the physical node that needs to deploy the target service control receives the configuration information of the target service control sent by the control node. The target service control is the service control selected by the user and has at least one direct child layer. The target service control is deployed according to the received configuration information of the target service control.

[0234] The initial sub-layer deployment request module 1002 is used to, if it is determined that the target service control has a direct sub-layer service control, place the identifier of the direct sub-layer service control of the target service control in the service deployment request and send it to the control node, so that the control node notifies the corresponding physical node to deploy the direct sub-layer service control of the target service control.

[0235] In an optional embodiment, the initial sub-layer deployment request module 1002 is further configured to:

[0236] If the runtime interface information of the service controls of each direct sub-layer of the target service control is received, the target service control is started, and a deployment completion notification of the target service control is sent to the control node.

[0237] In an optional embodiment, the control deployment module 901 is specifically used for:

[0238] Based on the received target configuration information, obtain the installer file of the service control to be deployed; the target configuration information includes the service image identifier and container startup boot parameters; the service image identifier is used by the physical node to obtain the installer file of the service control to be deployed based on the service image identifier included in the received target configuration information; the container startup boot parameters are used to boot and run the service control deployed based on the obtained installer file on the physical node where the service control to be deployed needs to be deployed.

[0239] Deploy the service controls to be deployed based on the obtained installer file.

[0240] In an optional embodiment, the service image identifier is further used to obtain preset dependency information of the service control to be deployed; the dependency information is the correspondence between the service control to be deployed and the service controls of its direct sub-layers; the sub-layer deployment request module 902 is further used to:

[0241] Based on the service image identifier included in the received target configuration information, obtain the dependency information of the service control to be deployed;

[0242] Based on the dependency information of the service control to be deployed, if it is determined that there is a service control with a direct sub-layer corresponding to the service control to be deployed, then it is determined that the deployed service control to be deployed has a service control with a direct sub-layer.

[0243] In an optional embodiment, the sub-layer deployment request module 902 is further configured to:

[0244] After deploying any service control, send the runtime interface information of any service control to the control node so that the control node can store the runtime interface information of any service control.

[0245] If any service control is detected to meet the preset conditions, then the service control is started and a deployment completion notification for the service control is sent to the control node. The preset conditions include any of the following: the service control does not have any directly subordinate service controls, or the service control has received the running interface information of the service controls of each of its directly subordinate service controls.

[0246] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. This electronic device can be used to build a big data cluster. In one embodiment, the electronic device can be a server, a terminal device, or other electronic equipment. In this embodiment, the structure of the electronic device can be as follows: Figure 11As shown, it includes a memory 101, a communication module 103, and one or more processors 102.

[0247] The memory 101 is used to store computer programs executed by the processor 102. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0248] Memory 101 may be volatile memory, such as random-access memory (RAM); memory 101 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 101 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 101 may be a combination of the above-described memories.

[0249] The processor 102 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 102 is used to implement the above-described method for building a big data cluster when it calls a computer program stored in the memory 101.

[0250] The communication module 103 is used to communicate with terminal devices and other servers.

[0251] This application does not limit the specific connection medium between the memory 101, communication module 103, and processor 102 described above. This disclosure embodiment... Figure 11 The memory 101 and the processor 102 are connected via a bus 104, and the bus 104 is in Figure 11 The connections between other components are shown in thick lines only and are not intended to be limiting. Bus 104 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0252] This application also provides a computer storage medium storing computer-executable instructions for implementing the method for building a big data cluster according to any embodiment of this application.

[0253] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for constructing a big data cluster as described in the above embodiments. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0254] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for constructing a big data cluster, characterized in that, The method is applied to a control node of the big data cluster, the big data cluster further including at least one physical node that communicates with the control node; the method includes: Upon receiving a service deployment request from a physical node, the target configuration information of the service control to be deployed is determined based on the identifier of the service control to be deployed included in the service deployment request; wherein, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has a direct sub-layer of service controls; the service control to be deployed is a direct sub-layer of the deployed service control. The target configuration information is sent to the physical node that needs to deploy the service control to be deployed, so that the physical node that receives the target configuration information can deploy the service control to be deployed; The determination of the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request includes: Based on the identifier of the service control to be deployed included in the service deployment request, the deployment parameters corresponding to the identifier of the service control to be deployed are obtained from the preset deployment settings information; the deployment settings information is the correspondence between the identifier of the service control and the deployment parameters. Based on the deployment parameters corresponding to the identifier of the service control to be deployed, determine the physical node where the service control to be deployed needs to be deployed and the target configuration information of the service control to be deployed. The method further includes: Before receiving a service deployment request sent by a physical node, respond to an initial service deployment request and determine the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request; the target service control is the service control selected by the user to be used, and has at least one service control of the directly subordinate sub-layer. The configuration information of the target service control is sent to the physical node that needs to deploy the target service control, so that the physical node that receives the configuration information of the target service control can deploy the target service control.

2. The method according to claim 1, characterized in that, The method further includes: When the deployment completion notification of the target service control is received, a deployment completion prompt for the big data cluster is generated.

3. The method according to claim 1, characterized in that, The target configuration information includes the service image identifier and container startup parameters; The service image identifier is used to enable physical nodes to obtain the installation program file of the service control to be deployed based on the service image identifier; the container startup boot parameters are used to boot and run the service control deployed based on the obtained installation program file on the physical node where the service control to be deployed needs to be deployed.

4. The method according to claim 1, characterized in that, The method further includes: If a deployment completion notification is received for a service control in a direct sublayer of any service control, the runtime interface information of the service control in the direct sublayer of any service control is sent to the physical node that deployed the service control, so that the physical node starts the service control based on the runtime interface information and sends a deployment completion notification of the service control to the control node after starting.

5. A method for constructing a big data cluster, characterized in that, The method is applied to physical nodes of the big data cluster, which further includes a control node that communicates with the physical nodes; the method includes: Upon receiving target configuration information for a service control to be deployed from a control node, the service control to be deployed is deployed according to the received target configuration information. If it is determined that the service control to be deployed has service controls with direct sub-layers, then: place the identifier of the service control with the direct sub-layers of the service control to be deployed in the service deployment request, and send the service deployment request to the control node, so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request; The method further includes: Before receiving the target configuration information of the service control to be deployed sent by the control node, the physical node that needs to deploy the target service control needs to receive the configuration information of the target service control sent by the control node; the target service control is the service control that the user selects to use, and has at least one service control of the direct sub-layer. Deploy the target service control based on the received configuration information of the target service control; If it is determined that the target service control has directly subordinate service controls, then the identifier of the directly subordinate service control of the target service control is placed in the service deployment request and sent to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate service control of the target service control.

6. The method according to claim 5, characterized in that, After sending the service deployment request, which includes the identifier of the service control of the direct sublayer of the target service control, to the control node, the method further includes: If the runtime interface information of the service controls of each of the direct sub-layers of the target service control is received, the target service control is started, and a deployment completion notification of the target service control is sent to the control node.

7. The method according to claim 5, characterized in that, The step of deploying the service control to be deployed according to the received target configuration information includes: Based on the received target configuration information, the installation program file of the service control to be deployed is obtained; the target configuration information includes a service image identifier and container startup boot parameters; the service image identifier is used by the physical node to obtain the installation program file of the service control to be deployed based on the service image identifier included in the received target configuration information after receiving the target configuration information; the container startup boot parameters are used to guide the running of the service control deployed according to the obtained installation program file on the physical node where the service control to be deployed needs to be deployed; Deploy the service control to be deployed based on the obtained installation program file.

8. The method according to claim 7, characterized in that, The service image identifier is also used to obtain the preset dependency information of the service control to be deployed; the dependency information is the correspondence between the service control to be deployed and the service controls of the direct sub-layer of the service control to be deployed. The step of determining that the service control to be deployed has service controls with direct child layers includes: Based on the service image identifier included in the received target configuration information, obtain the dependency information of the service control to be deployed; Based on the dependency information of the service control to be deployed, if it is determined that there is a service control in the direct sub-layer corresponding to the service control to be deployed, then it is determined that the service control to be deployed has a service control in the direct sub-layer.

9. The method according to claim 5, characterized in that, The method further includes: After deploying any service control, the runtime interface information of the service control is sent to the control node so that the control node stores the runtime interface information of the service control. If any of the service controls is detected to meet the preset conditions, then the service control is started and a deployment completion notification of the service control is sent to the control node; the preset conditions include any of the following: the service control does not have any directly subordinate service controls, or the service control has received the running interface information of the service controls of each of the directly subordinate service controls.

10. A device for building a big data cluster, characterized in that, A control node deployed in the big data cluster, the big data cluster further including at least one physical node communicating with the control node; the device includes: The information acquisition module is used to, upon receiving a service deployment request from a physical node, determine the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request; wherein, the service deployment request is sent by the physical node to the control node when it determines that the deployed service control has directly subordinate service controls; the service control to be deployed is a directly subordinate service control of the deployed service control; determining the target configuration information of the service control to be deployed based on the identifier of the service control to be deployed included in the service deployment request includes: obtaining deployment parameters corresponding to the identifier of the service control to be deployed from preset deployment setting information based on the identifier of the service control to be deployed included in the service deployment request; the deployment setting information is the correspondence between the identifier of the service control and the deployment parameters; and determining the physical node to which the service control to be deployed and the target configuration information of the service control to be deployed based on the deployment parameters corresponding to the identifier of the service control to be deployed. The node allocation module is used to send the target configuration information to the physical nodes that need to deploy the service control to be deployed, so that the physical nodes that receive the target configuration information can deploy the service control to be deployed. The device further includes: The initial information acquisition module is used to respond to the initial service deployment request before receiving the service deployment request sent by the physical node, and determine the configuration information of the target service control based on the identifier of the target service control included in the initial service deployment request; the target service control is the service control selected by the user to be used, and has at least one service control of the direct sub-layer. The initial node allocation module is used to send the configuration information of the target service control to the physical node that needs to deploy the target service control, so that the physical node that receives the configuration information of the target service control can deploy the target service control.

11. A device for building a big data cluster, characterized in that, The device comprises: physical nodes deployed in the big data cluster, the big data cluster further comprising control nodes that communicate with the physical nodes; and the device includes: The control deployment module is used to deploy the service control to be deployed according to the target configuration information received from the control node each time it receives the target configuration information of the service control to be deployed. The sub-layer deployment request module is used to, if it is determined that the service control to be deployed has a service control with a direct sub-layer, then: place the identifier of the service control of the direct sub-layer of the service control to be deployed in the service deployment request, and send the service deployment request to the control node, so that the control node notifies the corresponding physical node to deploy the service control corresponding to the identifier included in the service deployment request; The device further includes: The initial control deployment module is used to ensure that, before receiving the target configuration information of the service control to be deployed from the control node, the physical node that needs to deploy the target service control receives the configuration information of the target service control sent by the control node; the target service control is a service control selected by the user and has at least one service control of the directly subordinate sub-layer; and the target service control is deployed according to the received configuration information of the target service control. The initial sub-layer deployment request module is used to, if it is determined that the target service control has a directly subordinate sub-layer service control, place the identifier of the directly subordinate sub-layer service control in the service deployment request and send it to the control node, so that the control node notifies the corresponding physical node to deploy the directly subordinate sub-layer service control of the target service control.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4; or implements the method of any one of claims 5 to 9.

13. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it implements the method of any one of claims 1 to 4; or implements the method of any one of claims 5 to 9.

Images