Method, device, terminal and storage medium for deploying a big data cluster

Abstract

The embodiment of the application discloses a method and device for deploying a big data cluster, a terminal and a storage medium, and belongs to the technical field of big data. After obtaining metadata information of a target cluster, the application can detect whether the deployment environment meets the deployment condition when the component starts to be deployed. When the condition is met, the application can automatically generate a component configuration file according to the metadata information, and send the component configuration file and the installation package of the component to the target server corresponding to the component, so that the target server can normally deploy the component. Since the application can automatically detect whether the deployment environment meets the deployment condition, the staff does not need to manually judge the sequence of component deployment in the case that the deployment condition is complex, thereby improving the deployment efficiency of the big data cluster.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, apparatus, terminal and storage medium for deploying big data clusters. Background Technology

[0002] A big data cluster typically consists of several components deployed on different servers. Only when each component is deployed correctly can the big data cluster function properly and provide services.

[0003] In related technologies, when deploying a big data cluster, each component needs to be deployed in the correct order. For example, a cluster may include components A, B, and C. Component B can only be deployed correctly after component C is correctly deployed. Therefore, staff need to manually configure the configuration file and installation package for component C first, and then send the above data to the service responsible for configuring the cluster before sending the relevant data for component B. When there are many components in a big data cluster, the interdependencies between components are complex, making the configuration operations cumbersome and difficult for staff. Summary of the Invention

[0004] This application provides a method, apparatus, terminal, and storage medium for deploying a big data cluster. The technical solution is as follows:

[0005] According to one aspect of this application, a method for deploying a big data cluster is provided, the method comprising:

[0006] Obtain metadata information of the target cluster, the target cluster including at least one component, the metadata information being used to indicate the target server and the hardware information of the target server, the target server being used to deploy the component;

[0007] In response to the component initiating deployment, it is checked whether the deployment environment of the component meets the component deployment conditions;

[0008] If the deployment environment meets the component deployment conditions, a component configuration file is generated based on the metadata information. The component configuration file is used to support the operation of the component after deployment.

[0009] The installation package of the component and the component configuration file of the component are sent to the target server corresponding to the component;

[0010] Send a deployment instruction to the target server, the deployment instruction being used to instruct the target server to deploy the component according to the installation package and the component configuration file;

[0011] Once the component has been deployed, test the component.

[0012] Once all components in the target cluster have been deployed and passed testing, the deployment process for the target cluster ends.

[0013] According to another aspect of this application, an apparatus for deploying a big data cluster is provided, the apparatus comprising:

[0014] An acquisition unit is used to acquire metadata information of a target cluster, the target cluster including at least one component, the metadata information being used to indicate the target server and the hardware information of the target server, the target server being used to deploy the component;

[0015] The detection unit is used to detect whether the deployment environment of the component meets the deployment conditions in response to the component starting deployment;

[0016] A generation unit is configured to generate a component configuration file based on the metadata information when the deployment environment meets the component deployment conditions. The component configuration file is used to support the operation of the component after deployment.

[0017] The first sending unit is used to send the installation package of the component and the component configuration file of the component to the target server corresponding to the component;

[0018] The second sending unit is used to send a deployment instruction to the target server, the deployment instruction being used to instruct the target server to deploy the component according to the installation package and the component configuration file;

[0019] A testing unit is used to test the component after the component has been deployed.

[0020] The termination unit is used to end the deployment process of the target cluster after all components in the target cluster have been deployed and passed the test.

[0021] According to another aspect of this application, a terminal is provided, the terminal including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to implement a method for deploying a big data cluster as provided in various aspects of this application.

[0022] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the instruction being loaded and executed by a processor to implement a method for deploying a big data cluster as provided in various aspects of this application.

[0023] According to one aspect of this application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for deploying a big data cluster provided in the various alternative implementations described above.

[0024] The beneficial effects of the technical solutions provided in this application embodiment may include:

[0025] This application, after obtaining the target cluster's metadata information, can detect whether the deployment environment meets the deployment conditions when a component begins deployment. If the conditions are met, it automatically generates a component configuration file based on the metadata information and sends the configuration file and the component's installation package to the target server corresponding to the component, enabling the target server to deploy the component normally. Because this application can automatically detect whether the deployment environment meets the deployment conditions, staff no longer need to manually determine the order of component deployment, even in cases with complex deployment conditions, thereby improving the deployment efficiency of big data clusters. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application;

[0028] Figure 2 This is a flowchart illustrating a method for deploying a big data cluster, provided in an exemplary embodiment of this application.

[0029] Figure 3 This is a flowchart illustrating a method for deploying a big data cluster, provided in another exemplary embodiment of this application.

[0030] Figure 4 This is a flowchart illustrating a method for deploying a big data cluster, provided in another exemplary embodiment of this application.

[0031] Figure 5 Based on Figure 4 The illustrated embodiment provides a schematic diagram of an architecture for cluster-deployed applications;

[0032] Figure 6This is a structural block diagram of an apparatus for deploying a big data cluster, provided in an exemplary embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0036] As used herein, the term “if” may optionally be interpreted, depending on the context, as “when,” “in the event of,” “in response to determination,” or “in response to detection.” Similarly, depending on the context, the phrases “if it is determined that…” or “if (the stated condition or event) is detected” or “in response to the detection of (the stated condition or event).”

[0037] It should be noted that the use of personally identifiable information should comply with privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for protecting user privacy. Specifically, the nature of authorized use of personally identifiable information should be clearly explained to users during its management and processing to minimize the risk of unintentional or unauthorized access or use.

[0038] Please refer to Figure 1 , Figure 1 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. For example...Figure 1 As shown, the computer device includes a processor 120 and a memory 140, the memory 140 storing at least one instruction, which is loaded and executed by the processor 120 to implement the method of deploying a big data cluster as provided in the various method embodiments of this application.

[0039] In this application, computer device 100 is an electronic device capable of deploying a big data cluster. When computer device 100 obtains metadata information of a target cluster, it can respond to component deployment by detecting whether the deployment environment of the component meets the component deployment conditions. The target cluster includes at least one component, and the metadata information indicates the target server and its hardware information. The target server is used to deploy the component. If the deployment environment meets the component deployment conditions, a component configuration file is generated based on the metadata information. The component configuration file supports the operation of the component after deployment. The installation package of the component and the component configuration file are sent to the target server corresponding to the component. A deployment instruction is sent to the target server, instructing it to deploy the component based on the installation package and the component configuration file. If the component deployment is complete, the component is tested. If all components in the target cluster are deployed and pass the tests, the deployment process of the target cluster ends.

[0040] Processor 120 may include one or more processing cores. Processor 120 connects to various parts within the computer device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 140, and by calling data stored in memory 140. Optionally, processor 120 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 120 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 120 and may be implemented as a separate chip.

[0041] The memory 140 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 140 may include a non-transitory computer-readable storage medium. The memory 140 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 140 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data involved in the various method embodiments described below, etc.

[0042] It should be noted that the computer device 100 can be a single device or a group of devices, and this application embodiment does not limit this. For example, if the computer device 100 can independently complete the method for deploying a big data cluster provided in this application, then the computer device 100 can be just a single device. If the computer device 100 requires multiple devices to collaborate in completing the method for deploying a big data cluster provided in this application, then the computer device 100 can be multiple devices.

[0043] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for deploying a big data cluster according to an exemplary embodiment of this application. This method for deploying a big data cluster can be applied to the above-described... Figure 1 In the computer device shown. Figure 2 The method for deploying big data clusters includes:

[0044] Step 210: Obtain metadata information of the target cluster. The target cluster includes at least one component. The metadata information is used to indicate the target server and the hardware information of the target server. The target server is used to deploy the component.

[0045] In this embodiment, the computer device can determine the relevant data of the target server to which the target cluster will be deployed before deploying the big data cluster. In this embodiment, the relevant data of the target server is referred to as metadata information.

[0046] In one possible implementation, metadata information is used to indicate the target server and its hardware information. The target server is the server used to deploy the components. This application obtains the hardware information of the target server supporting the deployment of the target cluster by acquiring the target cluster's metadata information in advance, for processing in subsequent steps.

[0047] The metadata information may include the target server's identifier and its hardware information. This hardware information may include memory, external storage, processor, and bandwidth information. Each type of hardware information may include specific parameters, indicating the target server's performance in a particular aspect.

[0048] Step 220: In response to the component's deployment startup, check whether the component's deployment environment meets the component deployment conditions.

[0049] In this embodiment, if a target cluster needs to be deployed and the target cluster includes n components, the deployment process of the target cluster can be divided into the deployment process of the n components. Once the deployment of the n components is complete, the deployment of the target cluster is also completed accordingly. The deployment process of each component can be an independent process.

[0050] In this embodiment, the computer device, in response to a component deployment command, detects whether the component's deployment environment meets the deployment conditions. It should be noted that the component deployment conditions indicate whether the current component can be deployed immediately. That is, when the component's deployment environment meets the deployment conditions, the computer device will execute subsequent deployment-related tasks. When the component's deployment environment does not meet the deployment conditions, the computer device will stop executing subsequent deployment-related tasks. It should also be noted that the component can be any component in the target cluster.

[0051] In one possible approach, the computer device can determine whether the component's deployment environment meets the component deployment conditions by accessing external information sources or internally cached information. Specifically, the external information sources or internally cached information can store information about the component's deployment environment. After obtaining this information, the computer device compares it with preset component deployment conditions to determine whether the component's deployment environment meets the requirements.

[0052] Step 230: If the deployment environment meets the component deployment conditions, generate a component configuration file based on the metadata information. The component configuration file is used to support the operation of the component after deployment.

[0053] In this embodiment of the application, when the deployment environment meets the component deployment conditions, the computer device can generate a corresponding component configuration file according to the metadata information. The component configuration file can be used during component deployment and can also support the operation of the component after deployment is completed.

[0054] In detail, regarding the role of component configuration files in the deployment process, component configuration files can assist in the normal and orderly deployment process when components are deployed to the target server.

[0055] Regarding the role of component configuration files in the operation of a big data cluster, component configuration files can guide the amount of hardware resources that a component uses on the target server during component runtime, so that the component can run normally, thereby ensuring the normal operation of the entire big data cluster.

[0056] Step 240: Send the component's installation package and component configuration file to the target server corresponding to the component.

[0057] In this embodiment of the application, the computer device can send the component's installation package and the component's configuration file to the target server corresponding to the component.

[0058] In one possible approach, there are multiple target servers. The computer device can treat the component's installation package and component configuration file as a set of files. Multiple copies of this set of files are then sent to each of the target servers.

[0059] In another possible approach, the computer device can package the component's installation package and configuration file as a single file and send it to the target server. The target server can then decode this file back into the component's installation package and configuration file, allowing it to function correctly.

[0060] Step 250: Send a deployment instruction to the target server. The deployment instruction is used to instruct the target server to deploy the component according to the installation package and component configuration file.

[0061] In one possible approach, the computer device sends a deployment instruction to the target server, which can be sent before sending the component's installation package and component configuration file. That is, step 250 can be executed before step 240; step 250 can be executed after step 240; or step 250 can be executed at the same time as step 240.

[0062] Accordingly, after the target server receives the deployment instructions, as well as the component's installation package and component configuration file, the target server can install the component's installation package according to the component's configuration file under the guidance of the deployment instructions, thereby completing the component deployment.

[0063] Step 260: Once the component has been deployed, test the component.

[0064] In one possible approach, the computer equipment can test the component after deployment to ensure its performance meets preset targets. Furthermore, once the component is fully deployed on the target cluster, it can be verified that it can be correctly invoked by other components within the target cluster.

[0065] It should be noted that step 260 is used to perform performance testing on a single component after the individual component has been deployed.

[0066] Step 270: Once all components in the target cluster have been deployed and passed testing, the deployment process of the target cluster ends.

[0067] In this application, the computer device can terminate the deployment process of the target cluster after all components in the target cluster have been deployed and passed testing. The phrase "all components in the target cluster have been deployed and passed testing" has two meanings. The first meaning is that each component in the target cluster has passed performance testing; the second meaning is that each component in the target cluster can be correctly invoked after deployment. The second meaning will be explained through an exemplary embodiment.

[0068] For example, the target cluster includes component A, component B, component C, and component D. Component A requires components C and D to be invoked, component B requires component C, and components C and D can be invoked directly. After all four components are deployed, the computer device needs to invoke each component sequentially to check if each component can be invoked correctly. If every component in the target cluster is invoked correctly, the entire target cluster will run correctly. If any component in the target cluster cannot be invoked correctly, the target cluster will malfunction. For example, if component C malfunctions after deployment, components A and B will also be unable to be invoked correctly.

[0069] In summary, the method for deploying a big data cluster provided in this embodiment can, after obtaining the metadata information of the target cluster, detect whether the deployment environment meets the deployment conditions when the component starts deployment. If the conditions are met, it automatically generates a component configuration file based on the metadata information and sends the component configuration file and the component's installation package to the target server corresponding to the component, enabling the target server to deploy the component normally. Because this application can automatically detect whether the deployment environment meets the deployment conditions, staff do not need to manually determine the order of component deployment even in cases of complex deployment conditions, thereby improving the deployment efficiency of big data clusters.

[0070] Please see Figure 3 , Figure 3 This is a flowchart illustrating a method for deploying a big data cluster, provided in another exemplary embodiment of this application. This method for deploying a big data cluster can be applied to the above-mentioned... Figure 1 In the terminal shown. Figure 3 The method for deploying big data clusters includes:

[0071] Step 310: Obtain the metadata information of the target cluster.

[0072] In this application, the execution process of step 310 is the same as that of step 210, and will not be described again here.

[0073] Step 321: In response to the component deployment, detect whether the hardware resources and / or software environment of the target server used to deploy the component meet the first deployment conditions.

[0074] The first deployment condition is a component deployment condition.

[0075] In this application, a computer device can determine whether the hardware resources and / or software environment of the target server used to deploy the component meet the first deployment conditions by performing at least one of steps (a1), (a2), or (a3).

[0076] Step (a1) is to detect whether the operating system of the target server is compatible with the operating system required for the configuration in response to the deployment of the target cluster.

[0077] In this example, after the target cluster begins deployment, the computer device can read the identifier of the operating system required for configuration. Then, the computer device can access the external target server to obtain the identifier of the operating system running on the target server. The computer device will compare the identifier of the required operating system with the identifier of the operating system running on the target server. When the two identifiers are the same, the computer device will confirm that the operating system of the target server is compatible with the required operating system. When the two identifiers are different, the computer device will confirm that the operating system of the target server is incompatible with the required operating system.

[0078] It should be noted that the compatibility between the target server's operating system and the operating system required for configuration can include two scenarios. The first scenario is that the identifier of the target server's operating system is exactly the same as the identifier of the operating system required for configuration. The second scenario is that the identifier of the target server's operating system is the same as the identifier of the operating system required for configuration in the version portion. For example, the identifier of the target server's operating system is ABC2.0.12, and the identifier of the operating system required for configuration is ABC2.0.11. Here, the identifier used to identify the operating system is ABC2.0. Since both the identifier of the target server's operating system and the identifier of the operating system required for configuration have the version portion of ABC2.0, the target server's operating system is compatible with the operating system required for configuration.

[0079] Step (a2) is to detect whether the application software installed on the target server matches the application software required for the configuration in response to the deployment of the target cluster.

[0080] In this example, the computer device responds to the target cluster by initiating deployment and obtains a list of application software installed on the target server.

[0081] In one possible approach, the computer device obtains a list of application software installed on the target server by accessing the target server. After obtaining this list, the computer device verifies whether the application software on the list covers the application software required for the configuration. If the list of application software installed on the target server covers the application software required for the configuration, the computer device verifies that the application software installed on the target server matches the application software required for the configuration.

[0082] In another possible approach, the computer device would also access the target server to obtain a list of application software installed on the target server. After obtaining this list, the computer device would verify whether the application software in the list meets the requirements for configuration. For example, the required application software might be software A1, software B, and software C, or software A2, software B, and software C. If the list of application software installed on the target server includes software A2, software B, and software C, or if the list includes software A1, software B, and software C, then the application software on the target server can be considered to meet the requirements for configuration.

[0083] Step (a3) ​​involves detecting whether the dynamic-link libraries (DLLs) of the application software stored on the target server cover the dynamic-link libraries required for the configuration in response to the deployment of the target cluster.

[0084] In this application, the computer device can detect the dynamic link libraries (DLLs) of the application software stored in the target server after the target cluster has started deployment. If the items in the DLLs can cover the DLLs required for the configuration, the computer device confirms that the DLLs of the application software stored in the target server cover the DLLs required for the configuration.

[0085] In this application, when the hardware resources and software environment of the target server used for deploying the components meet the first deployment conditions, the computer device can execute step 322.

[0086] In this application, when the hardware resources and software environment of the target server used to deploy the components do not meet the first deployment conditions, the computer device may perform step (b).

[0087] Step (b): If the first deployment condition is not met, return the first unmet information. The first unmet information is used to indicate that the hardware resources of the target server do not meet the first deployment condition, and / or that the software environment of the target server does not meet the first deployment condition.

[0088] Step 322: If the first deployment condition is met, check whether the preceding components on which the component depends meet the second deployment condition.

[0089] In this example, once the target cluster starts deployment and the computer equipment confirms that the first deployment condition is met, the preceding components on which the currently deployed component depends are then checked to see if the preceding components meet the second deployment condition.

[0090] The second deployment condition is a component deployment condition. It can indicate that a component has been successfully deployed. When the preceding component meets the second deployment condition, it means that the preceding component has been successfully deployed.

[0091] Step 323: If the second deployment condition is met, confirm that the deployment environment of the component meets the component deployment condition.

[0092] In this application, if the computer device confirms that the second deployment condition is met, it will recognize that the deployment environment of the component being deployed or to be deployed meets the component deployment condition.

[0093] In this application, if the second deployment condition is not met, the computer device may perform at least one of steps (c1), (c2), and (c3).

[0094] Step (c1) returns a second unmet information if the second deployment condition is not met. The second unmet information is used to indicate that there are undeployed components in the front-end components.

[0095] In this application, if the second deployment condition is not met, the computer device returns a second unmet condition message, which indicates that there are components in the front-end components that are not deployed. That is, when the computer device generates the second unmet condition message, it means that one or more components in the front-end components are not deployed.

[0096] Step (c2) returns a second unmet information if the second deployment condition is not met. The second unmet information is used to indicate that there is a component with a deployment anomaly in the front-end components.

[0097] In this application, the second unmet requirement can also be used to indicate that there are one or more components in the preceding components that have deployment anomalies. In practical application scenarios, a component may have completed deployment, but an anomaly occurred during the deployment process, resulting in the component completing deployment but being in a deployment anomaly state.

[0098] Step (c3): If the second deployment condition is not met, return the second unmet information, which is used to indicate that there are components in the front-end components that have not been deployed.

[0099] In this application, the second unmet information can also be used to indicate that there are components in the preceding components that have been deployed but not yet completed. For example, the preceding components include components A, B, and C. Components A and B have been deployed, while component C is being deployed and has reached 60% completion. In this case, component C is the component in the preceding components that has not been fully deployed.

[0100] Step 331: Generate component configuration files based on metadata information.

[0101] In this application, the execution process of step 331 can be referred to the execution process of step 230, and will not be repeated here.

[0102] Step 332: Send the component's installation package and component configuration file to the target server corresponding to the component.

[0103] In this application, the execution process of step 332 can be referred to the execution process of step 240, and will not be repeated here.

[0104] Step 333: Send deployment instructions to the target server.

[0105] In this application, the execution process of step 333 can be referred to the execution process of step 250, and will not be repeated here.

[0106] Step 334: After the component has been deployed, test the component.

[0107] In this application, the execution process of step 334 can be referred to the execution process of step 260, and will not be repeated here.

[0108] Step 335: Once all components in the target cluster have been deployed and passed testing, the deployment process of the target cluster ends.

[0109] In this application, the execution process of step 335 can be referred to the execution process of step 270, and will not be repeated here.

[0110] In summary, this embodiment considers both the environment of the target server used to deploy the component and the deployment status of the component's preceding components simultaneously. When the environment of the target server meets the component's deployment requirements, the deployment status of the component's preceding components is further examined. Once the preceding components are deployed, the component itself is deployed, ensuring its normal deployment and operation. During the deployment of a big data cluster, this approach automates the deployment and operation of each component, improving deployment efficiency without consuming excessive hardware and software resources, thus enhancing the overall deployment efficiency of the big data cluster.

[0111] Please seeFigure 4 , Figure 4 This is a flowchart illustrating a method for deploying a big data cluster, provided in another exemplary embodiment of this application. This method for deploying a big data cluster can be applied to the above-mentioned... Figure 1 The computer device shown contains a cluster deployment application, which includes an environment monitoring module, a configuration management module, a function verification module, a resource management module, and an instruction execution module. Figure 4 The method for deploying big data clusters includes:

[0112] Step 411: Obtain the metadata information of the target cluster.

[0113] In this application, the execution process of step 411 can be referred to the execution process of step 210, and will not be repeated here.

[0114] Please see Figure 5 , Figure 5 Based on Figure 4 The illustrated embodiment provides a schematic diagram of an architecture for cluster-deployed applications. Figure 5 The cluster deployment application 500 includes an environment detection module 510, a configuration management module 520, a function verification module 530, a resource management module 540, and an instruction execution module 550. The instruction execution module 550 is used to compile data and / or instructions that other modules need to send, compiling them into information that the target server can interpret. The instruction execution module 550 is used to send data to the external target server.

[0115] Step 412: In response to the component deployment, the environment detection module checks whether the component's deployment environment meets the component deployment conditions.

[0116] In this application, the cluster deployment application in the computer device can, after the component starts deployment, detect whether the deployment environment of the component meets the component deployment conditions based on the environment detection module. The environment detection module can obtain the component's identifier and determine the component's deployment environment based on the identifier. Furthermore, the environment detection module can compare the component deployment conditions with the deployment environment to derive a final logical value. A logical value of true indicates that the deployment environment meets the component deployment conditions, while a logical value of false indicates that the deployment environment does not meet the component deployment conditions.

[0117] Optionally, clustered deployment of application 500 can be an Ansible-based application.

[0118] Step 413: If the deployment environment meets the component deployment conditions, generate the component configuration file based on the metadata information using the configuration management module.

[0119] In this application, when the deployment environment meets the component deployment conditions, the computer device, through the configuration management module in the cluster deployment application, generates configuration files based on metadata information. The configuration management module has a pre-configured program for processing metadata information. When metadata information is input into the configuration management module, it can automatically generate component configuration files based on that information.

[0120] Step 414: Based on the configuration management module, send the component configuration file to the target server corresponding to the component.

[0121] In this application, once the component configuration file has been generated, the cluster deployment application in the computer device can send the configuration file to the target server corresponding to the component through the configuration management module. Optionally, before the configuration file is sent to the target server, it can also be sent to the instruction execution module, which compiles it into data that the target server can recognize, and finally sends it to the target server.

[0122] Step 415: Based on the resource management module, send the component's installation package to the target server corresponding to the component.

[0123] In this example, the computer device can send the component's installation package to the target server corresponding to the component based on the resource management module. It should be noted that the resource management module manages the installation packages of each component; when a component needs to be deployed, the computer device sends the component's installation package to the target server.

[0124] Step 416: Based on the instruction execution module, a deployment instruction is sent to the target server. The instruction execution module is used to generate deployment instructions that the target server can recognize.

[0125] In this example, the cluster deployment application in the computer device can send deployment instructions to the target server based on the instruction execution module. The instructions issued by the instruction execution module can be used to generate deployment instructions that the target server can recognize.

[0126] Step 417: Generate the first functional verification instruction based on the functional verification module.

[0127] In this example, the cluster-deployed application in the computer device can generate a first functional verification instruction based on the functional verification module. This first functional verification instruction cannot be recognized by the target server. Therefore, the first functional verification instruction needs to be compiled within the cluster-deployed application.

[0128] Step 418: Based on the instruction execution module, the first functional verification instruction is compiled into a second functional verification instruction. The second functional verification instruction is used to execute the detection component process after being recognized by the target server.

[0129] In this example, the cluster-deployed application in the computer device will compile the first functional verification instruction into a second functional verification instruction based on the instruction execution module. This second functional verification instruction is used to execute the detection component's process after being identified by the target server.

[0130] Step 419: Once all components in the target cluster have been deployed and passed testing, the deployment process of the target cluster ends.

[0131] In this example, the cluster deployment of the application on the computer equipment can conclude the deployment process of the target cluster once all components in the target cluster have been deployed and passed testing. It should be noted that the target cluster deployed in this example can be a big data target cluster. After the big data target cluster is deployed, it can be put into operation online.

[0132] Step 420: Once the target cluster has been deployed, obtain the parameter reconfiguration instructions for the components.

[0133] In this example, the computer device performs operation and maintenance on the target cluster after its deployment is complete. During the operation and maintenance phase, components send parameter reconfiguration commands to the computer device when reconfiguration is required. Correspondingly, the computer device is able to receive these parameter reconfiguration commands from the components once the target cluster has been deployed.

[0134] Step 421: Based on the configuration management module, generate the component configuration file corresponding to the parameter reconfiguration command.

[0135] In this example, clustered applications in computer devices can generate component configuration files corresponding to parameter reconfiguration instructions based on the configuration management module.

[0136] Step 422: Based on the configuration management module, send the component configuration file corresponding to the parameter reconfiguration command to the target server so that the target server can update the component configuration file.

[0137] In this example, the cluster deployment application in the computer device can send the component configuration file corresponding to the parameter reconfiguration command to the target server based on the configuration management module, so that the target server can update the component's configuration file. Therefore, after installing the cluster deployment application, the computer device can not only successfully deploy the target cluster, but also use the cluster deployment application to maintain the target cluster, updating components that require updated configuration files, facilitating timely adjustments to the already deployed target cluster according to changing needs.

[0138] Based on the implementation method presented in this example, this application modularizes the big data cluster deployment process and utilizes Ansible's batch system configuration, batch program deployment, and batch command execution functions to automate the deployment process, improve the deployment efficiency of big data clusters, and reduce deployment costs. In this application, the cluster deployment application installed on the computer device includes an environment detection module, a configuration management module, a function verification module, a resource management module, and an instruction execution module. When the big data cluster deployment begins, the computer device needs to obtain metadata information from the outside world. This information includes the target servers for each component of the big data cluster and the corresponding CPU and memory information of the target servers. The environment detection module is responsible for detecting whether the software and hardware of each target server meet the requirements based on the cluster's metadata information. If the target server hardware or software does not meet the deployment requirements, the computer device exits the detection and outputs the exit reason. The environment detection module is also responsible for detecting whether the components meet the installation conditions. For example, if component A depends on components B and C, the computer device checks the status of components B and C before deploying component A. If the status of component B or component C is abnormal, the computer device automatically rolls back the deployment instruction or resets the instruction.

[0139] To illustrate, the configuration management module generates component configuration files required for the startup and runtime of each component based on the cluster's metadata. Furthermore, the configuration management module can push these component configuration files to the resource management module. The configuration management module is also responsible for regenerating the corresponding component configuration files after changes are made to the configuration of big data components.

[0140] The resource management module uploads the installation packages and configuration files of each component in the big data cluster to the corresponding target servers based on the cluster's metadata. The instruction execution module is responsible for executing specific instructions. For example, when the instruction execution module is an Ansible instruction execution module, it is responsible for executing specific Ansible instructions. The functional verification module is used to verify the functionality of a component after its deployment in the big data cluster; it is also responsible for performing regression testing and performance testing on the cluster after all components have been deployed to ensure the correctness of the big data cluster deployment.

[0141] In summary, this application integrates the functionality of deploying a big data cluster into a single cluster deployment application. This application includes an environment detection module, a configuration management module, a function verification module, a resource management module, and an instruction execution module. The instruction execution module compiles the function verification instructions from the cluster deployment application into instructions that the target server can recognize, thereby successfully deploying the components of the big data cluster to the target server. This automates the deployment and operation of each component, improving the efficiency of big data cluster deployment without consuming excessive hardware and software resources.

[0142] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0143] Please refer to Figure 6 , Figure 6 This is a structural block diagram of an apparatus for deploying a big data cluster, provided in an exemplary embodiment of this application. The apparatus for deploying the big data cluster can be implemented as all or part of a terminal through software, hardware, or a combination of both. The apparatus includes:

[0144] The acquisition unit 610 is used to acquire metadata information of a target cluster, the target cluster including at least one component, the metadata information being used to indicate the target server and the hardware information of the target server, the target server being used to deploy the component;

[0145] The detection unit 620 is used to detect whether the deployment environment of the component meets the deployment conditions in response to the component starting deployment;

[0146] The generation unit 630 is configured to generate a component configuration file based on the metadata information when the deployment environment meets the component deployment conditions. The component configuration file is used to support the operation of the component after deployment.

[0147] The first sending unit 640 is used to send the installation package of the component and the component configuration file of the component to the target server corresponding to the component;

[0148] The second sending unit 650 is used to send a deployment instruction to the target server, the deployment instruction being used to instruct the target server to deploy the component according to the installation package and the component configuration file;

[0149] Test unit 660 is used to test the component after the component has been deployed;

[0150] The termination unit 670 is used to terminate the deployment process of the target cluster when all components in the target cluster have been deployed and passed the test.

[0151] In an optional embodiment, the detection unit 620 is configured to detect whether the hardware resources and software environment of the target server used to deploy the component meet a first deployment condition, wherein the first deployment condition is a component deployment condition; if the first deployment condition is met, it detects whether the preceding components on which the component depends meet a second deployment condition, wherein the second deployment condition is a component deployment condition; if the second deployment condition is met, it confirms that the deployment environment of the component meets the component deployment condition.

[0152] In an optional embodiment, the detection unit 620 is configured to, in response to the deployment of the target cluster, detect whether the operating system of the target server is compatible with the operating system required for configuration; and / or, in response to the deployment of the target cluster, detect whether the application software installed on the target server matches the application software required for configuration; and / or, in response to the deployment of the target cluster, detect whether the dynamic link library of the application software stored in the target server covers the dynamic link library required for configuration.

[0153] In an optional embodiment, the apparatus further includes a first return unit, configured to return first non-metd information if the first deployment condition is not met, the first non-metd information indicating that the hardware resources of the target server do not meet the first deployment condition, and / or that the software environment of the target server does not meet the first deployment condition.

[0154] In an optional embodiment, the apparatus further includes a second return unit, configured to return a second unmet information if the second deployment condition is not met, the second unmet information indicating that there is an undeployed component in the front-end component; and / or, return a second unmet information if the second deployment condition is not met, the second unmet information indicating that there is a component with deployment anomalies in the front-end component; and / or, return a second unmet information if the second deployment condition is not met, the second unmet information indicating that there is a component in the front-end component that has not been fully deployed.

[0155] In an optional embodiment, the apparatus is applied to a cluster deployment application, which includes an environment detection module, a configuration management module, a function verification module, a resource management module, and an instruction execution module. The detection unit is used to detect, based on the environment detection module, whether the deployment environment of the component meets the component deployment conditions; the generation unit is used to generate a component configuration file based on the metadata information from the configuration management module; the first sending unit is used to send the component configuration file to the target server corresponding to the component from the configuration management module; the second sending unit is used to send the deployment instruction to the target server from the instruction execution module, the instruction execution module being used to generate the deployment instruction recognizable by the target server; the testing unit is used to generate a first function verification instruction from the function verification module; and to compile the first function verification instruction into a second function verification instruction from the instruction execution module, the second function verification instruction being used to execute a process to test the component after being recognized by the target server.

[0156] In an optional embodiment, the device further includes an operation and maintenance module, configured to: obtain a parameter reconfiguration instruction for the component after the target cluster has been deployed; generate a component configuration file corresponding to the parameter reconfiguration instruction based on the configuration management module; and send the component configuration file corresponding to the parameter reconfiguration instruction to the target server based on the configuration management module, so that the target server updates the component configuration file of the component.

[0157] In summary, the device for deploying big data clusters provided in this embodiment can, after obtaining the metadata information of the target cluster, detect whether the deployment environment meets the deployment conditions when the component starts deployment. If the conditions are met, it automatically generates a component configuration file based on the metadata information and sends the component configuration file and the component's installation package to the target server corresponding to the component, enabling the target server to deploy the component normally. Because this application can automatically detect whether the deployment environment meets the deployment conditions, staff do not need to manually determine the order of component deployment even in cases of complex deployment conditions, thereby improving the deployment efficiency of big data clusters.

[0158] This application also provides a computer-readable medium storing at least one instruction, which is loaded and executed by the processor to implement the method for deploying a big data cluster as described in the above embodiments.

[0159] It should be noted that the apparatus for deploying big data clusters provided in the above embodiments is only illustrated by the division of the above functional modules when executing the method for deploying big data clusters. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for deploying big data clusters and the method embodiments for deploying big data clusters provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0160] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0161] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0162] The above description is merely an exemplary embodiment that can be implemented in this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for deploying a big data cluster, characterized in that, The method includes: Obtain metadata information of the target cluster, the target cluster including at least one component, the metadata information being used to indicate the target server and the hardware information of the target server, the target server being used to deploy the component; the hardware information including memory information, external storage information, processor information, and bandwidth information; In response to the component initiating deployment, it is detected whether the hardware resources and software environment of the target server used to deploy the component meet the first deployment conditions; the first deployment conditions are used to detect: whether the operating system of the target server is compatible with the operating system required for configuration, whether the application software installed on the target server matches the application software required for configuration, and whether the dynamic link library of the application software stored in the target server covers the dynamic link library required for configuration. If the first deployment condition is met, it is detected whether the preceding components on which the component depends meet the second deployment condition; the second deployment condition is used to detect the deployment status of the preceding components on which the currently deployed component depends. If the second deployment condition is met, it is confirmed that the deployment environment of the component meets the component deployment condition; If the second deployment condition is not met, a second unmet information is returned; the second unmet information is used to indicate that there is an undeployed component in the front-end component, or to indicate that there is a component with a deployment error in the front-end component, or to indicate that there is a component in the front-end component that has been deployed but has not been completed. If the deployment environment meets the component deployment conditions, a component configuration file is generated based on the metadata information. The component configuration file is used to support the operation of the component after deployment. The configuration file is used to assist the normal and orderly deployment process during the deployment of the component to the target server, and to guide the amount of hardware resources occupied by the component on the target server during component operation. The installation package of the component and the component configuration file of the component are sent to the target server corresponding to the component; Send a deployment instruction to the target server, the deployment instruction being used to instruct the target server to deploy the component according to the installation package and the component configuration file; Once the component has been deployed, the component is tested; the testing includes performance testing of the individual component after its deployment; the testing is used to ensure that each component in the target cluster passes the performance test and that each component can be correctly invoked after deployment. Once all components in the target cluster have been deployed and passed testing, the deployment process for the target cluster ends.

2. The method according to claim 1, characterized in that, The method further includes: If the first deployment condition is not met, a first unmet condition is returned, which indicates that the hardware resources of the target server do not meet the first deployment condition, and / or that the software environment of the target server does not meet the first deployment condition.

3. The method according to claim 1, characterized in that, The method is applied to cluster deployment applications, which include an environment detection module, a configuration management module, a function verification module, a resource management module, and an instruction execution module. The method includes: Based on the environment detection module, it is detected whether the deployment environment of the component meets the deployment conditions of the component; The step of generating a component configuration file based on the metadata information includes: Based on the configuration management module, a component configuration file is generated according to the metadata information; Sending the installation package of the component and the component configuration file of the component to the target server corresponding to the component includes: Based on the configuration management module, the component configuration file is sent to the target server corresponding to the component; Based on the resource management module, the installation package of the component is sent to the target server corresponding to the component; Sending the deployment command to the target server includes: Based on the instruction execution module, the deployment instruction is sent to the target server, and the instruction execution module is used to generate the deployment instruction that can be recognized by the target server; The testing of the component includes: Based on the aforementioned function verification module, a first function verification instruction is generated; Based on the instruction execution module, the first functional verification instruction is compiled into a second functional verification instruction, which is used to execute the process of testing the component after being identified by the target server.

4. The method according to claim 3, characterized in that, The method further includes: Once the target cluster has been deployed, obtain the parameter reconfiguration instructions for the component. Based on the configuration management module, the component configuration file corresponding to the parameter reconfiguration instruction is generated; Based on the configuration management module, the component configuration file corresponding to the parameter reconfiguration instruction is sent to the target server so that the target server updates the component configuration file of the component.

5. An apparatus for deploying a big data cluster, characterized in that, The device includes: An acquisition unit is used to acquire metadata information of a target cluster, the target cluster including at least one component, the metadata information being used to indicate the target server and the hardware information of the target server, the target server being used to deploy the component; the hardware information includes memory information, external storage information, processor information, and bandwidth information; A detection unit is configured to, in response to the component initiating deployment, detect whether the hardware resources and software environment of the target server used to deploy the component meet a first deployment condition; the first deployment condition is configured to detect: whether the operating system of the target server is compatible with the operating system required for configuration; whether the application software installed on the target server matches the application software required for configuration; and whether the dynamic link libraries of the application software stored in the target server cover the dynamic link libraries required for configuration; if the first deployment condition is met, the unit detects whether the pre-components on which the component depends meet a second deployment condition; the second deployment condition is configured to detect the deployment status of the pre-components on which the currently deployed component depends; if the second deployment condition is met, the unit confirms that the deployment environment of the component meets the component deployment condition; if the second deployment condition is not met, the unit returns a second unmet information; the second unmet information is configured to indicate that there are undeployed components in the pre-components, or that there are components in the pre-components with deployment anomalies, or that there are components in the pre-components that have been deployed but have not completed deployment; The generation unit is configured to generate a component configuration file based on the metadata information when the deployment environment meets the component deployment conditions. The component configuration file is used to support the operation of the component after deployment. The configuration file is used to assist the normal and orderly deployment process during the deployment of the component to the target server, and to guide the amount of hardware resources occupied by the component on the target server during component operation. The first sending unit is used to send the installation package of the component and the component configuration file of the component to the target server corresponding to the component; The second sending unit is used to send a deployment instruction to the target server, the deployment instruction being used to instruct the target server to deploy the component according to the installation package and the component configuration file; A testing unit is used to test the component after the component has been deployed; the testing includes performance testing of the individual component after the individual component has been deployed; the testing is used to ensure that each component in the target cluster passes the performance test and that each component can be correctly invoked after deployment. The termination unit is used to end the deployment process of the target cluster after all components in the target cluster have been deployed and passed the test.

6. A terminal, characterized in that, The terminal includes a processor, a memory connected to the processor, and program instructions stored in the memory. When the processor executes the program instructions, it implements the method for deploying a big data cluster as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed by the processor, they implement the method for deploying a big data cluster as described in any one of claims 1 to 4.

Images