Big data cluster dynamic deployment method and device, equipment, product and storage medium

Abstract

The invention relates to the technical field of cluster deployment, and provides a big data cluster dynamic deployment method and device, equipment, a product and a storage medium. The method comprises the steps of obtaining a resource demand template based on a prediction type of a calculation task; generating a trigger event based on the monitoring signals in the sub-clusters of the plurality of calculation modes; generating a deployment rule based on the trigger event; based on a calculation mode required by the calculation task, constructing a big data cluster deployment model; and dynamically deploying the big data cluster based on the big data cluster deployment model and the deployment rule. Changes of computing resource requirements of big data in different computing modes can be sensed in real time and are sequentially associated with the trigger event, the deployment rule and the big data cluster deployment model, overall deployment of the big data cluster is achieved, the utilization rate of cluster resources is effectively increased, and the deployment efficiency is improved. And resource dynamic deployment requirements for different calculation modes in big data processing are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cluster deployment, and in particular to a big data cluster dynamic deployment method, device, equipment, product and storage medium. BACKGROUND

[0002] At present, big data cluster deployment schemes can be divided into static schemes and dynamic schemes from the perspective of elastic resource management.

[0003] 1. Static cluster scheme: Big data analysis tasks usually contain multiple computing modes. Traditional big data systems use Lambda architecture, use virtual machines and physical servers, and pre-establish real-time layer, batch processing layer and service layer sub-clusters through hierarchical design to process different modes of computing requirements. The static cluster node expansion and contraction operation and maintenance cost is high, and it cannot effectively respond to changes in cluster resource demand. The cluster load is high at some time, but the load is low at other times, causing resource waste and low cluster resource utilization.

[0004] 2. Dynamic cluster scheme: The cluster can be quickly expanded and contracted according to resource demand, thereby realizing fine management of cluster resources. Compared with the static cluster scheme, the dynamic cluster scheme reduces the cluster operation and maintenance cost, has stronger resource isolation, provides more flexible cluster elasticity, and can respond to changes in cluster resource demand to a certain extent, but this response does not consider the resource demand difference of different computing modes of big data, and the improvement of cluster resource utilization is very limited, which cannot meet the resource dynamic deployment demand in big data processing.

[0005] In summary, the existing big data cluster deployment schemes, whether static cluster schemes or dynamic cluster schemes, cannot effectively improve the cluster resource utilization, resulting in the inability to meet the resource dynamic deployment demand in big data processing. SUMMARY

[0006] The embodiments of the present application provide a big data cluster dynamic deployment method, device, equipment, product and storage medium to solve the technical problem that the existing big data cluster deployment schemes, whether static cluster schemes or dynamic cluster schemes, cannot effectively improve the cluster resource utilization, resulting in the inability to meet the resource dynamic deployment demand in big data processing.

[0007] In a first aspect, the embodiments of the present application provide a big data cluster dynamic deployment method, comprising: obtaining a resource demand template based on a predicted type of a computing task; generating a trigger event based on a monitoring signal in a sub-cluster of multiple computing modes; the monitoring signal includes computing resource demand for each sub-cluster in the resource demand template; generating a deployment rule based on the trigger event; construct a big data cluster deployment model based on the computing mode required by the computing task; perform dynamic deployment on the big data cluster based on the big data cluster deployment model and the deployment rule.

[0008] In one embodiment, the constructing the big data cluster deployment model based on the computing mode required by the computing task comprises: constructing a four-tuple based on the computing task, the computing mode, the computing mode sequence of the computing task, and the node requirement corresponding to the computing mode sequence; determining the four-tuple as the big data cluster deployment model.

[0009] In one embodiment, the performing dynamic deployment on the big data cluster based on the big data cluster deployment model and the deployment rule comprises: generating a big data cluster deployment plan based on the big data cluster deployment model and the deployment rule; converting the big data cluster deployment plan into a container command script; executing the container command script based on a lightweight container technology to perform dynamic deployment on the big data cluster.

[0010] In one embodiment, the generating the deployment rule based on the trigger event comprises: adding the trigger event to an event queue; traversing the event queue to generate corresponding deployment rules in sequence based on the trigger events in the event queue.

[0011] In one embodiment, the generating the trigger event based on the monitoring signal in the sub-cluster of the plurality of computing modes comprises: generating the trigger event based on a size relationship between the computing resource requirement and a resource threshold.

[0012] In one embodiment, the monitoring signal further comprises a computing resource state in each sub-cluster; and the generating the trigger event based on the monitoring signal in the sub-cluster of the plurality of computing modes comprises: generating the trigger event based on a size relationship between the computing resource state and a resource threshold.

[0013] In a second aspect, an embodiment of the present application provides a big data cluster dynamic deployment apparatus, comprising: a resource requirement obtaining module, configured to: obtain a resource requirement template based on a predicted type of a computing task; The trigger event generation module is configured to generate a trigger event based on a monitoring signal in a subset cluster of a plurality of computing modes, wherein the monitoring signal comprises a computing resource requirement of each subset cluster in the resource requirement template; The deployment rule generation module is configured to generate a deployment rule based on the trigger event; The deployment model construction module is configured to construct a big data cluster deployment model based on a computing mode required by the computing task. The cluster dynamic deployment module is configured to dynamically deploy the big data cluster based on the big data cluster deployment model and the deployment rule.

[0014] In a third aspect, an electronic device is provided, which includes a processor and a memory storing a computer program, and the processor implements the steps of the big data cluster dynamic deployment method in the first aspect when executing the program.

[0015] In a fourth aspect, a computer program product is provided, which includes a computer program, and the computer program implements the steps of the big data cluster dynamic deployment method in the first aspect when executed by a processor.

[0016] In a fifth aspect, a non-transitory computer-readable storage medium is provided, which includes a computer program, and the computer program implements the steps of the big data cluster dynamic deployment method in the first aspect when executed by a processor.

[0017] The big data cluster dynamic deployment method, device, equipment, product and storage medium provided by the present application are based on the predicted type of the computing task, obtain a resource requirement template, generate a trigger event based on a monitoring signal in a subset cluster of a plurality of computing modes, the monitoring signal comprises a computing resource requirement of each subset cluster in the resource requirement template, generate a deployment rule based on the trigger event, construct a big data cluster deployment model based on a computing mode required by the computing task, and dynamically deploy the big data cluster based on the big data cluster deployment model and the deployment rule. In the present application, each subset cluster corresponds to a computing mode, the computing resource requirement in each subset cluster is monitored, the change of the computing resource requirement under different computing modes is perceived in real time, and the change is quickly responded to generate a trigger event and a corresponding deployment rule, and then the big data cluster deployment model is constructed based on the computing mode required by the computing task, and the deployment rule corresponding to each computing mode in the model is combined to realize the overall dynamic deployment of the big data cluster. In summary, the present application can perceive the change of the computing resource requirement under different computing modes of big data in real time, and sequentially associate the change with a trigger event, a deployment rule and a big data cluster deployment model, so as to realize the overall deployment of the big data cluster, effectively improve the cluster resource utilization rate, and meet the dynamic deployment requirement of resources for different computing modes in big data processing. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is one of the flowcharts of the big data cluster dynamic deployment method provided by the embodiments of the present application; Figure 2 is the second flowchart of the big data cluster dynamic deployment method provided by the embodiments of the present application; Figure 3 is the third flowchart of the big data cluster dynamic deployment method provided by the embodiments of the present application; Figure 4 is the fourth flowchart of the big data cluster dynamic deployment method provided by the embodiments of the present application; Figure 5 is the fifth flowchart of the big data cluster dynamic deployment method provided by the embodiments of the present application; Figure 6 is the structural schematic diagram of the big data cluster dynamic deployment device provided by the embodiments of the present application; Figure 7 is the structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] Figure 1 is one of the flowcharts of the big data cluster dynamic deployment method provided by the embodiments of the present application. With reference to Figure 1 , the present application provides a big data cluster dynamic deployment method, which can include: Step 101, based on the predicted type of the computing task, obtaining a resource requirement template; Step 102, generating a trigger event based on the monitoring signal in the sub-cluster of the plurality of computing modes; The monitoring signal includes the computing resource requirement of each sub-cluster in the resource requirement template; Step 103, generating deployment rules based on the trigger event; Step 104, constructing a big data cluster deployment model based on the computing mode required by the computing task; Step 105, dynamically deploying the big data cluster based on the big data cluster deployment model and the deployment rules.

[0022] Referring to Figure 2 : In step 101, the computing task can be obtained from the task queue, and for each computing task, the corresponding resource requirement template is obtained based on its predicted type; the resource requirement template can be generated by the user after filling in the computing resource requirements based on the predicted type of the computing task, which realizes the quantification of the computing resource requirements, which can be as shown in the following table: Table 1 Resource requirement template

[0023] In step 102, the multiple computing modes can include batch computing, real-time computing, memory computing and graph computing, and for each computing mode, there is a corresponding sub-cluster in the big data dynamic cluster, that is, each sub-cluster is only responsible for executing computing tasks of one computing mode; Further, the computing resource requirements of each computing task for various computing modes can be obtained by analyzing the resource requirement template as a monitoring signal in the sub-cluster to which the corresponding computing mode belongs; wherein the computing resource requirements include CPU requirements, memory requirements, disk requirements and network requirements; Further, these computing resource requirements can be collected by a distributed collector and summarized in an event generator, and the trigger event can be generated by combining the event content in the event library, which can be as shown in the following table: Table 2 Trigger event table

[0024] In step 103, there is a mapping relationship between the trigger event and the deployment rules, and each trigger event can be queried to generate the corresponding deployment rules in the rule library based on the mapping relationship; The deployment rules can be generated by expert knowledge, which can be as shown in the following table: Table 3 Deployment rule table

[0025] In step 104, for all computing tasks and their required computing modes, a big data cluster deployment model is constructed, which reflects the initial form of the big data dynamic cluster required by all computing tasks.

[0026] In step 105, when the computing resource requirement in the computing mode changes, the computing resource requirement of the model involved in the computing mode needs to be adaptively adjusted according to the deployment rule to reflect the change, so as to complete the overall dynamic deployment of the big data cluster; compared with the traditional method of deploying only one computing mode corresponding to a sub-cluster, the embodiment can realize the deployment of multiple computing modes corresponding to sub-clusters at one time, dynamically adjust the form of the whole big data cluster, effectively improve the cluster resource utilization, and reduce the cluster maintenance cost.

[0027] In the traditional dynamic deployment scheme, due to the lack of consideration of the resource requirement difference of different computing modes of big data, the sub-cluster of a certain computing mode may be in high load at the same time, while the sub-cluster of other computing modes is in low load, which causes resource waste and low cluster resource utilization.

[0028] The big data cluster dynamic deployment method provided in the embodiment is based on the predicted type of the computing task, acquires the resource requirement template, generates the trigger event based on the monitoring signal in the sub-cluster of multiple computing modes, the monitoring signal includes the computing resource requirement of each sub-cluster in the resource requirement template, generates the deployment rule based on the trigger event, constructs the big data cluster deployment model based on the computing mode required by the computing task, and dynamically deploys the big data cluster based on the big data cluster deployment model and the deployment rule. In the embodiment, each sub-cluster corresponds to a computing mode, the computing resource requirement in each sub-cluster is monitored to realize real-time perception of the change of the computing resource requirement in different computing modes, and the change is quickly responded to generate the trigger event and the corresponding deployment rule. Then, the big data cluster deployment model is constructed based on the computing mode required by the computing task, and the overall dynamic deployment of the big data cluster is realized by combining the deployment rule corresponding to each computing mode in the model. In summary, the embodiment can realize real-time perception of the change of the computing resource requirement in different computing modes of big data, and sequentially associate the change with the trigger event, the deployment rule, and the big data cluster deployment model, so as to realize the overall deployment of the big data cluster, effectively improve the cluster resource utilization, and meet the resource dynamic deployment demand of different computing modes in big data processing.

[0029] Figure 3 FIG. 3 is a third flowchart of the big data cluster dynamic deployment method provided in the embodiment of the present application. Referring to FIG. 3, Figure 3 In one embodiment, step 104 can include: Step 301, constructing a four-tuple based on the computing task, the computing mode, the computing mode sequence of the computing task, and the node requirement corresponding to the computing mode sequence; Step 302, determining the four-tuple as the big data cluster deployment model.

[0030] The set of computing tasks can be denoted as J, the set of computing modes can be denoted as M, the computing mode sequence of the computing task can be denoted as I, and the node requirement corresponding to the computing mode sequence can be denoted as DC. A four-tuple DM = <J, M, I, DC> can be constructed, and the four-tuple is the big data cluster deployment model.

[0031] M = {mr, s, m, q}, where mr represents batch computing, s represents real-time computing, m represents memory computing, and q represents graph computing; and DC includes important attributes such as the computing mode and scale of the node.

[0032] Suppose J = {J1, J2, … Jn}, I = {I1, I2, … In}, and DC = {DC1, DC2, … DCn}. n J J For the computing task J1, the node requirement is to use 10 batch computing nodes, 5 real-time computing nodes, 2 batch computing nodes, and 10 memory computing nodes in sequence to complete the computing task.

[0033] The traditional method does not consider the characteristics of big data computing tasks as a whole, and cannot dynamically construct an execution cluster environment for the task.

[0034] The embodiment is based on the characteristics of big data computing tasks and multiple computing modes of the cluster, and constructs a big data cluster deployment model. The model includes all current computing tasks and their computing mode information, and therefore can realize task-level cluster customization based on the computing mode.

[0035] Figure 4 FIG. 4 is a flowchart of a big data cluster dynamic deployment method provided by an embodiment of the present application. As shown in FIG. 4, the method comprises the following steps. Figure 4 In one embodiment, step 105 can comprise: Step 401, generating a big data cluster deployment plan based on the big data cluster deployment model and the deployment rule; Step 402, converting the big data cluster deployment plan into a container command script; Step 403, executing the container command script based on the lightweight container technology to dynamically deploy the big data cluster.

[0036] In step 401, based on the deployment rule, a deployment plan can be generated for the node resources involved in different computing modes in the big data cluster deployment model, so as to generate a deployment plan for the entire big data cluster, so as to flexibly schedule and deploy the node resources subsequently.

[0037] In steps 402 to 403, the data cluster deployment plan is converted into a container command script, and the script is executed based on the lightweight container technology, so that the dynamic deployment of the big data cluster can be realized. ​​​​

[0038] The embodiment considers the characteristics of the big data computing mode, performs the big data cluster deployment plan based on the cloud native lightweight container technology, can realize the increase, deletion, update and the like of the cluster node, dynamically adjusts the cluster form, to create the optimal dynamic customization of the execution environment for the big data computing task, adapts to the dynamic change of the computing resource demand under each computing mode, and improves the cluster resource utilization.

[0039] Figure 5 Figure 5 is a flowchart of a fifth big data cluster dynamic deployment method provided by the embodiment of the application. Refer to Figure 5 In one embodiment, step 103 can include: Step 501, adding a trigger event to an event queue; Step 502, traversing the event queue, and generating corresponding deployment rules based on the trigger events in the event queue.

[0040] In step 501, each time a trigger event is generated, it is added to the event queue for subsequent ordered processing, so as to improve the processing efficiency and accuracy.

[0041] In step 502, for the current event queue, the queue is traversed, the corresponding deployment rule in the rule library is queried based on each trigger event, and the computing resource demand under the corresponding computing mode in the big data cluster deployment model is adjusted based on the deployment rule, so that the adjustment of the computing resource demand under all computing modes is gradually completed, that is, the overall deployment of the big data cluster.

[0042] It should be noted that when the deployment rule corresponding to the original trigger event in the event queue has been executed, the trigger event will be cleared; when a new trigger event is added to the event queue, step 501 can be returned, until the deployment rules of all trigger events in the event queue are generated again, and the overall deployment of the big data cluster is completed based on these deployment rules, to realize the dynamic deployment of the big data cluster.

[0043] The embodiment can orderly complete the overall deployment of the big data cluster by adding each trigger event to the event queue in turn according to the trigger time, querying the deployment rules of the trigger events in the event queue in turn by using the traversal mode, and then executing the deployment rules in turn, so as to improve the deployment efficiency and accuracy.

[0044] In one embodiment, step 102 can include: Generating a trigger event based on the size relationship between the computing resource demand and the resource threshold; Further, the monitoring signal further includes the computing resource state in each sub-cluster, and a trigger event is generated based on the size relationship between the computing resource state and the resource threshold.

[0045] Referring to Table 2, the trigger event is generated based on the size relationship between the computing resource demand and the resource threshold value, and the size relationship between the computing resource state and the resource threshold value, wherein the computing resource state is the usage of the current computing resource, which can include the usage of CPU, the usage of memory, the usage of disk and the usage of network utilization; the distributed collector is also responsible for collecting the computing resource state data, which is summarized to the event generator, and the trigger event is generated by combining the event content in the event library.

[0046] For example: events e1 and e2 are generated based on the size relationship between the computing resource demand and the resource threshold value, and events e4 and e5 are generated based on the size relationship between the computing resource state and the resource threshold value.

[0047] For each resource threshold value, it can be set by artificial experience or other algorithmic methods, which is not limited here.

[0048] Further, the trigger event can also be generated according to the size relationship between the computing resource demand and the usage of the computing resource, for example, e3 in Table 2, when the sum of the available CPU numbers of all computing mode sub-clusters is equal to the sum of the required CPU numbers of all computing mode sub-clusters, it means that the demand for computing resources reaches the upper limit of the cluster, so it is necessary to adjust the number of nodes and the configuration in each computing mode sub-cluster to improve the utilization of resources in each computing mode.

[0049] The embodiment generates the trigger event based on the size relationship between the computing resource demand and the resource threshold value, the size relationship between the computing resource state and the resource threshold value, and the size relationship between the computing resource demand and the usage of the computing resource, which can effectively represent the unreasonable demand for computing resources, the unreasonable use of computing resources, and the unreasonable relationship between demand and use, so as to generate the corresponding trigger event to obtain the deployment rule, dynamically deploy various resources in the cluster, correct the unreasonable situation, and ensure the stable operation of the big data cluster.

[0050] In one embodiment, the algorithm for dynamic deployment of the big data cluster is briefly described as follows: 1. Initialize the cluster to deploy appropriate nodes for different computing modes; 2. Initialize the event queue and add the start deployment event; 3. When the event queue is not empty: (3.1) Collect the monitoring signals in each computing mode sub-cluster; (3.2) Generate the trigger event according to the monitoring signals and add it to the event queue; (3.3) Traverse the trigger events in the event queue; (3.4) retrieving deployment rules based on the trigger event; (3.5) executing the deployment rules; 4. cyclically executing (3.1) to (3.5), and clearing the corresponding trigger event after executing the deployment rules, until the event queue is empty.

[0051] When a new trigger event is added to the event queue, return to step 3. After the algorithm generates a dynamic deployment plan of the big data cluster, the deployment plan is converted into a container command script, and the cluster is deployed based on the lightweight container technology.

[0052] The following describes a big data cluster dynamic deployment device provided by the embodiment of the application. The big data cluster dynamic deployment device described below can be correspondingly referred to the big data cluster dynamic deployment method described above.

[0053] Figure 6 is a structural schematic diagram of the big data cluster dynamic deployment device provided by the embodiment of the application. Referring to Figure 6 The embodiment of the application provides a big data cluster dynamic deployment device, which can include: The resource requirement acquisition module 601 is configured to acquire a resource requirement template based on a predicted type of a computing task. The trigger event generation module 602 is configured to generate a trigger event based on a monitoring signal in a sub-cluster of a plurality of computing modes; the monitoring signal includes a computing resource requirement of each sub-cluster in the resource requirement template. The deployment rule generation module 603 is configured to generate a deployment rule based on the trigger event. The deployment model construction module 604 is configured to construct a big data cluster deployment model based on a computing mode required by the computing task. The cluster dynamic deployment module 605 is configured to dynamically deploy a big data cluster based on the big data cluster deployment model and the deployment rule.

[0054] The big data cluster dynamic deployment device provided by the embodiment is based on a predicted type of a computing task, acquires a resource requirement template, generates a trigger event based on a monitoring signal in a subset cluster of a plurality of computing modes, the monitoring signal includes a computing resource requirement of each subset cluster in the resource requirement template, generates a deployment rule based on the trigger event, constructs a big data cluster deployment model based on a required computing mode of the computing task, and dynamically deploys the big data cluster based on the big data cluster deployment model and the deployment rule. In the embodiment, each subset cluster corresponds to a computing mode, the computing resource requirement in each subset cluster is monitored, the change of the computing resource requirement under different computing modes is perceived in real time, and the change is quickly responded to generate the trigger event and the corresponding deployment rule, the big data cluster deployment model is constructed based on the required computing mode of the computing task, the deployment rule corresponding to each computing mode in the model is combined, and the overall dynamic deployment of the big data cluster is realized. In summary, the embodiment can perceive the change of the computing resource requirement under different computing modes of big data in real time, and sequentially associate the change with the trigger event, the deployment rule, and the big data cluster deployment model, so that the overall deployment of the big data cluster is realized, the cluster resource utilization is effectively improved, and the dynamic deployment requirement of resources for different computing modes in big data processing is met.

[0055] In one embodiment, the deployment model construction module 604 is specifically configured to: construct a four-tuple based on the computing task, the computing mode, the computing mode sequence of the computing task, and the node requirement corresponding to the computing mode sequence; determine the four-tuple as a big data cluster deployment model.

[0056] In one embodiment, the cluster dynamic deployment module 605 is specifically configured to: generate a big data cluster deployment plan based on the big data cluster deployment model and the deployment rule; convert the big data cluster deployment plan into a container command script; execute the container command script based on a lightweight container technology to dynamically deploy the big data cluster.

[0057] In one embodiment, the deployment rule generation module 603 is specifically configured to: add the trigger event to an event queue; traverse the event queue, and sequentially generate a corresponding deployment rule based on the trigger event in the event queue.

[0058] In one embodiment, the trigger event generation module 602 is specifically configured to: generate a trigger event based on a size relationship between the computing resource requirement and a resource threshold.

[0059] In one embodiment, the monitoring signal further includes the computing resource status within each sub-cluster; the trigger event generation module 602 is specifically used for: Based on the relationship between the computing resource status and the resource threshold, a trigger event is generated.

[0060] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program stored in the memory 730 to execute steps of a dynamic deployment method for a big data cluster, such as: Based on the prediction type of the computing task, obtain the resource requirement template; Triggering events are generated based on monitoring signals within sub-clusters with multiple computing modes; the monitoring signals include the computing resource requirements for each sub-cluster in the resource requirement template; Based on the triggering event, deployment rules are generated; Based on the computing patterns required for the aforementioned computing tasks, a big data cluster deployment model is constructed. Based on the big data cluster deployment model and the deployment rules, the big data cluster is dynamically deployed.

[0061] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] In another aspect, the embodiments of the present application also provide a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to cause a computer to perform the steps of the method for dynamically deploying a big data cluster provided by any of the above embodiments, for example comprising: obtaining a resource requirement template based on a predicted type of the computing task; generating a trigger event based on a monitoring signal within a sub-cluster of the plurality of computing modes; the monitoring signal comprising a computing resource requirement for each sub-cluster in the resource requirement template; generating a deployment rule based on the trigger event; constructing a big data cluster deployment model based on a computing mode required by the computing task; dynamically deploying the big data cluster based on the big data cluster deployment model and the deployment rule.

[0063] In another aspect, the embodiments of the present application also provide a non-transitory computer-readable storage medium, which stores a computer program for causing a processor to perform the steps of the method for dynamically deploying a big data cluster provided by any of the above embodiments, for example comprising: obtaining a resource requirement template based on a predicted type of the computing task; generating a trigger event based on a monitoring signal within a sub-cluster of the plurality of computing modes; the monitoring signal comprising a computing resource requirement for each sub-cluster in the resource requirement template; generating a deployment rule based on the trigger event; constructing a big data cluster deployment model based on a computing mode required by the computing task; dynamically deploying the big data cluster based on the big data cluster deployment model and the deployment rule.

[0064] The non-transitory computer-readable storage medium can be any available medium or data storage device that a processor can access, including but not limited to a magnetic storage (e.g., floppy disks, hard disks, tape, MO, etc.), an optical storage (e.g., CD, DVD, BD, HVD, etc.), and a semiconductor storage (e.g., ROM, EPROM, EEPROM, NAND FLASH, SSD, etc.).

[0065] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0066] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0067] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for dynamically deploying a big data cluster, characterized in that, include: Based on the prediction type of the computing task, obtain the resource requirement template; Based on monitoring signals within sub-clusters with multiple computing modes, trigger events are generated; The monitoring signals include the computing resource requirements for each sub-cluster in the resource requirement template; Based on the triggering event, deployment rules are generated; Based on the computing patterns required for the aforementioned computing tasks, a big data cluster deployment model is constructed. Based on the big data cluster deployment model and the deployment rules, the big data cluster is dynamically deployed.

2. The method for dynamic deployment of big data clusters according to claim 1, characterized in that, The construction of a big data cluster deployment model based on the computing mode required by the computing task includes: Based on the computing task, the computing mode, the computing mode sequence of the computing task, and the node requirements corresponding to the computing mode sequence, a quadruple is constructed. The quadruple is defined as a big data cluster deployment model.

3. The method for dynamic deployment of big data clusters according to claim 1, characterized in that, The dynamic deployment of the big data cluster based on the big data cluster deployment model and the deployment rules includes: Based on the big data cluster deployment model and the deployment rules, a big data cluster deployment plan is generated; Convert the big data cluster deployment plan into a container command script; The container command script is executed using lightweight container technology to dynamically deploy the big data cluster.

4. The method for dynamic deployment of big data clusters according to claim 1, characterized in that, The generation of deployment rules based on the triggering event includes: Add the triggering event to the event queue; Traverse the event queue and generate corresponding deployment rules sequentially based on the triggering events in the event queue.

5. The method for dynamic deployment of big data clusters according to claim 1, characterized in that, The monitoring signals within the sub-cluster based on multiple computing modes generate trigger events, including: Based on the relationship between the computing resource requirements and the resource threshold, a trigger event is generated.

6. The method for dynamic deployment of big data clusters according to claim 1, characterized in that, The monitoring signals also include the computing resource status within each sub-cluster; The monitoring signals within the sub-cluster based on multiple computing modes generate trigger events, including: Based on the relationship between the computing resource status and the resource threshold, a trigger event is generated.

7. A dynamic deployment device for a big data cluster, characterized in that, include: The resource requirement acquisition module is used to: acquire resource requirement templates based on the prediction type of the computing task; The trigger event generation module is used to: generate trigger events based on monitoring signals within sub-clusters of multiple computing modes; the monitoring signals include the computing resource requirements for each sub-cluster in the resource requirement template; The deployment rule generation module is used to: generate deployment rules based on the triggering event; The deployment model building module is used to: build a big data cluster deployment model based on the computing mode required by the computing task; The cluster dynamic deployment module is used to dynamically deploy the big data cluster based on the big data cluster deployment model and the deployment rules.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the dynamic deployment method for big data clusters according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the dynamic deployment method for big data clusters according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the dynamic deployment method for big data clusters according to any one of claims 1 to 6.

Images