A method and system for graphical modeling of multi-cluster resource scheduling

Abstract

The application provides a multi-cluster resource scheduling method and system based on graphical modeling, comprising: selecting an algorithm package based on the demand of cluster resource management and establishing a business analysis model in a graphical manner by using a data set constructed by taking each cluster as a node; constructing a DAG task flow according to the business analysis model, and performing task instantiation on the DAG task flow and distributing the task to a big data platform of each cluster; and scheduling cluster resources based on the health state of each node cluster and the task instantiation distributed to each cluster; and the technical scheme of the application uniformly coordinates and allocates resources of a multi-cluster environment, which are heterogeneous and geographically dispersed.

Description

Technical Field

[0001] This invention relates to the field of graphics processing, and more specifically to a graphical modeling method and system for scheduling multi-cluster resources. Background Technology

[0002] Modeling and simulation technology is an important tool for analyzing the behavior of complex systems. Traditional manual programming-based modeling development requires modelers to have in-depth knowledge of the fundamental theories and frameworks related to system modeling, as well as proficiency in the programming languages ​​and interfaces of the corresponding computing platforms. This results in high development barriers, long development cycles, high maintenance costs, and difficulties in upgrading modules, making it difficult to meet the demands of frequent updates and efficient integration in business scenarios. Business-oriented graphical modeling platforms, however, offer a higher level of modeling than general-purpose programming languages, providing modeling analysts with an intuitive and rapid approach. They shield the underlying hardware and software platform technologies, lowering the barrier to entry for modeling development.

[0003] Research indicates that most current graphical modeling platforms are single-node / single-cluster systems operating within homogeneous networks. Their main solutions fall into the following categories:

[0004] ① A traditional graphical modeling and analysis system and its implementation method for a certain field, including a model resource management module, a model component primitive module, a model graphical construction module, and a model state analysis module;

[0005] ② A traditional graphical modeling and analysis system and its implementation method for a certain state, including a model building module, a state generation module, a state analysis module, and a statistical analysis module;

[0006] ③ A graphical modeling and analysis system for big data and its implementation method, including a big data model construction module, a model primitive generation module, and a module status analysis module.

[0007] Current graphical modeling solutions fail to consider the characteristics of heterogeneous and geographically dispersed resources in multi-cluster environments, unreliable network environments, and user needs. Furthermore, they do not consider unified model distribution requirements and parallel result collection and analysis functions, thus neglecting issues such as multi-node task concurrency, resource monitoring, and job scheduling. Summary of the Invention

[0008] To address the shortcomings of existing technologies, such as failing to consider the heterogeneous and geographically dispersed nature of resources in multi-cluster environments, unreliable network environments, and user-centric characteristics, as well as neglecting issues related to multi-node task concurrency, resource monitoring, and job scheduling, this invention provides a graphical modeling method for multi-cluster resource scheduling, comprising:

[0009] Based on the requirements of cluster resource management, select algorithm packages and build business analysis models using a graphical approach with each cluster as a node.

[0010] Based on the business analysis model, a DAG task flow is constructed, and the DAG task flow is instantiated and distributed to the big data platform of each cluster.

[0011] Cluster resources are scheduled based on the health status of the cluster at each node and the instantiation of tasks assigned to each cluster.

[0012] Preferably, the step of constructing a DAG task flow based on the business analysis model, instantiating the DAG task flow into tasks, and distributing it to the big data platform of each cluster includes:

[0013] The business analysis model is encapsulated to form an algorithm application;

[0014] The algorithm applications are managed using a tagging system;

[0015] The algorithm is applied to instantiate tasks based on parameters, and the instantiated tasks are distributed to the big data platforms of each cluster according to tags.

[0016] The parameters include: node, time / data, and operator.

[0017] Preferably, the scheduling of cluster resources based on the cluster health status at each node and the instantiation of tasks sent to each cluster includes:

[0018] The instantiated tasks issued are parsed into a DAG task flow that can be executed by the big data platform;

[0019] Select a scheduling algorithm based on the node's health status;

[0020] Resource nodes are partitioned based on executable DAG task flows, and cluster resource scheduling is performed by combining the selected scheduling algorithm and the labels applied by the algorithm.

[0021] Preferably, the step of parsing the issued instantiated task into a DAG task flow executable by the big data platform includes:

[0022] Based on the task request and model parameters in the instantiated task, the database is queried for the task ID. When the task ID is found, the workflow executor corresponding to the task ID is directly called. If not, a subtask is generated and the workflow executor is registered.

[0023] Preferably, the step of selecting a scheduling algorithm based on the node health status includes:

[0024] The overall health of a node is calculated by combining the node's attribute values, the corresponding filtering and comparison conditions, and the influencing factors of the attributes with the health formula.

[0025] When the overall health of the node is medium / poor, the perception scheduling algorithm is selected;

[0026] When the overall health of the node is excellent / good, the static scheduling algorithm is selected;

[0027] The scheduling algorithms include: static scheduling algorithm and perceptual scheduling algorithm.

[0028] Preferably, the health score formula is as follows:

[0029]

[0030] In the formula, H(v) represents the overall health of node v, p(k) represents the value of attribute k of node v, b(k) represents the filtering comparison condition corresponding to attribute k, the filter() function represents the influencing factors of attribute k, and T FL This represents the number of task flow levels.

[0031] Preferably, the step of establishing a business analysis model based on the algorithm package and dataset using a graphical method includes:

[0032] We encapsulate and configure statistical analysis, big data analysis, and professional algorithms to build an algorithm repository.

[0033] Select an algorithm package from the algorithm repository based on actual business needs;

[0034] For the computing nodes of different provincial companies, a unified data model and data index are constructed, and a data warehouse is established;

[0035] Select datasets from the data warehouse based on actual business needs;

[0036] Create a business analytics model based on the selected algorithm package and dataset.

[0037] Preferably, it also includes: real-time monitoring of task queuing progress and execution progress when scheduling cluster resources based on the health status of the cluster at each node and the task instantiation issued to each cluster, and reporting it to the server in real time.

[0038] Preferably, after scheduling cluster resources based on the cluster health status at each node and the instantiation of tasks distributed to each cluster, the method further includes:

[0039] After the task is completed, the task execution result is saved, a task history is generated, and the task result is returned to the server.

[0040] Based on the same inventive concept, this invention also provides a graphical modeling multi-cluster resource scheduling system, comprising:

[0041] A graphical algorithm modeler is used to select algorithm packages based on the needs of cluster resource management and to build business analysis models using a dataset constructed with each cluster as a node in a graphical manner; it is also used to construct DAG task flows based on the business analysis models, instantiate the DAG task flows, and distribute them to the big data platforms of each cluster.

[0042] Application-based sinking scheduling management is used to schedule cluster resources based on the health status of the cluster at each node and the instantiation of tasks sent to each cluster.

[0043] Preferably, the application sinking scheduling management includes: an interconnected application generation module, a task registration module, a task monitoring module, and a task history module;

[0044] The application generation module is used to query the database for the task ID from the task registration module. If the task ID is not found, a subtask is generated and the workflow executor is registered with the task registration module.

[0045] The task registration module is used to submit task workflows to the task monitoring module (Agent).

[0046] The task monitoring module is used to monitor the task queuing progress and execution progress, report the Agent status and task execution progress to the WebServer in a timely manner, and submit the results to the task history module after the task is completed.

[0047] The task history module is used to save task execution results, generate task history records, and return task results to the WebServer.

[0048] The task parsing submodule is used to parse the issued instantiated tasks into a DAG task flow that can be executed by the big data platform;

[0049] The algorithm selection submodule is used to select a scheduling algorithm based on the health status of the nodes;

[0050] The resource scheduling submodule is used to divide resource nodes based on executable DAG task flows and perform cluster resource scheduling in combination with the selected scheduling algorithm and the label applied by the algorithm.

[0051] The scheduling algorithms include: static scheduling algorithm and perceptual scheduling algorithm.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] 1. A graphical modeling method and system for multi-cluster resource scheduling, comprising: establishing a business analysis model graphically based on a cluster resource management demand selection algorithm package and a dataset constructed with each cluster as a node; constructing a DAG task flow according to the business analysis model, instantiating the DAG task flow into tasks, and distributing it to the big data platform of each cluster; scheduling cluster resources based on the health status of the cluster at each node and the task instantiations distributed to each cluster; the technical solution of this invention can uniformly coordinate and allocate heterogeneous and geographically dispersed resources in a multi-cluster environment. Attached Figure Description

[0054] Figure 1 This is a flowchart of a graphical modeling method for scheduling multi-cluster resources according to the present invention;

[0055] Figure 2 A functional flowchart of the multi-cluster resource scheduling system for the present invention, presented in a graphical model.

[0056] Figure 3 The interactive flowchart of the multi-cluster resource scheduling implementation method of the present invention is shown below.

[0057] Figure 4 This is a timing diagram for the task sinking calculation of the present invention;

[0058] Figure 5 A framework for translating and reorganizing graphical modeling models. Detailed Implementation

[0059] This solution comprehensively considers the functions of big data resource and algorithm manager, graphical algorithm modeler, application sinking scheduling management and resource information monitor in a multi-cluster environment. First, the model construction is abstracted into data index + algorithm description. Second, the model is translated into UXML through DAG task flow. Third, the UXML is reorganized into SXML according to node information. Finally, the workflow engine sends JSON to the multi-cluster local computation.

[0060] Objective 1: To change the traditional programming methods of big data mining and analysis by using graphical modules to replace command source code, thereby enabling the platform to empower business operations and allowing for one-stop modeling and multi-site execution.

[0061] Objective 2: For distributed multi-cluster architectures, utilize a unified model for multi-level and multi-method distribution and scheduling to standardize and streamline business indicator calculations, thereby accelerating the efficiency of big data analysis and processing.

[0062] Example 1:

[0063] This invention provides a graphical modeling method for scheduling multi-cluster resources, such as... Figure 1 and Figure 3 As shown, it includes:

[0064] Step 1: Based on the requirements of cluster resource management, select algorithm packages and build a business analysis model using a graphical method with each cluster as a node;

[0065] Step 2: Construct a DAG task flow based on the business analysis model, instantiate the DAG task flow into tasks, and distribute them to the big data platform of each cluster;

[0066] Step 3: Schedule cluster resources based on the health status of the cluster at each node and the instantiation of tasks sent to each cluster.

[0067] Step 1: Based on the requirements of cluster resource management, select algorithm packages and build a business analysis model using each cluster as a node, and then use a graphical approach to establish the model. This includes:

[0068] We encapsulate and configure statistical analysis, big data analysis, and professional algorithms to build an algorithm repository.

[0069] Select an algorithm package from the algorithm repository based on actual business needs;

[0070] For the computing nodes of different provincial companies, a unified data model and data index are constructed, and a data warehouse is established;

[0071] Select datasets from the data warehouse based on actual business needs;

[0072] Create a business analytics model based on the selected algorithm package and dataset.

[0073] Step 2: The step of constructing a DAG task flow based on the business analysis model, instantiating the DAG task flow into tasks, and distributing them to the big data platforms of each cluster specifically includes:

[0074] The business analysis model is encapsulated to form an algorithm application;

[0075] The algorithm applications are managed using a tagging system;

[0076] The algorithm is applied to instantiate tasks based on parameters, and the instantiated tasks are distributed to the big data platforms of each cluster according to tags.

[0077] The parameters include: node, time / data, and operator.

[0078] Step 3: The scheduling of cluster resources based on the cluster health status at each node and the instantiation of tasks assigned to each cluster specifically includes:

[0079] The instantiated tasks issued are parsed into a DAG task flow that can be executed by the big data platform;

[0080] Select a scheduling algorithm based on the node's health status;

[0081] Resource nodes are partitioned based on executable DAG task flows, and cluster resource scheduling is performed by combining the selected scheduling algorithm and the labels applied by the algorithm.

[0082] Preferably, the step of parsing the issued instantiated task into a DAG task flow executable by the big data platform includes:

[0083] Based on the task request and model parameters in the instantiated task, the database is queried for the task ID. When the task ID is found, the workflow executor corresponding to the task ID is directly called. If not, a subtask is generated and the workflow executor is registered.

[0084] Preferably, the step of selecting a scheduling algorithm based on the node health status includes:

[0085] The overall health of a node is calculated by combining the node's attribute values, the corresponding filtering and comparison conditions, and the influencing factors of the attributes with the health formula.

[0086] When the overall health of the node is medium / poor, the perception scheduling algorithm is selected;

[0087] When the overall health of the node is excellent / good, the static scheduling algorithm is selected;

[0088] The scheduling algorithms include: static scheduling algorithm and perceptual scheduling algorithm.

[0089] Preferably, the health score formula is as follows:

[0090]

[0091] In the formula, H(v) represents the overall health of node v, p(k) represents the value of attribute k of node v, b(k) represents the filtering comparison condition corresponding to attribute k, the filter() function represents the influencing factors of attribute k, and T FL This represents the number of task flow levels.

[0092] Preferably, it also includes: real-time monitoring of task queuing progress and execution progress when scheduling cluster resources based on the health status of the cluster at each node and the task instantiation sent to each cluster, and reporting it to the server in real time.

[0093] Preferably, after scheduling cluster resources based on the cluster health status at each node and the instantiation of tasks distributed to each cluster, the method further includes:

[0094] After the task is completed, the task execution result is saved, a task history is generated, and the task result is returned to the server.

[0095] Example 2:

[0096] Based on the same inventive concept, this invention also provides a graphical modeling multi-cluster resource scheduling system, such as... Figure 2 As shown, it includes:

[0097] A graphical algorithm modeler is used to select algorithm packages based on the needs of cluster resource management and to build business analysis models using a dataset constructed with each cluster as a node in a graphical manner; it is also used to construct DAG task flows based on the business analysis models, instantiate the DAG task flows, and distribute them to the big data platforms of each cluster.

[0098] Application-based sinking scheduling management is used to schedule cluster resources based on the health status of the cluster at each node and the instantiation of tasks sent to each cluster.

[0099] The application sinking scheduling management includes: an interconnected application generation module, a task registration module, a task monitoring module, and a task history module;

[0100] The application generation module is used to query the database for the task ID from the task registration module. If the task ID is not found, a subtask is generated and the workflow executor is registered with the task registration module.

[0101] The task registration module is used to submit task workflows to the task monitoring module (Agent).

[0102] The task monitoring module is used to monitor the task queuing progress and execution progress, report the Agent status and task execution progress to the WebServer in a timely manner, and submit the results to the task history module after the task is completed.

[0103] The task history module is used to save task execution results, generate task history records, and return task results to the WebServer.

[0104] The graphical algorithm modeler includes: a data resource manager and an algorithm resource manager;

[0105] First, the Data Explorer is used to manage multi-source data indexes, including dataset names, database types, table indexes, data exploration, and data tracing functions. {k1,k2,…,k n} is a collection with table names as keys, {{d 11 ,d 12 ,…,d 1m},{d 21 ,d 22 ,…,d 2m},…,{dn1 ,d n2 ,…,d nm}} is a multi-source data mapping matrix corresponding to the keywords. For example, d 11 It represents a series of related attributes {NodeIP, Port, Database, DatabaseType, DataSize, UpdateTime}, where NodeIP is the primary key.

[0106] The Algorithm Resource Manager is used to manage business algorithm packages, including algorithm package name, database type, shell script commands, interface parameters, and operator parameters.

[0107] The algorithm models in this scheme are divided into two categories: immutable parameter algorithms and variable parameter algorithms.

[0108] Immutable parameter algorithm: also known as static algorithm. Many classic algorithms have known defined parameter interfaces and numbers. The algorithm model generates a ZIP package containing a JAR algorithm package and corresponding HTML parameter descriptions. After uploading, the HTML is used to automatically parse the types and number of interfaces of input parameters, output parameters and operator parameters. This method is often used to define general-purpose algorithm models.

[0109] Variable parameter algorithm: also known as dynamic algorithm. For some algorithm models with custom or variable number of parameters, since the required parameters of the algorithm model cannot be determined in advance, its input parameter src_path_list, output parameter dst_path_list, and operator parameter script_path are defined as String type, and the parameter type and number of interfaces are dynamically adjusted during modeling and analysis.

[0110] A graphical modeling multi-cluster resource scheduling system, which also includes a resource information monitor;

[0111] The resource information monitor is used to perceive global status information, including channel health status perception, node running status perception, and cluster busy status perception.

[0112] A specific graphical modeling-based multi-cluster resource scheduling system is described below:

[0113] First, the Data Explorer is used to manage multi-source data indexes, including dataset names, database types, table indexes, data exploration, and data tracing functions. {k1,k2,…,k n} is a collection with table names as keys, {{d 11 ,d 12 ,…,d 1m},{d 21 ,d 22 ,…,d 2m},…,{dn1 ,d n2 ,…,d nm}} is a multi-source data mapping matrix corresponding to the keywords. For example, d 11 It represents a series of related attributes {NodeIP, Port, Database, DatabaseType, DataSize, UpdateTime}, where NodeIP is the primary key.

[0114] The Algorithm Resource Manager is used to manage business algorithm packages, including algorithm package name, database type, shell script commands, interface parameters, and operator parameters.

[0115] The algorithm models in this scheme are divided into two categories: immutable parameter algorithms and variable parameter algorithms.

[0116] Immutable parameter algorithm: also known as static algorithm. Many classic algorithms have known defined parameter interfaces and numbers. The algorithm model generates a ZIP package containing a JAR algorithm package and corresponding HTML parameter descriptions. After uploading, the HTML is used to automatically parse the types and number of interfaces of input parameters, output parameters and operator parameters. This method is often used to define general-purpose algorithm models.

[0117] Variable parameter algorithm: also known as dynamic algorithm. For some algorithm models with custom or variable number of parameters, since the required parameters of the algorithm model cannot be determined in advance, its input parameter src_path_list, output parameter dst_path_list, and operator parameter script_path are defined as String type, and the parameter type and number of interfaces are dynamically adjusted during modeling and analysis.

[0118] Secondly, the graphical algorithm modeler is used to construct and manage data primitives and algorithm primitives, including a model building module, a model translation module, and a model reorganization module. The overall framework is as follows: Figure 5 As shown, define a six-tuple: <Meta multi , Parameter, Translate RuleSet, UXML HPWF ,Transformation Engine,SXML HPWF >. Among them, Meta multi The index represents the multi-source data of the dataset; the parameter represents the application instantiation parameters, such as node parameters, trigger parameters, and operator parameters; the Translate RuleSet represents the set of translation rules; UXML HPWF The XML metamodel for Hadoop workflows is highly abstract and independent of data sources and node parameters, lacking actual task scheduling capabilities; SXML HPWFThe Hadoop workflow execution model enables task scheduling for a specific node; the Transformation Engine is the translation engine, utilizing predefined metadata and instantiation parameters, and employing translation rules in the RuleSet to translate UXML... HPWF Translated into SXML HPWF .

[0119] The formal description of its recombination process is as follows:

[0120] U(D,A,S)+P k +H k ->S k (d,a,s,p,h)

[0121] Where U(D,A,S) is UXML HPWF Meta-model description, P k H is the set of model parameters for node k. k S is used to calculate the health of node k. k (d,a,s,p,h) represents SXML HPWF Metamodel description.

[0122] Secondly, the application-based sinking scheduling management is used to manage model distribution, including the application generation module, task registration module, task monitoring module, and task history module. The application-based sinking scheduling management sequence diagram is as follows: Figure 4 As shown.

[0123] After the task is instantiated on the WebServer using the graphical algorithm modeler, the task request and model parameters are submitted to the task generation module. The application generation module queries the task registration module for the task ID in the database. If the task ID is not found, a subtask is generated, and the workflow executor is registered with the task registration module. The task registration module submits the task workflow to the task monitoring module (Agent). The task monitoring module monitors the task queuing progress and execution progress, and promptly reports the Agent status and task execution progress to the WebServer. Once the task is completed, the result is submitted to the task history module to save the task execution result, generate a task history record, and return the task result to the WebServer.

[0124] Finally, the resource information monitor is used to perceive global status information, including channel health status, node operating status, and cluster busy status. This solution proposes a comprehensive formula for determining node health:

[0125]

[0126] Where H(v) is the overall calculated health of node v, p(k) is the value of attribute k obtained by node v, b(k) is the filtering comparison condition corresponding to attribute k, the filter() function is the value of the influencing factor s × weight w of attribute k, and T FL The task flow hierarchy has a minimum of 3 levels. The global state attributes and their comprehensive health factors are defined in Table 1.

[0127] Table 1. Definition of Global Status Attributes and Their Comprehensive Calculation: Health Factors

[0128]

[0129]

[0130] Based on the comprehensive health score, the health status of nodes can be divided into four levels: excellent, good, average, and poor, as shown in Table 2.

[0131] Table 2 Node Health Status Description Table

[0132] Health description Comprehensive calculation of health status excellent >0.8 good <0.8 and >0.5 middle <0.5 and >0.3 Difference <0.3

[0133] The scheduling methods in this scheme are divided into two categories: static scheduling algorithms and perceptual scheduling algorithms.

[0134] Static scheduling, also known as pre-scheduling, involves requesting or reserving scheduling resources for a cluster after the task flow has been reorganized. When the cluster is relatively idle or has exclusive access to resources, this approach can quickly obtain the results of the sinking tasks; however, when the cluster is busy or lacks sufficient resources, the performance of the sinking tasks may be poor, and the tasks may fail due to insufficient resources.

[0135] Perceptive scheduling, also known as real-time scheduling, involves calculating the health status of tasks before task flow reorganization. Tasks are then delegating based on trigger conditions (immediate or timed). When a node's calculated health status is medium / poor, the task enters a waiting state for deployment. Tasks are deployed only when a node's health status recovers to excellent / good. This approach can achieve a high task execution success rate, but its efficiency is relatively lower than static scheduling.

[0136] The technical solution provided by this invention achieves the following effects:

[0137] 1. By combining graphical modeling with autonomous design, a cross-domain indicator linkage model is established at the data index layer, a fusion mode of professional algorithms and big data algorithms is constructed, and a flexible combination mode of algorithms is designed, thereby realizing multi-level penetration and sinking of algorithm applications, computational decomposition of sinking calculations, distributed parallel computing, and comprehensive scheduling of computing tasks.

[0138] 2. Through the graphical modeling and analysis system, the graphical representation of big data mining and analysis was realized, simplifying the complexity of big data analysis and achieving the goal of empowering business through the platform.

[0139] 3. By applying decentralized scheduling management, multi-mode parallel deployment scheduling with a unified algorithm model is realized in a distributed multi-cluster environment, which standardizes and streamlines the calculation of business indicators and improves the efficiency of big data analysis and processing.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A graphical modeling method for scheduling multi-cluster resources, characterized in that, include: Based on the requirements of cluster resource management, select algorithm packages and build business analysis models using a graphical approach with each cluster as a node. Based on the business analysis model, a DAG task flow is constructed, and the DAG task flow is instantiated and distributed to the big data platform of each cluster. Cluster resources are scheduled based on the health status of the cluster at each node and the instantiation of tasks sent to each cluster. The scheduling of cluster resources based on the health status of the cluster at each node and the instantiation of tasks assigned to each cluster includes: The instantiated tasks issued are parsed into a DAG task flow that can be executed by the big data platform; Select a scheduling algorithm based on the node's health status; Resource nodes are divided based on executable DAG task flows, and cluster resource scheduling is performed by combining the selected scheduling algorithm and the labels applied by the algorithm. The parsing of the issued instantiated tasks into a DAG task flow executable by the big data platform includes: Based on the task requests and model parameters in the instantiated task, the database task ID is queried. When the task ID is found, the workflow executor corresponding to the task ID is directly called. If not, a subtask is generated and the workflow executor is registered. The step of selecting a scheduling algorithm based on the node's health status includes: The overall health of a node is calculated by combining the node's attribute values, the corresponding filtering and comparison conditions, and the attribute's influence factors with the health formula. When the overall health of the node is medium / poor, the perception scheduling algorithm is selected; When the overall health of the node is excellent / good, the static scheduling algorithm is selected; The scheduling algorithms include: static scheduling algorithm and perceptual scheduling algorithm; The health score formula is shown below: In the formula, H(v) represents the overall health of node v, p(k) represents the value of attribute k of node v, b(k) represents the filtering comparison condition corresponding to attribute k, the filter() function represents the influencing factors of attribute k, and T FL This represents the number of task flow levels.

2. The scheduling method as described in claim 1, characterized in that, The step of constructing a DAG task flow based on the business analysis model, instantiating the DAG task flow into tasks, and distributing it to the big data platform of each cluster includes: The business analysis model is encapsulated to form an algorithm application; The algorithm applications are managed using tags; The algorithm is applied to instantiate tasks based on parameters, and the instantiated tasks are distributed to the big data platforms of each cluster according to tags. The parameters include: node, time / data, and operator.

3. The scheduling method as described in claim 1, characterized in that, The requirement algorithm package based on cluster resource management and the dataset constructed with each cluster as a node are used to establish a business analysis model in a graphical manner, including: We encapsulate and configure statistical analysis, big data analysis, and professional algorithms to build an algorithm repository. Select an algorithm package from the algorithm repository based on actual business needs; For the computing nodes of different provincial companies, a unified data model and data index are constructed, and a data warehouse is established; Select datasets from the data warehouse based on actual business needs; Create a business analytics model based on the selected algorithm package and dataset.

4. The scheduling method as described in claim 1, characterized in that, Also includes: Based on the health status of the cluster at each node and the scheduling of cluster resources when instantiating tasks sent to each cluster, the system monitors the queuing and execution progress of tasks in real time and reports them to the server in real time.

5. The scheduling method as described in claim 4, characterized in that, After scheduling cluster resources based on the cluster health status at each node and the instantiation of tasks assigned to each cluster, the process also includes: After the task is completed, the task execution result is saved, a task history is generated, and the task result is returned to the server.

6. A system for implementing the graphical modeling multi-cluster resource scheduling method as described in any one of claims 1-5, characterized in that, include: A graphical algorithm modeler is used to select algorithm packages based on the needs of cluster resource management and to build business analysis models using a dataset constructed with each cluster as a node in a graphical manner; it is also used to construct DAG task flows based on the business analysis models, instantiate the DAG task flows, and distribute them to the big data platforms of each cluster. Application-based sinking scheduling management is used to schedule cluster resources based on the health status of the cluster at each node and the instantiation of tasks sent to each cluster.

Images