A cluster data monitoring and evaluation system and method based on artificial intelligence

Abstract

The present invention discloses a cluster data monitoring and evaluation system and method based on artificial intelligence, which relates to the technical field of data cluster management, obtains historical records of computing tasks processed by a computing cluster, obtains task templates of different computing tasks, obtains fault records of collaborative work of computing nodes, obtains the stability coefficient of any two computing nodes in the computing cluster when running at the same time, aggregates the computing tasks currently running in the computing cluster, obtains node resources released by the computing cluster in different time periods, aggregates computing nodes that are idle in each time period into an idle node list, aggregates requests for computing tasks newly added in the computing cluster, matches computing nodes in the computing cluster that are in idle time periods according to the task templates of the newly added computing tasks, and calculates the matching degree of each computing task according to the matching degree between the task template and the computing node and the stability coefficient of the computing node.

Description

Technical Field

[0001] The present invention relates to the technical field of data cluster management, and in particular to an artificial intelligence-based cluster data monitoring and evaluation system and method. Background Art

[0002] The efficiency of computing tasks can be improved by forming a computing cluster with multiple computing nodes, which is widely used in scientific computing, deep learning, and training artificial intelligence models. For example, a computing cluster composed of multiple graphics processing units (GPUs) can handle the computational workload of larger neural network models.

[0003] In current application scenarios, there are defects in the orchestration and scheduling of computing tasks in computing clusters. The task perception ability and the ability to coordinate tasks across nodes are insufficient, and there is a lack of scheduling methods for fine-grained allocation of computing nodes. In the current management of computing clusters, nodes are mainly allocated based on existing operation plans. There is a lack of means to flexibly adjust computing resources for different tasks, which makes it impossible to fully utilize the computing resources in the cluster. Summary of the invention

[0004] The purpose of the present invention is to provide a cluster data monitoring and evaluation system and method based on artificial intelligence to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a cluster data monitoring and evaluation method based on artificial intelligence, the method comprising:

[0006] Step S100: in a computing cluster including at least two computing nodes, obtaining a history record of computing tasks processed by the computing cluster, where the computing tasks are computing tasks collaboratively processed by at least two computing nodes, and obtaining task templates of different computing tasks;

[0007] Step S200: Obtaining the fault records of the collaborative work of the computing nodes, and obtaining the stability coefficient when any two computing nodes in the computing cluster are running simultaneously;

[0008] Step S300: collect the computing tasks currently running in the computing cluster, obtain the node resources released by the computing cluster in different time periods, and collect the computing nodes that are idle in each time period into an idle node list;

[0009] Step S400: collect requests for newly added computing tasks in the computing cluster, match computing nodes in the computing cluster that are in an idle time period according to the task template of the newly added computing task, and calculate the matching degree of each computing task according to the matching degree between the task template and the computing node and the stability coefficient of the computing node;

[0010] Step S500: Arrange the newly added computing tasks according to the matching degree of each computing task, obtain the difference between the computing nodes provided by the computing cluster and the computing nodes corresponding to the computing tasks, and submit a computing node application to relevant management personnel.

[0011] Furthermore, step S100 includes:

[0012] Step S101: Obtain a certain computing task from the historical records as the target task, obtain the running process L of the target task in the computing cluster, and divide the running process into several computing steps according to the computing nodes occupied by different processing stages of the computing task, wherein sti represents the computing node running the target task in the i-th computing step in the running process L, and sti includes at least one computing node. When sti includes more than one computing node, sti also includes the connection relationship of the computing nodes;

[0013] Step S102: Get the length t 0 Unit time, extract the historical records of the target task, obtain the running time of each operation step, and divide the running time of the operation step into several unit times;

[0014] Step S103: Get the i-th operation step in the running process L, the corresponding number of unit time ni, obtain the sub-template (sti, ni) of the i-th operation step, and collect the sub-templates of all operation steps of the target task in the order of the operation steps to obtain the task template of the target task.

[0015] Further, step S200 includes:

[0016] Step S201: Obtain from historical records any two computing nodes that have serial collaboration or parallel collaboration to form a node pair, and obtain historical operation records of the node pair;

[0017] Step S202: Obtain the normal state operation time T1, abnormal state operation time T2 and total maintenance time T3 of the node pair, and calculate the stability coefficient η, η=(T1-T2-T3) / T1;

[0018] Among them, when the node pair is in parallel collaboration, the normal state operation time T1, abnormal state operation time T2 and total maintenance time T3 are respectively the time when the two computing nodes are in normal operation, abnormal operation or maintenance at the same time; when the node pair is in serial collaboration, the normal state operation time T1 is the total operation time when the two computing nodes have a serial operation relationship, the abnormal state operation time T2 and the total maintenance time T3 are respectively the abnormal state operation time and maintenance time of the latter computing node in the node pair;

[0019] It is generally believed that in a computing cluster, the normal operation time of the computing cluster accounts for the majority of the total operation time of the computing cluster, and the abnormal operation time and maintenance time account for a small part of the total operation time. In the scheme, T1, T2 and T3 meet the condition: T1>T2+T3;

[0020] The collaborative relationships between different computing nodes are collected from historical records, and the collaborative relationships are simplified and transformed into collaborative relationships between computing nodes. The collaborative records of computing nodes are traversed to the greatest extent possible to obtain the collaborative relationship between two computing nodes working at the same time. Computing nodes that are not suitable for collaborative work are screened out to reduce the adverse effects on computing tasks.

[0021] Furthermore, step S300 includes:

[0022] Step S301: Obtain a task template for each computing task in the task sequence, and obtain a computing node allocation plan for the computing task according to the total amount of computing nodes occupied by each computing step in the task template;

[0023] Step S302: Record the time period when the computing node performs computing tasks as the working time period, and record the time period when no computing tasks are performed as the idle time period, obtain the working time period and idle time period of each computing node, and collect them to obtain the task plan of each computing node;

[0024] Step S303: Taking the time when the first task in the task sequence starts to execute as the starting time, align the task plans of the computing nodes. After the starting time, gather the computing nodes in the same idle time period separately to obtain the computing nodes in the idle time period for each unit time.

[0025] Furthermore, step S400 includes:

[0026] Step S401: collect the newly added computing tasks into the first task set, and obtain any computing task taskA in the first task set;

[0027] Step S402: Obtain the computing units corresponding to each computing step and the number of unit times corresponding to each computing step from the task template of taskA, and obtain the computing node stAj corresponding to the j-th computing step of the computing task taskA;

[0028] Step S403: According to the running time of the j-th operation step, obtain the computing nodes that meet the connection mode in stAj from the idle node list. When the running time of the j-th operation step matches the computing nodes in at least one idle time period, add them into the node candidate set Hj of the j-th operation step. When the running time of the j-th operation step does not match the computing nodes in the idle time period, record the j-th operation step as an unmatched step.

[0029] Step S404: Get the number of computing nodes in Hj as NA, calculate the matching degree αj of the jth operation step, αj = NA / NAj, where NAj represents the number of corresponding computing nodes in stAj, and calculate the sum of the matching degrees of all operation steps in taskA as R A ;

[0030] Step S405: Obtain the connection mode of each node in Hj, and obtain the stability coefficient between any two computing nodes, sum the stability coefficients in Hj, and record the sum of the stability coefficients in Hj as K j ;

[0031] The stability coefficient describes the probability of stable operation between two nodes. The stability coefficients of any two connected nodes are accumulated to obtain the evaluation value of the overall stable working state of Hj;

[0032] Step S406: Calculate the matching degree C of taskA A , Among them, m A represents the total number of unmatched steps in taskA, M A Represents the total number of computation steps in taskA;

[0033] The matching degree calculation process is divided into three parts: A -m A ) / M A Indicates that there is a matching relationship among several computing steps in the computing template, that is, for a certain computing step, at least one computing node satisfies the matching relationship. When there is no matching computing node in the computing cluster for a certain computing step, A Generate counts in

[0034] R A The matching degree of each computing step, especially in the case of parallel computing through computing nodes, needs to obtain the satisfaction ratio of computing nodes connected in parallel, which will affect the computing nodes that need to be additionally applied for in the subsequent steps. For the current computing cluster, the existing computing nodes are used as much as possible, and computing tasks that require fewer additional computing nodes are given priority to the computing cluster;

[0035] It indicates that the stability coefficient is summed. When the stability coefficient is higher, it means that the computing nodes are more stable when working together. The computing units with stable node collaboration are preferred to perform computing tasks.

[0036] Furthermore, step S500 includes:

[0037] Step S501: Calculate the matching degree of each computing task in the first task set, and arrange them from high to low according to the matching degree to obtain a second task set;

[0038] Step S502: Get the first computing task task21 in the second task set, and get the computing node st corresponding to the pth computing step in task21. 21 p and node candidate set H 21 p , will st 21 p and H 21 p Compare and obtain st 21 p The missing computing nodes are denoted as st 21 p2 ;

[0039] Step S503: Gather the computing nodes missing from each step of computing task task21, obtain a list of nodes to be configured, and send an application for adding new computing nodes to relevant managers. When the computing nodes are added accordingly, input computing task task21 into the computing cluster.

[0040] In order to better implement the above method, a cluster data monitoring and evaluation system based on artificial intelligence is also proposed. The system includes: a task template management module, a stability coefficient management module, an idle node management module, a task matching management module and an information reminder module, wherein the task template management module is used to manage the task templates of the computing tasks, the stability coefficient management module is used to manage the stability coefficients between the computing nodes, the idle node management module is used to manage the idle nodes in each time period, the task matching management module is used to manage the task matching of the newly added tasks, and the information reminder module is used to provide information reminders to relevant management personnel;

[0041] Furthermore, the task template management module includes: a historical data management unit, an operation step management unit and a template collection unit, the historical data management unit is used to manage the historical operation records of the computing nodes in the computing cluster, the operation step management unit is used to manage the sub-templates corresponding to the operation steps, and the template collection unit is used to collect the sub-templates to obtain the task template;

[0042] Further, the stability coefficient management module includes: a fault management unit and a stability calculation unit, wherein the fault management unit is used to manage fault records in the collaborative process of computing nodes, and the stability calculation unit is used to calculate the stability coefficient between computing nodes;

[0043] Further, the idle node management module includes: a node plan management unit and an idle node management unit, wherein the node plan management unit is used to manage the computing node allocation plan of the computing task, and the idle node management unit is used to collect the computing nodes in the idle period to obtain an idle node list;

[0044] Further, the task matching management module includes: a task buffer unit, a step matching unit, a first matching calculation unit and a second matching calculation unit, wherein the task buffer unit is used to store the newly added computing task requests in the computing cluster, the step matching unit is used to obtain the task template of the computing task, obtain the computing unit corresponding to each computing step, and match them in the computing cluster, the first matching calculation unit is used to calculate the matching degree of the computing step, and the second matching calculation unit is used to obtain the matching degree of the computing task;

[0045] Furthermore, the information reminder module includes: a task sorting unit, a missing node management unit and a node application unit, wherein the task sorting unit is used to arrange the newly added computing tasks, the missing node management unit is used to obtain the missing computing units in each calculation step, and the node application unit is used to generate a list of newly added computing nodes based on the missing computing units and send information reminders to relevant managers.

[0046] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention forms task templates for different computing tasks by collecting historical data, finds idle computing nodes by auditing existing plans in the computing cluster, further compares the collaborative relationship of computing nodes, and allocates computing tasks to idle computing nodes. The present invention provides a computing task perception and computing node calling method based on fine-grainedness, forms a multi-mode computing power scheduling capability, and improves the utilization rate of host resources such as GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 It is a structural schematic diagram of a cluster data monitoring and evaluation system based on artificial intelligence of the present invention;

[0049] Figure 2 It is a flow chart of a cluster data monitoring and evaluation method based on artificial intelligence of the present invention;

[0050] Table 1 is a distribution table of idle nodes of a cluster data monitoring and evaluation method based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0053] Step S100: in a computing cluster including at least two computing nodes, obtaining a history record of computing tasks processed by the computing cluster, where the computing tasks are computing tasks collaboratively processed by at least two computing nodes, and obtaining task templates of different computing tasks;

[0054] Wherein, step S100 includes:

[0055] Step S101: Obtain a certain computing task from the historical records as the target task, obtain the running process L of the target task in the computing cluster, and divide the running process into several computing steps according to the computing nodes occupied by different processing stages of the computing task, wherein sti represents the computing node running the target task in the i-th computing step in the running process L, and sti includes at least one computing node. When sti includes more than one computing node, sti also includes the connection relationship of the computing nodes;

[0056] Step S102: Get the length t 0 Unit time, extract the historical records of the target task, obtain the running time of each operation step, and divide the running time of the operation step into several unit times;

[0057] Step S103: Get the i-th operation step in the running process L, the number of unit time ni corresponding to it, obtain the sub-template (sti, ni) of the i-th operation step, collect the sub-templates of all operation steps of the target task in the order of the operation steps, and obtain the task template of the target task;

[0058] In the embodiment, a computing task taskA, the operation process of taskA is LA, wherein LA includes three computing steps, namely, st1, st2, and st3, st1 includes two computing nodes, the two computing nodes are connected in parallel, st2 includes one computing node, st1 and st2 are connected in series, and st3 includes four computing nodes, the four computing nodes are connected in parallel;

[0059] It takes 4 time units to execute operation step st1, 2 time units to execute operation step st2, and 8 time units to execute st3. The template ModA of taskA is obtained by combining them. ModA: {(st1, 4), (st2, 2), (st3, 8)}.

[0060] Step S200: Obtaining the fault records of the collaborative work of the computing nodes, and obtaining the stability coefficient when any two computing nodes in the computing cluster are running simultaneously;

[0061] Wherein, step S200 includes:

[0062] Step S201: Obtain from historical records any two computing nodes that have serial collaboration or parallel collaboration to form a node pair, and obtain historical operation records of the node pair;

[0063] Step S202: Obtain the normal state operation time T1, abnormal state operation time T2 and total maintenance time T3 of the node pair, and calculate the stability coefficient η, η=(T1-T2-T3) / T1;

[0064] Among them, when the node pair is in parallel collaboration, the normal state operation time T1, abnormal state operation time T2 and total maintenance time T3 are respectively the time when the two computing nodes are in normal operation, abnormal operation or maintenance at the same time; when the node pair is in serial collaboration, the normal state operation time T1 is the total operation time when the two computing nodes have a serial operation relationship, the abnormal state operation time T2 and the total maintenance time T3 are respectively the abnormal state operation time and maintenance time of the latter computing node in the node pair;

[0065] In the embodiment, in computing task taskA, st1 includes computing nodes a1 and a2, st2 includes computing node b1, and st3 includes c1, c2, c3, and c4;

[0066] Among them, a1 and a2 form a parallel collaborative node pair; a1 and b1, a2 and b1 form serial node pairs respectively, b1 and c1, c2, c3 and c4 form 4 serial node pairs respectively; c1 and c2, c1 and c3, c1 and c4, c2 and c3, c2 and c4, c3 and c4 form 6 parallel node pairs respectively.

[0067] Step S300: collect the computing tasks currently running in the computing cluster, obtain the node resources released by the computing cluster in different time periods, and collect the computing nodes that are idle in each time period into an idle node list;

[0068] Wherein, step S300 includes:

[0069] Step S301: Obtain a task template for each computing task in the task sequence, and obtain a computing node allocation plan for the computing task according to the total amount of computing nodes occupied by each computing step in the task template;

[0070] Step S302: Record the time period when the computing node performs computing tasks as the working time period, and record the time period when no computing tasks are performed as the idle time period, obtain the working time period and idle time period of each computing node, and collect them to obtain the task plan of each computing node;

[0071] Step S303: taking the time when the first task in the task sequence starts to be executed as the time starting point, aligning the task plans of the computing nodes, and after the time starting point, respectively gathering the computing nodes in the same idle time period to obtain the computing nodes in the idle time period for each unit time;

[0072] Table 1 Distribution table of idle nodes

[0073]

[0074] For example, Table 1 shows a distribution table of idle nodes. In the table, U1, U2, U3, ... and U7 respectively represent the 1st, 2nd, 3rd, ... 7th computing nodes in the computing cluster. The unit time column represents statistics on whether the computing node is occupied in the unit time after the start time, where "00" indicates that the computing node is in an idle state in the corresponding unit time, and "0x" indicates that the computing node is in a working state in the corresponding unit time.

[0075] Step S400: collect requests for newly added computing tasks in the computing cluster, match computing nodes in the computing cluster that are in an idle time period according to the task template of the newly added computing task, and calculate the matching degree of each computing task according to the matching degree between the task template and the computing node and the stability coefficient of the computing node;

[0076] Wherein, step S400 includes:

[0077] Step S401: collect the newly added computing tasks into the first task set, and obtain any computing task taskA in the first task set;

[0078] Step S402: Obtain the computing units corresponding to each computing step and the number of unit times corresponding to each computing step from the task template of taskA, and obtain the computing node stAj corresponding to the j-th computing step of the computing task taskA;

[0079] Step S403: According to the running time of the j-th operation step, obtain the computing nodes that meet the connection mode in stAj from the idle node list. When the running time of the j-th operation step matches the computing nodes in at least one idle time period, add them into the node candidate set Hj of the j-th operation step. When the running time of the j-th operation step does not match the computing nodes in the idle time period, record the j-th operation step as an unmatched step.

[0080] Step S404: Get the number of computing nodes in Hj as NA, calculate the matching degree αj of the jth operation step, αj = NA / NAj, where NAj represents the number of corresponding computing nodes in stAj, and calculate the sum of the matching degrees of all operation steps in taskA as R A ;

[0081] Step S405: Obtain the connection mode of each node in Hj, and obtain the stability coefficient between any two computing nodes, sum the stability coefficients in Hj, and record the sum of the stability coefficients in Hj as K j ;

[0082] Step S406: Calculate the matching degree C of taskA A , Among them, m A represents the total number of unmatched steps in taskA, M A Indicates the total number of computation steps in taskA.

[0083] Step S500: Arrange the newly added computing tasks according to the matching degree of each computing task, obtain the difference between the computing nodes provided by the computing cluster and the computing nodes corresponding to the computing tasks, and submit a computing node application to the relevant management personnel;

[0084] Wherein, step S500 includes:

[0085] Step S501: Calculate the matching degree of each computing task in the first task set, and arrange them from high to low according to the matching degree to obtain a second task set;

[0086] Step S502: Get the first computing task task21 in the second task set, and get the computing node st corresponding to the pth computing step in task21. 21 p and node candidate set H 21 p , will st 21 p and H 21 p Compare and obtain st 21 p The missing computing nodes are denoted as st21 p2 ;

[0087] Step S503: collect the computing nodes missing in each step of the computing task task21, obtain a list of nodes to be configured, send an application for adding computing nodes to relevant managers, and when the computing nodes are added accordingly, input the computing task task21 into the computing cluster;

[0088] For example, the computing task taskB requires 4 parallel computing nodes to perform continuous computing for 2 unit times. Referring to Table 1, the possible solutions include: inserting at the 6th unit time after the starting time and calling U3 and U6; inserting at the 8th unit time after the starting time and calling U2 and U7, and obtaining the preferred solution by executing the comparison method in step S400; if inserting at the 4th unit time and calling U2, it is necessary to apply for an additional parallel connection with U2 at the 4th and 5th unit time.

[0089] The system includes: task template management module, stability coefficient management module, idle node management module, task matching management module and information reminder module;

[0090] The task template management module is used to manage the task templates of the computing tasks, wherein the task template management module includes: a historical data management unit, an operation step management unit and a template collection unit, the historical data management unit is used to manage the historical operation records of the computing nodes in the computing cluster, the operation step management unit is used to manage the sub-templates corresponding to the operation steps, and the template collection unit is used to collect the sub-templates to obtain the task template;

[0091] The stability coefficient management module is used to manage the stability coefficients between computing nodes, wherein the stability coefficient management module includes: a fault management unit and a stability calculation unit, wherein the fault management unit is used to manage the fault records in the collaborative process of computing nodes, and the stability calculation unit is used to calculate the stability coefficients between computing nodes;

[0092] The idle node management module is used to manage the idle nodes in each time period, wherein the idle node management module includes: a node plan management unit and an idle node management unit, wherein the node plan management unit is used to manage the computing node allocation plan of the computing task, and the idle node management unit is used to collect the computing nodes in the idle time period to obtain an idle node list;

[0093] Among them, the task matching degree management module is used to manage the task matching degree of the newly added task, wherein the task matching degree management module includes: a task buffer unit, a step matching unit, a first matching degree calculation unit and a second matching degree calculation unit, wherein the task buffer unit is used to store the newly added computing task request in the computing cluster, the step matching unit is used to obtain the task template of the computing task, obtain the computing unit corresponding to each computing step, and match them in the computing cluster, the first matching degree calculation unit is used to calculate the matching degree of the computing step, and the second matching degree calculation unit is used to obtain the matching degree of the computing task;

[0094] Among them, the information reminder module is used to provide information reminders to relevant management personnel, wherein the information reminder module includes: a task sorting unit, a missing node management unit and a node application unit, wherein the task sorting unit is used to arrange the newly added computing tasks, the missing node management unit is used to obtain the missing computing units in each calculation step, and the node application unit is used to generate a list of newly added computing nodes based on the missing computing units, and send information reminders to relevant management personnel.

[0095] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0096] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cluster data monitoring and evaluation method based on artificial intelligence, characterized in that: The method comprises the following steps: Step S100: in a computing cluster including at least two computing nodes, obtaining a history record of computing tasks processed by the computing cluster, wherein the computing tasks are computing tasks collaboratively processed by at least two computing nodes, and obtaining task templates of different computing tasks; Step S100 includes: Step S101: obtaining a certain computing task from the historical records as a target task, obtaining an operation process L of the target task in the computing cluster, and dividing the operation process into a plurality of operation steps according to the computing nodes occupied by different processing stages of the computing task, wherein sti represents the computing node that runs the target task in the i-th operation step in the operation process L, and sti includes at least one computing node. When sti includes more than one computing node, sti also includes the connection relationship of the computing nodes; Step S102: Obtain a unit time of length t0, extract the history record of the target task, obtain the running time of each operation step, and divide the running time of the operation step into a number of unit times; Step S103: Obtain the i-th operation step in the running process L, the number ni corresponding to the unit time, obtain the sub-template (sti, ni) of the i-th operation step, and collect the sub-templates of all operation steps of the target task in the order of the operation steps to obtain the task template of the target task; Step S200: Obtaining the fault records of the collaborative work of the computing nodes, and obtaining the stability coefficient when any two computing nodes in the computing cluster are running simultaneously; Step S200 includes: Step S201: Obtain from historical records any two computing nodes that have serial collaboration or parallel collaboration to form a node pair, and obtain historical operation records of the node pair; Step S202: Obtain the normal state operation time T1, abnormal state operation time T2 and total maintenance time T3 of the node pair, and calculate the stability coefficient η, η=(T1-T2-T3) / T1, where the condition T1>T2+T3 is satisfied; Among them, when the node pair is in parallel collaboration, the normal state operation time T1, abnormal state operation time T2 and total maintenance time T3 are respectively the time when the two computing nodes are in normal operation, abnormal operation or maintenance at the same time; when the node pair is in serial collaboration, the normal state operation time T1 is the total operation time when the two computing nodes have a serial operation relationship, the abnormal state operation time T2 and the total maintenance time T3 are respectively the abnormal state operation time and maintenance time of the latter computing node in the node pair; Step S300: collect the computing tasks currently running in the computing cluster, obtain the node resources released by the computing cluster in different time periods, and collect the computing nodes that are idle in each time period into an idle node list; Step S400: collect requests for newly added computing tasks in the computing cluster, match computing nodes in the computing cluster that are in an idle time period according to the task template of the newly added computing task, and calculate the matching degree of each computing task according to the matching degree between the task template and the computing node and the stability coefficient of the computing node; Step S400 includes: Step S401: collect the newly added computing tasks into the first task set, and obtain any computing task taskA in the first task set; Step S402: Obtain the computing units corresponding to each computing step and the number of unit times corresponding to each computing step from the task template of taskA, and obtain the computing node stAj corresponding to the j-th computing step of the computing task taskA; Step S403: According to the running time of the j-th operation step, obtain the computing nodes that meet the connection mode in stAj from the idle node list; when the running time of the j-th operation step matches the computing nodes in at least one idle time period, merge them into the node candidate set Hj of the j-th operation step; when the running time of the j-th operation step does not match the computing nodes in the idle time period, record the j-th operation step as an unmatched step; Step S404: Get the number of computing nodes in Hj as NA, calculate the matching degree αj of the jth operation step, αj = NA / NAj, where NAj represents the number of corresponding computing nodes in stAj, and calculate the sum of the matching degrees of all operation steps in taskA as R A ; Step S405: Obtain the connection mode of each node in Hj, and obtain the stability coefficient between any two computing nodes, sum the stability coefficients in Hj, and record the sum of the stability coefficients in Hj as K j ; Step S406: Calculate the matching degree C of taskA A , Among them, m A represents the total number of unmatched steps in taskA, M A Represents the total number of computation steps in taskA; Step S500: Arrange the newly added computing tasks according to the matching degree of each computing task, obtain the difference between the computing nodes provided by the computing cluster and the computing nodes corresponding to the computing tasks, and submit a computing node application to relevant management personnel.

2. The method for monitoring and evaluating cluster data based on artificial intelligence according to claim 1, characterized in that: Step S300 includes: Step S301: Obtain a task template for each computing task in the task sequence, and obtain a computing node allocation plan for the computing task according to the total amount of computing nodes occupied by each computing step in the task template; Step S302: Record the time period when the computing node performs computing tasks as the working time period, and record the time period when no computing tasks are performed as the idle time period, obtain the working time period and idle time period of each computing node, and collect them to obtain the task plan of each computing node; Step S303: Taking the time when the first task in the task sequence starts to execute as the starting time, align the task plans of the computing nodes. After the starting time, gather the computing nodes in the same idle time period separately to obtain the computing nodes in the idle time period for each unit time.

3. The method for monitoring and evaluating cluster data based on artificial intelligence according to claim 2, characterized in that: Step S500 includes: Step S501: Calculate the matching degree of each computing task in the first task set, and arrange them from high to low according to the matching degree to obtain a second task set; Step S502: Get the first computing task task21 in the second task set, and get the computing node st corresponding to the pth computing step in task21. 21 p and node candidate set H 21 p , will st 21 p and H 21 p Compare and obtain st 21 p The missing computing nodes are denoted as st 21 p2 ; Step S503: Gather the computing nodes missing from each step of computing task task21, obtain a list of nodes to be configured, send an application for adding computing nodes to relevant managers, and when the computing nodes are added accordingly, input the computing task task21 into the computing cluster.

4. A cluster data monitoring and evaluation system for the cluster data monitoring and evaluation method based on artificial intelligence according to any one of claims 1 to 3, characterized in that: The system includes the following modules: a task template management module, a stability coefficient management module, an idle node management module, a task matching degree management module and an information reminder module, wherein the task template management module is used to manage the task templates of the computing tasks, the stability coefficient management module is used to manage the stability coefficients between computing nodes, the idle node management module is used to manage the idle nodes in each time period, the task matching degree management module is used to manage the task matching degree of newly added tasks, and the information reminder module is used to provide information reminders to relevant management personnel.

5. The cluster data monitoring and evaluation system according to claim 4, characterized in that: The task template management module includes: a historical data management unit, an operation step management unit and a template collection unit. The historical data management unit is used to manage the historical operation records of the computing nodes in the computing cluster, the operation step management unit is used to manage the sub-templates corresponding to the operation steps, and the template collection unit is used to collect the sub-templates to obtain the task template.

6. The cluster data monitoring and evaluation system according to claim 4, characterized in that: The stability coefficient management module includes: a fault management unit and a stability calculation unit, wherein the fault management unit is used to manage the fault records in the collaborative process of the computing nodes, and the stability calculation unit is used to calculate the stability coefficient between the computing nodes; The idle node management module includes: a node plan management unit and an idle node management unit, wherein the node plan management unit is used to manage the computing node allocation plan of the computing task, and the idle node management unit is used to collect the computing nodes in the idle period to obtain an idle node list.

7. The cluster data monitoring and evaluation system according to claim 4, characterized in that: The task matching management module includes: a task buffer unit, a step matching unit, a first matching calculation unit and a second matching calculation unit, wherein the task buffer unit is used to store the newly added computing task requests in the computing cluster, the step matching unit is used to obtain the task template of the computing task, obtain the computing unit corresponding to each computing step, and match them in the computing cluster, the first matching calculation unit is used to calculate the matching degree of the computing step, and the second matching calculation unit is used to obtain the matching degree of the computing task; The information reminder module includes: a task sorting unit, a missing node management unit and a node application unit, wherein the task sorting unit is used to arrange the newly added computing tasks, the missing node management unit is used to obtain the missing computing units in each operation step, and the node application unit is used to generate a list of newly added computing nodes based on the missing computing units and send information reminders to relevant managers.

Images