A cross-data center computing power scheduling management method and system

Abstract

The present invention discloses a method and system for managing computing power scheduling across data centers, and specifically relates to the technical field of computing power scheduling management. The method divides a job task into a plurality of subtasks, extracts characteristic parameters of the subtasks, and transmits the characteristic parameters to a central management node; inputs the characteristic parameters of the subtasks into a computing power scheduling model, and outputs a supercomputing node combination and a scheduling target of computing power resources corresponding to the subtasks; generates execution instructions of the subtasks based on the supercomputing node combination to realize task allocation; monitors and stores status information of each supercomputing center and execution status information of the subtasks; triggers first-level and second-level warnings based on the execution status information of the subtasks to help users monitor the execution of job tasks, and solves the problem in the prior art that the abnormal progress of job task completion cannot be discovered in time in computing power scheduling across data centers; and achieves the global optimal computing power scheduling effect by evaluating the computing power scheduling quality between different data centers.

Description

Technical Field

[0001] The present invention relates to the field of computing power scheduling and management technology, and more specifically, to a cross-data center computing power scheduling and management method and system. Background Art

[0002] The cross-data center computing power scheduling and management method is widely used in cloud computing, big data analysis, artificial intelligence and other fields. For example, in the field of cloud computing, the cross-data center computing power scheduling and management method can realize the flexible deployment and efficient operation of cloud services. In the field of big data analysis, the data processing process can be optimized to improve the real-time and accuracy of data processing. In the field of artificial intelligence, the model training and reasoning process can be accelerated to improve the performance and effect of artificial intelligence applications.

[0003] In the existing computing power scheduling system, the cross-data center computing power scheduling management method aims to break the limitation of computing resources being planned according to a single data center. Through a unified scheduling system, computing tasks are assigned to the most suitable data center for processing. This method can make full use of the resources of each data center, improve the utilization rate of computing power resources, and reduce the delay and cost of network transmission.

[0004] However, it still has many shortcomings in actual use. For example, the existing technology cannot promptly detect abnormal progress of job completion in cross-data center computing power scheduling. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides a cross-data center computing power scheduling management method and system, which connects each supercomputing center through a dedicated network, realizes information sharing of computing resources and storage resources, establishes a stable network connection, and obtains a cross-data center platform. The cross-data center platform supports resource sharing and collaborative computing, and is characterized in that a central management node is set in the cross-data center platform, which is responsible for computing power resource scheduling, task allocation and status monitoring, so as to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a computing power scheduling management method across data centers, comprising the following steps:

[0007] Step 1: Divide the operation task into several subtasks, extract the characteristic parameters of the subtasks, and transmit the characteristic parameters to the central management node;

[0008] Step 2: Scheduling computing resources: Build a computing resource scheduling model, input the characteristic parameters of the subtask into the computing resource scheduling model, and output the supercomputing node combination and scheduling target of the computing resource corresponding to the subtask;

[0009] Step 3: Task allocation: Generate execution instructions for subtasks based on the combination of supercomputing nodes to achieve task allocation; the execution instructions include task start commands, data paths, and execution parameters; distribute the execution instructions to the corresponding supercomputing nodes to ensure that each supercomputing node receives and executes the execution instructions;

[0010] Step 4: Status monitoring and early warning: Monitor and store the status information of each supercomputing center and the execution status information of subtasks; take measures based on the execution status information of subtasks, including:

[0011] When the subtask completion progress is lower than expected, a first-level warning is triggered, and the triggering time and duration of the abnormal progress are recorded;

[0012] When the accumulated duration of the first-level warning exceeds the second threshold, the second-level warning is triggered.

[0013] Preferably, the preliminary preparations for building a computing power scheduling model include: defining a resource model of multiple supercomputing centers, the resource model including the computing power, storage capacity, and network bandwidth of each supercomputing center; establishing a scheduling goal as any one of minimizing completion time, maximizing resource utilization, and maximizing cost-effectiveness, or a combination thereof.

[0014] Preferably, the process of building the computing power scheduling model includes the following steps:

[0015] Step S11, initializing the model: During initialization, a certain number of particles are randomly generated based on the characteristic parameters of the subtasks and the resource model of multiple supercomputing centers. Each particle contains the allocation of subtasks and the resource usage of each supercomputing center.

[0016] Step S12, setting parameters, including the number of particles, the number of iterations, and the learning factor, where the learning factor refers to the degree of influence of individual and global optimal solutions on particle motion;

[0017] Step S13, optimization process: according to the scheduling target, the fitness value of each particle is calculated to evaluate the quality; the particle adjusts the speed and position according to the individual optimal solution and the global optimal solution to move closer to a better scheduling solution;

[0018] Step S14, iterative update: Repeat the above steps until the preset number of iterations is reached or converges to a stable solution; output the optimal scheduling plan corresponding to the subtask, the optimal scheduling plan includes the supercomputing node combination of each subtask and the scheduling target of the job task.

[0019] Preferably, the scheduling target refers to minimizing the completion time of the job task under the resource model constraint condition, so that the completion time of the last subtask to complete the calculation is the shortest, which is recorded as Min_Ttotal; the scheduling target is obtained as follows:

[0020] Assume that the job task includes n subtasks, and j represents the sequence number of the subtasks. The jth subtask is recorded as J j ;

[0021] Suppose there are several supercomputing centers, there are m computing nodes in total, i represents the number of the computing node, and the i-th computing node is recorded as SC i ;

[0022] The jth subtask is placed on the supercomputing node SC i The expected computation time is recorded as ;Transfer the data of the jth subtask from the current location to the computing power node SC i The expected transmission time is recorded as ;

[0023] The function to construct the scheduling target is as follows:

[0024] in, Used to obtain the completion time of the jth task; j∈[1,n], get the completion time set of all tasks, max is used to take the maximum value of the completion time set to get the completion time of the last subtask that completes the calculation.

[0025] Preferably, the computing power scheduling management method further includes:

[0026] Step 5: Status evaluation: obtain the actual transmission time and actual calculation time of each subtask's data from the current location to the computing power node SCi, process the subtask's execution status information, and output the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, it indicates that there are problems with computing power performance and network resource instability, triggering a third-level warning.

[0027] Preferably, the computing power stability optimizability index is obtained in the following manner:

[0028] The jth subtask is placed on the supercomputing node SC i The actual calculation time is recorded as ; Through the formula The optimized parameter Ya for computing power scheduling is calculated;

[0029] Transfer the data of the jth subtask from the current location to the computing power node SC i The actual transmission time is recorded as ; Through the formula The transmission scheduling optimization parameter Yb is calculated;

[0030] Combined analysis of the optimization parameters of computing power scheduling and transmission scheduling, through the formula The computing power stability can be optimized by calculating the index SY.

[0031] Preferably, the cross-data center computing power scheduling management method further includes:

[0032] Step 6: Abnormal response: Monitor and analyze the execution status of the job tasks, output the abnormal jamming impact coefficient, task execution quality assessment coefficient, and computing resource balance coefficient across data centers, and take measures based on the output results, including the following steps:

[0033] Step S601, obtain the triggering times and average duration of abnormal progress from the log, calculate the cumulative duration of abnormal progress, obtain the maximum value max_YK and the minimum value min_YK of the cumulative duration of abnormal progress, and use the linear normalization formula to calculate the abnormal jamming influence coefficient KY;

[0034] Step S602: Obtain the expected completion time TDy and the actual completion time TDs of the last completed subtask, calculate the loss rate ZS of the task, consider the success rate of the task, and obtain the task execution quality evaluation coefficient Rn;

[0035] Step S603: Obtain the average computing resource utilization, the average network resource utilization, and the average storage resource utilization, and obtain the computing resource balancing coefficient based on the difference between the average computing resource utilization and the expected utilization; the expected utilization includes the expected computing resource utilization, the expected network resource utilization, and the expected storage resource utilization;

[0036] Step S604, monitor and analyze the abnormal jamming impact coefficient, the task execution quality assessment coefficient and the computing power resource balance coefficient. If the abnormal jamming impact coefficient exceeds the ideal value, measures need to be taken to reduce jamming by optimizing the task scheduling algorithm, increasing hardware resources or adjusting the network configuration method. If the task execution quality assessment coefficient is lower than the ideal value, that is, the task execution quality is poor, it is necessary to check the task execution process, optimize task allocation or improve the task execution environment. If the computing power resource balance coefficient deviates from the ideal value, that is, resource utilization is unbalanced, it is necessary to reallocate tasks or adjust resource allocation strategies to achieve more efficient resource utilization.

[0037] Preferably, the abnormal jamming influence coefficient is obtained in the following manner:

[0038] Get the number of abnormal progress triggers N from the log error and the average duration T avg_erro , through the formula Calculate the cumulative duration of abnormal progress through the formula The abnormal jamming influence coefficient KY is calculated;

[0039] The task execution quality evaluation coefficient Rn is obtained by obtaining the expected completion time corresponding to the last completed subtask, recorded as TDy ; Get the actual completion time corresponding to the last completed subtask, recorded as TD s ; Through the formula Calculate the loss rate ZS of the task, The function is used to obtain the maximum value of two numbers in brackets; through the formula The task execution quality evaluation coefficient Rn is calculated, where Sr represents the success rate of the task;

[0040] The computing power resource balance coefficient SJ is obtained as follows:

[0041] Assume that there are K computing nodes in the cross-data center platform, and use s to represent the sequential number of the computing nodes;

[0042] The computing resource utilization, network resource utilization, and storage resource utilization of the sth computing node are recorded as js_s, wc_s, and cz_s, respectively; the amount of tasks processed by the sth computing node is recorded as krs, Represents the total amount of tasks processed, through the formula Calculate the average computing resource utilization through the formula Calculate the average network resource utilization through the formula Calculate the average storage resource utilization; by formula The computing power resource balance coefficient SJ is calculated, where δ10 represents the expected computing resource utilization, δ20 represents the expected network resource utilization, δ30 represents the expected storage resource utilization, and β1, β2, and β3 represent the weight coefficients of each item respectively.

[0043] To achieve the above object, the present invention provides the following technical solution: a computing power scheduling and management system across data centers, comprising:

[0044] The cross-data center platform building module connects each supercomputing center through a dedicated network, realizes information sharing of computing resources and storage resources, establishes a stable network connection, and obtains a cross-data center platform. The cross-data center platform supports resource sharing and collaborative computing, and sets a central management node in the cross-data center platform;

[0045] The job task division module is used to divide the job task into several subtasks, extract the characteristic parameters of the subtasks, and transmit the characteristic parameters to the central management node;

[0046] The computing resource scheduling module builds a computing scheduling model, inputs the characteristic parameters of the subtask into the computing scheduling model, and outputs the supercomputing node combination and scheduling target of the computing resources corresponding to the subtask;

[0047] The task allocation module generates execution instructions for subtasks based on the combination of supercomputing nodes to achieve task allocation; the execution instructions include task start commands, data paths, and execution parameters; the execution instructions are distributed to the corresponding supercomputing nodes to ensure that each supercomputing node receives and executes the execution instructions;

[0048] The status monitoring and warning module is used to monitor and store the status information of each supercomputing center and the execution status information of subtasks; take measures based on the execution status information of subtasks, including: when the completion progress of a subtask is lower than expected, trigger a first-level warning, and record the triggering time and duration of the abnormal progress; when the accumulated duration of the first-level warning exceeds the second threshold, trigger a second-level warning;

[0049] The status evaluation module obtains the actual transmission time and actual calculation time of each subtask's data from the current location to the computing power node SCi, processes the execution status information of the subtask, and outputs the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, a third-level warning is triggered.

[0050] Technical effects and advantages of the present invention:

[0051] The cross-data center computing power scheduling management method provided by the present invention divides the job task into a number of subtasks, extracts characteristic parameters of the subtasks, and transmits the characteristic parameters to the central management node; inputs the characteristic parameters of the subtasks into the computing power scheduling model, and outputs the supercomputing node combination and scheduling target of the computing power resources corresponding to the subtasks; generates execution instructions for the subtasks based on the supercomputing node combination to realize task allocation; monitors and stores the status information of each supercomputing center and the execution status information of the subtasks; triggers the first-level warning and the second-level warning based on the execution status information of the subtasks, helps users monitor the execution status of the job tasks, and facilitates timely adjustment of the computing power scheduling strategy, thereby solving the problem in the prior art that the abnormal progress of the job task completion cannot be discovered in time in the cross-data center computing power scheduling.

[0052] The cross-data center computing power scheduling management method provided by the present invention obtains the actual transmission time and actual calculation time of the data of each subtask from the current location to the computing power node SCi, processes the execution status information of the subtask, and outputs the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, it indicates that there are problems with computing power performance and network resources instability, triggering a three-level warning; based on the three-level warning, it helps users monitor the stability of computing power resources.

[0053] The cross-data center computing power scheduling management method provided by the present invention provides a comprehensive perspective to evaluate the quality of current computing power scheduling by jointly analyzing the abnormal jamming impact coefficient, the task execution quality evaluation coefficient and the computing power resource balance coefficient. It not only takes into account the stability of system operation, but also takes into account the efficiency and effect of task execution, as well as the balance of resource utilization. Based on the computing power scheduling quality evaluation index, system operation and maintenance personnel or automation systems can make more informed decisions. By evaluating the computing power scheduling quality between different data centers, tasks and resources can be allocated more intelligently to achieve the global optimal computing power scheduling effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the computing power scheduling management method based on status monitoring and early warning of the present invention.

[0055] Figure 2 This is a flow chart of the abnormal response process of computing power scheduling of the present invention.

[0056] Figure 3 This is a structural block diagram of the computing power scheduling management method based on state assessment and abnormal response of the present invention. DETAILED DESCRIPTION

[0057] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings; although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein; on the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0059] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0060] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0061] Background technology: Supercomputing centers are connected through a dedicated network to achieve information sharing of computing resources and storage resources, establish a stable network connection, and obtain a cross-data center platform. The scattered supercomputing center resources are organized into a unified virtual environment to support resource sharing and collaborative computing, support high-bandwidth, low-latency data transmission, and ensure that supercomputing centers in different geographical locations can collaborate efficiently; a central management node is set up in the cross-data center platform to be responsible for computing resource scheduling, task allocation, and status monitoring.

[0062] Example 1, see Figure 1 The present invention provides a flow chart of a computing power scheduling management method based on status monitoring and early warning. Figure 1 A cross-data center computing power scheduling management method is shown, comprising the following steps:

[0063] Step 1: Divide the job task into several subtasks, extract characteristic parameters of the subtasks, and transmit the characteristic parameters to the central management node; the characteristic parameters at least include job size, job type, and resource requirements; the job type includes computing-intensive tasks and data-intensive tasks;

[0064] Step 2: Scheduling computing resources: Build a computing resource scheduling model, input the characteristic parameters of the subtask into the computing resource scheduling model, and output the supercomputing node combination and scheduling target of the computing resource corresponding to the subtask;

[0065] Step 3: Task allocation: Generate execution instructions for subtasks based on the combination of supercomputing nodes to achieve task allocation; the execution instructions include task start commands, data paths, and execution parameters; distribute the execution instructions to the corresponding supercomputing nodes to ensure that each supercomputing node receives and executes the execution instructions;

[0066] Step 4: Status monitoring and early warning: Monitor and store the status information of each supercomputing center and the execution status information of subtasks; take measures based on the execution status information of subtasks, including:

[0067] When the subtask completion progress is lower than expected, a first-level warning is triggered, and the triggering time and duration of the abnormal progress are recorded;

[0068] When the accumulated duration of the first-level warning exceeds the second threshold, the second-level warning is triggered;

[0069] What needs to be further explained in the embodiments of the present invention is that if the first-level warning continues for a certain period of time, it will be upgraded to a second-level warning, and the cause will be further analyzed and preliminary measures will be taken; measures will be taken based on the first-level warning information and the second-level warning information to handle abnormal situations and screen out computing nodes with node failures and network congestion; tasks will be reallocated based on the scheduling target, and subtasks with delayed progress will be reallocated to other idle or lower-load nodes; computing resources or network bandwidth will be increased for subtasks with delayed progress; the order of tasks in the task execution queue will be adjusted to give priority to critical tasks or dependent tasks.

[0070] What needs to be further explained in the embodiments of the present invention is that the preliminary preparations for building a computing power scheduling model include: defining a resource model of multiple supercomputing centers, the resource model including the computing power, storage capacity, and network bandwidth of each supercomputing center; establishing a scheduling goal as any one of minimizing completion time, maximizing resource utilization, and maximizing cost-effectiveness, or a combination thereof.

[0071] It needs to be further explained in the embodiment of the present invention that the process of building the computing power scheduling model includes the following steps:

[0072] Step S11, initializing the model: During initialization, a certain number of particles are randomly generated based on the characteristic parameters of the subtasks and the resource model of multiple supercomputing centers. Each particle contains the allocation of subtasks and the resource usage of each supercomputing center.

[0073] Explanation: In the particle swarm algorithm, each particle represents a possible job scheduling scheme; the position of the particle is updated according to its speed, and this update is performed iteratively until a certain stopping condition is met; the initial position of the particle refers to a job scheduling scheme, which is an initial, random or heuristic rule-based job allocation and resource allocation strategy; for example, in the combined scheduling of computing power of multiple supercomputing centers, the initial position may represent the list of jobs allocated to each supercomputing center and the corresponding resource (such as CPU, memory, etc.) configuration;

[0074] Step S12, setting parameters, including the number of particles, the number of iterations, and the learning factor, where the learning factor refers to the degree of influence of individual and global optimal solutions on particle motion;

[0075] Step S13, optimization process: according to the scheduling goal such as minimizing the completion time, the fitness value of each particle is calculated and the quality is evaluated; the particle adjusts the speed and position according to the individual optimal solution and the global optimal solution to move closer to a better scheduling solution;

[0076] Step S14, iterative update: Repeat the above steps until the preset number of iterations is reached or converges to a stable solution; output the optimal scheduling plan corresponding to the subtask, that is, the particle position, the optimal scheduling plan performs best in the scheduling target and has the shortest overall completion time; the optimal scheduling plan includes the supercomputing node combination of each subtask and the scheduling target of the job task.

[0077] It needs to be further explained in the embodiment of the present invention that the scheduling target refers to minimizing the completion time of the job task under the resource model constraint condition, so that the completion time of the last subtask to complete the calculation is the shortest. This is because in a distributed computing environment, the completion time of the job task is usually determined by the last completed subtask, so optimizing this time can significantly improve the efficiency of job processing. The scheduling target is obtained as follows:

[0078] Assume that the job task includes n subtasks, and j represents the sequence number of the subtasks. The jth subtask is recorded as J j ;

[0079] Suppose there are several supercomputing centers, there are m computing nodes in total, i represents the number of the computing node, and the i-th computing node is recorded as SC i ;

[0080] The jth subtask is placed on the supercomputing node SC i The expected computation time is recorded as ;Transfer the data of the jth subtask from the current location to the computing power node SC i The expected transmission time is recorded as ;

[0081] The function to construct the scheduling target is as follows:

[0082] in, Used to obtain the completion time of the jth task; j∈[1,n], get the completion time set of all tasks, max is used to take the maximum value of the completion time set to get the completion time of the last subtask that completes the calculation.

[0083] What needs to be further explained in the embodiments of the present invention is that the resource model constraints include at least computing power constraints, storage resource constraints and network bandwidth constraints; the computing power constraints mean that the total computing power of each supercomputing center does not exceed its computing power; the storage resource constraints mean that all subtask data allocated to the supercomputing center can be stored in its storage resources; the network bandwidth constraints mean that the total data transmission rate of all transmission jobs does not exceed the network bandwidth to avoid network congestion.

[0084] Embodiment 2: The difference between the embodiment of the present invention and embodiment 1 is that the computing power scheduling management method described in the embodiment of the present invention further includes:

[0085] Step 5: Status evaluation: obtain the actual transmission time and actual calculation time of each subtask's data from the current location to the computing power node SCi, process the subtask's execution status information, and output the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, it indicates that there are problems with computing power performance and network resources instability, triggering a third-level warning;

[0086] It needs to be further explained in the embodiment of the present invention that the computing power stability optimizability index is obtained in the following manner:

[0087] The jth subtask is placed on the supercomputing node SC i The actual calculation time is recorded as ; Through the formula The optimized parameter Ya for computing power scheduling is calculated;

[0088] Transfer the data of the jth subtask from the current location to the computing power node SC i The actual transmission time is recorded as ; Through the formula The transmission scheduling optimization parameter Yb is calculated;

[0089] Combined analysis of the optimization parameters of computing power scheduling and transmission scheduling, through the formula The computing power stability can be optimized by calculating the index SY.

[0090] Example 3, see Figure 2 The abnormal response process flow chart of computing power scheduling and Figure 3 The structure block diagram of the computing power scheduling management method based on state evaluation and abnormal response is shown in FIG. The difference between the embodiment of the present invention and embodiment 1 and embodiment 2 is that the cross-data center computing power scheduling management method further includes:

[0091] Step 6: Abnormal response: Monitor and analyze the execution status of the job tasks, output the abnormal jamming impact coefficient, task execution quality assessment coefficient, and computing resource balance coefficient across data centers, and take measures based on the output results, including the following steps:

[0092] Step S601, obtain the triggering times and average duration of abnormal progress from the log, calculate the cumulative duration of abnormal progress, obtain the maximum value max_YK and the minimum value min_YK of the cumulative duration of abnormal progress, and use the linear normalization formula to calculate the abnormal jamming influence coefficient KY;

[0093] Step S602: Obtain the expected completion time TDy and the actual completion time TDs of the last completed subtask, calculate the loss rate ZS of the task, consider the success rate of the task, and obtain the task execution quality evaluation coefficient Rn;

[0094] Step S603: Obtain the average computing resource utilization, the average network resource utilization, and the average storage resource utilization, and obtain a computing resource balance coefficient based on the difference between the average utilization and the expected utilization;

[0095] The expected utilization includes expected computing resource utilization, expected network resource utilization, and expected storage resource utilization;

[0096] Step S604: monitor and analyze the abnormal jamming impact coefficient, task execution quality assessment coefficient, and computing power resource balance coefficient;

[0097] Step S605: taking measures based on the difference between the monitoring result and the ideal value;

[0098] If the impact coefficient of abnormal jamming exceeds the ideal value, measures need to be taken to reduce jamming by optimizing task scheduling algorithms, increasing hardware resources, or adjusting network configuration methods.

[0099] If the task execution quality assessment coefficient is lower than the ideal value, that is, the task execution quality is poor, it is necessary to check the task execution process, optimize task allocation, or improve the task execution environment;

[0100] If the computing power resource balance coefficient deviates from the ideal value, that is, resource utilization is uneven, it is necessary to reallocate tasks or adjust resource allocation strategies to achieve more efficient resource utilization.

[0101] What needs to be further explained in the embodiments of the present invention is that weight coefficients are matched for the abnormal jamming impact coefficient, the task execution quality assessment coefficient and the computing power resource balance coefficient, and the weighted sum is used to obtain the computing power scheduling quality assessment index, which provides a comprehensive perspective to evaluate the quality of the current computing power scheduling; by calculating the computing power scheduling quality assessment index periodically or in real time, continuously monitoring its performance, and continuously adjusting and optimizing the scheduling strategy based on the evaluation results, so as to ensure that the system is always kept in the best state.

[0102] It needs to be further explained in the embodiment of the present invention that the abnormal jamming influence coefficient is obtained in the following manner:

[0103] Get the number of abnormal progress triggers N from the log error and the average duration T avg_erro , through the formula Calculate the cumulative duration of abnormal progress through the formula The abnormal jamming influence coefficient KY is calculated;

[0104] The task execution quality evaluation coefficient Rn is obtained by obtaining the expected completion time corresponding to the last completed subtask, recorded as TD y ; Get the actual completion time corresponding to the last completed subtask, recorded as TD s ; Through the formula Calculate the loss rate ZS of the task, The function is used to obtain the maximum value of two numbers in brackets; through the formula The task execution quality evaluation coefficient Rn is calculated, where Sr represents the success rate of the task;

[0105] The computing power resource balance coefficient SJ is obtained as follows:

[0106] Assume that there are K computing nodes in the cross-data center platform, and use s to represent the sequential number of the computing nodes;

[0107] The computing resource utilization, network resource utilization, and storage resource utilization of the sth computing node are recorded as js_s, wc_s, and cz_s, respectively; the amount of tasks processed by the sth computing node is recorded as krs, Represents the total amount of tasks processed, through the formula Calculate the average computing resource utilization through the formula Calculate the average network resource utilization through the formula Calculate the average storage resource utilization; by formula The computing power resource balance coefficient SJ is calculated, where δ10 represents the expected computing resource utilization, δ20 represents the expected network resource utilization, δ30 represents the expected storage resource utilization, and β1, β2, and β3 represent the weight coefficients of each item respectively.

[0108] What needs to be further explained in the embodiments of the present invention is that the weight coefficients are matched for the abnormal jamming impact coefficient, the task execution quality assessment coefficient and the computing power resource balance coefficient, and the weighted sum is obtained to obtain the computing power scheduling quality assessment index, which provides a comprehensive perspective to evaluate the quality of the current computing power scheduling; by calculating the computing power scheduling quality assessment index regularly or in real time, continuously monitoring its performance, and continuously adjusting and optimizing the scheduling strategy according to the evaluation results, so as to ensure that the system is always kept in the best state.

[0109] Embodiment 4: This embodiment of the present invention provides a cross-data center computing power scheduling and management system, including:

[0110] The cross-data center platform building module connects each supercomputing center through a dedicated network, realizes information sharing of computing resources and storage resources, establishes a stable network connection, and obtains a cross-data center platform. The cross-data center platform supports resource sharing and collaborative computing, and sets a central management node in the cross-data center platform;

[0111] The job task division module is used to divide the job task into several subtasks, extract the characteristic parameters of the subtasks, and transmit the characteristic parameters to the central management node;

[0112] The computing resource scheduling module builds a computing scheduling model, inputs the characteristic parameters of the subtask into the computing scheduling model, and outputs the supercomputing node combination and scheduling target of the computing resources corresponding to the subtask;

[0113] The task allocation module generates execution instructions for subtasks based on the combination of supercomputing nodes to achieve task allocation; the execution instructions include task start commands, data paths, and execution parameters; the execution instructions are distributed to the corresponding supercomputing nodes to ensure that each supercomputing node receives and executes the execution instructions;

[0114] The status monitoring and warning module is used to monitor and store the status information of each supercomputing center and the execution status information of subtasks; take measures based on the execution status information of subtasks, including: when the completion progress of a subtask is lower than expected, trigger a first-level warning, and record the triggering time and duration of the abnormal progress; when the accumulated duration of the first-level warning exceeds the second threshold, trigger a second-level warning;

[0115] The status evaluation module obtains the actual transmission time and actual calculation time of each subtask's data from the current location to the computing power node SCi, processes the execution status information of the subtask, and outputs the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, a third-level warning is triggered.

[0116] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A computing power scheduling management method across data centers, characterized in that: Setting up a central management node in the cross-data center platform is responsible for computing resource scheduling, task allocation, and status monitoring, including the following steps: Step 1: Divide the operation task into several subtasks, extract the characteristic parameters of the subtasks, and transmit the characteristic parameters to the central management node; Step 2: Scheduling computing resources: Build a computing resource scheduling model, input the characteristic parameters of the subtask into the computing resource scheduling model, and output the supercomputing node combination and scheduling target of the computing resource corresponding to the subtask; Step 3: Task allocation: Generate execution instructions for subtasks based on the combination of supercomputing nodes to achieve task allocation; the execution instructions include task start commands, data paths, and execution parameters; Distribute the execution instructions to the corresponding supercomputing nodes to ensure that each supercomputing node receives and executes the execution instructions; Step 4: Status monitoring and early warning: monitor and store the status information of each supercomputing center and the execution status information of subtasks; Take actions based on the execution status information of the subtask, including: When the subtask completion progress is lower than expected, a first-level warning is triggered, and the triggering time and duration of the abnormal progress are recorded; When the accumulated duration of the first-level warning exceeds the second threshold, the second-level warning is triggered; The process of building the computing power scheduling model includes the following steps: Step S11, initializing the model: During initialization, a certain number of particles are randomly generated based on the characteristic parameters of the subtasks and the resource model of multiple supercomputing centers. Each particle contains the allocation of subtasks and the resource usage of each supercomputing center. Step S12, setting parameters, including the number of particles, the number of iterations, and the learning factor, where the learning factor refers to the degree of influence of individual and global optimal solutions on particle motion; Step S13, optimization process: according to the scheduling target, the fitness value of each particle is calculated to evaluate the pros and cons; the particle adjusts the speed and position according to the individual optimal solution and the global optimal solution to move closer to a better scheduling solution; Step S14, iterative update: repeat the above steps until the preset number of iterations is reached or converges to a stable solution; output the optimal scheduling solution corresponding to the subtask, the optimal scheduling solution includes the supercomputing node combination of each subtask and the scheduling target of the job task; The scheduling target is obtained as follows: Assume that the job task includes n subtasks, and j represents the sequence number of the subtasks. The jth subtask is recorded as J j ; Suppose there are several supercomputing centers, there are m computing nodes in total, i represents the number of the computing node, and the i-th computing node is recorded as SC i ; The jth subtask is placed on the supercomputing node SC i The expected computation time is recorded as ;Transfer the data of the jth subtask from the current location to the computing power node SC i The expected transmission time is recorded as ; The function to construct the scheduling target is as follows: in, Used to obtain the completion time of the jth task; j∈[1,n], get the completion time set of all tasks, max is used to take the maximum value of the completion time set to get the completion time of the last subtask that has completed the calculation; The computing power scheduling management method also includes: Step 5: Status evaluation: obtain the actual transmission time and actual calculation time of each subtask's data from the current location to the computing power node SCi, process the subtask's execution status information, and output the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, trigger the third-level warning; The cross-data center computing power scheduling management method further includes: Step 6: Abnormal response: Monitor and analyze the execution status of the job tasks, output the abnormal jamming impact coefficient, task execution quality assessment coefficient, and computing resource balance coefficient across data centers, and take measures based on the output results, including the following steps: Step S601, obtain the triggering times and average duration of abnormal progress from the log, calculate the cumulative duration of abnormal progress, obtain the maximum value max_YK and the minimum value min_YK of the cumulative duration of abnormal progress, and use the linear normalization formula to calculate the abnormal jamming influence coefficient KY; Step S602: Obtain the expected completion time TD of the last completed subtask y and actual completion time TD s , calculate the loss rate ZS of the task; consider the success rate of the task, and get the task execution quality evaluation coefficient Rn; Step S603: Obtain the average computing resource utilization, the average network resource utilization, and the average storage resource utilization, and obtain a computing resource balance coefficient based on the difference between the average utilization and the expected utilization; Step S604: monitor and analyze the abnormal jamming impact coefficient, task execution quality assessment coefficient, and computing power resource balance coefficient; If the abnormal jamming impact coefficient exceeds the ideal value, measures need to be taken to reduce jamming; If the task execution quality assessment coefficient is lower than the ideal value, it is necessary to check the task execution process, optimize task allocation, or improve the task execution environment; If the computing power resource balance coefficient deviates from the ideal value, it is necessary to reallocate tasks or adjust resource allocation strategies to achieve more efficient resource utilization; The abnormal jamming influence coefficient is obtained as follows: Get the number of abnormal progress triggers N from the log error and the average duration T avg_erro , through the formula Calculate the cumulative duration of abnormal progress through the formula The abnormal jamming influence coefficient KY is calculated; The task execution quality evaluation coefficient Rn is obtained by: Calculate the loss rate ZS of the task, The function is used to obtain the maximum value of two numbers in brackets; through the formula The task execution quality evaluation coefficient Rn is calculated, where Sr represents the success rate of the task; The computing power resource balance coefficient SJ is obtained as follows: Assume that there are K computing nodes in the cross-data center platform, and use s to represent the sequential number of the computing nodes; The computing resource utilization, network resource utilization, and storage resource utilization of the sth computing node are recorded as js_s, wc_s, and cz_s, respectively; the amount of tasks processed by the sth computing node is recorded as krs, Represents the total amount of tasks processed, through the formula Calculate the average computing resource utilization through the formula Calculate the average network resource utilization through the formula Calculate the average storage resource utilization; by formula The computing power resource balance coefficient SJ is calculated, where δ10 represents the expected computing resource utilization, δ20 represents the expected network resource utilization, δ30 represents the expected storage resource utilization, and β1, β2, and β3 represent the weight coefficients of each item respectively.

2. The cross-data center computing power scheduling management method according to claim 1 is characterized in that: The preliminary preparations for building a computing power scheduling model include: defining the resource model of multiple supercomputing centers, which includes the computing power, storage capacity, and network bandwidth of each supercomputing center; establishing the scheduling goal as any one of minimizing completion time, maximizing resource utilization, and maximizing cost-effectiveness, or a combination thereof.

3. The cross-data center computing power scheduling management method according to claim 2 is characterized in that: The scheduling goal refers to minimizing the completion time of the job task under the resource model constraints, so that the completion time of the last subtask to complete the calculation is the shortest, which is recorded as Min_Ttotal.

4. The cross-data center computing power scheduling management method according to claim 3 is characterized in that: The method for obtaining the computing power stability optimizability index is as follows: The jth subtask is placed on the supercomputing node SC i The actual calculation time is recorded as ; Through the formula The optimized parameter Ya for computing power scheduling is calculated; Transfer the data of the jth subtask from the current location to the computing power node SC i The actual transmission time is recorded as ; Through the formula The transmission scheduling optimization parameter Yb is calculated; Combined analysis of the optimization parameters of computing power scheduling and transmission scheduling, through the formula The computing power stability can be optimized by calculating the index SY.

5. A cross-data center computing power scheduling management system, used to implement the cross-data center computing power scheduling management method described in claim 4, characterized in that: include: The cross-data center platform building module connects each supercomputing center through a dedicated network, realizes information sharing of computing resources and storage resources, establishes a stable network connection, and obtains a cross-data center platform. The cross-data center platform supports resource sharing and collaborative computing, and sets a central management node in the cross-data center platform; The job task division module is used to divide the job task into several subtasks, extract the characteristic parameters of the subtasks, and transmit the characteristic parameters to the central management node; The computing resource scheduling module builds a computing scheduling model, inputs the characteristic parameters of the subtask into the computing scheduling model, and outputs the supercomputing node combination and scheduling target of the computing resources corresponding to the subtask; The task allocation module generates execution instructions for subtasks based on the combination of supercomputing nodes to achieve task allocation; Status monitoring and early warning module, used to monitor and store the status information of each supercomputing center and the execution status information of subtasks; Take measures based on the execution status information of the subtask, including: when the subtask completion progress is lower than expected, trigger the first-level warning and record the triggering time and duration of the abnormal progress; when the cumulative duration of the first-level warning exceeds the second threshold, trigger the second-level warning; The status assessment module obtains the actual transmission time and actual calculation time of each subtask's data from the current location to the computing power node, processes the execution status information of the subtask, and outputs the computing power stability optimization index; when the computing power stability optimization index exceeds the preset third threshold, a third-level warning is triggered.

Images