Energy-saving operation and maintenance decision system and method based on big data intelligent computing center

Abstract

The application discloses an energy-saving operation and maintenance decision system and method of a big-data-based intelligent computing center, and particularly relates to the field of intelligent computing center operation and maintenance management, and comprises the following steps: writing training logs, power logs, synchronization logs, cache logs and checkpoint logs of a distributed training task into round records according to round numbers; extracting completion time, waiting power, resident data and checkpoint states of each training node to generate a round table; and performing pattern recognition on round operation records, synchronization waiting relationships, checkpoint states and resident data states of the distributed training task, and performing dual update determination of energy consumption reduction, waiting energy consumption increase and data rework energy consumption increase on candidate energy-saving actions, so as to solve the problem that single-point energy-saving actions in existing energy-saving operation and maintenance cause whole-group training waiting energy consumption increase and data rework energy consumption increase.

Description

Technical Field

[0001] This invention relates to the field of intelligent computing center operation and maintenance management technology, and more specifically, to an energy-saving operation and maintenance decision-making system and method for intelligent computing centers based on big data. Background Technology

[0002] In energy-saving operation and maintenance decision-making in intelligent computing centers, existing technologies mainly focus on reducing the energy consumption of servers, GPU clusters and cooling systems during operation. Typically, by collecting data such as GPU utilization, server power, rack temperature, task queue, PUE, electricity price period and historical energy consumption curves, and combining load prediction, threshold determination, pattern recognition or task priority sorting, energy-saving operation and maintenance actions such as frequency reduction, migration, pause, off-peak startup, release of video memory or shutdown of idle nodes are generated. However, in intelligent computing center scenarios where large model training, fine-tuning, and evaluation tasks run in parallel, a single task often occupies multiple servers and multiple GPUs simultaneously. After the computation phase, each node also needs to perform parameter synchronization, gradient synchronization, or checkpoint writing. Furthermore, the datasets, model weights, cached data, and intermediate results already loaded during the operation will continuously occupy GPU memory, RAM, and storage read / write states. On the business side, it is also subject to the joint constraints of task completion time limits, training interruption restrictions, and reserved computing power windows. Under the above constraints, the existing method of generating energy-saving actions based on the status of a single server, a single GPU, or a single rack will consistently expose the problem of fragmented decision-making objects. On-site observations show that after the utilization of a certain GPU decreases, it is downclocked or migrated, but the synchronization waiting time of other nodes in the same training task increases. Or, after a task is paused and GPU memory is released, the dataset is reread, model weights are reloaded, and the cache is rebuilt when the task resumes, which increases storage access volume, network transmission volume, total task runtime, and cooling duration. As a result, the power reduction at a single point does not translate into a reduction in the energy consumption of the entire task group. Therefore, the technical problem to be solved by this application is: how to determine whether a candidate energy-saving action will cause distributed training synchronization waiting or data state rework in the energy-saving operation and maintenance decision-making of intelligent computing centers based on the pattern recognition of task operation stage and data status, so as to avoid the single point energy-saving action from causing the entire group of tasks to increase consumption. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an energy-saving operation and maintenance decision-making system and method for intelligent computing centers based on big data. This system solves the problems mentioned in the background art by performing pattern recognition on the round-run records of distributed training tasks, synchronous waiting relationships, checkpoint status, and resident data status, and by performing dual update judgments on candidate energy-saving actions such as reducing energy consumption, increasing energy consumption while waiting, and increasing energy consumption by reworking data.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a big data-based intelligent computing center energy-saving operation and maintenance decision-making method, comprising: S1. Write the training log, power log, synchronization log, cache log and checkpoint log of the distributed training task into the round record according to the round number, extract the completion time, waiting power, resident data and checkpoint status of each training node, and generate the round table. S2. Perform rule matching on the round table, generate closing time and waiting segment according to the same round number, and generate prohibited segment and rework table based on checkpoint status and dwell data to obtain rule matching results; S3. Generate candidate actions for the affected training nodes that do not hit the prohibited segment in the rule matching results, write the candidate actions for the affected training nodes in the round table, recalculate the completion time, dwell state and checkpoint state according to the action type, update the closure time with the recalculated completion time, and generate the action table. S4. After fixing the round number, training node and action type, independently decompose the execution conditions of the action table, generate a power reduction item based on the power difference and action duration, generate a waiting item based on the closing shift amount and waiting power, generate a rework item based on the recovery power and recovery duration of the released data, and generate the decomposition results. S5. Perform dual updates with the energy reduction item as the main item and the waiting item and rework item as the dual items to generate net item, time period mark, data mark and dual update result; when the net item is positive, the time period mark is passed and the data mark is passed, output the energy saving action; otherwise, write the candidate action as the rejection action and maintain the node power, resident data and checkpoint status of the training node.

[0005] In a preferred embodiment, S1 includes: S1-1. In the training log and synchronization log, the calculation end time of the same training node within the same round number is written as the completion time, and the sampling period between the completion time and the synchronization start time is written as the waiting period. S1-2. For the power log during the waiting period, multiply each sampled power by the time difference between adjacent samples, sum them up, and then divide by the duration of the waiting period to obtain the waiting power of the training node. S1-3. Based on the cache log and checkpoint log, write the data that has been loaded but not written to the completed checkpoint as resident data, write the checkpoint completion status or checkpoint incomplete status as checkpoint status, and generate a round table according to round number, training node, completion time, waiting power, resident data and checkpoint status.

[0006] In a preferred embodiment, S2 includes: S2-1. Read the round table using the round number as the grouping key, sort the completion times of each training node within the same round number in ascending order of time, take the completion time of the last sorted node as the closing time, and write the closing time into the record of each training node within the same round number. S2-2. Subtract the completion time from the closing time of each training node within the same round number to obtain the node time difference. When the node time difference is positive, use the completion time as the waiting start point and the closing time as the waiting end point to generate a waiting segment from the waiting start point, waiting end point, round number, and training node.

[0007] In a preferred embodiment, S2 further includes: S2-3. Based on the checkpoint status of each training node within the same round number, take the writing period before the checkpoint status changes from incomplete to complete as the prohibited period, and generate a prohibited segment by the start point, end point, round number, and training node of the prohibited period. S2-4. Perform a difference operation on the resident data of each training node within the same round number and the completed checkpoint data. Use the difference result as the resident data outside the checkpoint. Generate a rework table according to the round number, training node, and resident data outside the checkpoint. Then, write the closing time, waiting segment, forbidden segment, and rework table into the same rule matching record according to the same round number to generate the rule matching result.

[0008] In a preferred embodiment, S3 includes: S3-1. Read the prohibited segment hit results according to the training node in the rule matching results, and generate action node records for the training nodes that did not hit the prohibited segment, their round number, closing time, waiting segment and rework table. S3-2. For action node records, write the period from the start of the waiting segment to the end of the waiting segment as the frequency reduction action period and generate candidate actions with a frequency reduction flag; when there is no data outside the checkpoint belonging to the training node in the rework table, generate candidate actions with a memory release flag; when the checkpoint status is complete, generate candidate actions with a pause flag. S3-3. Write the candidate actions into the round table of the affected training nodes. When the candidate action carries a frequency reduction flag, update the completion time. When the candidate action carries a memory release flag, update the resident state. When the candidate action carries a pause flag, update the checkpoint state. Recalculate the closure time with the updated completion time to generate the action table.

[0009] In a preferred embodiment, S4 includes: S4-1. After fixing the round number, training node and candidate action label, perform segmented subtraction on the round table and action table according to the action start point and action end point. Subtract the node power in the action table from the node power in the round table to generate the power difference before and after the action, and subtract the action start point from the action end point to generate the action duration. S4-2. Write the time period when the power difference before and after the action is negative as the invalid power reduction period, and write the time period when the power difference before and after the action is non-negative as the effective power reduction period. Then, multiply the power difference before and after the action in the effective power reduction period by the action duration and sum them up to generate the power reduction item.

[0010] In a preferred embodiment, S4 further includes: S4-3. Subtract the closing time in the rule matching result from the closing time in the action table to generate the closing time shift. When the closing time shift is positive, multiply the closing time shift by the waiting power of each training node in the same round number (excluding the affected training node) and sum them up to generate a waiting item. S4-4. Write the resident data released by the candidate action in the rework table as release data. Subtract the release time from the recovery end time of the release data to generate the recovery duration. Multiply the recovery power of the release data by the recovery duration and sum them to generate rework items. Then generate decomposition results with the cost reduction items, waiting items and rework items.

[0011] In a preferred embodiment, S5 includes: S5-1. Read the cost reduction item, waiting item, and rework item from the decomposition results. Add the waiting item and the rework item to get the deduction item for this round. Subtract the cost reduction item from the deduction item for this round to get the net value for this round. Write the net value for this round into the rule matching results according to the round number, training node, and candidate action. S5-2. The later arrival time between the candidate action time period start and the prohibited segment start is recorded as the time period intersection start. The earlier arrival time between the candidate action time period end and the prohibited segment end is recorded as the time period intersection end. When the time period intersection end is later than the time period intersection start, the time period intersection value is obtained by subtracting the time period intersection start from the time period intersection end and written into the time period rejection flag. Otherwise, the time period intersection value is recorded as zero and written into the time period pass flag. S5-3. Record the set of identifiers for candidate action release data as the release set, and the set of identifiers for data remaining outside the checkpoint in the rework table as the rework set. Write the data identifiers that belong to both the release set and the rework set into the data intersection. When the data intersection is empty, record the intersection data volume as zero and write it into the data pass flag. Otherwise, sum the data volumes of each data identifier in the data intersection to obtain the intersection data volume and write it into the data rejection flag.

[0012] In a preferred embodiment, S5 further includes: S5-4. Read the historical deduction items that have been written for the same training node and the same candidate action in the rule matching results, sum the historical deduction items to obtain the stock deduction items, and record the stock deduction items as zero when there are no historical deduction items. Subtract the stock deduction items from the net value of this round to obtain the net item, and generate the dual update result with the deduction items of this round, the stock deduction items and the net item. S5-5: When the net item is positive, the time period exists through the marker, and the data exists through the marker, output the energy-saving action, and write the net item to offset the stock deduction item and then write it into the dual update result; otherwise, write the candidate action as the rejection action, keep the node power, resident data, and checkpoint status of the training node as the values ​​before the candidate action was written, and add the current round of deduction item to the stock deduction item and then write it into the dual update result.

[0013] In a preferred embodiment, the big data-based intelligent computing center energy-saving operation and maintenance decision-making system includes: The round collection module is used to write the training logs, power logs, synchronization logs, cache logs and checkpoint logs of the distributed training task into the round record according to the round number, extract the completion time, waiting power, resident data and checkpoint status of each training node, and generate a round table. The rule matching module is used to perform rule matching on the round table, generate closing moments and waiting segments according to the same round number, and generate prohibited segments and rework tables based on checkpoint status and dwell data to obtain the rule matching results; The action recalculation module is used to generate candidate actions for the affected training nodes that do not hit the prohibited segment in the rule matching results, write the candidate actions for the affected training nodes in the round table, recalculate the completion time, dwell state and checkpoint state according to the action type, update the closure time with the recalculated completion time, and generate the action table. The decomposition measurement module is used to independently decompose the execution conditions of the action table after fixing the round number, training node and action type. It generates energy reduction items based on power difference and action duration, waiting items based on closure shift amount and waiting power, and rework items based on recovery power and recovery duration of released data, and generates decomposition results. The dual reverse write module is used to perform dual updates with the energy reduction item as the main item and the waiting item and rework item as the dual items, generating net items, time period markers, data markers and dual update results; when the net item is positive, the time period marker is passed and the data marker is passed, the energy saving action is output; otherwise, the candidate action is written as a rejection action, and the node power, resident data and checkpoint status of the training node are maintained.

[0014] The technical effects and advantages of this invention are as follows: 1. Identify the consequences of candidate actions through the round table, waiting period, and rework table. First, deduct waiting items and rework items, and then output energy-saving actions to relatively suppress the problem of increased consumption of the whole group caused by energy saving at a single point. 2. By constructing waiting segments through closing moments and completion moments, the synchronous waiting of early completion nodes is quantified into rounds, making it easier to distinguish between actual energy-saving space and synchronous entanglement risks; 3. By generating prohibited segments based on checkpoint states and determining the execution time of candidate actions, pauses, migrations, and memory releases can avoid the period when checkpoints are not completed, reducing the risk of training interruptions and rework. 4. Generate a rework table by the difference between the resident data and the completed checkpoint data, so that the resident data outside the checkpoint is retained in the action judgment, which relatively reduces data rereading and cache reconstruction; 5. By forming a dual update result through consumption reduction items, waiting items, and rework items, the historical deduction items can have a reverse influence on similar candidate actions, thereby relatively improving the problem of repeated misjudgments and repeated ineffective energy saving. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method steps of the present invention.

[0016] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Refer to the instruction manual appendix Figure 1-2 The present invention provides a big data-based intelligent computing center energy-saving operation and maintenance decision-making method, comprising: S1. Write the training log, power log, synchronization log, cache log and checkpoint log of the distributed training task into the round record according to the round number, extract the completion time, waiting power, resident data and checkpoint status of each training node, and generate the round table. In this embodiment, S1 is used to organize the running process of the distributed training task within the same training round into a round table for subsequent rule matching and reading. During processing, the round number and training node are first used as indexes to attach the calculation end record in the training log, the synchronization start record in the synchronization log, the sampling power in the power log, the loaded data in the cache log, and the write status in the checkpoint log to the same training node. Then, the completion time, waiting period, waiting power, resident data, and checkpoint status are determined to form the basic data items required for subsequent closure time, waiting period, rework table, and candidate action calculation. The implementation process includes the following steps: In S1-1, records of the same training node within the same round number are read from the training log and synchronization log. The round number is preferentially taken from the training iteration number in the training log. If the training log does not carry the training iteration number, the round number is generated from the computation interval between two adjacent synchronization start times within the same distributed training task. The training node is jointly determined by the task identifier, server identifier, GPU identifier, and training process identifier. Only one completion time record is retained for the same training node within the same round number. The computation end time is taken from the time when the last GPU computation segment of the training node ends within the round and written as the completion time. The synchronization start time is taken from the time when the training node first enters parameter synchronization or gradient synchronization within the round. If the completion time is earlier than the synchronization start time, the sampling period between the completion time and the synchronization start time is written as the waiting period. If the synchronization start time is not later than the completion time, the waiting period duration is recorded as zero, and the completion time of the training node is retained in the round record, without generating a valid waiting period. In S1-2, for the power log during the waiting period, the node power of the training node is first read according to the sampling time. When the start or end of the waiting period falls between two sampling times, the sampling duration of the first and last segments is trimmed according to the waiting period boundary. The power record within the waiting period is based on the difference between adjacent sampling times as the duration. During calculation, each sampled power is multiplied by the sampling duration covered by that sampled power to obtain the segmented power. Then, the power of each segment is summed and divided by the waiting period duration to obtain the waiting power of the training node. When the waiting period duration is zero, the waiting power of the training node is recorded as zero. The waiting power obtained in this way only represents the power level of the training node before entering synchronization after completing the calculation, and is not mixed with the power of the calculation segment. In S1-3, the resident data and checkpoint status are determined based on the cache log and checkpoint log. Loaded data in the cache log is written according to its data identifier, which includes data category, data number, version number, training node, and data volume. Data categories include training sample cache, model weights, optimizer status, intermediate activation cache, and intermediate results. Data with completed checkpoints in the checkpoint log is considered completed checkpoint data, while data in the loaded data that has not yet been written to a completed checkpoint is considered resident data. When a completion record appears in the checkpoint log, the checkpoint status of that training node in that round is written as "checkpoint completed." When a start-writing record exists in the checkpoint log but no completion record appears, the checkpoint status is written as "checkpoint incomplete." Finally, a round table is generated based on round number, training node, completion time, waiting power, resident data, and checkpoint status. Records for each training node under the same round number are written to this round table. Through the above processing, the training log, synchronization log, power log, cache log and checkpoint log are compressed into the necessary record items in the round table, so that the subsequent S2 can directly generate the closing time and waiting segment based on the completion time, generate the forbidden segment based on the checkpoint state, and generate the rework table based on the resident data, avoiding repeated parsing of the original log in subsequent steps. In practical applications: For example, in a distributed training task with 4 training nodes in round 128, node A completes computation at 10:00:03.200 and enters synchronization at 10:00:03.600. Then, 10:00:03.200 is written as the completion time of node A, and the 400 milliseconds between them are written as the waiting period. If the sampling power of the two segments during the waiting period is 420W for 200 milliseconds and 390W for 200 milliseconds respectively, then the waiting power is 405W. If node A has loaded model weights W1 and cache block C1, where W1 has been written to the completion checkpoint but C1 has not been written to the completion checkpoint, then C1 is written as resident data, and the completion or incomplete record in the checkpoint log is combined to write the checkpoint status.

[0019] S2. Perform rule matching on the round table, generate closing time and waiting segment according to the same round number, and generate prohibited segment and rework table based on checkpoint status and dwell data to obtain rule matching results; In this implementation, S2 is used to convert the completion time, checkpoint status, and dwell data in the round table into rule matching results, so that subsequent candidate actions can only be generated within the range defined by the closing time, waiting segment, prohibited segment, and rework table. During processing, the closing time of the same training round is first determined according to the round number, and then the waiting segment is formed according to the difference between the closing time and the completion time of each training node. Subsequently, the prohibited segment is formed according to the checkpoint status, and the rework table is formed by the difference between the dwell data and the completed checkpoint data. Finally, the closing time, waiting segment, prohibited segment, and rework table are written into the same rule matching record. The implementation process includes the following steps: In S2-1, the round number is used as the grouping key to read the round table. Training node records belonging to the same round number are placed into the same round group. The completion times of each training node in the round group are sorted in ascending order of time, and the completion time of the last sorted node is used as the closing time. If multiple training nodes have the same last sorted completion time, the closing time is taken from that time value, and it is not generated repeatedly according to the number of training nodes. After the closing time is determined, it is written into the records of each training node in the same round number, so that each training node in the same training round carries the same closing time, providing the same reference for subsequent node time difference calculation. In S2-2, the node time difference is obtained by subtracting the completion time of each training node from its closure time within the same round number. The node time difference is measured in milliseconds or seconds, with the unit determined by the time unit of the training log. When the node time difference is positive, it means that the training node completed the calculation of this round earlier than the closure time. The completion time is used as the waiting start time, and the closure time is used as the waiting end time. A waiting segment is generated by combining the waiting start time, the waiting end time, the round number, and the training node. When the node time difference is zero, it means that the training node completed the calculation of this round at the closure time, and no waiting segment is generated. When the node time difference is less than zero, the round is recorded as an abnormal round record and is not included in the candidate action generation. In S2-3, a forbidden segment is generated based on the checkpoint status of each training node within the same round number. The write period before the checkpoint is completed is taken as the forbidden period. The start time of the forbidden period is the checkpoint write start time in the checkpoint log. If the checkpoint log does not record the write start time, the first occurrence time of the checkpoint incomplete state is taken. The end time of the forbidden period is the first occurrence time of the checkpoint completed state. If the checkpoint completed state does not occur in the current round, the closing time of the round is taken as the end time of the forbidden period. Subsequently, the start time, end time, round number, and training node of the forbidden period are used to generate a forbidden segment, so that migration, pause, and memory release actions are subject to subsequent judgment during the checkpoint incomplete period. In S2-4, the resident data of each training node within the same round number is subjected to a difference operation with the data of the completed checkpoints. The difference operation uses the data identifier as the key, and the data identifier consists of the data category, data number, version number, training node, and data volume. Data identifiers that do not exist in the completed checkpoint data are regarded as resident data outside the checkpoints. Subsequently, a rework table is generated according to the round number, training node, and resident data outside the checkpoints. After the waiting segment, prohibited segment, and rework table are generated, the closing time, waiting segment, prohibited segment, and rework table are written into the same rule matching record according to the same round number to generate the rule matching result. Through this process, the completion time in the round table is transformed into a closed time and a waiting segment, the checkpoint status is transformed into a prohibited segment, and the resident data is transformed into a rework table, so that the subsequent S3 can filter the affected training nodes based on the rule matching results and prevent candidate actions from entering the incomplete period of the checkpoint or releasing resident data outside the checkpoint. In a practical application: In the 128th round of a distributed training task, there are nodes A, B, C, and D, with four completion times of 10:00:03.200, 10:00:03.350, 10:00:03.600, and 10:00:03.600 respectively. Therefore, the closing time is taken as 10:00:03.600. Node A generates a waiting segment with a starting point of 10:00:03.200 and an ending point of 10:00:03.600, while node B generates a waiting segment with a starting point of 10:00:03.350 and an ending point of 10:00:03.600. The waiting segment is not generated for nodes C and D. If node A's checkpoint starts writing at 10:00:03.450 and completes at 10:00:03.700, then a forbidden segment is generated from 10:00:03.450 to 10:00:03.700. If node A's resident data includes model weights W1, cache blocks C1, and intermediate results R1, where W1 has been written to the completion checkpoint, and C1 and R1 have not been written to the completion checkpoint, then C1 and R1 are written to the rework table as resident data outside the checkpoint, and the results are matched with the closing time, waiting segment, and forbidden segment writing rules of this round.

[0020] S3. Generate candidate actions for the affected training nodes that do not hit the prohibited segment in the rule matching results, write the candidate actions for the affected training nodes in the round table, recalculate the completion time, dwell state and checkpoint state according to the action type, update the closure time with the recalculated completion time, and generate the action table. In this embodiment, S3 is used to generate candidate actions within the range defined by the rule matching results, and write the candidate actions into the round table to deduce the training node state after the action is executed. During processing, training nodes that hit the prohibited segment are first excluded, and then three types of candidate actions are generated according to the waiting segment, rework table and checkpoint state, respectively: frequency reduction, memory release and pause. Then, the candidate actions are written into the affected training node, and the completion time, dwell state and checkpoint state are rewritten according to the markers carried by the candidate actions to obtain the action table for S4 to decompose and calculate. The implementation process includes the following steps: In S3-1, the prohibited segment hit result is read from the rule matching result by training node. The prohibited segment hit result is obtained by the cross-judgment of the candidate action time period of the training node and the prohibited segment. When there is an intersection, it is recorded as a prohibited segment hit. When there is no intersection, it is recorded as a prohibited segment not hit. The training node that hits the prohibited segment does not enter the candidate action generation. The training node that does not hit the prohibited segment is written as the action training node, and the round number, closing time, waiting segment and rework table of the training node are generated into an action node record. The action node record retains the training node index. Subsequent steps read the checkpoint status from the round table according to the training node index, and read the checkpoint out-of-station data belonging to the training node from the rework table according to the training node index. In S3-2, for an action node record, first determine if the training node has a waiting segment. If a waiting segment exists, write the period from the start to the end of the waiting segment as the frequency reduction action period and generate a candidate action with a frequency reduction flag. The candidate action contains the round number, training node, frequency reduction action period, and frequency reduction flag. Then, read the rework table. If there is no resident data outside the checkpoint belonging to the training node in the rework table, it means that all resident data of the training node has been written to the completed checkpoint, and a candidate action with a memory release flag is generated. If there is resident data outside the checkpoint belonging to the training node in the rework table, no memory release candidate action is generated. Then, read the checkpoint status of the training node from the round table. If the checkpoint status is completed, a candidate action with a pause flag is generated. If the checkpoint status is not completed, no pause candidate action is generated. When the same training node meets multiple generation conditions, multiple candidate actions are generated, and each candidate action carries only one flag. In S3-3, candidate actions are written into the round table of affected training nodes, and the completion time, dwell state, and checkpoint state of the affected training nodes are rewritten according to the markers carried by the candidate actions. When a candidate action carries a frequency reduction marker, the frequency reduction action period is located within the waiting segment, and the calculation process has ended before the start of the waiting segment. Therefore, the completion time is the same as the completion time before the candidate action was written, and the node power after the action is written as the node power after frequency reduction. When a candidate action carries a memory release marker, the data to be released is removed from the dwell data, and the dwell state is written as released. When a candidate action carries a pause marker, the training progress from the pause start time to the resume time is recorded as zero, and the checkpoint state remains the checkpoint state before the candidate action was written. After the completion time, dwell state, and checkpoint state of the affected training nodes are updated, the completion time of each training node under the same round number is reread, the completion time of the last sorted node is taken as the updated closure time, and the candidate action, the updated completion time, dwell state, checkpoint state, and updated closure time are written into the action table. Through the above processing, S3 generates candidate actions only on training nodes that have not hit the prohibited segment, and each type of candidate action is directly triggered by the waiting segment, rework table, or checkpoint state, avoiding the direct use of any low-power node as an energy-saving target; the action table retains the state changes before and after writing the candidate action, enabling S4 to continue to split the power reduction items, waiting items, and rework items according to the power difference, closing shift amount, and release data recovery process; In practical application: In round 128, if node A misses the forbidden segment and has a waiting segment from 10:00:03.200 to 10:00:03.600, then node A generates a candidate action with a down-frequency flag, and the down-frequency action period is the waiting segment; if there is still cache block C1 in node A's rework table, then no candidate action to release video memory is generated; if node A's checkpoint status is complete, then another candidate action with a pause flag is generated; after the candidate action is written, the down-frequency candidate action retains node A's original completion time but writes the down-frequency power, and the pause candidate action rewrites the training advance amount according to the pause period, and then recalculates the closure time using the completion times of each training node in the same round and generates the action table.

[0021] S4. After fixing the round number, training node and action type, independently decompose the execution conditions of the action table, generate a power reduction item based on the power difference and action duration, generate a waiting item based on the closing shift amount and waiting power, generate a rework item based on the recovery power and recovery duration of the released data, and generate the decomposition results. In this embodiment, S4 is used to decompose the results of candidate actions in the action table into energy saving items, waiting items, and rework items, so that S5 can read the energy saving of the candidate action itself, the energy increase caused by other training nodes being forced to wait in the same round, and the energy increase caused by the recovery after releasing the resident data. During processing, after fixing the round number, training node, and candidate action label, the round table and the action table are subdivided in the same segment to obtain the power difference before and after the action and the action duration. Then, the time period that does not generate energy saving is excluded, and the energy saving items are calculated. Subsequently, the waiting items caused by the shift of the closing time to other training nodes are calculated, and the rework items caused by the data release recovery process are calculated. The implementation process includes the following steps: In S4-1, after the round number, training node, and candidate action label are fixed, the start and end points of the candidate action are read, and the node power records in the round table and action table are trimmed according to the start and end points respectively. If the sampling times of the round table and action table are different, the overlapping time period of the two power sampling intervals is used as the same segment difference unit. In each overlapping time period, the node power in the round table is read as the pre-action power, and the node power in the action table is read as the post-action power. The power difference before and after the action is obtained by subtracting the post-action power from the pre-action power. The action duration of each overlapping time period is the overlap time period end point minus the overlap time period start point. If the action start point is equal to the action end point, the action duration is recorded as zero and is not included in the subsequent power reduction accumulation. In S4-2, the power difference before and after the action obtained in S4-1 is used to divide the energy reduction segment. The time period with a negative power difference before and after the action indicates that the node power after the candidate action is executed is higher than the node power before the candidate action is written. It is written as an invalid energy reduction segment and this time period is not included in the energy reduction term accumulation. The time period with a non-negative power difference before and after the action is written as an effective energy reduction segment. For each effective energy reduction segment, the power difference before and after the action is multiplied by the action duration of the effective energy reduction segment to obtain the segmented energy reduction amount. Then, the segmented energy reduction amounts under the same round number, the same training node, and the same candidate action label are accumulated to generate an energy reduction term. Thus, the energy reduction term only represents the power reduction amount formed by the candidate action in the affected training node itself, without mixing in the energy consumption changes formed by other training nodes waiting to form. In S4-3, the closing time in the action table is subtracted from the closing time in the rule matching result to generate the closing time shift. When the closing time shift is less than or equal to zero, it means that the candidate action has not extended the closing time of this round of training, and the waiting item is recorded as zero. When the closing time shift is positive, the waiting power of each training node in the same round number, except for the affected training node, is read. The closing time shift is multiplied by the waiting power of these training nodes to obtain the new waiting amount of each training node. Then, the new waiting amounts are accumulated to generate the waiting item. The affected training node does not participate in the waiting item calculation because the power change of the training node itself has been included in the power reduction item, so as to avoid the same training node being repeatedly measured in the power reduction item and the waiting item. In S4-4, the resident data released by candidate actions in the rework table is read and written as release data; the release time of the release data is taken as the execution time of the candidate action; the recovery end time is taken as the time when the release data is reloaded, the cache hit is restored, or the model weights are re-resident; the recovery power is obtained by adding the storage read power, network transmission power, and node additional power during the recovery process of historical release data of the same type; when there are historical recovery records of release data of the same type on the same training node, the most recent recovery record is taken; when there are no records on the same training node, the average of historical recovery records of the same data category is taken; the recovery duration is obtained by subtracting the release time from the recovery end time; the recovery power of each release data is multiplied by the recovery duration to obtain the single rework amount, and then the single rework amounts are accumulated to generate a rework item; finally, the decomposition results are generated with the cost reduction item, the waiting item, and the rework item, and written according to the round number, training node, and candidate action tag for S5 to perform dual update; Through the above processing, the impact of the candidate action is broken down into three independent quantities: the power reduction term records the power reduction of the affected training node itself, the waiting term records the waiting power increase of other training nodes in the same round due to the delayed closing time, and the rework term records the recovery power increase after releasing the resident data; in this way, S5 no longer makes energy saving judgments based solely on the power reduction of a single training node, but also deducts the costs of synchronization waiting and data recovery at the same time. In practical application: In round 128, the candidate action period for frequency reduction of node A is from 10:00:03.200 to 10:00:03.600. The power of node A in the round table is 420W, the power of node A in the action table is 300W, and the action duration is 0.4 seconds. Therefore, the power reduction item for this period is 48J. If the closing time in the action table is moved from 10:00:03.600 to 10:00:03.750, the closing time is moved by 0.15 seconds. The waiting power of nodes B and C is 390W and 410W respectively. Therefore, the waiting item is 120J. If the candidate action for releasing video memory releases cache block C1, the recovery power of C1 is 260W, and the recovery duration is 0.3 seconds. Therefore, the rework item is 78J. The decomposition result is written as power reduction item 48J, waiting item 120J, and rework item 78J.

[0022] S5. Perform dual updates with the energy reduction item as the main item and the waiting item and rework item as the dual items to generate net item, time period mark, data mark and dual update result; when the net item is positive, the time period mark is passed and the data mark is passed, output the energy saving action; otherwise, write the candidate action as the rejection action and maintain the node power, resident data and checkpoint status of the training node. In this implementation, S5 is used to determine whether a candidate action should be executed based on the decomposition results, and to convert the waiting items and rework items generated in this round into deduction records for the same training node and the same candidate action in the future. During processing, the net value of this round is first obtained from the cost reduction items, waiting items and rework items, and then it is determined whether the candidate action time period falls into the prohibited segment and whether the candidate action release data belongs to the data resident outside the checkpoint. Then, the historical deduction items are read to form the stock deduction items, and the net items are used to determine whether to execute or reject. The implementation process includes the following steps: In S5-1, the energy consumption reduction items, waiting items, and rework items are read from the decomposition results according to the round number, training node, and candidate action. The waiting item represents the increased waiting energy consumption of other training nodes in the same round caused by the candidate action, and the rework item represents the energy consumption generated after the candidate action releases the resident data. Therefore, the waiting item and the rework item are added to obtain the deduction item for this round. Then, the energy consumption reduction item is subtracted from the deduction item for this round to obtain the net value for this round. The net value for this round, the deduction item for this round, the energy consumption reduction item, the waiting item, and the rework item are all written to the rule matching results according to the round number, training node, and candidate action, so that subsequent historical deduction items can be read from the same write position. In S5-2, the candidate action time period and the prohibited segment are interleaved. The moment with the larger time value between the start of the candidate action time period and the start of the prohibited segment is written as the start of the time period intersection, and the moment with the smaller time value between the end of the candidate action time period and the end of the prohibited segment is written as the end of the time period intersection. If the end of the time period intersection is later than the start of the time period intersection, the time period intersection value is obtained by subtracting the start of the time period intersection from the end of the time period intersection, and this value is written to the time period rejection flag. Otherwise, the time period intersection value is recorded as zero, and this value is written to the time period pass flag. The time period intersection value, the time period pass flag, or the time period rejection flag are all written to the rule matching result to distinguish whether the candidate action occupies the incomplete period of the checkpoint. In S5-3, the data identifiers of the candidate action release data are grouped into a release set, and the data identifiers of the data remaining outside the checkpoint in the rework table are grouped into a rework set. The data identifiers in the release set are read one by one and compared with the data identifiers in the rework set. Data identifiers that exist in both the release set and the rework set are written into the data intersection. When the data intersection is empty, the intersection data volume is recorded as zero and a data pass flag is written. Otherwise, the data volume of each data identifier in the data intersection is read, the data volumes are summed to obtain the intersection data volume, and a data rejection flag is written. The intersection data volume, the data pass flag, or the data rejection flag are all written into the rule matching result to distinguish whether the candidate action releases the checkpoint remaining data that still needs to be restored. In S5-4, the historical deduction items already written for the same training node and the same candidate action in the rule matching results are read. The historical deduction items are derived from the current round deduction items written for the same candidate action that was previously rejected. When there are multiple historical deduction items, the sum of each historical deduction item is used to obtain the existing deduction item. When there are no historical deduction items, the existing deduction item is recorded as zero. Then, the net value of the current round is subtracted from the existing deduction item to obtain the net item, and the current round deduction item, the existing deduction item, and the net item are written into the dual update result. Thus, waiting items and rework items will not be deducted only once in the current round, but will be transformed into the basis for subsequent deductions for the same training node and the same candidate action. In S5-5, the action result is output based on the net item, time period marker, and data marker. When the net item is positive, the time period exists through the marker, and the data exists through the marker, the energy-saving action is output, and the net item is subtracted from the stock deduction item to obtain the deduction item after reduction. If the deduction item after reduction is less than zero, it is recorded as zero and written to the dual update result. Otherwise, the candidate action is written as a rejection action, the node power, resident data, and checkpoint status of the training node in the action table are restored to the values ​​before the candidate action is written, and the deduction item of this round is added to the stock deduction item and written to the dual update result. This rejection action does not enter the energy-saving execution queue. Through the above processing, S5 does not only compare the single cost reduction, but transforms the waiting items and rework items into the historical burden of the same training node and the same candidate action, so that the candidate action still needs to offset the waiting and rework costs caused in the previous rounds in subsequent rounds; the time period marker is used to exclude actions that occupy prohibited segments, the data marker is used to exclude actions that release data outside the checkpoint, and the net item is used to determine whether the current action's benefit can cover the deduction items in this round and the existing deduction items; In practical application: In the 128th round, the frequency reduction candidate action of node A generates a power reduction item of 48J, a waiting item of 120J, and a rework item of 0J. The deduction item for this round is 120J, and the net value for this round is -72J. This action is written as a veto action, and 120J is written into the historical deduction item. In the next round, the same node A generates a frequency reduction candidate action again. If the net value for this round is 160J and both the time period and the data pass the mark exist, the net item is 40J. The energy saving execution action is output, and 40J is used to offset the 120J stock deduction item. After the offset, the deduction item is 80J, and the dual update result is continued.

[0023] Furthermore, the big data-based intelligent computing center energy-saving operation and maintenance decision-making system includes: The round collection module is used to write the training logs, power logs, synchronization logs, cache logs and checkpoint logs of the distributed training task into the round record according to the round number, extract the completion time, waiting power, resident data and checkpoint status of each training node, and generate a round table. The rule matching module is used to perform rule matching on the round table, generate closing moments and waiting segments according to the same round number, and generate prohibited segments and rework tables based on checkpoint status and dwell data to obtain the rule matching results; The action recalculation module is used to generate candidate actions for the affected training nodes that do not hit the prohibited segment in the rule matching results, write the candidate actions for the affected training nodes in the round table, recalculate the completion time, dwell state and checkpoint state according to the action type, update the closure time with the recalculated completion time, and generate the action table. The decomposition measurement module is used to independently decompose the execution conditions of the action table after fixing the round number, training node and action type. It generates energy reduction items based on power difference and action duration, waiting items based on closure shift amount and waiting power, and rework items based on recovery power and recovery duration of released data, and generates decomposition results. The dual reverse write module is used to perform dual updates with the energy reduction item as the main item and the waiting item and rework item as the dual items, generating net items, time period markers, data markers and dual update results; when the net item is positive, the time period marker is passed and the data marker is passed, the energy saving action is output; otherwise, the candidate action is written as a rejection action, and the node power, resident data and checkpoint status of the training node are maintained.

[0024] The working principle of this scheme is as follows: First, the training logs, power logs, synchronization logs, cache logs, and checkpoint logs in the distributed training task are organized into a round table according to the round number, obtaining the completion time, waiting power, resident data, and checkpoint status of each training node. Then, through pattern recognition and rule matching, the closing time, waiting segment, prohibited segment, and rework table within the same round are identified to determine which nodes have energy-saving action space and which nodes cannot be directly processed due to incomplete checkpoints or resident data outside the checkpoints. Subsequently, candidate actions are written into the round table, and the completion time, resident status, checkpoint status, and closing time after the action are recalculated. The impact of the candidate action is broken down into power reduction items, waiting items, and rework items. Finally, dual updates are used to determine whether the action truly saves energy. If the power reduction can cover the cost of synchronization waiting and data rework, and the prohibited segment is not hit and the rework data is not released, then the energy-saving action is output; otherwise, the action is rejected and the original state of the node is maintained. In practical applications, for example, a large model training task is executed by multiple GPU nodes. Node A completes its current round of computation earlier and enters a waiting state. Traditional energy-saving methods might directly reduce the frequency of node A or release its video memory. However, this solution first determines whether node A's waiting period is within the checkpoint completion period and checks whether node A's cached data, model weights, or intermediate results have not yet been written to the completion checkpoint. If reducing the frequency of node A would cause the closing time of this round to be delayed, increasing the waiting time of other nodes, or releasing video memory would cause subsequent data reloading, this solution will deduct these additional waiting energy consumption and rework energy consumption before deciding whether to execute. In this way, the system does not immediately save energy when it sees a decrease in the power of a single node, but first determines the impact of the action on the entire training round and the subsequent recovery process, thereby avoiding the situation where a single point seems to save power, but the entire group of tasks actually increases power consumption. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A big data-based intelligent computing center energy-saving operation and maintenance decision-making method, characterized in that, include: S1. Write the training log, power log, synchronization log, cache log and checkpoint log of the distributed training task into the round record according to the round number, extract the completion time, waiting power, resident data and checkpoint status of each training node, and generate the round table. S2. Perform rule matching on the round table, generate closing time and waiting segment according to the same round number, and generate prohibited segment and rework table based on checkpoint status and dwell data to obtain rule matching results; S3. Generate candidate actions for the affected training nodes that do not hit the prohibited segment in the rule matching results, write the candidate actions for the affected training nodes in the round table, recalculate the completion time, dwell state and checkpoint state according to the action type, update the closure time with the recalculated completion time, and generate the action table. S4. After fixing the round number, training node and action type, independently decompose the execution conditions of the action table, generate a power reduction item based on the power difference and action duration, generate a waiting item based on the closing shift amount and waiting power, generate a rework item based on the recovery power and recovery duration of the released data, and generate the decomposition results. S5. Perform dual updates with the energy reduction item as the main item and the waiting item and rework item as the dual items to generate net item, time period mark, data mark and dual update result; when the net item is positive, the time period mark is passed and the data mark is passed, output the energy saving action; otherwise, write the candidate action as the rejection action and maintain the node power, resident data and checkpoint status of the training node.

2. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data according to claim 1, characterized in that: S1 includes: S1-1. In the training log and synchronization log, the calculation end time of the same training node within the same round number is written as the completion time, and the sampling period between the completion time and the synchronization start time is written as the waiting period. S1-2. For the power log during the waiting period, multiply each sampled power by the time difference between adjacent samples, sum them up, and then divide by the duration of the waiting period to obtain the waiting power of the training node. S1-3. Based on the cache log and checkpoint log, write the data that has been loaded but not written to the completed checkpoint as resident data, write the checkpoint completion status or checkpoint incomplete status as checkpoint status, and generate a round table according to round number, training node, completion time, waiting power, resident data and checkpoint status.

3. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data according to claim 2, characterized in that: S2 includes: S2-1. Read the round table using the round number as the grouping key, sort the completion times of each training node within the same round number in ascending order of time, take the completion time of the last sorted node as the closing time, and write the closing time into the record of each training node within the same round number. S2-2. Subtract the completion time from the closing time of each training node within the same round number to obtain the node time difference. When the node time difference is positive, use the completion time as the waiting start point and the closing time as the waiting end point to generate a waiting segment from the waiting start point, waiting end point, round number, and training node.

4. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data according to claim 3, characterized in that: S2 further includes: S2-3. Based on the checkpoint status of each training node within the same round number, take the writing period before the checkpoint status changes from incomplete to complete as the prohibited period, and generate a prohibited segment by the start point, end point, round number, and training node of the prohibited period. S2-4. Perform a difference operation on the resident data of each training node within the same round number and the completed checkpoint data. Use the difference result as the resident data outside the checkpoint. Generate a rework table according to the round number, training node, and resident data outside the checkpoint. Then, write the closing time, waiting segment, forbidden segment, and rework table into the same rule matching record according to the same round number to generate the rule matching result.

5. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data according to claim 4, characterized in that: S3 includes: S3-1. Read the prohibited segment hit results according to the training node in the rule matching results, and generate action node records for the training nodes that did not hit the prohibited segment, their round number, closing time, waiting segment and rework table. S3-2. For action node records, write the period from the start of the waiting segment to the end of the waiting segment as the frequency reduction action period and generate candidate actions with a frequency reduction flag; when there is no data outside the checkpoint belonging to the training node in the rework table, generate candidate actions with a memory release flag; when the checkpoint status is complete, generate candidate actions with a pause flag. S3-3. Write the candidate actions into the round table of the affected training nodes. When the candidate action carries a frequency reduction flag, update the completion time. When the candidate action carries a memory release flag, update the resident state. When the candidate action carries a pause flag, update the checkpoint state. Recalculate the closure time with the updated completion time to generate the action table.

6. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data according to claim 5, characterized in that: S4 includes: S4-1. After fixing the round number, training node and candidate action label, perform segmented subtraction on the round table and action table according to the action start point and action end point. Subtract the node power in the action table from the node power in the round table to generate the power difference before and after the action, and subtract the action start point from the action end point to generate the action duration. S4-2. Write the time period when the power difference before and after the action is negative as the invalid power reduction period, and write the time period when the power difference before and after the action is non-negative as the effective power reduction period. Then, multiply the power difference before and after the action in the effective power reduction period by the action duration and sum them up to generate the power reduction item.

7. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data as described in claim 6, characterized in that: S4 further includes: S4-3. Subtract the closing time in the rule matching result from the closing time in the action table to generate the closing time shift. When the closing time shift is positive, multiply the closing time shift by the waiting power of each training node in the same round number (excluding the affected training node) and sum them up to generate a waiting item. S4-4. Write the resident data released by the candidate action in the rework table as release data. Subtract the release time from the recovery end time of the release data to generate the recovery duration. Multiply the recovery power of the release data by the recovery duration and sum them to generate rework items. Then generate decomposition results with the cost reduction items, waiting items and rework items.

8. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data according to claim 7, characterized in that: S5 includes: S5-1. Read the cost reduction item, waiting item, and rework item from the decomposition results. Add the waiting item and the rework item to get the deduction item for this round. Subtract the cost reduction item from the deduction item for this round to get the net value for this round. Write the net value for this round into the rule matching results according to the round number, training node, and candidate action. S5-2. The later arrival time between the candidate action time period start and the prohibited segment start is recorded as the time period intersection start. The earlier arrival time between the candidate action time period end and the prohibited segment end is recorded as the time period intersection end. When the time period intersection end is later than the time period intersection start, the time period intersection value is obtained by subtracting the time period intersection start from the time period intersection end and written into the time period rejection flag. Otherwise, the time period intersection value is recorded as zero and written into the time period pass flag. S5-3. Record the set of identifiers for candidate action release data as the release set, and the set of identifiers for data remaining outside the checkpoint in the rework table as the rework set. Write the data identifiers that belong to both the release set and the rework set into the data intersection. When the data intersection is empty, record the intersection data volume as zero and write it into the data pass flag. Otherwise, sum the data volumes of each data identifier in the data intersection to obtain the intersection data volume and write it into the data rejection flag.

9. The energy-saving operation and maintenance decision-making method for intelligent computing centers based on big data as described in claim 8, characterized in that: The S5 also includes: S5-4. Read the historical deduction items that have been written for the same training node and the same candidate action in the rule matching results, sum the historical deduction items to obtain the stock deduction items, and record the stock deduction items as zero when there are no historical deduction items. Subtract the stock deduction items from the net value of this round to obtain the net item, and generate the dual update result with the deduction items of this round, the stock deduction items and the net item. S5-5: When the net item is positive, the time period exists through the marker, and the data exists through the marker, output the energy-saving action, and write the net item to offset the stock deduction item and then write it into the dual update result; otherwise, write the candidate action as the rejection action, keep the node power, resident data, and checkpoint status of the training node as the values ​​before the candidate action was written, and add the current round of deduction item to the stock deduction item and then write it into the dual update result.

10. A big data-based intelligent computing center energy-saving operation and maintenance decision-making system, characterized in that: include: The round collection module is used to write the training logs, power logs, synchronization logs, cache logs and checkpoint logs of the distributed training task into the round record according to the round number, extract the completion time, waiting power, resident data and checkpoint status of each training node, and generate a round table. The rule matching module is used to perform rule matching on the round table, generate closing moments and waiting segments according to the same round number, and generate prohibited segments and rework tables based on checkpoint status and dwell data to obtain the rule matching results; The action recalculation module is used to generate candidate actions for the affected training nodes that do not hit the prohibited segment in the rule matching results, write the candidate actions for the affected training nodes in the round table, recalculate the completion time, dwell state and checkpoint state according to the action type, update the closure time with the recalculated completion time, and generate the action table. The decomposition measurement module is used to independently decompose the execution conditions of the action table after fixing the round number, training node and action type. It generates energy reduction items based on power difference and action duration, waiting items based on closure shift amount and waiting power, and rework items based on recovery power and recovery duration of released data, and generates decomposition results. The dual reverse write module is used to perform dual updates with the energy reduction item as the main item and the waiting item and rework item as the dual items, generating net items, time period markers, data markers and dual update results; when the net item is positive, the time period marker is passed and the data marker is passed, the energy saving action is output; otherwise, the candidate action is written as a rejection action, and the node power, resident data and checkpoint status of the training node are maintained.

Images