Model training resource scheduling method and device, computer device, readable storage medium and program product

Abstract

This application relates to a method, apparatus, computer device, computer-readable storage medium, and computer program product for scheduling model training resources. The method includes acquiring monitoring data of the target model during training; inputting the monitoring data into a prediction model to obtain third time-series data; performing weighted fusion processing on the first and third time-series data to obtain a weighted value of a preset monitoring indicator within a future target time window; and scheduling the training resources of the target model within the future target time window based on the weighted value of the preset monitoring indicator within the future target time window. This method can improve the optimization rate of monitoring indicators corresponding to resource consumption during the target model's training process.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for scheduling model training resources. Background Technology

[0002] With the continuous development of computer technology and the continuous improvement of intelligence, artificial intelligence (AI) is gradually participating in all aspects of production and life. AI can help users efficiently process massive amounts of data and automate repetitive tasks, greatly improving the efficiency of production and life. It is a cutting-edge technology research direction at present.

[0003] In practical applications, artificial intelligence usually needs to be trained to achieve corresponding functions. During the training process, a large amount of basic computing resources are required, including hardware resources and power resources. Existing technologies often do not pay attention to the resource consumption problem during the training process, which leads to the problem of poor optimization rate of resource consumption-related monitoring indicators. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for scheduling model training resources that can improve the optimization rate of resource consumption corresponding to detection indicators during model training, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for scheduling model training resources, comprising: acquiring monitoring data of a target model during training, the monitoring data including first time-series data and / or second time-series data, the first time-series data being determined based on the real-time monitoring value of a preset monitoring indicator within the current time window, and the second time-series data being the historical monitoring value of the preset monitoring indicator; inputting the monitoring data into a prediction model to obtain third time-series data, the third time-series data being the predicted value of the preset monitoring indicator output by the prediction model within a future target time window; performing weighted fusion processing on the first time-series data and the third time-series data to obtain a weighted value of the preset monitoring indicator within the future target time window; and scheduling the training resources of the target model within the future target time window according to the weighted value of the preset monitoring indicator within the future target time window.

[0006] In one embodiment, the method further includes: determining a confidence index of the third time series data by means of the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window based on the prediction model; and determining the weights for weighted fusion based on the confidence index.

[0007] In one embodiment, the monitoring data is input into the prediction model to obtain the third time series data, including: under the condition of meeting the confidence downgrade conditions, the monitoring data is input into the basic prediction model for prediction to obtain the third time series data; wherein, the prediction model includes the basic prediction model; the monitoring data also includes environmental data; the type of confidence downgrade conditions includes at least one of the following: the data of the first time series data is empty, the data volume of the second time series data is less than the data volume threshold, and the environmental data is missing.

[0008] In one embodiment, the confidence index of the third time series data is determined by a prediction model based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the mean absolute percentage error of the historical time window. This includes: determining candidate confidence indices of the third time series data based on the interval width, the historical maximum interval width, and the mean absolute percentage error; determining preset coefficients according to the type of confidence downgrade conditions; reducing the candidate confidence indices based on the preset coefficients to obtain the confidence index.

[0009] In one embodiment, determining the weights for weighted fusion based on the confidence index includes: determining a first weight for the first time series data and a second weight for the third time series data based on the confidence index; performing weighted fusion processing on the first time series data and the third time series data to obtain a weighted value of a preset monitoring indicator in a future target time window, including: calculating a first product of the first weight and the first time series data, and a second product of the second weight and the third time series data; and calculating the sum of the first product and the second product to obtain the weighted value.

[0010] In one embodiment, acquiring monitoring data of the target model during training includes at least one of the following: acquiring candidate environment data for the current time window, sampling and normalizing the candidate environment data to obtain environment data; acquiring real-time monitoring values ​​for the current time window, aggregating and normalizing the real-time monitoring values ​​to obtain first time-series data; and encoding the current time window to obtain time features.

[0011] In one embodiment, monitoring data is input into a prediction model to obtain third time-series data, including: inputting first time-series data, environmental data, time features, and second time-series data into the prediction model to obtain third time-series data.

[0012] In one embodiment, the training resources of the target model are scheduled within the future target time window based on the weighted value of a preset monitoring indicator within the future target time window, including: determining the target number of training units of the target model within the future target time window based on the weighted value; and performing resource expansion or resource reduction processing on the number of training units of the target model within the future target time window based on the target number.

[0013] In one embodiment, determining the target number of training units for the target model within a future target time window based on a weighted value includes: determining a function for determining the number of training units for the target model based on the weighted value and the training consumption parameters of the training units of the target model; and solving for the minimum value of the function based on the number of remaining training units of the target model to obtain the target number of training units within the future target time window.

[0014] In one embodiment, the determination function for determining the number of training units of the target model based on the weighted value and the training consumption parameter of the training units of the target model includes: determining at least one sub-determining function based on the weighted value and / or the training consumption parameter of the training units of the target model; and performing weighted calculation based on the preset weights corresponding to the at least one sub-determining function to obtain the determining function.

[0015] In one embodiment, at least one sub-determining function is determined based on the weighting value and / or the training consumption parameters of the training units of the target model, including at least one of the following: determining a first sub-function based on the weighting value and the unit energy consumption included in the training consumption parameters, wherein the value of the first sub-function is proportional to the number of training units; determining a second sub-function, wherein the value of the second sub-function is inversely proportional to the number of training units; and determining a third sub-function based on the unit energy consumption and the resource consumption parameters included in the training consumption parameters, wherein the value of the third sub-function is proportional to the number of training units.

[0016] In one embodiment, the method further includes: saving the training backup data of the target model when the training backup conditions are detected to be met; wherein the training backup conditions include the expiration of a preset backup time, or the scheduling quantity of training resources of the target model exceeds a scheduling quantity threshold.

[0017] In one embodiment, the method further includes: performing training consistency guarantee processing for the target model when a change in the number of training resources for the target model is detected; wherein the training consistency guarantee processing includes at least one of global synchronization of parameter values, recovery of the optimizer's internal state, gradient normalization compensation, and warm start of the learning rate.

[0018] In one embodiment, the method further includes at least one of the following: when a prediction failure of the prediction model is detected, performing weighted fusion processing on the first time series data and the third time series data to obtain a weighted value of the preset monitoring index in the future target time window, including determining the first time series data as the weighted value; when a network failure is detected, scheduling the training resources of the target model in the future target time window based on the weighted value, including maintaining the training resources at the current moment; and when the scheduling of training resources in the future target time window is identified, rolling back to the training backup data at the target moment, wherein the time series of the target moment is prior to all moments in the future target time window.

[0019] In one embodiment, the preset monitoring indicators include at least one of carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators.

[0020] Secondly, this application also provides a model training resource scheduling device, comprising: a monitoring data acquisition module for acquiring monitoring data of the target model during training; a data prediction module for inputting the monitoring data into a prediction model to obtain a weighted value of a preset monitoring indicator within a future target time window; a fusion module for performing weighted fusion processing on the first time series data and the third time series data to obtain the third time series data, wherein the third time series data is the predicted value of the preset monitoring indicator output by the prediction model within the future target time window; and a resource scheduling module for scheduling the training resources of the target model within the future target time window according to the weighted value of the preset monitoring indicator within the future target time window.

[0021] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect.

[0022] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0023] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0024] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for scheduling model training resources, during the training of the target model, first acquire monitoring data. This monitoring data includes first time-series data determined based on real-time monitoring values ​​of preset monitoring indicators within the current time window, and / or second time-series data corresponding to historical monitoring values ​​of the preset monitoring indicators. The monitoring data is then input into a prediction model to obtain third time-series data for the future target time window. This third time-series data represents the predicted value of the preset monitoring indicators output by the prediction model within the future target time window, thus achieving data prediction for the future time window. The first and third time-series data are weighted and fused to obtain a weighted value. Based on this weighted value of the preset monitoring indicators within the future target time window, the training resources of the target model are scheduled within that window. The weighted value of the preset monitoring indicators within the future target time window serves as the basis for scheduling training resources. By fusing the current first time-series data and the future third time-series data, a reliable weighted value is obtained, and the training resources are adjusted accordingly. This improves the optimization rate of the monitoring indicators corresponding to resource consumption during the target model's training process. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an application environment diagram of a model training resource scheduling method in one embodiment;

[0027] Figure 2 This is a flowchart illustrating a method for scheduling model training resources in one embodiment;

[0028] Figure 3 This is a flowchart illustrating step 201 in one embodiment;

[0029] Figure 4 This is a flowchart illustrating the confidence index determination step in one embodiment;

[0030] Figure 5 This is a flowchart illustrating step 202 in one embodiment;

[0031] Figure 6 This is a flowchart illustrating step 401 in one embodiment;

[0032] Figure 7 This is a flowchart illustrating step 202 in another embodiment;

[0033] Figure 8 This is a flowchart illustrating steps 402 and 203 in one embodiment;

[0034] Figure 9 This is a flowchart illustrating step 204 in one embodiment;

[0035] Figure 10 This is a flowchart illustrating step 901 in one embodiment;

[0036] Figure 11 This is a flowchart illustrating step 1001 in one embodiment;

[0037] Figure 12 This is a flowchart illustrating step 1101 in one embodiment;

[0038] Figure 13(a) is a flowchart illustrating the data backup steps in one embodiment;

[0039] Figure 13(b) is a schematic diagram of a training state snapshot in one embodiment;

[0040] Figure 14(a) is a schematic diagram of the consistency guarantee process in one embodiment;

[0041] Figure 14(b) is a detailed schematic diagram of consistency guarantee in one embodiment;

[0042] Figure 15 This is a flowchart illustrating the training and maintenance steps in one embodiment;

[0043] Figure 16 This is a flowchart illustrating the scheduling method for model training resources in another embodiment;

[0044] Figure 17 This is a structural block diagram of a model training resource scheduling device in one embodiment;

[0045] Figure 18 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0048] The model training resource scheduling method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown includes at least target model 101 and server 102.

[0049] The target model 101 can be a large language model or an expert model. The target model 101 is used to perform corresponding data processing according to the user's needs and / or instructions in order to realize the function of the target model 101. For example, the target model 101 can perform training processing based on the training dataset according to the user's training instructions to obtain the trained model.

[0050] Server 102 is used to acquire monitoring data of the target model 101 during the training process within the current time window, predict the third time-series data for the future target time window based on the monitoring data, obtain the weighted value of the preset monitoring indicators for the future target time window based on the first and third time-series data, and schedule the training resources of the target model 101 based on the weighted value. Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Server 102 can be a server corresponding to any one of the following: a large-scale generative artificial intelligence model training platform, a cloud computing GPU resource scheduling platform, a data center green computing control system, a multi-task collaborative training scheduling system, and an artificial intelligence computing hosting platform.

[0051] In real-world scenarios, models continuously utilize significant computing resources during training. This consumption of computing power and electricity leads to substantial carbon emissions. Existing AI training processes typically rely on workload scheduling systems. These systems manage computing resources through job allocation, application and allocation of computing resources, training task management, GPU resource allocation and release, and communication coordination during distributed training. However, this scheduling approach focuses primarily on traditional metrics such as GPU utilization, task priority, task completion time, and resource rental costs. It rarely incorporates environmental factors such as electricity carbon intensity, future carbon emission trend predictions, and the effective utilization of low-carbon energy periods into the scheduling decision-making system. Consequently, it has significant limitations in achieving green computing and carbon efficiency optimization.

[0052] Overall, the relevant technologies still have the following limitations and shortcomings:

[0053] First, existing technologies cannot achieve elastic resource scheduling at the computational unit level within the training task. Most existing mechanisms remain at the task level, making it difficult to adjust the scale of parallel computing in real time according to carbon intensity fluctuations. The resulting technical problem is: how to achieve dynamic resource scheduling of computational units within the training task to respond to changes in carbon intensity and optimize resource consumption.

[0054] Second, the relevant technologies are lagging, relying solely on the real-time state of the system to respond, and failing to introduce a predictive mechanism to achieve proactive scheduling. The technical problem to be solved is: how to construct a predictive model and proactively plan training resources based on predictive information.

[0055] Third, during dynamic resource adjustments, such as scaling up or down computing nodes, related technologies can easily disrupt gradient synchronization and state consistency in distributed training, affecting training continuity. The corresponding technical challenge is: how to maintain the consistency of the training state during dynamic scaling up or down of computing resources.

[0056] Fourth, the relevant technologies do not provide user-configurable multi-objective weight adjustment functions, making it impossible to flexibly balance time costs, carbon emissions, and economic costs. The technical problem that needs to be solved is: how to support users in autonomously adjusting the balance between training speed and resource consumption.

[0057] Fifth, the relevant technologies lack robust fault tolerance mechanisms, failing to reliably degrade and roll back the system when the prediction model fails or the scheduling decision is abnormal. This leads to the technical challenge of designing a highly fault-tolerant scheduling architecture with degradation and rollback capabilities.

[0058] To address this, this application obtains time-series data of the target model corresponding to the preset monitoring indicators during the training process by setting preset monitoring indicators, and determines the weighted value of the preset monitoring indicators within the future target time window based on the time-series data. The training resources of the target model are then scheduled based on the weighted value to improve the optimization rate of the monitoring indicators corresponding to resource consumption during model training. Furthermore, the above-mentioned technical problems are solved by combining prediction and real-time methods, setting backups, and adjusting training resources.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a method for scheduling model training resources is provided, which can be applied to... Figure 1 The following steps, 201 to 204, are used as an example to illustrate the process of using a server in the example.

[0060] Step 201: Obtain monitoring data of the target model during the training process.

[0061] In real-world scenarios, a model can be pre-set with at least one monitoring indicator during training. This indicator represents key metrics related to resource consumption during training, thereby monitoring the resource consumption of the target model. Furthermore, resource consumption can be represented by more intuitive data, such as electricity consumption. It can also be represented by software-level data, such as computing power consumption / operator occupancy. In addition, resource consumption can also be represented by carbon emission data.

[0062] Optionally, the preset monitoring indicators include at least one of the following: carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators. Specifically, the carbon emission monitoring indicator represents the monitoring of carbon emissions during the target model training process; the electricity consumption monitoring indicator represents the monitoring of electricity consumption during the target model training process; and the computing power consumption monitoring indicator represents the monitoring of operator occupancy and / or computing power consumption during the target model training process. Correspondingly, real-time monitoring values ​​can be: real-time carbon intensity data, real-time electricity consumption data, and / or real-time computing power occupancy data; historical monitoring values ​​can be: historical carbon intensity data, historical electricity consumption data, and / or historical computing power occupancy data.

[0063] The target model consumes a significant amount of power during training. This power consumption may come from the operation of computing resources, such as GPUs, or from the operation of configured components, such as cooling components. To address this, energy consumption data of the target model during training can be collected and converted into monitoring data. Furthermore, the frequency of data integration can be set, a fixed time window can be established, and the data acquired within that time window can be integrated.

[0064] During implementation, the server acquires monitoring data of the target model for preset monitoring indicators during the training process. In this application, the monitoring data can be data related to the resource consumption of the target model during training, or data related to the resource consumption of the target model during training. The monitoring data can be real-time monitoring data and / or historical detection data of the target model; wherein, the monitoring data includes first time-series data and / or second time-series data, the first time-series data is determined based on the real-time monitoring value of the preset monitoring indicator within the current time window, and the second time-series data is the historical monitoring value of the preset monitoring indicator.

[0065] In addition, the monitoring data can also be current environmental data, which may include at least one of meteorological data, power grid operation information, and macroeconomic control information. Correspondingly, the monitoring data can be feature data after feature transformation of the above data, or it can be multidimensional feature data. Optionally, the monitoring data includes at least one of second time series data, first time series data, environmental data of the current time window, and time features corresponding to the current time window.

[0066] During execution, the current time window can be the time window from the current moment to the past x minutes. Accordingly, the monitoring data can include at least one of the following: the monitoring values ​​corresponding to the preset monitoring indicators within the past x minutes, the environmental data within the past x minutes, and the time characteristics of the past x minutes / current moment. The first time series data can include the monitoring values ​​corresponding to the preset monitoring indicators within the past x minutes. It should be noted that the second time series data can be the monitoring values ​​corresponding to the preset monitoring indicators within a longer past time window, such as carbon intensity data, electricity consumption data, and / or computing power consumption data over the past x months.

[0067] Step 202: Input the monitoring data into the prediction model to obtain the third time series data.

[0068] During implementation, the server inputs monitoring data into the prediction model, which then uses this model to predict data for preset monitoring indicators, obtaining third-time-series data within the future target time window. This third-time-series data represents the predicted value of the preset monitoring indicators within the future target time window, output by the prediction model. Furthermore, the prediction model can also output the confidence level of the third-time-series data obtained from this prediction.

[0069] During execution, the prediction model may include a multi-layer network structure. The prediction model may first predict the third time series data based on the monitoring data, and then determine the confidence level of this prediction based on the monitoring data, that is: determine the confidence index of the third time series data, and output the third time series data and the confidence index together; optionally, the third time series data includes the third time series data and the confidence index within the future target time window.

[0070] It should be noted that the prediction model module supports flexible model replacement. Optional models include Long Short-Term Memory Networks, LightGBM, Transformer architecture, and any time series prediction model that can output point predictions and uncertainty intervals.

[0071] Step 203: Perform weighted fusion processing on the first time series data and the third time series data to obtain the weighted values ​​of the preset monitoring indicators in the future target time window.

[0072] During implementation, the server performs weighted fusion processing on the first and third time-series data in the monitoring data to obtain the weighted values ​​of the preset monitoring indicators in the future target time window.

[0073] During execution, the weights of the first and third time series data can be determined based on their data reliability. Different confidence levels can be set, and the weights of the first and third time series data can be determined based on their data reliability and confidence levels. Alternatively, the weights of the first and third time series data can be determined based on the amount of data in the first time series data, and the sum of the two weights can be 1.

[0074] For example, if the first time series data is not empty, the weight of the third time series data is determined based on the confidence level of the third time series data, and the weight of the first time series data is obtained by subtracting the weight of the third time series data from 1; if the first time series data is empty, the weight of the first time series data is 0, and correspondingly, the weight of the third time series data is 1, so the third time series data can be determined as the weighted value, and training resource scheduling can be performed independently based on the third time series data; if the confidence level of the third time series data is lower than the threshold, the weight of the third time series data is set to 0, and the weight of the first time series data is set to 1, so the first time series data can be determined as the weighted value.

[0075] Step 204: Based on the weighted values ​​of preset monitoring indicators within the future target time window, schedule the training resources of the target model within the future target time window.

[0076] During implementation, the server determines the training resources that match the weighted values ​​based on the weighted values, and adjusts the current training resources based on the matched training resources to complete the scheduling of training resources.

[0077] During execution, the server can compare the weighted value with the preset monitoring indicator threshold. If the weighted value is greater than or equal to the preset monitoring indicator threshold, it indicates that the indicator has exceeded the standard, and the training resources of the target model can be reduced, i.e., shrinking. If the weighted value is less than the preset monitoring indicator threshold, it indicates that the indicator has not exceeded the standard or the indicator is stable, and the training resources of the target model can be maintained or increased, i.e., maintaining or expanding.

[0078] In this application, the training resources of the target model can be GPU resources used for model training. Reducing the training resources of the target model can be done by reducing the number of GPUs used for model training, i.e., shrinking the scale. Increasing the training resources of the target model can be done by increasing the number of GPUs used for model training, i.e., expanding the scale.

[0079] For example, the initial computing resources for a training task are set to 32 GPU units. When the system detects a period of high resource consumption, it automatically and gradually reduces the computing resources to 4 GPUs to reduce resource consumption. When the system enters a period of low resource consumption, it restores the resources to the initial configuration of 32 GPUs to ensure training efficiency. This achieves a 35% reduction in overall resource consumption for the training task while maintaining model training accuracy and convergence characteristics, and no abnormal training loss or model performance degradation occurs during the entire scaling process.

[0080] In the above-mentioned method for scheduling training resources, during the training of the target model, monitoring data is first acquired. The monitoring data includes first time-series data determined based on the real-time monitoring values ​​of preset monitoring indicators within the current time window, and / or second time-series data corresponding to the historical monitoring values ​​of the preset monitoring indicators. The monitoring data is input into the prediction model to obtain third time-series data for the future target time window. The third time-series data is the predicted value of the preset monitoring indicators output by the prediction model within the future target time window, thus realizing data prediction for the future time window. The first and third time-series data in the monitoring data are weighted and fused to obtain a weighted value. Based on the weighted value of the preset monitoring indicators within the future target time window, the training resources of the target model are scheduled within the future target time window. The weighted value of the preset monitoring indicators within the future target time window is used as the basis for scheduling training resources. By fusing the current first time-series data and the future third time-series data, a reliable weighted value is obtained, and then the training resources are adjusted, thereby improving the optimization rate of the monitoring indicators corresponding to the resource consumption of the target model during the training process.

[0081] Based on the above exemplary embodiment, the following provides a method for scheduling model training resources in one or more exemplary embodiments, which is applied to... Figure 1 Taking the server in the example, the following content will be used for explanation.

[0082] In the process of acquiring monitoring data, the monitoring values ​​of preset monitoring indicators in multiple time windows can be acquired first, and feature processing can be performed on each monitoring value to obtain monitoring data. For the monitoring values, they can be acquired in layers, which can be divided into a core necessary data layer (L1 layer) and an auxiliary enhancement data layer (L2 layer). The L1 layer may include: first time-series data (feature importance approximately 45%), second time-series data (feature importance approximately 35%), and time features (feature importance approximately 10%). When L1 layer data is missing, a degradation strategy is triggered, and the system cannot be scheduled normally. The L2 layer data may include meteorological data (feature importance approximately 6%), power grid operation information (feature importance approximately 3%), and macro-schedule factors (feature importance approximately 1%). When L2 layer data is missing, default values ​​can be used to fill the gaps, and the prediction confidence level is correspondingly lowered. Here, the confidence level can be the confidence level of the third time-series data. In one optional implementation provided in this application, such as… Figure 3 As shown, step 201 includes at least one of steps 301 to 303:

[0083] Step 301: Obtain candidate environment data for the current time window, perform sampling and normalization processing on the candidate environment data to obtain the environment data.

[0084] During implementation, the server can acquire candidate environmental data / raw environmental data for the current time window through sensors, meteorological services, or external applications. It can then perform sampling and normalization processing on the candidate environmental data / raw environmental data to obtain environmental data. This environmental data can be used to compose monitoring data, which includes meteorological data and / or power grid structure data.

[0085] During the sampling process, environmental data may be updated at a low frequency, such as hourly or daily. To ensure that all types of data have the same time granularity, linear interpolation can be used to upsample to the same granularity as the current time window, which can be 5 minutes. For example, meteorological data and power grid structure data can be upsampled to a 5-minute granularity using linear interpolation. For holiday and policy indicator data, a broadcast filling method can be used to copy the daily scalar value to all 5-minute time slots of the day.

[0086] During the normalization process, a sliding window dynamic normalization strategy can be used, rather than static statistics based on the entire historical data. See the formula for details. Perform; where μ is the current mean and σ is the standard deviation.

[0087] For example, the system can default to using historical data from the past 30 days as a sliding window to calculate the current mean μ(t) and standard deviation σ(t) in real time, with the normalized statistics automatically refreshed every 24 hours. If the first time series data deviates from the current μ(t) by more than 3σ(t) for 3 consecutive hours, an immediate update of the statistics is triggered. When the model performs incremental parameter updates, the normalized statistics are refreshed synchronously to ensure that the model input distribution remains consistent with the training distribution. If there is less than 30 days of historical data, the system calculates the statistics based on all available historical data, while downgrading the confidence flag and biasing κ(t) towards the real-time data.

[0088] It should be noted that the other normalization processes in this application can also be carried out with reference to the above normalization process. In addition, other existing normalization methods can also be used, and no limitation is made here.

[0089] Step 302: Obtain the real-time monitoring value of the current time window, perform aggregation and normalization processing on the real-time monitoring value to obtain the first time series data.

[0090] During implementation, the server can acquire and record the real-time monitoring values ​​of preset monitoring indicators in the current time window through real-time acquisition. The real-time monitoring values ​​can include monitoring data from multiple moments in the current time window. The real-time monitoring values ​​are then aggregated and normalized to obtain the first time series data.

[0091] During execution, noise interference can be suppressed by aggregating data to the granularity of the current time window using a moving average. Thus, the highest acquisition frequency for the first time series data is on the minute level, and scheduling execution and state synchronization take approximately 2 to 3 minutes. With 5 minutes as the feasible minimum granularity, scaling operations take an average of 1 to 2 minutes. The 5-minute window ensures sufficient execution margin, enabling more robust training resource scheduling operations.

[0092] Step 303: Encode the current time window to obtain time features.

[0093] During implementation, the server performs sine / cosine time encoding on the current time window based on the time scale to obtain the time characteristics corresponding to the current time window.

[0094] During execution, please refer to the formula. The system performs sine / cosine dual-channel periodic encoding on four time scales: hour, weekday, month, and days in a year. The hourly scale uses a 24-period cycle to capture intraday peak and valley data changes; the weekday scale uses a 7-period cycle to capture load differences between weekdays and weekends; the monthly scale uses a 12-period cycle to capture seasonal differences in renewable energy output; and the days in a year scale uses a 365-period cycle to capture continuous trends during transitional periods across months.

[0095] In addition, for non-periodic features such as holidays and policy interventions, the system can use one-hot encoding, appending a 0 / 1 binary vector to the feature matrix. The holiday encoding dimension is automatically generated based on the statutory holiday calendar of the target region, and supports the holiday systems of China, the United States, and the European Union by default.

[0096] In addition, other encoding methods in the existing technology can also be used to encode the current time window to obtain time features, and no limitation is made here.

[0097] One optional implementation method provided in this application obtains monitoring data corresponding to preset monitoring indicators by acquiring and processing various types of monitoring data, thereby achieving the unification of various types of monitoring data on the time scale and improving the reliability of the third time series data obtained in subsequent predictions.

[0098] After obtaining the monitoring data, data prediction processing for future target time windows can be performed. Prediction processing can be performed based on the first time series data using a pre-trained prediction model. The prediction model in this application can output third time series data in the form of binary or triplet sets. For binary sets, the third time series data may include the third time series data and its confidence index. The point prediction value sequence represents the estimated data values ​​for the next H 5-minute steps, with a default H=48, corresponding to the next 4 hours, and supports adjustments from 6 steps (30 minutes) to 288 steps (24 hours). The κ(t) coefficient can be automatically adjusted according to the time window to reduce dependence on long-term prediction results. The confidence index is a scalar value within the range [0,1], calculated jointly by the interval width and historical prediction error. For triplet sets, the third time series data may include the point prediction value sequence, the uncertainty interval, and the confidence index. The uncertainty interval is based on quantile regression, outputting the lower bound of P10 and the upper bound of P90, forming an 80% confidence interval. In one optional implementation provided by this application, such as... Figure 4 As shown, it also includes steps 401 to 402:

[0099] Step 401: Determine the confidence index of the third time series data by using the prediction model based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window.

[0100] During implementation, the prediction model can determine the confidence index of the third time series data based on the interval width of the uncertainty interval of the target model in the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window.

[0101] During execution, the uncertainty interval can be based on the P10 lower bound and P90 upper bound sequence output by quantile regression to form an 80% confidence interval; the confidence index is a scalar value in the range of [0,1], which is calculated by combining the interval width and historical prediction error.

[0102] During the confidence index calculation, the prediction model can determine the confidence index based on the formula: c(t) = 0.6·c_interval(t) + 0.4·c_history(t). Here, c_interval(t) is calculated based on the current prediction interval width, expressed as 1 minus the interval width to the historical maximum interval width CI_max. c_history(t) is calculated based on the past 24-hour rolling average absolute percentage error MAPE_rolling(t), expressed as max(0, 1-MAPE_rolling(t) / 0.15), with a default threshold of 15%.

[0103] Furthermore, the specific form of setting the weights and quantile parameters in the quantile loss function during the training process of the target model is as follows: Furthermore, the training loss function adopts a weighted combination of mean squared error and quantile loss, expressed as L_total = 0.6·L_MSE + 0.4·L_Quantile; the quantile loss term is calculated by taking the average of the three quantile parameters τ∈{0.1,0.5,0.9}, expressed as follows: Wherein, ρ_τ(u) is a piecewise function: when u≥0, it takes u·τ, and when u<0, it takes u·(τ-1). The physical meanings of the three quantiles are as follows: τ=0.1 corresponds to the lower bound of the prediction, representing an optimistic estimate; τ=0.5 corresponds to the median, which is used as the principal prediction value; τ=0.9 corresponds to the upper bound of the prediction, representing a pessimistic estimate. The weight ratio of the mean squared error to the quantile loss in the loss function, 0.6:0.4, is determined by performing a grid search on the validation set, with a search step size set to 0.1.

[0104] In addition, the target model can also have an incremental learning mechanism. The triggering conditions for this mechanism include the following three conditions, and an update is triggered if any one of them is met: Condition A is error triggering, i.e., the rolling average absolute percentage error over the past 12 hours continuously exceeds 12% for more than 3 hours. Condition B is time triggering, i.e., more than 7 days have passed since the last incremental update. Condition C is distribution drift triggering, i.e., the mean of the latest 1000 samples deviates from the historical mean by more than 2.5 times the standard deviation. To prevent catastrophic forgetting, the incremental update only fine-tunes the last two layers of the model, i.e., the output layer and the penultimate hidden layer, while the parameters of the previous layers remain frozen. The learning rate is set to one-tenth of the original training learning rate. Simultaneously, a random 5% subset of data from the past 30 days is mixed in during the update process to maintain the model's ability to remember historical distributions. For service continuity assurance, the incremental update is executed asynchronously in the background, without interrupting the main prediction service. After the update is completed, it is deployed in Shadow mode, i.e., the average absolute percentage error of the new and old models is compared on the validation set, and the switch is only completed when the new model performs better.

[0105] Step 402: Determine the weights for weighted fusion based on the confidence index.

[0106] During implementation, the server determines the index range corresponding to the confidence index based on the confidence index and selects the weight value corresponding to the index range.

[0107] One optional implementation method provided in this application determines the confidence index through a prediction model, which ensures the reliability and accuracy of the prediction data, thereby improving the reliability of training resource scheduling.

[0108] During the use of the prediction model, the server inputs at least one-dimensional feature data from the monitoring data into the prediction model. That is, the server can input the first time series data from the monitoring data into the prediction model, or it can input the second time series data into the prediction model, or it can input the first time series data and the second time series data together into the prediction model. The prediction model makes predictions based on at least one-dimensional monitoring data to obtain the third time series data for the future target time window.

[0109] In this application, the monitoring data can be composed of multi-dimensional data spliced ​​together, and can be divided into multiple feature groups. The monitoring data may include the first group of second time series data, which contains the monitoring time series values ​​of the past 72 hours, corresponding to 864 5-minute steps, with a dimension of 864; the second group of time-coded features, including 2-dimensional sine / cosine codes for hours, days of the week, months, and days of the year, as well as 1-dimensional unique hot codes for holidays, for a total of 9 dimensions; the third group of meteorological features, including temperature, wind speed, solar radiation intensity, and precipitation probability, with each field being Z-score normalized, and a dimension of 4; and the fourth group of power grid structure features, including power generation structure proportion vectors such as the proportion of renewable energy, thermal power, and nuclear power, with a default dimension of 5. When L2 layer data is missing, the corresponding feature dimensions are filled with zero vectors, and a one-bit data availability mask vector is appended to the end of the input tensor for the model to distinguish and process in attention or gating mechanisms.

[0110] Furthermore, during the prediction process, the third time series data can be further adjusted based on the current prediction confidence level to ensure the reliability of the prediction data; in one optional implementation provided by this application, such as Figure 5 As shown, step 202 includes step 501:

[0111] Step 501: If the confidence level downgrade condition is met, input the monitoring data into the basic prediction model for prediction to obtain the third time series data.

[0112] In this application, since the monitoring values ​​of the preset monitoring indicators may contain errors, such as the possibility of obtaining empty data, which would lead to insufficient prediction accuracy, a confidence reduction condition can be set. When the confidence reduction condition is met, data prediction is performed using the basic prediction model to obtain the third time series data. The confidence reduction condition may include the monitoring data including the second time series data having a data volume threshold of less than a certain threshold.

[0113] During implementation, when the confidence level downgrade condition is met—specifically, when the amount of data in the second time series is less than a data volume threshold—the prediction model inputs the monitoring data into the basic prediction model for prediction, resulting in the third time series data. The basic prediction model is a sub-model within the prediction model. During execution, when historical data accumulation is less than 30 days, the system activates transfer learning, loads the regionally general basic model, lowers the prediction confidence by 15%, and adjusts κ(t) to increase.

[0114] One optional implementation provided in this application is to load a regional general basic model through transfer learning when the amount of data in the second time series is insufficient. This avoids the problem of large prediction errors caused by incomplete data and improves the reliability of the third time series data.

[0115] In practical scenarios, if the confidence reduction condition is met, the confidence index of the third time series data obtained from this prediction can also be reduced. Optionally, the confidence reduction condition includes at least one of the following: the first time series data included in the monitoring data is empty; the amount of data in the second time series data included in the monitoring data is less than the data amount threshold; meteorological data is missing in the environmental data included in the monitoring data; and power grid structure data is missing in the environmental data included in the monitoring data. In one optional implementation provided by this application, such as... Figure 6 As shown, step 401 includes steps 601 to 602:

[0116] Step 601: Determine the candidate confidence index of the third time series data based on the interval width, the historical maximum interval width, and the mean absolute percentage error using the prediction model.

[0117] During implementation, the prediction model first determines the candidate confidence index of the third time series data based on the interval width, the historical maximum interval width, and the mean absolute percentage error. The specific determination method can be referred to the above embodiment, and will not be repeated here.

[0118] Step 602: Determine the preset coefficients according to the type of confidence downgrade conditions, and reduce the candidate confidence index based on the preset coefficients to obtain the confidence index.

[0119] During implementation, the preset coefficient for this reduction is determined according to the type of confidence reduction condition corresponding to this prediction. Based on the preset coefficient, the candidate confidence index is reduced to obtain the confidence index.

[0120] During execution, when the first time-series data is interrupted, the system completely replaces it with the predicted value. At this time, κ approaches zero, the prediction confidence is reduced by 20%, and the adjustment direction of κ(t) is decreasing. When the historical data accumulation is less than 30 days, the system enables transfer learning, loads the regional general basic model, reduces the prediction confidence by 15%, and the adjustment direction of κ(t) is increasing. When meteorological data is missing, the system uses the historical average for the same period to fill in the gaps and marks the L2 degradation flag, reducing the prediction confidence by 5%, and the adjustment direction of κ(t) is slightly increasing. When power grid structure data is missing, the system uses the regional fixed default energy structure ratio as a substitute, reduces the prediction confidence by 3%, and keeps the direction of κ(t) unchanged. When macro-dispatch factors are missing, the system ignores this feature dimension, does not trigger degradation, and the prediction confidence and κ(t) remain unchanged.

[0121] Furthermore, if both the first and second time-series data in the L1 layer are missing, the system enters a fully conservative mode: all training resource scheduling operations are stopped, the current training scale remains unchanged, and an alarm notification is sent to the operation and maintenance platform.

[0122] One optional implementation provided in this application avoids the data processing flow from being affected by poor-performing data by setting confidence degradation conditions, and at the same time avoids the prediction results from being affected by poor-performing data, thus ensuring the smooth data processing flow while ensuring the reliability of the predicted third-time series data.

[0123] In addition, the prediction model can also directly make predictions based on the monitoring data obtained above to obtain third time-series data; another optional implementation provided in this application is as follows: Figure 7 As shown, step 202 includes step 701:

[0124] Step 701: Input the first time series data, environmental data, time features, and the second time series data into the prediction model to obtain the third time series data.

[0125] During implementation, the server extracts the first time series data, environmental data, time features, and second time series data from the monitoring data, and inputs the first time series data, environmental data, time features, and second time series data into the prediction model to obtain the third time series data.

[0126] During execution, step 701 can also be performed if the confidence degradation condition is not met. Step 701 can be replaced by: if the confidence degradation condition is not met, inputting the first time series data, environmental data, time features, and second time series data into the prediction model to obtain the third time series data. Furthermore, the monitoring data in step 501 can also include the first time series data, environmental data, time features, and second time series data.

[0127] In one optional implementation provided by this application, if the confidence degradation condition is not met, the third time series data can be predicted by combining the first time series data, environmental data, time features, and the second time series data, thereby improving the reliability of the third time series data.

[0128] In the weighted fusion process, the weights of the first time series data and the third time series data can be determined through the confidence index obtained above. The confidence index represents the reliability of the third time series data, thereby ensuring the rationality of the weight allocation. The third time series data may include the confidence index. In one optional implementation provided by this application, such as... Figure 8 As shown, steps 402 and 203 include steps 801 to 803:

[0129] Step 801: Determine the first weight of the first time series data and the second weight of the third time series data based on the confidence index included in the third time series data.

[0130] During implementation, the server determines the second weight of the third time series data based on the confidence index, and obtains the first weight by subtracting the second weight from 1; or, the server determines the first weight of the first time series data based on the confidence index, and obtains the second weight by subtracting the first weight from 1.

[0131] During execution, the fusion weight κ(t) is analytically calculated based on the prediction confidence index c(t) using the inverse Sigmoid function, ensuring that κ changes smoothly within the interval [κ_min, κ_max]. The calculation formula is as follows: κ(t) = κ_max - (κ_max - κ_min) · σ(β · (c(t) - 0.5)); where σ(x) = 1 / (1 + exp(-x)) is the standard Sigmoid function. The default parameter values ​​are: κ_min = 0.05, corresponding to the lower bound when the prediction is highly reliable; κ_max = 0.95, corresponding to the upper bound when the prediction is completely unreliable; β = 8, used to control the steepness of the transition curve. The typical correspondence between the confidence index c(t) and the fusion weight κ(t) is as follows: when c(t) > 0.90, κ ≈ 0.05~0.10, the system is highly dependent on the predicted value, and resources can be pre-adjusted 2 to 4 hours in advance. When 0.60 ≤ c(t) ≤ 0.90, κ ≈ 0.10~0.50, and the system adopts a normal scheduling mode that integrates prediction and real-time balanced operation. When 0.30 ≤ c(t) < 0.60, κ ≈ 0.50~0.80, the system is biased towards real-time data, and the predicted value is only used as a reference. When c(t) < 0.30, κ ≈ 0.80~0.95, the system is almost completely dependent on real-time data, and the prediction service is degraded. The prediction error is calculated based on the rolling average absolute percentage error within the past 24-hour sliding window, rather than the single-step absolute error, to avoid occasional outliers from disturbing κ.

[0132] Step 802: Calculate the first product of the first weight and the first time series data, and the second product of the second weight and the third time series data.

[0133] During implementation, the server calculates the first product of the first weight and the first time series data to obtain the fusion weight of the first time series data, and calculates the second product of the second weight and the third time series data to obtain the fusion weight of the third time series data.

[0134] During execution, the formula can be used. It is determined that the weighting coefficient κ(t) is dynamically adjusted according to the prediction error: when the prediction reliability is high, κ takes a smaller value, and the scheduling decision depends more on the predicted value; when the prediction is unstable, κ takes a larger value, and the scheduling decision depends more on the real-time data.

[0135] For example, when the prediction error is less than 5%, κ can be 0.1, and the system will prioritize using the predicted value for advance scheduling; when the prediction error is between 5% and 15%, κ can be 0.5 to achieve a balance between prediction and real-time data; when the prediction error exceeds 15%, κ can be 0.9, and the system will degrade to a scheduling mode that prioritizes real-time data.

[0136] Step 803: Calculate the sum of the first product and the second product to obtain the weighted value.

[0137] During implementation, the first and second products are summed, and the sum is used as the weighted value.

[0138] One optional implementation provided in this application is to output a weighted value by weighting and fusing first time-series data and third time-series data, thereby achieving forward-looking resource planning. By combining real-time and prediction methods, the reliability of the weighted value is improved, thus ensuring the effectiveness of training resource scheduling.

[0139] In the process of scheduling training resources, the number of training units best suited to the weighted value of a preset monitoring indicator within a future target time window can be determined. Based on the number of training units, resource expansion or contraction can be performed to achieve training resource scheduling. In one optional implementation provided in this application, such as... Figure 9 As shown, step 204 includes steps 901 to 902:

[0140] Step 901: Determine the target number of training units for the target model within the future target time window based on the weighted values.

[0141] In real-world scenarios, the more training units a target model has during training, the faster the training speed usually is, but the more resources it consumes. To address this, a weighted value can be used as the upper limit of resource consumption corresponding to a preset monitoring indicator, thereby limiting the number of training units in the target model.

[0142] During implementation, the server performs data conversion based on weighted values ​​to obtain the target number of training units for the target model in the future target time window.

[0143] Step 902: Based on the target quantity, perform resource expansion or resource reduction processing on the number of training units of the target model within the future target time window.

[0144] During implementation, the number of training units of the target model in the future target time window is expanded or reduced based on the target number, so that the number of training units of the target model in the future target time window is less than the target number of training units of the target model in the future target time window.

[0145] During the scaling-up process, the server can increase the number of training units of the target model over time, increasing the number of training units in the future target time window to the target number; during the scaling-down process, the server can decrease the number of training units of the target model over time, reducing the number of training units in the future target time window to the target number.

[0146] During execution, generally speaking, when the weighting value is at a low level, the system performs a scaling-up operation, gradually increasing the number of training computing units, for example, expanding from 1 GPU to 8 or 64 GPUs; when the weighting value rises to a high level, the system performs a scaling-down operation, gradually reducing the number of computing units, for example, reducing from 64 GPUs to 8 or 1 GPU, or simply maintaining a keep-alive state to sustain the basic operation of the training task. Specific thresholds can be determined by the user through interaction.

[0147] For example, the quantization thresholds for "lower level" and "higher level" are defined using a segmented interval format: when the weighted value CI_eff(t) is less than the first threshold L_low, an expansion operation is triggered; when CI_eff(t) is between the first threshold L_low and the second threshold L_high (inclusive), the current resource size remains unchanged; when CI_eff(t) is greater than the second threshold L_high, a shrinkage operation is triggered, and L_low can be set to... L_high= It should be noted that L_low and L_high are both user-configurable parameters and can be dynamically adjusted through a graphical user interface or application programming interface (API).

[0148] In one optional implementation provided by this application, a weighted value is used as a constraint to control the number of training units of the target model, which effectively ensures the resource consumption of the target model during the training process, while ensuring the reliability of the scheduling of the training resources of the target model.

[0149] In determining the target number of training units for the target model, a conversion relationship between the weighted values ​​of preset monitoring indicators over a future target time period and the number of training units for the target model can be constructed as a deterministic function. The target number is obtained by solving the deterministic function. In one optional implementation provided in this application, such as... Figure 10 As shown, step 901 includes steps 1001 to 1002:

[0150] Step 1001: Determine the function for determining the number of training units of the target model based on the weighting value and the training consumption parameters of the training units of the target model.

[0151] During implementation, the server can construct a function to determine the number of training units for the target model based on the weighted values ​​and the training consumption parameters of the training units of the target model. The training consumption parameters may include the efficiency of the training units of the target model and / or the resource consumption parameters of the training units of the target model, where the resource consumption parameters may be the power consumption per training unit during operation.

[0152] During execution, the server can also construct a deterministic function based on weights. This function determines the number of training units in the target model based on the weighted values, the corresponding weights, the efficiency of the target model's training units, the weights corresponding to the efficiency of the target model's training units, and the resource consumption parameters of the current time window, along with their corresponding weights. This deterministic function can be obtained by combining three sub-functions: carbon emissions, training duration, and energy cost.

[0153] Step 1002: Minimize the determination function based on the number of remaining training units of the target model to obtain the target number of training units for the future target time window.

[0154] During implementation, the server uses the number of remaining training units of the target model as a variable to solve for the minimum value of a deterministic function, thereby obtaining the target number of training units for the future target time window.

[0155] During the solution process, the solution can be based on the number of remaining training units available in the current target model. If the number of remaining training units is small, for example, less than 128, exhaustive search is used, that is, 0-128 is substituted into the objective function for calculation to determine the number of training units corresponding to the minimum value. If the number of remaining training units is large, for example, greater than 128, greedy search is used. After the solution is completed, scheduling is performed based on the obtained optimal number of training units.

[0156] One optional implementation provided in this application sets an objective function and solves for its minimum value, using the weighted values ​​of preset monitoring indicators within the future target window as constraints. This ensures the training of the target model while improving the processing efficiency of monitoring indicators during the training process.

[0157] In determining the objective function, it can be obtained by setting at least one sub-function, or by combining two or more sub-functions. The sub-functions can characterize carbon emissions, training time, and energy costs. One optional implementation provided in this application is as follows: Figure 11 As shown, step 1001 includes steps 1101 to 1102:

[0158] Step 1101: Determine at least one sub-determining function based on the weighted values ​​and / or the training consumption parameters of the training units of the target model.

[0159] During implementation, the server can determine at least one sub-determining function based on the weighted value, or based on the training consumption parameters of the training unit of the target model, or based on the weighted value and the training consumption parameters of the training unit of the target model.

[0160] The training consumption parameters of the training units of the target model can be unit consumption parameters, such as the electricity consumed by a training unit per unit time, the computing power used by a training unit per unit time, or the carbon emissions emitted by a training unit per unit time. Alternatively, the training consumption parameters of the training units of the target model can be the training consumption parameters of all training units of the target model.

[0161] Step 1102: Perform weighted calculation based on the preset weights corresponding to at least one sub-determined function to obtain the determined function.

[0162] During implementation, the server can perform weighted calculations based on preset weights corresponding to at least one sub-determined function to combine at least one sub-determined function and obtain a determined function.

[0163] During execution, the server can perform a weighted summation of each sub-function based on its corresponding weight to obtain a final function. The weights of each sub-function can be set by the user through interactive settings.

[0164] During the execution process, you can refer to the formula. The objective function is obtained by adding the three sub-functions mentioned above, where the decision variable N represents the number of GPUs, and its hard constraints are as follows: The lower bound constraint is N ≥ N_min, with a default N_min = 1, corresponding to a single-GPU keep-alive mode. The upper bound constraint is N ≤ N_max, determined by the available GPU capacity reported in real-time by the scheduling system. The step size constraint is |ΔN| ≤ 50% × current N, to avoid system oscillation caused by excessively large single scheduling increments. The cooling constraint is the minimum interval T_cool = 15 minutes between two consecutive scheduling operations, used to prevent frequent scaling up and down. The solution method is selected based on the upper bound. When N_max ≤ 128, an exhaustive search of the integer field is used; when N_max > 128, a heuristic greedy search is used.

[0165] One optional implementation provided in this application obtains a determining function by determining multiple sub-determining functions and weighting at least one sub-determining function, thereby ensuring the reliability of the objective function and thus ensuring the effectiveness of the calculated number of training units.

[0166] In determining the sub-determining functions, three sub-determining functions can be preset, and at least one sub-determining function can be determined based on the weighting value and / or the training consumption parameters of the training units of the target model. The sub-determining functions may include at least one of a first sub-function, a second sub-function, and a third sub-function; in one optional implementation provided by this application, such as Figure 12 As shown, step 1101 includes at least one of steps 1201 to 1203:

[0167] Step 1201: Determine the first sub-function based on the weighting value and the unit energy consumption of the training unit.

[0168] During implementation, the server constructs a first sub-function based on the product of the unit energy consumption of the training unit, the number of training units, the weighting value, and time, representing the carbon emissions corresponding to the number of training units. The first sub-function can also be used to calculate the carbon emissions during the training process of the target model.

[0169] During execution, it can be expressed by the formula C_carbon(N)=CI_eff(t)·P_gpu·N·Δt, where CI_eff(t) is the effective carbon intensity, and the unit is Δt. P_gpu represents the average power consumption of a single GPU, typically ranging from 0.3 to 0.5 kW, and Δt is a value of 1 / 12 of an hour. The value of the first sub-function is proportional to the number of training units.

[0170] Step 1202: Determine the second sub-function.

[0171] During implementation, the server constructs a second sub-function based on a preset efficiency factor and the number of training units; the value of the second sub-function is inversely proportional to the number of training units. The second sub-function can also be used to calculate the training time during the target model's training process.

[0172] During execution, the second sub-function can be represented by the formula C_time(N)=T_total / (N·η(N)), where η(N)=1 / (s+(1-s) / N+γ·log(N)) is the Amdahl parallel efficiency factor, the serial ratio s is 0.05, and the communication overhead coefficient γ is 0.02.

[0173] Step 1203: Determine the third sub-function based on the resource consumption parameters and the unit energy consumption of the training unit.

[0174] During implementation, the server constructs a third sub-function based on the product of resource consumption parameters, the unit energy consumption of training units, and the number of training units; the value of the third sub-function is proportional to the number of training units. The third sub-function can also be used to calculate the power consumption during the training process of the target model.

[0175] During execution, the third sub-function can be represented by the formula C_energy(N)=EP(t)·P_gpu·N·Δt, where EP(t) is the electricity price for the current period. When the data is unavailable, a fixed default value is used instead.

[0176] One optional implementation provided in this application ensures the reliability of the objective function by setting sub-functions for different scenarios, thereby ensuring the effectiveness of the calculated number of training units.

[0177] In real-world scenarios, the target model can be backed up during training. To address this, in one optional implementation provided by this application, as shown in Figure 13(a), the method further includes step 1301:

[0178] Step 1301: If the training backup conditions are met, save the training backup data of the target model.

[0179] During implementation, the server can save the training backup data of the target model when it detects that the training backup conditions are met. Alternatively, the training device of the target model can also save the training backup data of the target model when it detects that the training backup conditions are met. These training backup conditions include the expiration of a preset backup time or the scheduling quantity of training resources for the target model exceeding a scheduling quantity threshold.

[0180] During execution, the trigger frequency can be once every 5 minutes or after a fixed number of training steps. The saved content is the increment Δθ of the model parameters relative to the previous standard snapshot; the optimizer state and gradient accumulation state are not saved. A backup stored every 5 minutes can be called a lightweight snapshot, characterized by its minimal size and shortest write time. A backup stored every 30 minutes or forcibly triggered before each scaling operation can be used as the trigger condition. The saved content includes: the model parameter increment Δθ (relative to the previous complete snapshot), the complete optimizer state (for example, the Adam optimizer includes the first moment m, the second moment v, and the time step t), and the gradient accumulation state. A backup stored every 30 minutes can be called a standard snapshot, characterized by its moderate size, making it the preferred snapshot type for regular recovery operations. A backup stored every 4 hours or forcibly triggered before significant scaling (ΔN>50%) can be used as the trigger condition. The stored content includes: complete model parameters θ (non-incremental form), complete optimizer state, gradient accumulation state, current learning rate scheduler state, and random number generator state (to ensure reproducible randomness). A snapshot taken every 4 hours can be called a complete snapshot. Its storage characteristics are the largest volume and the lowest write frequency.

[0181] For example, as shown in Figure 13(b), a lightweight snapshot only retains the parameter increment (Δθ), a standard snapshot retains the parameter values ​​and optimizer state, and a full snapshot retains all model parameters, optimizer state, gradient cache, random seed, and execution context.

[0182] It should be noted that the system prioritizes loading the latest standard snapshot; if the hash verification of the standard snapshot fails, it will automatically roll back to the most recent complete snapshot; if the complete snapshot is also corrupted, the fault tolerance mechanism will be triggered and an alarm will be sent to the operation and maintenance platform.

[0183] An optional implementation provided in this application also includes saving a snapshot of the target model to improve the stability and reliability of the target model's training and enhance the training security of the target model.

[0184] In practical scenarios, the training consistency of the target model can also be guaranteed. In one optional implementation provided in this application, as shown in Figure 14(a), the method further includes step 1401:

[0185] Step 1401: If a change in the number of training resources for the target model is detected, perform training consistency assurance processing for the target model.

[0186] During implementation, if the server or training device of the target model detects a change in the number of training resources for the target model, it will perform training consistency assurance processing. This training consistency assurance processing includes at least one of the following: global synchronization of parameter values, restoration of the optimizer's internal state, gradient normalization compensation, and warm start-up of the learning rate.

[0187] During execution, global synchronization of parameter values ​​is triggered after the scaling operation is completed and new nodes are ready, but before any forward propagation begins. The master node distributes the current complete parameters θ to all computing nodes via global broadcast. The communication backend supports MCCL / NCCL, Gloo, and MPI. Each node returns an acknowledgment signal after receiving the signal, and the master node summarizes these signals before continuing training to verify the synchronization is complete. Restoring the optimizer's internal state involves deserializing the optimizer state from a standard snapshot or a complete snapshot, including the first moment m, second moment v, and time step t (using Adam as an example). For newly added nodes, since they have no historical optimizer state, the current state is cloned from an existing node for initialization, avoiding the break in the optimization trajectory caused by initialization from zero. The optimizer state of exiting nodes is already saved in the most recent snapshot and requires no additional processing. Gradient normalization compensation refers to the fact that when the number of GPUs changes from N to N', the equivalent batch size changes. Without compensation, this implicitly changes the learning rate and affects convergence. The compensation method is to multiply the gradient of each node by the compensation coefficient N / N' to offset the impact of batch change on the gradient magnitude. This compensation is performed after parameter synchronization is complete and before gradient aggregation during the first backpropagation. The warm start strategy for the learning rate is automatically triggered after each scaling up or down operation. Within the first K steps after scaling up or down, the learning rate is linearly or cosinely increased from the initial value η_start to the target value η_target. The typical value of K ranges from 100 to 500 steps and can be configured by the user. It is recommended that η_start be 10%–20% of η_target. This strategy aims to avoid training loss oscillations caused by parameter synchronization errors or gradient compensation residuals, ensuring convergence stability. After warm start, it seamlessly integrates with the original scheduler (e.g., cosine annealing or linear decay) to continue execution.

[0188] For example, as shown in Figure 14(b), if the number of training resources changes after resource scheduling, the training consistency of the target model is guaranteed by saving training state snapshots, parameter synchronization, optimizer state recovery, gradient normalization compensation, and learning rate warm start. Training execution is resumed after the consistency guarantee is completed.

[0189] One optional implementation method provided in this application ensures the consistency of training of the target model during the training process, thereby improving the reliability and stability of the target model training.

[0190] In real-world scenarios, various unforeseen circumstances can arise during the training of the target model. One optional implementation method provided in this application is as follows: Figure 15 As shown, the method further includes at least one of steps 1501 to 1503:

[0191] Step 1501: If the prediction model fails, the first time series data is determined as the weighted value.

[0192] During implementation, if the server detects that the prediction model has failed, that is, when the confidence level of the third time series data output by the prediction model is lower than the confidence level threshold and cannot be used, the server can determine the weight of the third time series data to 0 during the weighted fusion processing of the first time series data and the third time series data, and the server will determine the first time series data as the weighted value. That is, the first time series data will be used as the weighted value of the preset monitoring indicator in the future target time window.

[0193] It should be noted that step 203 can be replaced by step 1501.

[0194] Step 1502: In the event of a network failure, maintain the training resources at the current moment.

[0195] During implementation, if the server detects a network failure, it schedules training resources for the target model in future target time windows based on weighted values, including maintaining the training resources at the current moment. If a network partitioning failure occurs during execution, the system will suspend all elastic scaling operations of computing resources to maintain stability.

[0196] It should be noted that step 204 can be replaced by step 1502.

[0197] Step 1503: In the case of identifying the scheduling of training resources in the future target time window, roll back to the training backup data at the target time.

[0198] During implementation, when the scheduling of training resources in the future target time window is identified, the server rolls back to the training backup data at the target time; where the target time is earlier than the earliest time in the future target time window.

[0199] During execution, if a corrupted training snapshot is detected, the system automatically switches to the previous available snapshot for recovery. When a GPU resource allocation failure occurs, the system performs a resource state rollback operation to the most recent successfully allocated state.

[0200] One or more optional implementation methods provided in this application supplement error / fault handling methods during target model training, ensuring the training stability of the target model and thus guaranteeing the training effect of the target model.

[0201] In one embodiment, see Figure 16 The document illustrates a flowchart of a model training resource scheduling method provided in an embodiment of this application. This model training resource scheduling method can be applied to... Figure 1 In the server shown. For example... Figure 16 As shown, the method for scheduling training resources for this model may include the following steps:

[0202] Step 1601: Obtain candidate environment data for the current time window, perform sampling and normalization processing on the candidate environment data to obtain the environment data.

[0203] Step 1602: Obtain the real-time monitoring value of the current time window, perform aggregation and normalization processing on the real-time monitoring value to obtain the first time series data.

[0204] Step 1603: Encode the current time window to obtain time features.

[0205] Step 1604: Obtain the second time series data and construct monitoring data based on environmental data, the second time series data, the first time series data, and time characteristics.

[0206] Step 1605: Input the monitoring data into the prediction model to obtain the third time series data.

[0207] Step 1606: Determine the confidence index of the third time series data by using the prediction model based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window.

[0208] Step 1607: Determine the first weight of the first time series data and the second weight of the third time series data based on the confidence index included in the third time series data.

[0209] Step 1608: Calculate the first product of the first weight and the first time series data, and the second product of the second weight and the third time series data.

[0210] Step 1609: Calculate the sum of the first product and the second product to obtain the weighted value of the preset monitoring indicator within the future target time window.

[0211] Step 1610: Determine the function for determining the number of training units of the target model based on the weighted value, the efficiency of the training units of the target model, the resource consumption parameters of the current time window, and their respective weights.

[0212] Step 1611: Minimize the determination function based on the number of remaining training units in the target model to obtain the target number of training units for the future target time window.

[0213] Step 1612: Based on the target quantity, perform resource expansion or resource reduction processing on the number of training units of the target model in the future target time window.

[0214] It should be noted that any one or more of steps 1601 to 1612 can be combined to form a new implementation method according to the needs of implementation and deployment. Furthermore, any one or more technical features in the technical solution composed of steps 1601 to 1612 can also be combined to form a new implementation method according to the actual deployment needs, or technical features in one or more optional implementation methods provided by one or more of the above embodiments can be combined to form a new implementation method. These will not be elaborated on here.

[0215] The present invention has produced at least the following beneficial technical effects:

[0216] (1) To realize the elastic scheduling of training resources based on the dynamic changes of the monitoring values ​​corresponding to the monitoring indicators, thereby reducing the resource consumption level of training tasks.

[0217] (2) Introduce a prediction-driven active scheduling mechanism, use prediction data and real-time data to output a weighted value, and realize forward-looking resource planning instead of passive response.

[0218] (3) Maintain consistency of training state and model convergence during resource adjustment, and ensure that model parameters, optimizer state and gradient are consistent during scaling up and down to avoid deterioration of training performance.

[0219] (4) Construct an adaptive fusion decision strategy, dynamically adjust the weights based on the prediction error, so that the scheduling strategy can adapt to the changes in prediction accuracy at different stages.

[0220] (5) Provide users with configurable optimization target weights, support users in configuring the weights of each sub-function, and make the scheduling behavior adjustable and adaptable.

[0221] (6) Provides a reliable fault tolerance mechanism and system recovery capability, which can ensure that training is not interrupted or recovers with minimal loss in abnormal situations.

[0222] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0223] Based on the same inventive concept, this application also provides a model training resource scheduling apparatus for implementing the model training resource scheduling method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more model training resource scheduling apparatus embodiments provided below can be found in the limitations of the model training resource scheduling method described above, and will not be repeated here.

[0224] In one exemplary embodiment, such as Figure 17 As shown, a model training resource scheduling device is provided, including: a monitoring data acquisition module 1701, a data prediction module 1702, a fusion module 1703, and a resource scheduling module 1704, wherein: the monitoring data acquisition module 1701 is used to acquire monitoring data of the target model during the training process; the data prediction module 1702 is used to input the monitoring data into the prediction model to obtain the weighted value of the preset monitoring index within the future target time window; the fusion module 1703 is used to perform weighted fusion processing on the first time series data and the third time series data to obtain the third time series data; and the resource scheduling module 1704 is used to schedule the training resources of the target model within the future target time window according to the weighted value of the preset monitoring index within the future target time window.

[0225] In one embodiment, the apparatus further includes a confidence prediction unit and a weight determination unit, wherein: the confidence prediction unit is used to determine the confidence index of the third time series data by using the prediction model based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window; the weight determination unit is used to determine the weights for weighted fusion based on the confidence index.

[0226] In one embodiment, the data prediction module 1702 includes a first prediction unit, which is used to input monitoring data into a basic prediction model for prediction to obtain third time series data when the confidence level downgrade condition is met; wherein the confidence level downgrade condition includes at least one of the following: the first time series data is empty, the amount of data in the second time series data is less than the data amount threshold, and the environmental data is missing.

[0227] In one embodiment, the confidence prediction unit includes a first confidence prediction unit and a second confidence prediction unit, wherein: the first confidence prediction unit is used to determine the candidate confidence index of the third time series data based on the interval width, the historical maximum interval width and the mean absolute percentage error through the prediction model; the second confidence prediction unit is used to determine a preset coefficient according to the type of confidence downgrade condition, and reduce the candidate confidence index based on the preset coefficient to obtain the confidence index.

[0228] In one embodiment, the fusion module 1703 includes a weight determination unit, a product determination unit, and a weighted summation unit, wherein: the weight determination unit is used to determine the first weight of the first time series data and the second weight of the third time series data according to the confidence index; the product determination unit is used to calculate the first product of the first weight and the first time series data, and the second product of the second weight and the third time series data; the weighted summation unit is used to calculate the sum of the first product and the second product to obtain a weighted value.

[0229] In one embodiment, the monitoring data acquisition module 1701 includes at least one of an environmental data acquisition unit, a first time-series data acquisition unit, and a time feature acquisition unit, wherein: the environmental data acquisition unit is used to acquire candidate environmental data of the current time window, perform sampling processing and normalization processing on the candidate environmental data to obtain environmental data; the first time-series data acquisition unit is used to acquire real-time monitoring values ​​of the current time window, perform aggregation processing and normalization processing on the real-time monitoring values ​​to obtain first time-series data; and the time feature acquisition unit is used to encode the current time window to obtain time features.

[0230] In one embodiment, the data prediction module 1702 is further configured to input the first time series data, environmental data, time features and the second time series data into the prediction model to obtain the third time series data.

[0231] In one embodiment, the resource scheduling module 1704 includes a training unit determination unit and a scheduling unit, wherein: the training unit determination unit is used to determine the target number of training units of the target model in the future target time window according to the weighting value; the scheduling unit is used to perform resource expansion or resource reduction processing on the number of training units of the target model in the future target time window based on the target number.

[0232] In one embodiment, the training unit determination unit includes a function determination unit and a function solving unit, wherein: the function determination unit is used to determine a determination function for the number of training units of the target model based on the weighting value and the training consumption parameter of the training units of the target model; the function solving unit is used to solve the determination function for the minimum value based on the number of remaining training units of the target model to obtain the target number of training units in the future target time window.

[0233] In one embodiment, the function determination unit includes a sub-function determination unit and a weighted calculation unit, wherein: the sub-function determination unit is used to determine at least one sub-determined function based on the weighted value and / or the training consumption parameters of the training unit of the target model; the weighted calculation unit is used to perform weighted calculation based on the preset weights corresponding to the at least one sub-determined function to obtain the determined function.

[0234] In one embodiment, the function determination unit includes at least one of a first sub-function determination unit, a second sub-function determination unit, and a third sub-function determination unit, wherein: the first sub-function determination unit is used to determine a first sub-function based on a weighted value and the unit energy consumption of the training unit, the value of the first sub-function being proportional to the number of training units; the second sub-function determination unit is used to determine a second sub-function, the value of the second sub-function being inversely proportional to the number of training units; and the third sub-function determination unit is used to determine a third sub-function based on resource consumption parameters and the unit energy consumption of the training unit, the value of the third sub-function being proportional to the number of training units.

[0235] In one embodiment, the apparatus further includes a data backup module for saving training backup data of the target model when training backup conditions are detected to be met; wherein the training backup conditions include the expiration of a preset backup time or the scheduling quantity of training resources of the target model exceeding a scheduling quantity threshold.

[0236] In one embodiment, the apparatus further includes a training consistency module for performing training consistency assurance processing of the target model when a change in the number of training resources for the target model is detected; wherein the training consistency assurance processing includes at least one of global synchronization of parameter values, recovery of the optimizer's internal state, gradient normalization compensation, and warm start of the learning rate.

[0237] In one embodiment, the apparatus further includes at least one of a first target data determination module, a training resource maintenance module, and a rollback module, wherein: the first target data determination module is used to perform weighted fusion processing on first time series data and third time series data to obtain a weighted value of a preset monitoring indicator in a future target time window when a prediction failure of the prediction model is detected, including determining the first time series data as the weighted value; the training resource maintenance module is used to schedule the training resources of the target model in the future target time window based on the weighted value when a network failure is detected, including maintaining the training resources at the current moment; the rollback module is used to roll back to the training backup data at the target moment when the scheduling of training resources in the future target time window is identified, wherein the target moment is earlier than the earliest moment in the future target time window.

[0238] In one embodiment, the preset monitoring indicators include at least one of carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators.

[0239] Each module in the aforementioned model training resource scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0240] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 18 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores scheduling data for model training resources. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for scheduling model training resources.

[0241] Those skilled in the art will understand that Figure 18The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0242] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring monitoring data of a target model during training; inputting the monitoring data into a prediction model to obtain third time-series data; performing weighted fusion processing on the first time-series data and the third time-series data to obtain a weighted value of a preset monitoring indicator in a future target time window; and scheduling training resources of the target model within the future target time window according to the weighted value of the preset monitoring indicator within the future target time window.

[0243] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a confidence index of the third time series data based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window using a prediction model; and determining the weights for weighted fusion based on the confidence index.

[0244] In one embodiment, when the processor executes the computer program, it further performs the following steps: under the condition of confidence degradation, the monitoring data is input into the basic prediction model for prediction to obtain the third time series data; wherein, the type of confidence degradation condition includes at least one of the following: the first time series data is empty, the amount of data in the second time series data is less than the data amount threshold, and the environmental data is missing.

[0245] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a candidate confidence index for the third time series data based on the interval width, the historical maximum interval width, and the mean absolute percentage error using a prediction model; determining a preset coefficient based on the type of confidence downgrade condition; reducing the candidate confidence index based on the preset coefficient to obtain the confidence index.

[0246] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first weight of the first time-series data and a second weight of the third time-series data based on the confidence index included in the third time-series data; calculating a first product of the first weight and the first time-series data, and a second product of the second weight and the third time-series data; and calculating the sum of the first product and the second product to obtain a weighted value.

[0247] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring candidate environment data for the current time window, sampling and normalizing the candidate environment data to obtain environment data; acquiring real-time monitoring values ​​for the current time window, aggregating and normalizing the real-time monitoring values ​​to obtain first time-series data; and encoding the current time window to obtain time features.

[0248] In one embodiment, when the processor executes the computer program, it further performs the following steps: inputting first time-series data, environmental data, time features, and second time-series data into the prediction model to obtain third time-series data.

[0249] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the target number of training units for the target model within a future target time window based on the weighted value; and performing resource expansion or resource reduction processing on the number of training units for the target model within the future target time window based on the target number.

[0250] In one embodiment, when the processor executes the computer program, it further implements the following steps: determining a function for the number of training units of the target model based on the weighting value and the training consumption parameter of the training units of the target model; and solving for the minimum value of the determining function based on the number of remaining training units of the target model to obtain the target number of training units in the future target time window.

[0251] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining at least one sub-determining function based on the weighted values ​​and / or the training consumption parameters of the training units of the target model; and performing weighted calculations based on the preset weights corresponding to the at least one sub-determining function to obtain the determining function.

[0252] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first sub-function based on a weighted value and a unit energy consumption parameter included in the training consumption parameters, wherein the value of the first sub-function is proportional to the number of training units; determining a second sub-function, wherein the value of the second sub-function is inversely proportional to the number of training units; and determining a third sub-function based on a unit energy consumption parameter and a resource consumption parameter included in the training consumption parameters, wherein the value of the third sub-function is proportional to the number of training units.

[0253] In one embodiment, when the processor executes the computer program, it further performs the following steps: if training backup conditions are detected, the training backup data of the target model is saved; wherein the training backup conditions include the expiration of a preset backup time, or the scheduling quantity of training resources for the target model exceeds a scheduling quantity threshold.

[0254] In one embodiment, when the processor executes the computer program, it further implements the following steps: when a change in the number of training resources for the target model is detected, it performs training consistency guarantee processing for the target model; wherein the training consistency guarantee processing includes at least one of global synchronization of parameter values, recovery of the optimizer's internal state, gradient normalization compensation, and warm start of the learning rate.

[0255] In one embodiment, when the processor executes the computer program, it further performs the following steps: in the case of detecting a prediction failure of the prediction model, performing weighted fusion processing on the first time series data and the third time series data to obtain a weighted value of the preset monitoring index in the future target time window, including determining the first time series data as the weighted value; in the case of detecting a network failure, scheduling the training resources of the target model in the future target time window based on the weighted value, including maintaining the training resources at the current moment; and in the case of identifying the scheduling of training resources in the future target time window, rolling back to the training backup data at the target moment, where the target moment is earlier than the earliest moment in the future target time window.

[0256] In one embodiment, the preset monitoring indicators include at least one of carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators.

[0257] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: acquiring monitoring data of the target model during the training process; inputting the monitoring data into a prediction model to obtain third time-series data; performing weighted fusion processing on the first time-series data and the third time-series data to obtain a weighted value of a preset monitoring indicator in a future target time window; and scheduling the training resources of the target model in the future target time window according to the weighted value of the preset monitoring indicator in the future target time window.

[0258] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a confidence index of the third time series data based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window using a prediction model; and determining the weights for weighted fusion based on the confidence index.

[0259] In one embodiment, when the processor executes the computer program, it further performs the following steps: under the condition of confidence degradation, the monitoring data is input into the basic prediction model for prediction to obtain the third time series data; wherein, the type of confidence degradation condition includes at least one of the following: the first time series data is empty, the amount of data in the second time series data is less than the data amount threshold, and the environmental data is missing.

[0260] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a candidate confidence index for the third time series data based on the interval width, the historical maximum interval width, and the mean absolute percentage error using a prediction model; determining a preset coefficient based on the type of confidence downgrade condition; reducing the candidate confidence index based on the preset coefficient to obtain the confidence index.

[0261] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first weight of the first time-series data and a second weight of the third time-series data based on the confidence index included in the third time-series data; calculating a first product of the first weight and the first time-series data, and a second product of the second weight and the third time-series data; and calculating the sum of the first product and the second product to obtain a weighted value.

[0262] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring candidate environment data for the current time window, sampling and normalizing the candidate environment data to obtain environment data; acquiring real-time monitoring values ​​for the current time window, aggregating and normalizing the real-time monitoring values ​​to obtain first time-series data; and encoding the current time window to obtain time features.

[0263] In one embodiment, when the processor executes the computer program, it further performs the following steps: inputting first time-series data, environmental data, time features, and second time-series data into the prediction model to obtain third time-series data.

[0264] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the target number of training units for the target model within a future target time window based on the weighted value; and performing resource expansion or resource reduction processing on the number of training units for the target model within the future target time window based on the target number.

[0265] In one embodiment, when the processor executes the computer program, it further implements the following steps: determining a function for the number of training units of the target model based on the weighting value and the training consumption parameter of the training units of the target model; and solving for the minimum value of the determining function based on the number of remaining training units of the target model to obtain the target number of training units in the future target time window.

[0266] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining at least one sub-determining function based on the weighted values ​​and / or the training consumption parameters of the training units of the target model; and performing weighted calculations based on the preset weights corresponding to the at least one sub-determining function to obtain the determining function.

[0267] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first sub-function based on a weighted value and a unit energy consumption parameter included in the training consumption parameters, wherein the value of the first sub-function is proportional to the number of training units; determining a second sub-function, wherein the value of the second sub-function is inversely proportional to the number of training units; and determining a third sub-function based on a unit energy consumption parameter and a resource consumption parameter included in the training consumption parameters, wherein the value of the third sub-function is proportional to the number of training units.

[0268] In one embodiment, when the processor executes the computer program, it further performs the following steps: if training backup conditions are detected, the training backup data of the target model is saved; wherein the training backup conditions include the expiration of a preset backup time, or the scheduling quantity of training resources for the target model exceeds a scheduling quantity threshold.

[0269] In one embodiment, when the processor executes the computer program, it further implements the following steps: when a change in the number of training resources for the target model is detected, it performs training consistency guarantee processing for the target model; wherein the training consistency guarantee processing includes at least one of global synchronization of parameter values, recovery of the optimizer's internal state, gradient normalization compensation, and warm start of the learning rate.

[0270] In one embodiment, when the processor executes the computer program, it further performs the following steps: in the case of detecting a prediction failure of the prediction model, performing weighted fusion processing on the first time series data and the third time series data to obtain a weighted value of the preset monitoring index in the future target time window, including determining the first time series data as the weighted value; in the case of detecting a network failure, scheduling the training resources of the target model in the future target time window based on the weighted value, including maintaining the training resources at the current moment; and in the case of identifying the scheduling of training resources in the future target time window, rolling back to the training backup data at the target moment, where the target moment is earlier than the earliest moment in the future target time window.

[0271] In one embodiment, the preset monitoring indicators include at least one of carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators.

[0272] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring monitoring data of a target model during training; inputting the monitoring data into a prediction model to obtain third time-series data; performing weighted fusion processing on the first and third time-series data to obtain a weighted value of a preset monitoring indicator within a future target time window; and scheduling training resources for the target model within the future target time window based on the weighted value of the preset monitoring indicator within the future target time window.

[0273] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a confidence index of the third time series data based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window using a prediction model; and determining the weights for weighted fusion based on the confidence index.

[0274] In one embodiment, when the processor executes the computer program, it further performs the following steps: under the condition of confidence degradation, the monitoring data is input into the basic prediction model for prediction to obtain the third time series data; wherein, the type of confidence degradation condition includes at least one of the following: the first time series data is empty, the amount of data in the second time series data is less than the data amount threshold, and the environmental data is missing.

[0275] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a candidate confidence index for the third time series data based on the interval width, the historical maximum interval width, and the mean absolute percentage error using a prediction model; determining a preset coefficient based on the type of confidence downgrade condition; reducing the candidate confidence index based on the preset coefficient to obtain the confidence index.

[0276] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first weight of the first time-series data and a second weight of the third time-series data based on the confidence index included in the third time-series data; calculating a first product of the first weight and the first time-series data, and a second product of the second weight and the third time-series data; and calculating the sum of the first product and the second product to obtain a weighted value.

[0277] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring candidate environment data for the current time window, sampling and normalizing the candidate environment data to obtain environment data; acquiring real-time monitoring values ​​for the current time window, aggregating and normalizing the real-time monitoring values ​​to obtain first time-series data; and encoding the current time window to obtain time features.

[0278] In one embodiment, when the processor executes the computer program, it further performs the following steps: inputting first time-series data, environmental data, time features, and second time-series data into the prediction model to obtain third time-series data.

[0279] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the target number of training units for the target model within a future target time window based on the weighted value; and performing resource expansion or resource reduction processing on the number of training units for the target model within the future target time window based on the target number.

[0280] In one embodiment, when the processor executes the computer program, it further implements the following steps: determining a function for the number of training units of the target model based on the weighting value and the training consumption parameter of the training units of the target model; and solving for the minimum value of the determining function based on the number of remaining training units of the target model to obtain the target number of training units in the future target time window.

[0281] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining at least one sub-determining function based on the weighted values ​​and / or the training consumption parameters of the training units of the target model; and performing weighted calculations based on the preset weights corresponding to the at least one sub-determining function to obtain the determining function.

[0282] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first sub-function based on a weighted value and a unit energy consumption parameter included in the training consumption parameters, wherein the value of the first sub-function is proportional to the number of training units; determining a second sub-function, wherein the value of the second sub-function is inversely proportional to the number of training units; and determining a third sub-function based on a unit energy consumption parameter and a resource consumption parameter included in the training consumption parameters, wherein the value of the third sub-function is proportional to the number of training units.

[0283] In one embodiment, when the processor executes the computer program, it further performs the following steps: if training backup conditions are detected, the training backup data of the target model is saved; wherein the training backup conditions include the expiration of a preset backup time, or the scheduling quantity of training resources for the target model exceeds a scheduling quantity threshold.

[0284] In one embodiment, when the processor executes the computer program, it further implements the following steps: when a change in the number of training resources for the target model is detected, it performs training consistency guarantee processing for the target model; wherein the training consistency guarantee processing includes at least one of global synchronization of parameter values, recovery of the optimizer's internal state, gradient normalization compensation, and warm start of the learning rate.

[0285] In one embodiment, when the processor executes the computer program, it further performs the following steps: in the case of detecting a prediction failure of the prediction model, performing weighted fusion processing on the first time series data and the third time series data to obtain a weighted value of the preset monitoring index in the future target time window, including determining the first time series data as the weighted value; in the case of detecting a network failure, scheduling the training resources of the target model in the future target time window based on the weighted value, including maintaining the training resources at the current moment; and in the case of identifying the scheduling of training resources in the future target time window, rolling back to the training backup data at the target moment, where the target moment is earlier than the earliest moment in the future target time window.

[0286] In one embodiment, the preset monitoring indicators include at least one of carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators.

[0287] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0288] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0289] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0290] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for scheduling model training resources, characterized in that, The method includes: Acquire monitoring data of the target model during training. The monitoring data includes first time-series data and / or second time-series data. The first time-series data is determined based on the real-time monitoring value of a preset monitoring indicator within the current time window. The second time-series data is the historical monitoring value of the preset monitoring indicator. The monitoring data is input into the prediction model to obtain the third time series data, which is the predicted value of the preset monitoring index output by the prediction model within the future target time window. The first time series data and the third time series data are weighted and fused to obtain the weighted value of the preset monitoring indicator within the future target time window; Based on the weighted values ​​of the preset monitoring indicators within the future target time window, the training resources of the target model are scheduled and processed within the future target time window.

2. The method according to claim 1, characterized in that, The method further includes: The confidence index of the third time series data is determined by the prediction model based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window. The weights of the weighted fusion are determined based on the confidence index.

3. The method according to claim 2, characterized in that, The prediction model includes a basic prediction model, and the monitoring data also includes environmental data. The step of inputting the monitoring data into the prediction model to obtain the third time series data includes: If the confidence level downgrade condition is met, the monitoring data is input into the basic prediction model for prediction to obtain the third time series data; The confidence downgrade conditions include at least one of the following: the first time series data is empty, the amount of the second time series data is less than the data amount threshold, and the environmental data is missing.

4. The method according to claim 2, characterized in that, The step of determining the confidence index of the third time series data using the prediction model, based on the interval width of the uncertainty interval of the future target time window, the historical maximum interval width, and the average absolute percentage error of the historical time window, includes: The candidate confidence index of the third time series data is determined by the prediction model based on the interval width, the historical maximum interval width, and the mean absolute percentage error. A preset coefficient is determined based on the type of confidence downgrade condition, and the candidate confidence index is reduced based on the preset coefficient to obtain the confidence index.

5. The method according to claim 2, characterized in that, Determining the weights of the weighted fusion based on the confidence index includes: Based on the confidence index, determine the first weight of the first time series data and the second weight of the third time series data; The step of weighted fusion processing of the first time-series data and the third time-series data to obtain the weighted value of the preset monitoring indicator within the future target time window includes: Calculate the first product of the first weight and the first time series data, and the second product of the second weight and the third time series data; The sum of the first product and the second product is calculated to obtain the weighted value.

6. The method according to claim 1, characterized in that, The monitoring data also includes at least one of environmental data and time features, and the acquisition of monitoring data of the target model during the training process includes at least one of the following: Acquire candidate environment data within the current time window, and perform sampling and normalization processing on the candidate environment data to obtain the environment data; The real-time monitoring values ​​are obtained, and the real-time monitoring values ​​are aggregated and normalized to obtain the first time-series data. The time period corresponding to the current time window is encoded to obtain the time feature.

7. The method according to claim 6, characterized in that, The step of inputting the monitoring data into the prediction model to obtain the third time series data includes: The first time series data, the environmental data, the time features, and the second time series data are input into the prediction model to obtain the third time series data.

8. The method according to claim 1, characterized in that, The step of scheduling training resources for the target model within the future target time window based on the weighted value of the preset monitoring indicators within the future target time window includes: Based on the weighted values, determine the target number of training units for the target model within the future target time window; Based on the target number, the number of training units of the target model within the future target time window is expanded or reduced in terms of resources.

9. The method according to claim 8, characterized in that, Determining the target number of training units for the target model within the future target time window based on the weighted value includes: A function is used to determine the number of training units of the target model based on the weighting value and the training consumption parameters of the training units of the target model. The minimum value of the determined function is calculated based on the number of remaining training units in the target model to obtain the target number.

10. The method according to claim 9, characterized in that, The function for determining the number of training units of the target model based on the weighted value and the training consumption parameters of the training units of the target model includes: Based on the weighted values ​​and / or the training consumption parameters of the training units of the target model, at least one sub-determining function is determined; The determining function is obtained by performing a weighted calculation based on the preset weights corresponding to the at least one sub-determining function.

11. The method according to claim 10, characterized in that, The step of determining at least one sub-determining function based on the weighted value and / or the training consumption parameters of the training units of the target model includes at least one of the following: Based on the weighted value and the unit energy consumption included in the training consumption parameters, a first sub-function is determined, and the value of the first sub-function is proportional to the number of training units; Determine a second sub-function, the value of which is inversely proportional to the number of training units; A third sub-function is determined based on the unit energy consumption and the resource consumption parameters included in the training consumption parameters. The value of the third sub-function is proportional to the number of training units.

12. The method according to claim 1, characterized in that, The method further includes: If the training backup conditions are met, the training backup data of the target model is saved. The training backup conditions include the expiration of a preset backup time, or the scheduling quantity of training resources for the target model exceeding a scheduling quantity threshold.

13. The method according to claim 1, characterized in that, The method further includes: If a change in the number of training resources for the target model is detected, training consistency of the target model is guaranteed. The training consistency guarantee process includes at least one of the following: global synchronization of parameter values, recovery of the optimizer's internal state, gradient normalization compensation, and warm start of the learning rate.

14. The method according to claim 1, characterized in that, The method further includes at least one of the following: In the event that the prediction model fails, the step of performing weighted fusion processing on the first time series data and the third time series data to obtain the weighted value of the preset monitoring indicator within the future target time window includes: determining the first time series data as the weighted value; In the event of a network failure, the step of scheduling the training resources of the target model within the future target time window based on the weighted value of the preset monitoring indicators within the future target time window includes: maintaining the training resources at the current moment; In the event of a failure to schedule training resources within the future target time window, the training backup data is rolled back to the target time, which is prior to all times within the future target time window.

15. The method according to any one of claims 1-14, characterized in that, The preset monitoring indicators include at least one of the following: carbon emission monitoring indicators, electricity consumption monitoring indicators, and computing power consumption monitoring indicators.

16. A scheduling device for model training resources, characterized in that, The device includes: The monitoring data acquisition module is used to acquire monitoring data of the target model during the training process. The monitoring data includes first time series data and / or second time series data. The first time series data is determined based on the real-time monitoring value of a preset monitoring indicator within the current time window, and the second time series data is the historical monitoring value of the preset monitoring indicator. The data prediction module is used to input the monitoring data into the prediction model to obtain the third time series data, wherein the third time series data is the predicted value of the preset monitoring indicator output by the prediction model within the future target time window; The fusion module is used to perform weighted fusion processing on the first time series data and the third time series data to obtain the weighted value of the preset monitoring indicator within the future target time window; The resource scheduling module is used to schedule the training resources of the target model within the future target time window based on the weighted value of the preset monitoring indicators within the future target time window.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 15.

Images