Partition priority adjustment method and device, equipment and storage medium

Abstract

The invention relates to a partition priority adjustment method and device, equipment and a storage medium. The method comprises the steps that a job increasing time period of a target partition in a scheduling system is obtained, the job increasing time period is obtained by processing historical job data of the target partition in a reference time period, the historical job data comprises the job number and resource data used for processing the job, and the job increasing time period is obtained by processing the historical job data of the target partition in the reference time period; and when it is detected that the current time reaches the start time in the job growth period, adjusting the priority of the target interval according to the current priority parameters of other partitions in the scheduling system. By adopting the method, the efficiency of job processing in the target partition can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a partition priority adjustment method, device, equipment and storage medium. Background Art

[0002] To ensure the reliability of job execution, the Slurm job scheduling system (Simple Linux Utility for Resource Management) classifies resources such as the central processing unit (CPU) and system memory, and runs jobs with different requirements on different computing nodes.

[0003] In the Slurm job scheduling system, at the beginning of each round of the main scheduling process, the system will summarize the jobs in the queue in all partitions and build a list of jobs to be scheduled; then, the jobs in the list of jobs to be scheduled will be sorted according to the priority of each partition, and the jobs will be processed according to the sorting results.

[0004] However, when a partition has a lower priority but there are many jobs in the queue in the partition, the efficiency of job processing in the partition will be reduced due to the limitation of the partition priority. Summary of the invention

[0005] Based on this, it is necessary to provide a partition priority adjustment method, device, equipment and storage medium that can improve the efficiency of job processing within a partition in response to the above technical problems.

[0006] In a first aspect, the present application provides a partition priority adjustment method, comprising:

[0007] Obtaining a job growth period of a target partition in a scheduling system; wherein the job growth period is obtained by processing historical job data of the target partition within a reference period, and the historical job data includes the number of jobs and resource data used to process the jobs;

[0008] When it is detected that the current time reaches the start time in the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system.

[0009] In an embodiment of the present application, on the one hand, the historical job data of the target partition is processed to obtain the job growth period, which can accurately determine the priority adjustment time and ensure the timeliness of the priority adjustment; on the other hand, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system, which can ensure the rationality of the priority adjustment and thereby improve the efficiency of job processing in the target partition.

[0010] In one embodiment, the historical job data of the target partition within a reference period is processed, including:

[0011] Determine the change trend of the workload of the target partition within the reference period according to the historical operation data of the target partition within the reference period;

[0012] Determine the job growth period of the target partition based on the job volume change trend.

[0013] In an embodiment of the present application, by determining the workload change trend of the target partition within a reference period based on historical job data, and then determining the job growth period of the target partition, the accuracy of the job growth period can be ensured.

[0014] In one embodiment, determining the change trend of the workload of the target partition within the reference period according to the historical operation data of the target partition within the reference period includes:

[0015] Splitting the historical operation data of the target partition within the reference period to obtain the historical operation data corresponding to each sub-period within the reference period; wherein the duration of each sub-period is the same;

[0016] For each sub-period, extracting target operation data from the historical operation data corresponding to the sub-period; wherein the data features corresponding to the target operation data can be used to represent the data features corresponding to the historical operation data;

[0017] According to the target operation data corresponding to each sub-period, the operation volume change trend of the target partition within the historical period is determined.

[0018] In an embodiment of the present application, by determining the workload change trend of the target partition within the historical time period based on the target operation data extracted from the historical operation data corresponding to each sub-time period, the accuracy of the workload change trend can be ensured while ensuring the efficiency of determining the workload change trend.

[0019] In one embodiment, determining the change trend of the workload of the target partition in the historical period according to the target operation data corresponding to each sub-period includes:

[0020] Compare the target operation data corresponding to two adjacent sub-periods to obtain the operation volume change rate corresponding to the remaining sub-periods except the first sub-period in each sub-period;

[0021] According to the workload change rates corresponding to the remaining sub-periods, the workload change trend of the target partition within the historical period is determined.

[0022] In the embodiment of the present application, by comparing the target operation data corresponding to two adjacent sub-periods, the workload change rate of each sub-period is obtained, and then the workload change trend is determined, which can ensure the accuracy of the workload change trend.

[0023] In one embodiment, determining the job growth period of the target partition according to the job volume change trend includes:

[0024] Input the workload change trends into different base learners respectively to obtain the reference growth periods output by different base learners;

[0025] Input the reference growth periods output by different base learners into the main learner to obtain the job growth period of the target partition;

[0026] Among them, the different base learners include at least two of the seasonal time series SARIMA base learner, exponential smoothing base learner, time series prediction prophet base learner, random forest base learner, and long short-term memory network LSTM base learner.

[0027] In an embodiment of the present application, by inputting the workload change trends into different base learners respectively, obtaining various reference growth periods, and then determining the workload growth period, the comprehensiveness of data analysis can be improved, thereby ensuring the accuracy of the workload growth period.

[0028] In one embodiment, adjusting the priority of the target interval according to current priority parameters of other partitions in the scheduling system includes:

[0029] Selecting a maximum priority parameter and a minimum priority parameter from the current priority parameters of each partition in the scheduling system;

[0030] Determine a target priority parameter of the target partition according to the maximum priority parameter and the minimum priority parameter;

[0031] Based on the target priority parameter, the priority of the target interval is adjusted.

[0032] In the embodiment of the present application, by determining the target priority parameter of the target partition according to the maximum priority parameter and the minimum priority parameter to adjust the priority of the target partition, the rationality of the priority adjustment of the target partition can be ensured.

[0033] In one of the embodiments, before adjusting the priority of the target interval, the method further includes:

[0034] Save the current priority parameters of each partition in the scheduling system;

[0035] Accordingly, the method further includes:

[0036] When it is detected that the current time reaches the end time of the job growth period, the priority of each partition in the scheduling system is restored according to the current priority parameters of each partition in the scheduling system.

[0037] In an embodiment of the present application, by restoring the priority of each partition in the scheduling system when it is detected that the current time reaches the end time of the job growth period, the rationality of the priority allocation of each partition can be ensured, thereby improving the resource utilization of the scheduling system.

[0038] In a second aspect, the present application further provides a partition priority adjustment device, comprising:

[0039] A time period acquisition module is used to obtain the job growth period of the target partition in the scheduling system; wherein the job growth period is obtained by processing the historical job data of the target partition within the reference period, and the historical job data includes the number of jobs and the resource data used to process the jobs;

[0040] The adjustment module is used to adjust the priority of the target interval according to the current priority parameters of other partitions in the scheduling system when it is detected that the current time reaches the start time of the job growth period.

[0041] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0042] Obtaining a job growth period of a target partition in a scheduling system; wherein the job growth period is obtained by processing historical job data of the target partition within a reference period, and the historical job data includes the number of jobs and resource data used to process the jobs;

[0043] When it is detected that the current time reaches the start time in the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0045] Obtaining a job growth period of a target partition in a scheduling system; wherein the job growth period is obtained by processing historical job data of the target partition within a reference period, and the historical job data includes the number of jobs and resource data used to process the jobs;

[0046] When it is detected that the current time reaches the start time in the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system.

[0047] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0048] Obtaining a job growth period of a target partition in a scheduling system; wherein the job growth period is obtained by processing historical job data of the target partition within a reference period, and the historical job data includes the number of jobs and resource data used to process the jobs;

[0049] When it is detected that the current time reaches the start time in the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system.

[0050] The above partition priority adjustment method, device, equipment and storage medium obtain the job growth period obtained by processing the historical job data of the target partition in the reference period in the scheduling system, and when it is detected that the current time reaches the start time of the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system. Compared with the related art, in which the jobs in each partition are processed only based on the initially configured partition priority, the above method, on the one hand, processes the historical job data of the target partition to obtain the job growth period, and can accurately determine the priority adjustment time, thereby ensuring the timeliness of the priority adjustment; on the other hand, according to the current priority parameters of other partitions in the scheduling system, the priority of the target interval is adjusted, which can ensure the rationality of the priority adjustment, thereby improving the efficiency of job processing in the target partition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram of a flow chart of a partition priority adjustment method in one embodiment;

[0053] Figure 2 A schematic diagram of a process for determining a job growth period in one embodiment;

[0054] Figure 3 A schematic diagram of a process for determining a workload change trend in one embodiment;

[0055] Figure 4A schematic diagram of a process for determining a job growth period in another embodiment;

[0056] Figure 5 A schematic diagram of model training in one embodiment;

[0057] Figure 6 A schematic diagram of a flow chart of a partition priority adjustment method in another embodiment;

[0058] Figure 7 A structural block diagram of a partition priority adjustment device in one embodiment;

[0059] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] To ensure the reliability of job execution, the Slurm job scheduling system classifies resources such as the central processing unit (CPU) and system memory, and runs jobs with different requirements on different computing nodes.

[0062] In the Slurm job scheduling system, at the beginning of each round of the main scheduling process, the system will summarize the jobs in the queue in all partitions and build a list of jobs to be scheduled; then, the jobs in the list of jobs to be scheduled will be sorted according to the priority of each partition, and the jobs will be processed according to the sorting results.

[0063] However, when a partition has a lower priority but there are many jobs in the queue in the partition, the efficiency of job processing in the partition will be reduced due to the limitation of the partition priority.

[0064] Based on this, in an exemplary embodiment, Figure 1 As shown, a partition priority adjustment method is provided, and the method is applied to a priority adjustment device deployed in a scheduling system as an example for explanation, and specifically includes the following steps:

[0065] S101, obtaining a job growth period of a target partition in a scheduling system.

[0066] The target partition is a partition with priority adjustment requirements; the job growth period is obtained by processing the historical job data of the target partition within the reference period, and the historical job data includes the number of jobs and resource data used for processing the jobs, for example, the resource data used for processing the jobs may be the CPU utilization rate. The reference period is any period before the current period.

[0067] In order to reduce the restriction of partition priority, when a growth prediction event for any partition is detected, the job growth period of the partition can be predetermined according to the number of jobs of the partition in a reference period and the resource data used for processing jobs.

[0068] Among them, the growth prediction event can be automatically generated by the event generation device in each partition according to a preset prediction period, or it can be generated when the event generation device detects that there is a fluctuation in the workload in the partition. Alternatively, the growth prediction event can also be generated based on a priority adjustment instruction sent by the user. This is not limited in the present application.

[0069] For example, the CPU utilization and the number of jobs of the target partition in the reference period can be simultaneously input into a trained period determination model, and the period determination model outputs the job growth period of the target partition based on the CPU utilization, the number of jobs and the model parameters.

[0070] In an optional implementation, in order to ensure the timeliness of time period acquisition, after determining that the target partition has a corresponding job growth time period, a time period acquisition time can be automatically generated based on the start time in the job growth time period, and when the time period acquisition time is reached, the job growth time period of the target partition is acquired. The time period acquisition time is earlier than the start time in the job growth time period.

[0071] Exemplarily, the time before the start time of the job growth period can be used as the period acquisition time. For example, 5 minutes before the start time of the job growth period can be used as the period acquisition time.

[0072] S102: When it is detected that the current time reaches the start time of the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system.

[0073] Among them, other partitions are partitions other than the target partition in the scheduling system; the so-called priority parameter is a value that can characterize the priority of the partition. The larger the priority parameter, the higher the priority of the partition; the current priority parameter is the unadjusted priority parameter at the current moment.

[0074] In an optional implementation, after obtaining the job growth period, it is necessary to make a real-time comparison between the current time and the start time in the job growth period. When it is detected that the current time reaches the start time in the job growth period, the target priority parameter to which the target partition needs to be adjusted is determined based on the current priority parameters of other partitions in the scheduling system; then, the priority parameter of the target interval is adjusted to the target priority parameter to adjust the priority of the target interval.

[0075] After adjusting the priority of the target interval, the adjusted priority of the target partition can be made effective by restarting the configuration of the scheduling system; then, the scheduling system can sort the pending jobs in each partition based on the adjusted priority of each partition, and process the jobs based on the sorting results.

[0076] In the above partition priority adjustment method, the job growth period is obtained by processing the historical job data of the target partition in the reference period in the scheduling system, and when it is detected that the current time reaches the start time of the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system. Compared with the related art, in which the jobs in each partition are processed only based on the initially configured partition priority, the above method, on the one hand, processes the historical job data of the target partition to obtain the job growth period, which can accurately determine the priority adjustment time and ensure the timeliness of the priority adjustment; on the other hand, according to the current priority parameters of other partitions in the scheduling system, the priority of the target interval is adjusted, which can ensure the rationality of the priority adjustment, thereby improving the efficiency of job processing in the target partition.

[0077] In order to ensure the accuracy of the job growth period, based on the above embodiment, in the embodiment of the present application, an optional method for determining the job growth period is provided, such as Figure 2 As shown, the specific steps include:

[0078] S201, determining a change trend of the workload of a target partition within a reference period according to historical operation data of the target partition within a reference period.

[0079] The workload change trend is used to characterize the workload change in the target partition.

[0080] It is understandable that there is a clear correlation between the number of jobs and the resource data used by the processing jobs (for example, CPU utilization): as the number of jobs in a partition increases, the resource data used by the processing jobs tends to increase accordingly; conversely, when the number of jobs decreases, the resource data used by the processing jobs may decrease. Therefore, combining the number of jobs with the resource data used by the processing jobs can more accurately capture the resource change trend of the target partition during the reference period, and then determine the trend of job volume changes.

[0081] After obtaining the original real operation data of the target partition within the reference period, the real operation data needs to be preprocessed to obtain historical operation data. Exemplarily, data cleaning operations can be performed on the real operation data, including but not limited to filling missing value operations, removing abnormal points operations, and smoothing operations.

[0082] For example, linear interpolation can be used to fill in missing value data in the actual operation data, and the standard score evaluation Z-score method and interquartile range (IQR) method can be used to replace outliers in the actual operation data with the data mean.

[0083] After determining the historical operation data of the target partition within the reference time period, in an optional implementation, the historical operation data can be input into a trained first trend determination model, and the first trend determination model outputs the workload change trend of the target partition within the reference time period based on the historical operation data and model parameters.

[0084] In another optional implementation, the historical job data within the reference time period can be split into historical job data in multiple sub-time periods according to preset time intervals; then, based on the changes in the historical job data in each sub-time period in each dimension, the workload change trend of the target partition within the reference time period is determined.

[0085] S202: Determine the job growth period of the target partition according to the job volume change trend.

[0086] In an optional implementation, the workload variation trend may be input into a trained time period prediction model, and the time period prediction model outputs the workload growth time period of the target partition based on the workload variation trend and model parameters.

[0087] In another optional implementation, the predicted operation data of the target partition in the future period can be predicted based on the change trend of the operation volume; then, the predicted operation data at each moment is compared with the preset operation volume threshold, and the operation growth period of the target partition is determined according to the comparison result. The operation volume threshold can be determined by relevant technicians based on historical experience or based on a large number of experiments, which is not limited in this application.

[0088] Exemplarily, the first moment when the first predicted job data is higher than the job volume threshold can be used as the start time of the job growth period, and the second moment after the first moment when the first predicted job data is lower than the job volume threshold can be used as the end time of the job growth period, thereby obtaining the job growth period of the target partition.

[0089] In an embodiment of the present application, by determining the workload change trend of the target partition within a reference period based on historical job data, and then determining the job growth period of the target partition, the accuracy of the job growth period can be ensured.

[0090] It is understandable that, since the amount of historical operation data in the reference period is large, if the operation volume change trend is determined directly based on all historical operation data, it is easy to reduce the efficiency of determining the operation volume change trend. Based on this, in order to ensure the efficiency and accuracy of determining the operation volume change trend, on the basis of the above embodiment, in the embodiment of the present application, an optional method for determining the operation volume change trend is provided, such as Figure 3 As shown, the specific steps include:

[0091] S301, splitting the historical operation data of the target partition within the reference period to obtain the historical operation data corresponding to each sub-period within the reference period.

[0092] The duration of each sub-period is the same.

[0093] In order to ensure the reliability of trend analysis, in an optional implementation, a standard splitting duration can be determined based on the duration of the reference period; then, based on the standard splitting duration, the historical operation data of the target partition within the reference period is split one by one in the time dimension to obtain the historical operation data corresponding to each sub-period. At this time, the sub-periods do not overlap.

[0094] For example, if the reference time period is 4:00-7:00, and the standard split time period is 1 hour, the sub-time periods 4:00-5:00, 5:00-6:00 and 6:00-7:00 can be obtained.

[0095] In another optional implementation, when the reference period is short, in order to increase the number of sub-periods, the historical operation data of the target partition within the reference period can be split in the time dimension based on the standard splitting period and the offset period to obtain the historical operation data corresponding to each sub-period. In this case, the sub-periods overlap.

[0096] For example, if the reference time period is 4:00-7:00, when the standard split time period is 1 hour and the offset time period is 30 minutes, the sub-time periods 4:00-5:00, 4:30-5:30, 5:00-6:00, 5:30-6:30 and 6:00-7:00 can be obtained.

[0097] S302 : For each sub-period, extract target operation data from the historical operation data corresponding to the sub-period.

[0098] Among them, the data features corresponding to the target operation data can be used to represent the data features corresponding to the historical operation data.

[0099] In an optional implementation, in order to improve data processing efficiency, for each sub-period, target operation data that can characterize data changes in the period can be extracted from the historical operation data based on data fluctuations in the historical operation data corresponding to the sub-period.

[0100] For example, the weighted moving average method can be used to process the historical operation data corresponding to the sub-period, and the processed data can be screened to obtain the target operation data. For example, for the historical operation data within one hour, the historical operation data within 15 minutes that can characterize the data changes within the period can be used as the target operation data.

[0101] S303: Determine the change trend of the workload of the target partition in the historical period according to the target operation data corresponding to each sub-period.

[0102] In an optional embodiment, the target operation data corresponding to each sub-period can be input into the trained second trend determination model in chronological order, and the second trend determination model outputs the workload change trend of the target partition in the historical period based on the target operation data corresponding to each sub-period and the model parameters.

[0103] In another optional implementation, data fluctuations between target job data corresponding to two adjacent sub-periods may be analyzed in sequence, thereby determining a change trend of the job volume of the target partition within the historical period.

[0104] In an embodiment of the present application, by determining the workload change trend of the target partition within the historical time period based on the target operation data extracted from the historical operation data corresponding to each sub-time period, the accuracy of the workload change trend can be ensured while ensuring the efficiency of determining the workload change trend.

[0105] In order to ensure the accuracy of the workload change trend, based on the above embodiment, in the embodiment of the present application, another optional method for determining the workload change trend is provided, specifically, the target operation data corresponding to two adjacent sub-periods are compared to obtain the workload change rate corresponding to the remaining sub-periods except the first sub-period in each sub-period; according to the workload change rate corresponding to the remaining sub-periods, the workload change trend of the target partition in the historical period is determined.

[0106] The so-called workload change rate is used to characterize the change in workload of the target workload data in each sub-period relative to the target workload data in the previous period.

[0107] In an optional implementation, two adjacent sub-periods can be used as a data comparison group. For example, the first sub-period and the second sub-period are used as a data comparison group, the second sub-period and the third sub-period are used as a data comparison group, and so on, to obtain various data comparison groups.

[0108] For each data comparison group, the target operation data corresponding to the two sub-periods in the data comparison group can be compared under the dimensions of each data type to obtain the operation volume change rate corresponding to the sub-period at the later time sequence in the data comparison group.

[0109] For example, for the data comparison group consisting of the first sub-period and the second sub-period, the target job data of the first sub-period in the two dimensions of CPU utilization and job quantity can be compared with the target job data of the second sub-period in the two dimensions of CPU utilization and job quantity to obtain the job volume change rate of the second sub-period.

[0110] It is understandable that since there is no corresponding previous sub-period for the first sub-period, it is impossible to determine the workload change rate of the first sub-period by data comparison. At this time, the workload change trend of the target partition in the historical period can be determined only by the workload change rates corresponding to the remaining sub-periods.

[0111] Alternatively, the target operation data of the first sub-period may be input into a trained change rate determination model, and the change rate determination model outputs the operation volume change rate of the first sub-period based on the target operation data and model parameters.

[0112] After determining the workload change rate of each sub-period, in an optional implementation, the workload change rate of each sub-period may be concatenated to obtain the workload change trend of the target partition in the historical period.

[0113] In another optional implementation, the workload change rate of each sub-period may be input into a trained third trend determination model, which may output the workload change trend of the target partition within the historical period based on the workload change rate of each sub-period and model parameters.

[0114] In the embodiment of the present application, by comparing the target operation data corresponding to two adjacent sub-periods, the workload change rate of each sub-period is obtained, and then the workload change trend is determined, which can ensure the accuracy of the workload change trend.

[0115] In order to ensure the accuracy of the job growth period, based on the above embodiment, in the embodiment of the present application, another optional method for determining the job growth period is provided, such as Figure 4 As shown, the specific steps include:

[0116] S401, inputting workload variation trends into different base learners respectively to obtain reference growth periods output by different base learners.

[0117] The so-called reference growth period is the period prediction result output by each base learner. The different base learners include at least two of seasonal time series SARIMA base learner, exponential smoothing base learner, time series prediction prophet base learner, random forest base learner, and long short-term memory network LSTM base learner.

[0118] SARIMA-based learners, exponential smoothing-based learners, and prophet-based learners can capture trends and periodicity in data and are suitable for stationary data; random forest-based learners can handle complex nonlinear relationships and have high flexibility and accuracy; LSTM-based learners can capture long-term dependencies and complex patterns and are suitable for complex CPU utilization predictions.

[0119] In order to ensure the accuracy of model prediction, a prediction model including multiple base learners and a main learner can be trained by using the sample workload change trend and the actual workload growth period corresponding to the sample workload change trend.

[0120] Exemplary, reference Figure 5 As shown in the model training diagram, the SARIMA base learner, exponential smoothing base learner, prophet base learner, random forest base learner and LSTM base learner can be selected as the base learners of the prediction model at the same time, and the logistic regression meta-learner can be selected as the meta-learner of the prediction model.

[0121] The sample workload change trend set (sample set) can be randomly divided into 5 subsets according to the 5-fold cross-validation method and input into 5 base learners, and the 5 base learners output the corresponding 5 sample reference growth periods.

[0122] Furthermore, the five sample reference growth periods can be merged and input into the logistic regression meta-learner for regression processing to output the sample predicted job growth period; subsequently, the prediction model can be optimized based on the deviation value between the predicted job growth period and the actual job growth period to obtain the final prediction model.

[0123] In an optional implementation, for each base learner, the workload change trend can be input into the base learner, and the base learner outputs a reference growth period corresponding to the base learner based on the workload change trend and the model parameters corresponding to the base learner.

[0124] S402, inputting the reference growth periods output by different base learners into the main learner to obtain the job growth period of the target partition.

[0125] In an optional implementation, after determining the reference growth period output by each base learner, each reference growth period may be simultaneously input into the main learner for logistic regression processing to obtain the job growth period of the target partition.

[0126] In an embodiment of the present application, by inputting the workload change trends into different base learners respectively, obtaining various reference growth periods, and then determining the workload growth period, the comprehensiveness of data analysis can be improved, thereby ensuring the accuracy of the workload growth period.

[0127] In order to ensure the rationality of the priority adjustment, based on the above embodiments, in the embodiments of the present application, an optional method of priority adjustment is provided, specifically, selecting a maximum priority parameter and a minimum priority parameter from the current priority parameters of each partition in the scheduling system; determining the target priority parameter of the target partition according to the maximum priority parameter and the minimum priority parameter; and adjusting the priority of the target interval based on the target priority parameter.

[0128] Among them, the so-called priority parameter is a parameter that can characterize the priority of a partition. The larger the priority parameter, the higher the priority of the partition. The current priority parameter is the original priority parameter of each partition before the priority is adjusted. The target priority parameter is the priority parameter of the target partition after adjustment.

[0129] In order to ensure the rationality of the priority adjustment, the current priority parameters of each partition may be sorted in descending order, and according to the sorting result, the maximum priority parameter and the minimum priority parameter may be selected from the current priority parameters.

[0130] In an optional implementation, the maximum priority parameter and the minimum priority parameter may be simultaneously input into a trained parameter determination model, and the parameter determination model outputs a target priority parameter based on the maximum priority parameter, the minimum priority parameter and the model parameters.

[0131] In another optional implementation, the sum of the maximum priority parameter and the minimum priority parameter may be used as a reference priority parameter, and then the reference priority parameter may be optimized based on the number of jobs and CPU utilization of each partition to obtain a target priority parameter.

[0132] After determining the target priority parameter, the priority parameter of the target interval can be adjusted to the target priority parameter to complete the adjustment of the target interval priority. At this time, the job of the target partition can have a higher priority and run first.

[0133] In the embodiment of the present application, by determining the target priority parameter of the target partition according to the maximum priority parameter and the minimum priority parameter to adjust the priority of the target partition, the rationality of the priority adjustment of the target partition can be ensured.

[0134] It is understandable that after the job growth period, the target partition does not need a higher priority, and the priority needs to be restored at this time. Based on this, on the basis of the above embodiment, in the embodiment of the present application, an optional method for priority restoration is provided, specifically:

[0135] Before adjusting the priority of the target interval, the current priority parameters of each partition in the scheduling system are saved; accordingly, when it is detected that the current time reaches the end time of the job growth period, the priority of each partition in the scheduling system is restored according to the current priority parameters of each partition in the scheduling system.

[0136] In order to ensure the reliability of priority recovery, when it is detected that an operation to adjust the priority of the target interval is about to be performed, it is necessary to obtain the current priority parameters of each partition from each partition in the scheduling system; or, directly obtain the current priority parameters of each partition from the configuration information of the scheduling system.

[0137] After saving the current priority parameters of each partition, you can refer to the above steps to perform the priority adjustment process normally and detect in real time whether the job growth period has ended.

[0138] When it is detected that the current time reaches the end time of the job growth period, the priority parameters of each partition can be adjusted with reference to the current priority parameters of each partition saved in advance to restore the priority of each partition in the scheduling system. It is understandable that after adjusting the priority parameters of each partition, the scheduling system configuration needs to be restarted to ensure that the scheduling system is restored to a normal resource allocation state.

[0139] In an embodiment of the present application, by restoring the priority of each partition in the scheduling system when it is detected that the current time reaches the end time of the job growth period, the rationality of the priority allocation of each partition can be ensured, thereby improving the resource utilization of the scheduling system.

[0140] Figure 6 FIG. 2 is a flow chart of a partition priority adjustment method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a partition priority adjustment method. Figure 6 The specific implementation process is as follows:

[0141] S601, splitting the historical operation data of the target partition within the reference period to obtain the historical operation data corresponding to each sub-period within the reference period.

[0142] The duration of each sub-period is the same; the historical job data includes the number of jobs and the resource data used to process the jobs.

[0143] S602: extract target operation data from the historical operation data corresponding to each sub-period.

[0144] Among them, the data features corresponding to the target operation data can be used to represent the data features corresponding to the historical operation data.

[0145] S603: Compare the target operation data corresponding to two adjacent sub-periods to obtain the operation volume change rates corresponding to the remaining sub-periods except the first sub-period in each sub-period.

[0146] S604: Determine the workload change trend of the target partition within the historical period according to the workload change rates corresponding to the remaining sub-periods.

[0147] S605, inputting the workload variation trends into different base learners respectively to obtain reference growth periods output by different base learners.

[0148] S606, inputting the reference growth periods output by different base learners into the main learner to obtain the job growth period of the target partition.

[0149] Among them, the different base learners include at least two of a SARIMA base learner, an exponential smoothing base learner, a prophet base learner, a random forest base learner, and a LSTM base learner.

[0150] S607: When it is detected that the current time reaches the start time of the job growth period, the current priority parameters of each partition in the scheduling system are saved.

[0151] S608: Select a maximum priority parameter and a minimum priority parameter from the current priority parameters of each partition in the scheduling system.

[0152] S609: Determine a target priority parameter of the target partition according to the maximum priority parameter and the minimum priority parameter.

[0153] S610: Adjust the priority of the target interval based on the target priority parameter.

[0154] S611, when it is detected that the current time reaches the end time of the job growth period, the priority of each partition in the scheduling system is restored according to the current priority parameters of each partition in the scheduling system.

[0155] The specific process of S601-S611 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a partition priority adjustment device for implementing the partition priority adjustment method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more partition priority adjustment device embodiments provided below can refer to the limitations on the partition priority adjustment method above, and will not be repeated here.

[0158] In an exemplary embodiment, Figure 7As shown, a partition priority adjustment device 1 is provided, comprising: a time period acquisition module 10 and an adjustment module 20, wherein:

[0159] The time period acquisition module 10 is used to acquire the job growth period of the target partition in the scheduling system; wherein the job growth period is obtained by processing the historical job data of the target partition within the reference period, and the historical job data includes the number of jobs and the resource data used to process the jobs;

[0160] The adjustment module 20 is used to adjust the priority of the target interval according to the current priority parameters of other partitions in the scheduling system when it is detected that the current time reaches the start time of the job growth period.

[0161] In an exemplary embodiment, the partition priority adjustment device 1 further includes a time period prediction module, wherein the time period prediction module includes:

[0162] A trend prediction unit, used to determine the change trend of the operation volume of the target partition within the reference period according to the historical operation data of the target partition within the reference period;

[0163] The time period prediction unit is used to determine the job growth period of the target partition according to the job volume change trend.

[0164] In an exemplary embodiment, the trend prediction unit includes:

[0165] The first sub-unit is used to split the historical operation data of the target partition within the reference period to obtain the historical operation data corresponding to each sub-period within the reference period; wherein the duration of each sub-period is the same;

[0166] A second subunit is used to extract target operation data from the historical operation data corresponding to each sub-period for each sub-period; wherein the data features corresponding to the target operation data can be used to represent the data features corresponding to the historical operation data;

[0167] The third sub-unit is used to determine the workload change trend of the target partition within the historical period according to the target operation data corresponding to each sub-period.

[0168] In an exemplary embodiment, the third subunit is specifically configured to:

[0169] The target operation data corresponding to two adjacent sub-periods are compared to obtain the operation volume change rates corresponding to the remaining sub-periods except the first sub-period; according to the operation volume change rates corresponding to the remaining sub-periods, the operation volume change trend of the target partition in the historical period is determined.

[0170] In an exemplary embodiment, the time period prediction unit is specifically used to:

[0171] The workload change trends are input into different base learners respectively to obtain reference growth periods output by different base learners; the reference growth periods output by different base learners are input into the main learner to obtain the job growth periods of the target partition; wherein the different base learners include at least two of the seasonal time series SARIMA base learner, exponential smoothing base learner, time series prediction prophet base learner, random forest base learner, and long short-term memory network LSTM base learner.

[0172] In an exemplary embodiment, the adjustment module 20 is specifically configured to:

[0173] A maximum priority parameter and a minimum priority parameter are selected from the current priority parameters of each partition in the scheduling system; a target priority parameter of the target partition is determined according to the maximum priority parameter and the minimum priority parameter; and the priority of the target interval is adjusted based on the target priority parameter.

[0174] In an exemplary embodiment, before adjusting the priority of the target interval, the partition priority adjustment device 1 further includes a parameter saving module, wherein the parameter saving module is specifically used to:

[0175] Save the current priority parameters of each partition in the scheduling system;

[0176] Accordingly, the adjustment module 20 is also used for:

[0177] When it is detected that the current time reaches the end time of the job growth period, the priority of each partition in the scheduling system is restored according to the current priority parameters of each partition in the scheduling system.

[0178] Each module in the above partition priority adjustment device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0179] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store historical operation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a partition priority adjustment method is implemented.

[0180] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0183] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0184] It should be noted that the data involved in this application (including but not limited to historical operation data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A partition priority adjustment method, characterized in that: The method comprises: Obtaining a job growth period of a target partition in a scheduling system; wherein the job growth period is obtained by processing historical job data of the target partition within a reference period, wherein the historical job data includes the number of jobs and resource data used to process the jobs; When it is detected that the current time reaches the start time of the job growth period, the priority of the target interval is adjusted according to the current priority parameters of other partitions in the scheduling system.

2. The method according to claim 1, characterized in that: The historical operation data of the target partition within the reference period is processed, including: Determine the workload change trend of the target partition within the reference period according to the historical operation data of the target partition within the reference period; According to the workload variation trend, determine a workload growth period of the target partition.

3. The method according to claim 2, characterized in that The determining, based on the historical operation data of the target partition within the reference period, the operation volume change trend of the target partition within the reference period includes: Splitting the historical operation data of the target partition within the reference period to obtain the historical operation data corresponding to each sub-period within the reference period; wherein the duration of each sub-period is the same; For each sub-period, extracting target operation data from the historical operation data corresponding to the sub-period; wherein the data features corresponding to the target operation data can be used to represent the data features corresponding to the historical operation data; According to the target operation data corresponding to each sub-period, the operation volume change trend of the target partition in the historical period is determined.

4. The method according to claim 3, characterized in that: The determining, according to the target operation data corresponding to each sub-period, the operation volume change trend of the target partition in the historical period includes: Compare the target operation data corresponding to two adjacent sub-periods to obtain the operation volume change rate corresponding to the remaining sub-periods except the first sub-period in each sub-period; The workload change trend of the target partition within the historical period is determined according to the workload change rates corresponding to the remaining sub-periods.

5. The method according to claim 1, characterized in that The step of determining the job growth period of the target partition according to the job volume change trend includes: Inputting the workload variation trend into different base learners respectively to obtain reference growth periods output by different base learners; Inputting the reference growth periods output by different base learners into the main learner to obtain the job growth period of the target partition; Among them, the different base learners include at least two of the seasonal time series SARIMA base learner, exponential smoothing base learner, time series prediction prophet base learner, random forest base learner, and long short-term memory network LSTM base learner.

6. The method according to claim 1, characterized in that The adjusting the priority of the target interval according to the current priority parameters of other partitions in the scheduling system includes: Selecting a maximum priority parameter and a minimum priority parameter from the current priority parameters of each partition in the scheduling system; Determining a target priority parameter of the target partition according to the maximum priority parameter and the minimum priority parameter; Based on the target priority parameter, the priority of the target interval is adjusted.

7. The method according to claim 1, characterized in that Before adjusting the priority of the target interval, the method further includes: Saving the current priority parameters of each partition in the scheduling system; Accordingly, the method further includes: When it is detected that the current time reaches the end time of the job growth period, the priority of each partition in the scheduling system is restored according to the current priority parameters of each partition in the scheduling system.

8. A partition priority adjustment device, characterized in that: The device comprises: A time period acquisition module, used to acquire a job growth period of a target partition in the scheduling system; wherein the job growth period is obtained by processing historical job data of the target partition within a reference period, wherein the historical job data includes the number of jobs and resource data used to process the jobs; The adjustment module is used to adjust the priority of the target interval according to the current priority parameters of other partitions in the scheduling system when it is detected that the current time reaches the start time of the job growth period.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images