A task scheduling method, device, equipment and computer readable storage medium

Abstract

The present invention discloses a task scheduling method, device, equipment and computer-readable storage medium, which relates to the technical field of management software, including: slicing real-time data acquisition tasks to obtain various real-time data acquisition jobs, and using various real-time executors to schedule various real-time data acquisition jobs; slicing periodic data acquisition tasks to obtain various periodic data acquisition jobs; obtaining the deadline, waiting time and estimated execution time corresponding to each periodic data acquisition job; scoring each periodic data acquisition job according to the deadline, waiting time and estimated execution time to obtain a scoring result; using each periodic executor to schedule each periodic data acquisition job according to each scoring result. The present invention solves the problem that the execution period of periodic tasks is relatively large, which easily causes performance degradation and hardware performance waste, effectively improves resource utilization, and improves system performance.

Description

Technical Field

[0001] The present invention relates to the field of management software technology, and in particular to a task scheduling method, device, equipment, computer-readable storage medium and computer program product. Background Art

[0002] In order to better reflect the actual operating status of the data center, it is usually necessary to collect a variety of data, such as equipment asset information, equipment performance information, etc.

[0003] Currently, task scheduling in data centers mainly uses polling-based scheduling. Data collection tasks are usually classified into real-time collection and periodic collection. For real-time collection tasks, when a collection task is received, the task is immediately placed in the execution queue. For periodic scheduling tasks, different collection cycles are set for different collection tasks, and timer polling is used. When the timer meets the time condition, the task is placed in the execution queue.

[0004] Due to the characteristics of the data collection task itself, the execution cycles of periodic tasks may vary greatly, and the execution cycles of different periodic tasks inevitably overlap. When intensive task scheduling occurs, performance will be reduced, and when sparse task scheduling occurs, hardware performance will be wasted.

[0005] In summary, how to effectively solve the problem that the execution period difference of periodic tasks is large, which easily causes performance degradation and waste of hardware performance, is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] The purpose of the present invention is to provide a task scheduling method, which effectively improves resource utilization and improves system performance; another purpose of the present invention is to provide a task scheduling device, equipment, computer-readable storage medium and computer program product.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] A task scheduling method, comprising:

[0009] The real-time data collection task is processed in pieces to obtain various real-time data collection jobs, and each real-time executor is used to schedule each real-time data collection job;

[0010] The periodic data collection task is processed in pieces to obtain the data collection jobs for each period;

[0011] Get the deadline, waiting time and estimated execution time corresponding to each periodic data collection job;

[0012] Scoring each periodic data collection job according to the deadline, the waiting time, and the estimated execution time to obtain a scoring result;

[0013] Each periodic executor is used to schedule the data collection jobs of each period according to the scoring results.

[0014] In a specific implementation of the present invention, obtaining the deadline, waiting time, and estimated execution time corresponding to each periodic data collection job includes:

[0015] Add each periodic data collection job to the periodic job queue;

[0016] Obtain the periodic job type to which each periodic data collection job belongs;

[0017] Determine the deadline corresponding to each periodic data collection operation according to the periodic operation type;

[0018] Respectively obtain the cache time length of each periodic data collection job in the periodic job queue, and determine the cache time length as the waiting time;

[0019] The estimated execution time corresponding to each periodic data collection job is determined according to the historical scheduling data of each periodic data collection job.

[0020] In a specific implementation of the present invention, determining the deadlines corresponding to the respective periodic data collection jobs according to the periodic job type includes:

[0021] When it is determined according to the periodic job type that the periodic data collection job is a flow-limited data collection job, obtaining the total number of flow-limited jobs included in the periodic job queue;

[0022] Obtaining the current limiting operation number of the periodic data collection operation;

[0023] Obtaining a task execution period corresponding to a periodic data collection task to which the periodic data collection job belongs;

[0024] The deadline corresponding to the periodic data collection job is calculated according to the total number of the current limiting jobs, the current limiting job number and the task execution period.

[0025] In a specific implementation of the present invention, determining the deadlines corresponding to the respective periodic data collection jobs according to the periodic job type includes:

[0026] When it is determined according to the periodic job type that the periodic data collection job is a discrete data collection job, obtaining a task execution period corresponding to the periodic data collection task to which the periodic data collection job belongs;

[0027] Determine the deadline corresponding to the periodic data collection task according to the task execution cycle;

[0028] The deadline corresponding to the periodic data collection task is determined as the deadline corresponding to the periodic data collection job.

[0029] In a specific implementation of the present invention, determining the estimated execution time corresponding to each periodic data collection job according to the historical scheduling data of each periodic data collection job includes:

[0030] When it is determined that there is historical scheduling data, determining the actual execution time corresponding to each historical scheduling order of the periodic data collection job according to the historical scheduling data;

[0031] Determine the total scheduling times of the periodic data collection operation according to the historical scheduling data;

[0032] The estimated execution time is calculated according to each historical scheduling order, the actual execution time corresponding to each historical scheduling order, and the total scheduling times.

[0033] In a specific implementation of the present invention, determining the estimated execution time corresponding to each periodic data collection job according to the historical scheduling data of each periodic data collection job includes:

[0034] When it is determined that there is no historical scheduling data, obtaining a preset experience value corresponding to the periodic data collection operation;

[0035] The preset experience value is determined as the estimated execution time corresponding to the periodic data collection operation.

[0036] In a specific implementation of the present invention, each periodic executor is used to schedule each periodic data collection job according to each scoring result, including:

[0037] According to the scoring results, the cycle data collection task with the highest score is selected from each cycle data collection task;

[0038] Obtain the sum of the estimated execution time of each periodic data collection job in the execution queue corresponding to each periodic executor;

[0039] Select the maximum value of the sum of the estimated execution time of each job;

[0040] Calculate the difference between the sum of the estimated execution time of each job and the maximum value respectively;

[0041] Calculate the absolute value of the difference between the estimated execution time corresponding to the periodic data collection job with the highest score and the difference;

[0042] Allocating the periodic data collection job with the highest score to the periodic executor corresponding to the smallest absolute value of the difference;

[0043] The step of selecting the periodic data collection job with the highest score from each periodic data collection job according to each score result is repeatedly performed until there is no idle periodic executor or each periodic data collection job is scheduled to be completed.

[0044] In a specific implementation of the present invention, allocating the highest-scoring periodic data collection job to the periodic executor corresponding to the smallest absolute value of the difference includes:

[0045] The periodic data collection job with the highest score is cached in the execution queue of the periodic executor corresponding to the smallest absolute difference value, so that the periodic executor corresponding to the smallest absolute difference value performs job processing according to the arrangement order of each periodic data collection job in the execution queue.

[0046] In a specific implementation of the present invention, after each periodic executor is used to schedule each periodic data collection job according to each scoring result, the method further includes:

[0047] Statistics are collected on the actual execution time corresponding to each periodic data collection operation;

[0048] The actual execution time counted is recorded.

[0049] In a specific implementation of the present invention, after obtaining each real-time data collection operation, it also includes:

[0050] Generate real-time operation identifiers corresponding to each real-time data collection operation;

[0051] Obtaining the real-time task identifier of the real-time data collection task to which each real-time data collection job belongs;

[0052] The real-time job identifier and the real-time task identifier are filled into each real-time data collection job.

[0053] In a specific implementation of the present invention, before using each real-time executor to schedule each real-time data collection job, it also includes:

[0054] Searching whether the real-time data collection job exists in the real-time job queue according to the real-time job identifier and the real-time task identifier respectively corresponding to each real-time data collection job;

[0055] When it is determined that the real-time data collection job exists in the real-time job queue, determining the real-time data collection job as a repeated job;

[0056] When it is determined that the real-time data collection job does not exist in the real-time job queue, the real-time data collection job is added to the real-time job queue.

[0057] In a specific implementation of the present invention, after obtaining each periodic data collection operation, it also includes:

[0058] Generate periodic operation identifiers corresponding to each periodic data collection operation;

[0059] Obtaining the periodic task identifier of the periodic data collection task to which each periodic data collection job belongs;

[0060] The periodic job identifier and the periodic task identifier are filled into each periodic data collection job.

[0061] In a specific implementation of the present invention, after the periodic job identifier and the periodic task identifier are filled into each periodic data collection job, the method further includes:

[0062] Searching whether the periodic data collection job exists in the periodic job queue according to the periodic job identifier and the periodic task identifier respectively corresponding to each periodic data collection job;

[0063] If yes, migrating the periodic data collection job in the periodic job queue to the real-time job queue;

[0064] If not, the periodic data collection job is added to the periodic job queue.

[0065] In a specific implementation of the present invention, each real-time executor is used to schedule each real-time data collection job, including:

[0066] Add each real-time data collection job to the real-time job queue;

[0067] The real-time data collection jobs in the real-time job queue are scheduled using the real-time executors according to the load balancing principle.

[0068] In a specific implementation of the present invention, the periodic data collection task is processed in slices, including:

[0069] The periodic data collection task is processed in slices according to resource dimensions.

[0070] In a specific implementation of the present invention, after each periodic executor is used to schedule each periodic data collection job according to each scoring result, the method further includes:

[0071] Monitor the processing of data collection operations in each cycle;

[0072] When a periodic data collection operation is detected to be in an abnormal state, the abnormal state record is made.

[0073] A task scheduling device, comprising:

[0074] The real-time job scheduling module is used to process the real-time data collection tasks in slices to obtain various real-time data collection jobs, and use various real-time executors to schedule various real-time data collection jobs;

[0075] The periodic job acquisition module is used to process the periodic data collection tasks in slices to obtain the data collection jobs of each period;

[0076] The time acquisition module is used to obtain the deadline, waiting time and estimated execution time corresponding to each periodic data collection job;

[0077] A scoring result obtaining module, used to score each periodic data collection job according to the deadline, the waiting time and the estimated execution time to obtain a scoring result;

[0078] The job scheduling module is used to schedule the data collection jobs of each period according to the scoring results by using the executors of each period.

[0079] A task scheduling device, comprising:

[0080] Memory for storing computer programs;

[0081] The processor is used to implement the steps of the task scheduling method as described above when executing the computer program.

[0082] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task scheduling method described above are implemented.

[0083] A computer program product includes a computer program, which implements the steps of the task scheduling method as described above when executed by a processor.

[0084] The task scheduling method provided by the present invention obtains a data acquisition task set; performs slice processing on the real-time data acquisition tasks in the data acquisition task set to obtain each real-time data acquisition job, and uses each real-time executor to schedule each real-time data acquisition job; performs slice processing on the periodic data acquisition tasks in the data acquisition task set to obtain each periodic data acquisition job; obtains the deadline, waiting time and estimated execution time corresponding to each periodic data acquisition job; scores each periodic data acquisition job according to the deadline, waiting time and estimated execution time to obtain a scoring result; and uses each periodic executor to schedule each periodic data acquisition job according to each scoring result.

[0085] It can be seen from the above technical solution that by slicing the data collection tasks and setting the corresponding deadline, waiting time and estimated execution time for each periodic data collection job obtained by the slicing, the tasks can be evenly distributed in their scheduling cycles, reducing the load pressure of task operation, reducing the impact of uneven task running time, improving the rationality of task scheduling, and effectively improving resource utilization. By uniformly scoring different tasks, unified management of different tasks is achieved. By uniformly managing data collection tasks, balanced scheduling of data collection tasks is achieved. Under the same data center scale, the system's hardware conditions can be more fully utilized, reducing the system's operating pressure and improving system performance.

[0086] Correspondingly, the present invention also provides a task scheduling device, equipment, computer-readable storage medium and computer program product corresponding to the above-mentioned task scheduling method, which have the above-mentioned technical effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0088] Figure 1 A flowchart of an implementation of the task scheduling method in an embodiment of the present invention;

[0089] Figure 2 Another implementation flow chart of the task scheduling method in the embodiment of the present invention;

[0090] Figure 3 is a structural block diagram of a task scheduling system in an embodiment of the present invention;

[0091] Figure 4 is a structural block diagram of a task scheduling device in an embodiment of the present invention;

[0092] Figure 5 is a structural block diagram of a task scheduling device in an embodiment of the present invention;

[0093] Figure 6 A schematic diagram of the specific structure of a task scheduling device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0094] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0095] See also Figure 1 , Figure 1 This is an implementation flow chart of a task scheduling method in an embodiment of the present invention. The method may include the following steps:

[0096] S101: Slice the real-time data collection task to obtain various real-time data collection jobs, and use various real-time executors to schedule the various real-time data collection jobs.

[0097] The data center includes data collection tasks such as collecting equipment asset information, equipment performance information, etc.

[0098] The asset information of the device may include the manufacturer, model, component type, component status, etc. The performance information of the device may include temperature information, fan speed, device operating power consumption, etc.

[0099] After obtaining the real-time data collection task, the real-time data collection task is sliced ​​and processed to obtain each real-time data collection job. A real-time executor is pre-set, and each real-time executor is used to schedule each real-time data collection job. After the task is divided, the start time is assigned to each independent job in an equal interval division. In this way, the decentralized execution of tasks is achieved, reducing the possibility of intensive scheduling of tasks.

[0100] Real-time data collection tasks refer to tasks that need to be executed immediately, such as asset refresh tasks, alarm synchronization tasks, etc. Real-time data collection tasks are one-time tasks and will not be executed repeatedly.

[0101] S102: Slice the periodic data collection task to obtain each periodic data collection job.

[0102] After obtaining the periodic data collection task, the periodic data collection task is segmented to obtain each periodic data collection job. This is done by splitting an overall periodic data collection task into multiple independent jobs that can be executed separately, such as dividing the asset collection task according to the resource dimension, and dividing the performance collection task according to the indicator dimension.

[0103] Both the current-limited data collection task and the discrete data collection task are periodic data collection tasks, and you need to set the execution cycle when you create them. The difference is that the current-limited data collection task has a short collection time and a short task execution time, and the task has high requirements for the periodicity of data collection, such as the performance data collection task and the equipment inspection task. The discrete data collection task has a long collection cycle and a long collection time, and the task has low requirements for the periodicity of data collection, such as the asset information collection task and the log download task. The current-limited data collection task and the discrete data collection task will be repeatedly executed according to the execution cycle specified when the task is created.

[0104] When splitting a job, the necessary attributes will be filled in the job, and the job is defined as , ,in Indicates the task ID through which the source of the job can be identified; Indicates the job ID, which can be used to identify different jobs after task slicing. In the periodic data collection task, the job IDs after slicing in different rounds are consistent; Indicates the execution cycle of the task. For real-time tasks Set to -1; Indicates the start time of the job; Indicates the deadline for the assignment; Indicates the estimated execution time of the job, provided by the data statistics module.

[0105] When the task center receives a real-time task, it splits the task according to the slicer specified by the task, sets the job start time and deadline to -1, and adds the job to the real-time queue of the job queue module. When the task center receives a discrete task, it determines whether the time meets the scheduling period of the discrete task. If the time meets the time, it slices the task, sets the job start time to -1, sets the job deadline to the next execution time of the discrete task, and adds the sliced ​​job to the periodic queue of the job queue module. When the task center receives a flow-limiting task, it also determines whether the time meets the scheduling period of the flow-limiting task, and slices the task if the time meets the time.

[0106] S103: Obtain the deadline, waiting time and estimated execution time corresponding to each periodic data collection job.

[0107] After each periodic data collection job is divided, the deadline, waiting time and estimated execution time corresponding to each periodic data collection job are obtained.

[0108] The deadline is the deadline for dividing the execution cycle corresponding to the periodic data collection task into the corresponding deadlines of each periodic data collection job. The waiting time is the timing duration corresponding to each periodic data collection job from the time when the periodic data collection job is divided. The estimated execution time is the time required to execute each periodic data collection job.

[0109] S104: Score each periodic data collection job according to the deadline, waiting time and estimated execution time to obtain a scoring result.

[0110] After obtaining the deadline, waiting time, and estimated execution time corresponding to each periodic data collection job, score each periodic data collection job according to the deadline, waiting time, and estimated execution time to obtain a scoring result. For example, a scoring rule can be set so that the closer the deadline of the periodic data collection job is to the current time and the longer the waiting time is, the higher the corresponding score is.

[0111] S105: Utilize each period executor to schedule each period data collection job according to each scoring result.

[0112] A periodic executor is pre-set. After obtaining the scoring results corresponding to each periodic data collection job, each periodic executor is used to schedule each periodic data collection job according to each scoring result. For example, the scheduling priority can be set to be proportional to the scoring result, that is, the higher the score, the higher the corresponding scheduling priority. After dividing the periodic data collection tasks, according to the execution cycle of the periodic data collection tasks, the start time is assigned to each independent job by equal interval division, thereby realizing the decentralized execution of tasks and reducing the possibility of intensive scheduling of tasks.

[0113] It should be noted that the real-time executor and the periodic executor can be set according to actual needs. Usually, the number of real-time executors is smaller than the number of periodic executors. Each executor contains an execution queue and an execution thread. The execution thread will execute tasks in the order of tasks in the execution queue.

[0114] It can be seen from the above technical solution that by slicing the data collection tasks and setting the corresponding deadline, waiting time and estimated execution time for each periodic data collection job obtained by the slicing, the tasks can be evenly distributed in their scheduling cycles, reducing the load pressure of task operation, reducing the impact of uneven task running time, improving the rationality of task scheduling, and effectively improving resource utilization. By uniformly scoring different tasks, unified management of different tasks is achieved. By uniformly managing data collection tasks, balanced scheduling of data collection tasks is achieved. Under the same data center scale, the system's hardware conditions can be more fully utilized, reducing the system's operating pressure and improving system performance.

[0115] It should be noted that, based on the above embodiment, the embodiment of the present invention also provides corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0116] In a specific implementation of the present invention, after step S106, the method may further include the following steps:

[0117] Step 1: Count the actual execution time corresponding to each periodic data collection operation;

[0118] Step 2: Record the actual execution time.

[0119] For the convenience of description, the above two steps can be combined for explanation.

[0120] After scheduling each periodic data collection job according to each scoring result using each periodic executor, the actual execution time corresponding to each periodic data collection job is counted and the counted actual execution time is recorded. By recording the actual execution time corresponding to each periodic data collection job, a reference is provided for predicting the execution time of the subsequent same periodic data collection jobs.

[0121] In a specific implementation of the present invention, after obtaining each real-time data collection operation, the method may further include the following steps:

[0122] Step 1: Generate real-time job identifiers corresponding to each real-time data collection job;

[0123] Step 2: Obtain the real-time task identifier of the real-time data collection task to which each real-time data collection operation belongs;

[0124] Step 3: Fill the real-time job ID and real-time task ID into each real-time data collection job.

[0125] For the convenience of description, the above three steps can be combined for explanation.

[0126] After obtaining each real-time data collection job, a real-time job identifier corresponding to each real-time data collection job is generated. Each real-time data collection task has a corresponding real-time task identifier, and the real-time task identifier of the real-time data collection task to which each real-time data collection job belongs is obtained. After obtaining the real-time job identifier corresponding to each real-time data collection job and the real-time task identifier of the real-time data collection task to which each real-time data collection job belongs, the real-time job identifier and the real-time task identifier are filled into each real-time data collection job. By filling the real-time task identifier into each real-time data collection job, the source information of each real-time data collection job is effectively identified, and by jointly identifying each real-time data collection job with the real-time job identifier and the real-time task identifier, each real-time data collection job is uniquely identified.

[0127] In a specific implementation of the present invention, before using each real-time executor to schedule each real-time data collection job, the method may further include the following steps:

[0128] Step 1: Find whether there is a real-time data collection job in the real-time job queue according to the real-time job identifier and the real-time task identifier corresponding to each real-time data collection job;

[0129] Step 2: when it is determined that there is a real-time data collection job in the real-time job queue, the real-time data collection job is determined as a repeated job;

[0130] Step 3: When it is determined that there is no real-time data collection job in the real-time job queue, add the real-time data collection job to the real-time job queue.

[0131] For the convenience of description, the above three steps can be combined for explanation.

[0132] Before using each real-time executor to schedule each real-time data collection job, check whether there is a real-time data collection job in the real-time job queue according to the real-time job identifier and real-time task identifier corresponding to each real-time data collection job. When it is determined that there is a real-time data collection job in the real-time job queue, the real-time data collection job is determined to be a repeated job, that is, it may be a repeated job generated by job refresh, and the real-time data collection job is not added to the real-time data collection job. When it is determined that there is no real-time data collection job in the real-time job queue, it means that the real-time data collection job is a newly generated real-time job that needs to be scheduled, and the real-time data collection job is added to the real-time job queue. By searching the real-time job queue according to the real-time job identifier and the real-time task identifier, effective filtering of repeated jobs is achieved, repeated execution of real-time data collection jobs is avoided, and resource utilization is improved.

[0133] In a specific implementation of the present invention, after obtaining each periodic data collection operation, the method may further include the following steps:

[0134] Step 1: Generate periodic operation identifiers corresponding to each periodic data collection operation;

[0135] Step 2: Obtain the periodic task identifier of the periodic data collection task to which each periodic data collection job belongs;

[0136] Step 3: Fill the periodic job ID and periodic task ID into each periodic data collection job.

[0137] For the convenience of description, the above three steps can be combined for explanation.

[0138] After obtaining each periodic data collection job, a periodic job identifier corresponding to each periodic data collection job is generated. Each periodic data collection task has a corresponding periodic task identifier, and the periodic task identifier of the periodic data collection task to which each periodic data collection job belongs is obtained. After obtaining the periodic job identifier corresponding to each periodic data collection job and the periodic task identifier of the periodic data collection task to which each periodic data collection job belongs, the periodic job identifier and the periodic task identifier are filled into each periodic data collection job. By filling the periodic task identifier into each periodic data collection job, the source information of each periodic data collection job is effectively identified. By jointly identifying each periodic data collection job with the periodic job identifier and the periodic task identifier, each periodic data collection job is uniquely identified.

[0139] In a specific implementation of the present invention, after filling the periodic job identifier and the periodic task identifier into each periodic data collection job, the method may further include the following steps:

[0140] Step 1: Check whether there is a periodic data collection job in the periodic job queue according to the periodic job identifier and periodic task identifier corresponding to each periodic data collection job. If yes, execute step 2; if no, execute step 3;

[0141] Step 2: Migrate the periodic data collection job in the periodic job queue to the real-time job queue;

[0142] Step 3: Add the periodic data collection job to the periodic job queue.

[0143] For the convenience of description, the above three steps can be combined for explanation.

[0144] After filling the periodic job identifier and periodic task identifier into each periodic data collection job, check whether there is a periodic data collection job in the periodic job queue according to the periodic job identifier and periodic task identifier corresponding to each periodic data collection job. If yes, it means that the frequency of generation of the periodic data collection job is relatively high, and the timeliness requirement for execution is relatively high, and it is relatively important. The periodic data collection job in the periodic job queue is migrated to the real-time job queue. If not, it means that the frequency of generation of the periodic data collection job is relatively low, and the timeliness requirement for execution is relatively low. The periodic data collection job is added to the periodic job queue. By migrating the periodic data collection jobs that are generated relatively frequently from the periodic job queue to the real-time job queue for execution, queue reuse is achieved, and the processing efficiency of the periodic data collection jobs that are generated relatively frequently is improved.

[0145] In a specific implementation of the present invention, scheduling each real-time data collection job using each real-time executor may include the following steps:

[0146] Step 1: Add each real-time data collection job to the real-time job queue;

[0147] Step 2: Use each real-time executor to schedule each real-time data collection job in the real-time job queue according to the load balancing principle.

[0148] For the convenience of description, the above two steps can be combined for explanation.

[0149] After obtaining each real-time data collection job, each real-time data collection job is added to the real-time job queue, and each real-time executor is used to schedule each real-time data collection job in the real-time job queue according to the load balancing principle. By using each real-time executor to schedule the real-time data collection job according to the load balancing principle, the resource utilization rate is improved and the execution efficiency of the real-time data collection job is improved.

[0150] In a specific implementation of the present invention, the slicing process of the periodic data collection task may include the following steps:

[0151] The periodic data collection tasks are segmented according to the resource dimension.

[0152] When the periodic data collection task is sliced, the periodic data collection task is sliced ​​according to the resource dimension. By slicing the periodic data collection task according to the resource dimension, the periodic data collection task is effectively split.

[0153] In a specific implementation of the present invention, after step S106, the method may further include the following steps:

[0154] Step 1: Monitor the processing of each cycle data collection operation;

[0155] Step 2: When a periodic data collection operation is detected to be in an abnormal state, the abnormal state is recorded.

[0156] For the convenience of description, the above two steps can be combined for explanation.

[0157] After scheduling each periodic data collection job according to each scoring result using each periodic executor, the processing process of each periodic data collection job is monitored, and when a periodic data collection job in an abnormal state is detected, the abnormal state is recorded. By recording the abnormal state, it is convenient to track the execution results of each periodic data collection job in the future.

[0158] See also Figure 2 , Figure 2 Another implementation flow chart of the task scheduling method in an embodiment of the present invention is shown below. The method may include the following steps:

[0159] S201: Slice the real-time data collection task to obtain various real-time data collection jobs, and use various real-time executors to schedule the various real-time data collection jobs.

[0160] S202: Slice the periodic data collection task to obtain each periodic data collection job.

[0161] S203: Add each periodic data collection job to the periodic job queue.

[0162] A periodic job queue is created in advance, and after each periodic data collection job is obtained, each periodic data collection job is added to the periodic job queue. By adding each periodic data collection job to the periodic job queue, unified management of each periodic data collection job is achieved.

[0163] S204: Obtain the periodic job type to which each periodic data collection job belongs.

[0164] After each periodic data collection job is added to the periodic job queue, the periodic job type to which each periodic data collection job belongs is obtained.

[0165] Periodic job types can include limited-flow data collection jobs and discrete data collection jobs. Limited-flow data collection jobs have a short collection time and a short execution time, and the job has high requirements for the periodicity of data collection, such as performance data collection jobs and equipment inspection jobs. Discrete data collection jobs have a long collection cycle and a long collection time, and the job has low requirements for the periodicity of data collection, such as asset information collection jobs and log download jobs.

[0166] S205: Determine the deadline corresponding to each periodic data collection job according to the periodic job type.

[0167] Different types of periodic jobs have different deadline determination methods. After the periodic job types to which each periodic data collection job belongs are acquired, the deadlines corresponding to each periodic data collection job are determined according to the periodic job types.

[0168] In a specific implementation of the present invention, step S205 may include the following steps:

[0169] Step 1: when it is determined that the periodic data collection job is a flow-limited data collection job according to the periodic job type, the total number of flow-limited jobs included in the periodic job queue is obtained;

[0170] Step 2: Get the flow-limiting job number of the periodic data collection job;

[0171] Step 3: Obtain the task execution period corresponding to the periodic data collection task to which the periodic data collection job belongs;

[0172] Step 4: Calculate the deadline corresponding to the periodic data collection job based on the total number of current limiting jobs, the current limiting job number, and the task execution period.

[0173] For the convenience of description, the above four steps can be combined for explanation.

[0174] After obtaining the data collection tasks for each cycle After the periodic job types are respectively determined, when the periodic data collection job is determined to be a flow-limited data collection job according to the periodic job type, the total number of flow-limited jobs n contained in the periodic job queue is obtained, and the flow-limited job number of the periodic data collection job is obtained , obtain the task execution period T corresponding to the periodic data collection task to which the periodic data collection job belongs, according to the total number of current limiting jobs n and the current limiting job number The deadline corresponding to the task execution period T calculation period data collection job For example, the following formula can be used to calculate the total number of flow-limiting jobs n and the flow-limiting job number. The deadline corresponding to the task execution period T calculation period data collection job :

[0175] ;

[0176] in, Represents the proportional coefficient, which is used to control the proportion of the sharded tasks in the entire cycle. The default value is 0.8. When there are fewer sharded tasks, it can be appropriately reduced. Indicates the start time of the periodic data collection job.

[0177] After calculating the start and end times of each periodic data collection job and splitting the tasks, the task center will use a countdown method to regularly reduce the start time of the current limiting job through timed polling. ,when When the job is completed, the job is added to the periodic queue of the queue module. The flow of each periodic data collection job is limited according to the calculated start time and end time. The impact of uneven task running time is reduced and the rationality of task scheduling is improved.

[0178] In a specific implementation of the present invention, step S205 may include the following steps:

[0179] Step 1: when it is determined according to the periodic job type that the periodic data collection job is a discrete data collection job, obtaining a task execution period corresponding to the periodic data collection task to which the periodic data collection job belongs;

[0180] Step 2: Determine the deadline corresponding to the periodic data collection task according to the task execution cycle;

[0181] Step 3: Determine the deadline corresponding to the periodic data collection task as the deadline corresponding to the periodic data collection job.

[0182] For the convenience of description, the above three steps can be combined for explanation.

[0183] When it is determined that the periodic data collection job is a discrete data collection job according to the periodic job type, the task execution cycle corresponding to the periodic data collection task to which the periodic data collection job belongs is obtained, the deadline corresponding to the periodic data collection task is determined according to the task execution cycle, and the deadline corresponding to the periodic data collection task is determined as the deadline corresponding to the periodic data collection job. If the task execution cycle of the periodic data collection task to which the periodic data collection job belongs is 1 hour, then when the periodic data collection job is a discrete data collection job, the deadline corresponding to the periodic data collection job is determined to be the moment 1 hour after the execution of the periodic data collection task begins. By determining the deadline corresponding to the periodic data collection task to which the periodic data collection job belongs as the deadline corresponding to the periodic data collection job when the periodic data collection job is a discrete data collection job, while satisfying the smooth execution of discrete data collection jobs that do not have high periodicity requirements, the impact of uneven task running time is reduced, and the rationality of task scheduling is improved.

[0184] S206: Obtain the cache duration of each periodic data collection job in the periodic job queue respectively, and determine the cache duration as the waiting time.

[0185] A periodic job queue is created in advance. After each periodic data collection job is obtained by sharding, each periodic data collection job is cached in the periodic job queue. The cache duration of each periodic data collection job in the periodic job queue is obtained respectively, and the cache duration is determined as the waiting time.

[0186] S207: Determine the estimated execution time corresponding to each periodic data collection job according to the historical scheduling data of each periodic data collection job.

[0187] Since the periodic data collection operation is executed by polling, when it is not the first time to execute the periodic data collection operation, there will be corresponding historical scheduling data, and the historical scheduling data corresponding to each periodic data collection operation is obtained.

[0188] After obtaining the historical scheduling data corresponding to each periodic data collection job, the estimated execution time corresponding to each periodic data collection job is determined according to the historical scheduling data of each periodic data collection job. By using the historical scheduling data of each periodic data collection job as a reference to calculate the estimated execution time, an effective basis is provided for calculating the estimated execution time of each periodic data collection job, thereby improving the accuracy of the calculated estimated execution time.

[0189] In a specific implementation of the present invention, step S207 may include the following steps:

[0190] Step 1: When it is determined that there is historical scheduling data, the actual execution time corresponding to each historical scheduling order of the periodic data collection job is determined according to the historical scheduling data;

[0191] Step 2: Determine the total scheduling times of periodic data collection jobs based on historical scheduling data;

[0192] Step 3: Calculate the estimated execution time based on each historical scheduling order, the actual execution time corresponding to each historical scheduling order, and the total number of scheduling times.

[0193] For the convenience of description, the above three steps can be combined for explanation.

[0194] When it is determined that there is historical scheduling data, the historical scheduling order of each periodic data collection job is determined according to the historical scheduling data The actual execution time corresponding to each , determine the total scheduling times n of periodic data collection jobs based on historical scheduling data, and , the actual execution time corresponding to each historical scheduling order And the total number of scheduling times n calculate the estimated execution time. For example, the estimated execution time can be calculated using the following formula :

[0195] ;

[0196] ;

[0197] Among them, when When , it indicates the first scheduling of the job. At this time, there is no statistical record of the job in the job statistics center. Therefore, the set empirical value N is used as the estimated execution time of the job. N is a constant and can be specified in the form of a configuration file. When When , linear regression is used to predict the expected execution time, where is the linear regression coefficient.

[0198] By using the historical scheduling order of each periodic data collection job as a reference to calculate the estimated execution time, an effective basis is provided for calculating the estimated execution time of each periodic data collection job, thereby improving the accuracy of the calculated estimated execution time.

[0199] In a specific implementation of the present invention, step S207 may include the following steps:

[0200] Step 1: When it is determined that there is no historical scheduling data, a preset experience value corresponding to the periodic data collection operation is obtained;

[0201] Step 2: Determine the preset experience value as the estimated execution time corresponding to the periodic data collection job.

[0202] For the convenience of description, the above two steps can be combined for explanation.

[0203] When it is determined that there is no historical scheduling data, it means that the periodic data collection job is executed for the first time, and the preset experience value corresponding to the periodic data collection job is obtained, and the preset experience value is determined as the estimated execution time corresponding to the periodic data collection job. By setting the estimated execution time of the periodic data collection job when it is first executed based on experience, the accuracy of determining the estimated execution time of the periodic data collection job executed for the first time is improved.

[0204] S208: Score each periodic data collection job according to the deadline, waiting time and estimated execution time to obtain a scoring result.

[0205] After obtaining the deadlines corresponding to each periodic data collection task , Waiting time and estimated execution time Afterwards, score each periodic data collection job according to the deadline, waiting time, and estimated execution time to obtain a scoring result. For example, the following formula can be used to score each periodic data collection job:

[0206] ;

[0207] ;

[0208] ;

[0209] in, represents the estimated execution time, R represents the scoring result, T represents the task execution cycle, Indicates the deadline, Indicates the waiting time. Represents the time coefficient of the task, which is represented by the above piecewise function.

[0210] S209: Selecting the periodic data collection task with the highest score from among the periodic data collection tasks according to the scoring results.

[0211] After obtaining the scoring results corresponding to each periodic data collection operation, the periodic data collection operation with the highest score is selected from the periodic data collection operations according to the scoring results.

[0212] S210: Obtain the sum of the estimated execution time of each periodic data collection job in the execution queue corresponding to each periodic executor.

[0213] Each periodic executor has its own corresponding execution queue. The sum of the estimated execution time of each periodic data collection job in the execution queue corresponding to each periodic executor is obtained, which is recorded as .

[0214] S211: Select the maximum value of the sum of the estimated execution time of each job.

[0215] After obtaining the sum of the estimated execution time of the jobs corresponding to each periodic executor After that, the maximum value of the sum of the estimated execution time of each job is selected and recorded as .

[0216] S212: Calculate the difference between the total estimated execution time of each job and the maximum value.

[0217] The maximum value of the sum of the estimated execution time of each job is selected After that, calculate the total estimated execution time of each job With the maximum value The difference is recorded as .

[0218] S213: Calculate the absolute value of the difference between the estimated execution time corresponding to the periodic data collection job with the highest score and the difference.

[0219] Calculate the total estimated execution time of each job separately With the maximum value After calculating the difference between the two, we can calculate the estimated execution time of the periodic data collection job with the highest score. With each difference The absolute value of the difference between .

[0220] S214: Allocate the periodic data collection job with the highest score to the periodic executor corresponding to the smallest absolute value of the difference.

[0221] After calculating the absolute value of the difference between the estimated execution time corresponding to the highest-scoring periodic data collection job and each difference, the highest-scoring periodic data collection job is assigned to the periodic executor corresponding to the smallest absolute value of the difference. By assigning the periodic data collection job to the periodic executor with the smallest estimated execution time difference, the compatibility between the periodic executor and the periodic data collection job is improved, and resource utilization is improved.

[0222] In a specific implementation of the present invention, step S214 may include the following steps:

[0223] The periodic data collection job with the highest score is cached to the execution queue of the periodic executor corresponding to the smallest absolute difference value, so that the periodic executor corresponding to the smallest absolute difference value processes the job according to the arrangement order of the periodic data collection jobs in the execution queue.

[0224] By caching the highest-scoring periodic data collection job to the execution queue of the periodic executor corresponding to the smallest absolute difference value, the periodic executor corresponding to the smallest absolute difference value can process the job in the order of arrangement of the periodic data collection jobs in the execution queue. By processing the job in the order of arrangement of the periodic data collection jobs in the execution queue, the orderliness of the execution of the periodic data collection jobs is improved, resource competition among the periodic data collection jobs is avoided, and the task scheduling efficiency is improved.

[0225] S215: Determine whether there is no idle periodic executor or whether all periodic data collection jobs have been scheduled. If so, end the current task scheduling; if not, return to step S211.

[0226] After assigning the current highest-scoring periodic data collection job to the periodic executor corresponding to the smallest absolute value of the difference, determine whether there are no idle periodic executors or whether all periodic data collection jobs have been scheduled. If so, it means that there are currently no periodic data collection jobs to be executed or no idle periodic executors, and the current task scheduling is terminated. If not, it means that there are still periodic data collection jobs to be executed, and the screening and execution of the periodic data collection job with the highest score continues.

[0227] See also Figure 3 , Figure 3The structure diagram of a task scheduling system in an embodiment of the present invention is shown in FIG. The task scheduling system includes four modules, namely: 1) Task center module. The task center module is the entrance to the entire data acquisition task, which is used to create a data acquisition task and divide it into separately executable data acquisition jobs; 2) Job queue module. The job queue module is used to store jobs to be executed. After the data acquisition task is divided into acquisition jobs, the job queue will be managed uniformly. The job queue module includes two queues, a real-time queue and a periodic queue, and a job scoring module. The real-time queue and the periodic queue are used to store real-time jobs and periodic jobs waiting to be executed after segmentation, respectively. The job scoring module scores according to the time parameters of the tasks in the two queues. 3) Scheduling center module. The scheduling center module assigns jobs in the job queue to the executor according to the set scheduling strategy. The executor is the actual executor of the job. In addition to executing the task, the executor is also responsible for monitoring the execution status of the job and recording the actual execution time of the job. 4) Data statistics module. The data statistics module is used to record the execution time of the job in the historical scheduling and the abnormal status of the job in the entire functional process. It can also record the status changes of the job in the entire scheduling process, such as logging the creation of tasks, task slicing, job distribution, etc. When the data statistics module is recording, it is based on the task representation. Record the same tasks uniformly and sort them according to the job ID Keep unified records for the same operations.

[0228] Corresponding to the above method embodiment, the present invention further provides a task scheduling device. The task scheduling device described below and the task scheduling method described above can be referred to each other.

[0229] See also Figure 4 , Figure 4 This is a structural block diagram of a task scheduling device in an embodiment of the present invention, and the device may include:

[0230] The real-time job scheduling module 41 is used to process the real-time data collection tasks in slices to obtain various real-time data collection jobs, and schedule the various real-time data collection jobs using various real-time executors;

[0231] The periodic job acquisition module 42 is used to perform segmentation processing on the periodic data collection task to obtain each periodic data collection job;

[0232] The time acquisition module 43 is used to obtain the deadline, waiting time and estimated execution time corresponding to each periodic data collection job;

[0233] The scoring result obtaining module 44 is used to score each periodic data collection job according to the deadline, waiting time and estimated execution time to obtain a scoring result;

[0234] The job scheduling module 45 is used to schedule the data collection jobs of each period according to the scoring results by using each period executor.

[0235] It can be seen from the above technical solution that by slicing the data collection tasks and setting the corresponding deadline, waiting time and estimated execution time for each periodic data collection job obtained by the slicing, the tasks can be evenly distributed in their scheduling cycles, reducing the load pressure of task operation, reducing the impact of uneven task running time, improving the rationality of task scheduling, and effectively improving resource utilization. By uniformly scoring different tasks, unified management of different tasks is achieved. By uniformly managing data collection tasks, balanced scheduling of data collection tasks is achieved. Under the same data center scale, the system's hardware conditions can be more fully utilized, reducing the system's operating pressure and improving system performance.

[0236] In a specific implementation of the present invention, the time acquisition module 43 may include:

[0237] The job adding submodule is used to add each periodic data collection job to the periodic job queue;

[0238] The job type acquisition submodule is used to obtain the periodic job type to which each periodic data collection job belongs;

[0239] The deadline determination submodule is used to determine the deadline corresponding to each periodic data collection job according to the periodic job type;

[0240] The waiting time determination submodule is used to respectively obtain the cache time length of each periodic data collection job in the periodic job queue and determine the cache time length as the waiting time;

[0241] The estimated execution time determination submodule is used to determine the estimated execution time corresponding to each periodic data collection job according to the historical scheduling data of each periodic data collection job.

[0242] In a specific implementation of the present invention, the deadline determination submodule may include:

[0243] A flow-limiting job total number acquisition unit, used to acquire the total number of flow-limiting jobs included in the periodic job queue when the periodic data collection job is determined to be a flow-limiting data collection job according to the periodic job type;

[0244] A flow-limiting operation number acquisition unit, used to acquire the flow-limiting operation number of the periodic data collection operation;

[0245] A first task execution cycle acquisition unit is used to acquire a task execution cycle corresponding to a periodic data collection task to which a periodic data collection job belongs;

[0246] The deadline calculation unit is used to calculate the deadline corresponding to the periodic data collection job according to the total number of current limiting jobs, the current limiting job number and the task execution period.

[0247] In a specific implementation of the present invention, the deadline determination submodule may include:

[0248] A second task execution cycle acquisition unit is used to acquire a task execution cycle corresponding to the periodic data collection task to which the periodic data collection job belongs when the periodic data collection job is determined to be a discrete data collection job according to the periodic job type;

[0249] A task deadline determination unit, used to determine the deadline corresponding to the periodic data collection task according to the task execution cycle;

[0250] The job deadline determination unit is used to determine the deadline corresponding to the periodic data collection task as the deadline corresponding to the periodic data collection job.

[0251] In a specific implementation of the present invention, the estimated execution time determination submodule may include:

[0252] An actual execution time acquisition unit, for determining the actual execution time corresponding to each historical scheduling order of the periodic data collection job according to the historical scheduling data when it is determined that there is historical scheduling data;

[0253] A total scheduling times acquisition unit is used to determine the total scheduling times of periodic data collection operations according to historical scheduling data;

[0254] The estimated execution time calculation unit is used to calculate the estimated execution time according to each historical scheduling order, the actual execution time corresponding to each historical scheduling order and the total scheduling times.

[0255] In a specific implementation of the present invention, the estimated execution time determination submodule may include:

[0256] A preset experience value acquisition unit, used to acquire a preset experience value corresponding to the periodic data collection operation when it is determined that there is no historical scheduling data;

[0257] The estimated execution time determination unit is used to determine the preset experience value as the estimated execution time corresponding to the periodic data collection job.

[0258] In a specific embodiment of the present invention, the job scheduling module 45 may include:

[0259] The periodic job selection submodule is used to select the periodic data collection job with the highest score from each periodic data collection job according to each scoring result;

[0260] The estimated execution time sum acquisition submodule is used to obtain the estimated execution time sum of each periodic data collection job in the execution queue corresponding to each periodic executor;

[0261] The maximum sum selection submodule is used to select the maximum value of the sum of the estimated execution time of each job;

[0262] The difference calculation submodule is used to calculate the difference between the sum of the estimated execution time of each job and the maximum value;

[0263] The difference absolute value calculation submodule is used to calculate the absolute value of the difference between the estimated execution time corresponding to the periodic data collection job with the highest score and the difference;

[0264] The job allocation submodule is used to allocate the periodic data collection job with the highest score to the periodic executor corresponding to the smallest absolute value of the difference;

[0265] The repeated execution submodule is used to repeatedly execute the step of selecting the periodic data collection job with the highest score from each periodic data collection job according to each scoring result until there is no idle periodic executor or each periodic data collection job is scheduled to be completed.

[0266] In a specific embodiment of the present invention, the job allocation submodule is specifically a module that caches the periodic data collection job with the highest score to the execution queue of the periodic executor corresponding to the smallest absolute value of the difference, so that the periodic executor corresponding to the smallest absolute value of the difference performs job processing according to the arrangement order of each periodic data collection job in the execution queue.

[0267] In a specific embodiment of the present invention, the device may further include:

[0268] The actual execution time statistics module is used to count the actual execution time corresponding to each periodic data collection job after scheduling each periodic data collection job according to each scoring result using each periodic executor;

[0269] The actual execution time recording module is used to record the actual execution time counted.

[0270] In a specific embodiment of the present invention, the device may further include:

[0271] A real-time operation identification generation module is used to generate real-time operation identifications corresponding to each real-time data collection operation after obtaining each real-time data collection operation;

[0272] A real-time task identification acquisition module is used to acquire the real-time task identification of the real-time data collection task to which each real-time data collection operation belongs;

[0273] The real-time identification filling module is used to fill the real-time operation identification and the real-time task identification into each real-time data collection operation.

[0274] In a specific embodiment of the present invention, the device may further include:

[0275] The job search module is used to search whether there is a real-time data collection job in the real-time job queue according to the real-time job identifier and the real-time task identifier corresponding to each real-time data collection job before using each real-time executor to schedule each real-time data collection job;

[0276] A repeated job determination module, used for determining the real-time data collection job as a repeated job when it is determined that there is a real-time data collection job in the real-time job queue;

[0277] The first job adding module is used to add the real-time data collection job to the real-time job queue when it is determined that there is no real-time data collection job in the real-time job queue.

[0278] In a specific embodiment of the present invention, the device may further include:

[0279] A periodic operation identifier generation module is used to generate periodic operation identifiers corresponding to each periodic data collection operation after obtaining each periodic data collection operation;

[0280] A periodic task identification acquisition module is used to acquire the periodic task identification of the periodic data collection task to which each periodic data collection job belongs;

[0281] The cycle identification filling module is used to fill the cycle job identification and cycle task identification into each cycle data collection operation.

[0282] In a specific embodiment of the present invention, the device may further include:

[0283] A periodic job search module is used to search whether there is a periodic data collection job in the periodic job queue according to the periodic job identifier and periodic task identifier respectively corresponding to each periodic data collection job after filling the periodic job identifier and periodic task identifier into each periodic data collection job;

[0284] The job migration module is used to migrate the periodic data collection jobs in the periodic job queue to the real-time job queue when there are periodic data collection jobs in the periodic job queue;

[0285] The second job adding module is used to add the periodic data collection job to the periodic job queue when there is no periodic data collection job in the periodic job queue.

[0286] In a specific implementation of the present invention, the real-time job scheduling module 41 includes:

[0287] The real-time job adding submodule is used to add each real-time data collection job to the real-time job queue;

[0288] The real-time job scheduling submodule is used to schedule the real-time data collection jobs in the real-time job queue using the real-time executors according to the load balancing principle.

[0289] In a specific implementation of the present invention, the periodic job acquisition module 42 is specifically a module for performing slice processing on the periodic data collection tasks according to resource dimensions.

[0290] In a specific embodiment of the present invention, the device may further include:

[0291] The job processing process monitoring module is used to monitor the processing process of each periodic data collection job after scheduling each periodic data collection job according to each scoring result using each periodic executor;

[0292] The abnormal status recording module is used to record the abnormal status when a periodic data collection operation in an abnormal status is detected.

[0293] Corresponding to the above method embodiment, see Figure 5 , Figure 5 This is a schematic diagram of a task scheduling device provided by the present invention, and the device may include:

[0294] A memory 332, for storing computer programs;

[0295] The processor 322 is used to implement the steps of the task scheduling method of the above method embodiment when executing a computer program.

[0296] For details, please refer to Figure 6 , Figure 6 A specific structural diagram of a task scheduling device provided for this embodiment, which may have relatively large differences due to different configurations or performances, may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, wherein the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 may be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the task scheduling device 301.

[0297] The task scheduling device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0298] The steps in the task scheduling method described above can be implemented by the structure of the task scheduling device.

[0299] Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0300] The real-time data collection task is processed in slices to obtain each real-time data collection job, and each real-time executor is used to schedule each real-time data collection job; the periodic data collection task is processed in slices to obtain each periodic data collection job; the deadline, waiting time and estimated execution time corresponding to each periodic data collection job are obtained; each periodic data collection job is scored according to the deadline, waiting time and estimated execution time to obtain the scoring result; each periodic executor is used to schedule each periodic data collection job according to the scoring result.

[0301] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0302] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0303] Corresponding to the above method embodiment, the present invention also provides a computer program product, including a computer program, which implements the steps of the above task scheduling method when executed by a processor.

[0304] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment, computer-readable storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0305] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A task scheduling method, It is characterized in that include: The real-time data collection task is processed in pieces to obtain various real-time data collection jobs, and each real-time executor is used to schedule each real-time data collection job; The periodic data collection task is processed in pieces to obtain the data collection jobs for each period; Get the deadline, waiting time and estimated execution time corresponding to each periodic data collection job; Scoring each periodic data collection job according to the deadline, the waiting time, and the estimated execution time to obtain a scoring result; Use each periodic executor to schedule the data collection jobs of each period according to the scoring results; The deadline, waiting time, and estimated execution time corresponding to each periodic data collection job are obtained, including: Add each periodic data collection job to the periodic job queue; Obtain the periodic job type to which each periodic data collection job belongs; Determine the deadline corresponding to each periodic data collection operation according to the periodic operation type; Respectively obtain the cache time length of each periodic data collection job in the periodic job queue, and determine the cache time length as the waiting time; Determine the estimated execution time corresponding to each periodic data collection job according to the historical scheduling data of each periodic data collection job; Each periodic executor is used to schedule the data collection jobs of each period according to the scoring results, including: According to the scoring results, the cycle data collection task with the highest score is selected from each cycle data collection task; Obtain the sum of the estimated execution time of each periodic data collection job in the execution queue corresponding to each periodic executor; Select the maximum value of the sum of the estimated execution time of each job; Calculate the difference between the sum of the estimated execution time of each job and the maximum value respectively; Calculate the absolute value of the difference between the estimated execution time corresponding to the periodic data collection job with the highest score and the difference; Allocating the periodic data collection job with the highest score to the periodic executor corresponding to the smallest absolute value of the difference; Repeat the step of selecting the periodic data collection job with the highest score from each periodic data collection job according to each score result until there is no idle periodic executor or each periodic data collection job is scheduled to be completed; Also includes: Searching whether the periodic data collection job exists in the periodic job queue according to the periodic job identifier and the periodic task identifier respectively corresponding to each periodic data collection job; If yes, migrating the periodic data collection job in the periodic job queue to the real-time job queue; If not, the periodic data collection job is added to the periodic job queue.

2. The task scheduling method according to claim 1, It is characterized in that Determining the deadlines corresponding to the respective periodic data collection operations according to the periodic operation type includes: When it is determined according to the periodic job type that the periodic data collection job is a flow-limited data collection job, obtaining the total number of flow-limited jobs included in the periodic job queue; Obtaining the current limiting operation number of the periodic data collection operation; Obtaining a task execution period corresponding to a periodic data collection task to which the periodic data collection job belongs; The deadline corresponding to the periodic data collection job is calculated according to the total number of the current limiting jobs, the current limiting job number and the task execution period.

3. The task scheduling method according to claim 1, It is characterized in that Determining the deadlines corresponding to the respective periodic data collection operations according to the periodic operation type includes: When it is determined according to the periodic job type that the periodic data collection job is a discrete data collection job, obtaining a task execution period corresponding to the periodic data collection task to which the periodic data collection job belongs; Determine the deadline corresponding to the periodic data collection task according to the task execution cycle; The deadline corresponding to the periodic data collection task is determined as the deadline corresponding to the periodic data collection job.

4. The task scheduling method according to claim 1, It is characterized in that The estimated execution time corresponding to each periodic data collection job is determined based on the historical scheduling data of each periodic data collection job, including: When it is determined that there is historical scheduling data, determining the actual execution time corresponding to each historical scheduling order of the periodic data collection job according to the historical scheduling data; Determine the total scheduling times of the periodic data collection operation according to the historical scheduling data; The estimated execution time is calculated according to each historical scheduling order, the actual execution time corresponding to each historical scheduling order, and the total scheduling times.

5. The task scheduling method according to claim 1, It is characterized in that The estimated execution time corresponding to each periodic data collection job is determined based on the historical scheduling data of each periodic data collection job, including: When it is determined that there is no historical scheduling data, obtaining a preset experience value corresponding to the periodic data collection operation; The preset experience value is determined as the estimated execution time corresponding to the periodic data collection operation.

6. The task scheduling method according to claim 1, It is characterized in that Allocating the periodic data collection job with the highest score to the periodic executor corresponding to the smallest absolute value of the difference includes: The periodic data collection job with the highest score is cached in the execution queue of the periodic executor corresponding to the smallest absolute difference value, so that the periodic executor corresponding to the smallest absolute difference value performs job processing according to the arrangement order of each periodic data collection job in the execution queue.

7. The task scheduling method according to claim 1, It is characterized in that After scheduling the data collection jobs of each period according to the scoring results by using each period executor, it also includes: Statistics are collected on the actual execution time corresponding to each periodic data collection operation; The actual execution time counted is recorded.

8. The task scheduling method according to claim 1, It is characterized in that After obtaining each real-time data collection operation, it also includes: Generate real-time operation identifiers corresponding to each real-time data collection operation; Obtaining the real-time task identifier of the real-time data collection task to which each real-time data collection job belongs; The real-time job identifier and the real-time task identifier are filled into each real-time data collection job.

9. The task scheduling method according to claim 1, It is characterized in that Before using each real-time executor to schedule each real-time data collection job, it also includes: Searching whether the real-time data collection job exists in the real-time job queue according to the real-time job identifier and the real-time task identifier respectively corresponding to each real-time data collection job; When it is determined that the real-time data collection job exists in the real-time job queue, determining the real-time data collection job as a repeated job; When it is determined that the real-time data collection job does not exist in the real-time job queue, the real-time data collection job is added to the real-time job queue.

10. The task scheduling method according to claim 1, It is characterized in that After obtaining the data collection operations for each cycle, it also includes: Generate periodic operation identifiers corresponding to each periodic data collection operation; Obtaining the periodic task identifier of the periodic data collection task to which each periodic data collection job belongs; The periodic job identifier and the periodic task identifier are filled into each periodic data collection job.

11. The task scheduling method according to claim 1, It is characterized in that Use each real-time executor to schedule each real-time data collection job, including: Add each real-time data collection job to the real-time job queue; The real-time data collection jobs in the real-time job queue are scheduled using the real-time executors according to the load balancing principle.

12. The task scheduling method according to claim 1, It is characterized in that The periodic data collection task is processed in slices, including: The periodic data collection task is processed in slices according to resource dimensions.

13. The task scheduling method according to claim 1, It is characterized in that After scheduling the data collection jobs of each period according to the scoring results by using each period executor, it also includes: Monitor the processing of data collection operations in each cycle; When a periodic data collection operation is detected to be in an abnormal state, the abnormal state record is made.

14. A task scheduling device, It is characterized in that include: The real-time job scheduling module is used to process the real-time data collection tasks in slices to obtain various real-time data collection jobs, and use various real-time executors to schedule various real-time data collection jobs; The periodic job acquisition module is used to process the periodic data collection tasks in slices to obtain the data collection jobs of each period; The time acquisition module is used to obtain the deadline, waiting time and estimated execution time corresponding to each periodic data collection job; A scoring result obtaining module, used to score each periodic data collection job according to the deadline, the waiting time and the estimated execution time to obtain a scoring result; The job scheduling module is used to schedule the data collection jobs of each period according to the scoring results by using the executors of each period; The time acquisition module includes: The job adding submodule is used to add each periodic data collection job to the periodic job queue; The job type acquisition submodule is used to obtain the periodic job type to which each periodic data collection job belongs; The deadline determination submodule is used to determine the deadline corresponding to each periodic data collection job according to the periodic job type; The waiting time determination submodule is used to respectively obtain the cache time length of each periodic data collection job in the periodic job queue and determine the cache time length as the waiting time; The estimated execution time determination submodule is used to determine the estimated execution time corresponding to each periodic data collection job according to the historical scheduling data of each periodic data collection job; The job scheduling module includes: The periodic job selection submodule is used to select the periodic data collection job with the highest score from each periodic data collection job according to each scoring result; The estimated execution time sum acquisition submodule is used to obtain the estimated execution time sum of each periodic data collection job in the execution queue corresponding to each periodic executor; The maximum sum selection submodule is used to select the maximum value of the sum of the estimated execution time of each job; The difference calculation submodule is used to calculate the difference between the sum of the estimated execution time of each job and the maximum value; The difference absolute value calculation submodule is used to calculate the absolute value of the difference between the estimated execution time corresponding to the periodic data collection job with the highest score and the difference; The job allocation submodule is used to allocate the periodic data collection job with the highest score to the periodic executor corresponding to the smallest absolute value of the difference; A repeated execution submodule is used to repeatedly execute the step of selecting the periodic data collection job with the highest score from each periodic data collection job according to each scoring result, until there is no idle periodic executor or each periodic data collection job is scheduled to be completed; Also includes: A periodic job search module is used to search whether there is a periodic data collection job in the periodic job queue according to the periodic job identifier and periodic task identifier corresponding to each periodic data collection job; The job migration module is used to migrate the periodic data collection jobs in the periodic job queue to the real-time job queue when there are periodic data collection jobs in the periodic job queue; The second job adding module is used to add the periodic data collection job to the periodic job queue when there is no periodic data collection job in the periodic job queue.

15. A task scheduling device, It is characterized in that include: Memory for storing computer programs; A processor, configured to implement the steps of the task scheduling method according to any one of claims 1 to 13 when executing the computer program.

16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task scheduling method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the task scheduling method as claimed in any one of claims 1 to 13 are implemented.

Images