A method, device, equipment and storage medium for resource scheduling

By obtaining and utilizing job scores and preset machine configuration strategies, the target jobs and configuring the executors are determined, which solves the problems of starvation of low-priority jobs and low resource scheduling efficiency, and achieves timely scheduling of candidate jobs and efficient resource resources use.

CN112181645BActive Publication Date: 2024-11-19CHINA CONSTRUCTION BANK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010994712.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-21
Publication Date
2024-11-19
Estimated Expiration
2040-09-21

AI Technical Summary

Technical Problem

In the multi-tenant environment of public cloud platform, low-priority jobs are easily starved to death, and the existing technology is difficult to allocate jobs based on waiting time, resulting in low resource scheduling efficiency and inability to process all jobs in a timely manner.

Method used

By obtaining the job scores of the pending task queue, combining priority and time information for inserting the queue, the target job is determined, and the target execution machine is configured according to the preset machine configuration strategy, the timely scheduling of candidate jobs and the rational use of resources is achieved. .

Benefits of technology

It solves the problem of low-priority jobs starving to death, reduces job waiting time, improves resource scheduling efficiency, makes full use of machine resources, and ensures timely processing of all jobs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112181645B_ABST
    Figure CN112181645B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for resource scheduling. The method includes: obtaining a job score of a pending task queue; wherein the job score is used to represent the job priority of a candidate job to be processed and the time information of inserting the candidate job into the pending task queue; determining a target job to be processed according to the job score; and determining a target execution machine of the target job according to a preset machine configuration policy to complete resource scheduling of the target job. The embodiment of the present invention determines the target job by obtaining the job score of the candidate job, and considers the priority and time in the job score to avoid that jobs with high priority or long waiting time are not processed in time, and performs resource scheduling according to the preset machine configuration policy to improve the efficiency of resource scheduling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and in particular to a resource scheduling method, apparatus, device and storage medium. Background Art

[0002] In a multi-tenant environment of a public cloud platform, as the number of tenants increases, the number of tasks running daily can reach more than one million. Due to limited computing resources, a large number of jobs will be waiting for resources. Therefore, it is necessary to store a large number of jobs while being able to process them quickly.

[0003] In the prior art, jobs can be assigned priorities, and high-priority jobs are processed first, while low-priority jobs are left in a pending state for a long time. Low-priority jobs are easily starved, and it is impossible to schedule jobs according to waiting time, making it difficult to ensure that all jobs are processed in a timely manner. There is no targeted machine configuration when scheduling each job, and the efficiency of resource scheduling is low. Summary of the invention

[0004] The embodiments of the present invention provide a resource scheduling method, apparatus, device and storage medium to improve the efficiency of resource scheduling.

[0005] In a first aspect, an embodiment of the present invention provides a resource scheduling method, the method comprising:

[0006] Obtaining a job score of a pending task queue; wherein the job score is used to indicate the job priority of the pending candidate job and the time information of inserting the pending task queue;

[0007] Determine the target job to be processed according to the job score;

[0008] According to a preset machine configuration strategy, a target execution machine of the target job is determined to complete resource scheduling of the target job.

[0009] In a second aspect, an embodiment of the present invention further provides a resource scheduling device, the device comprising:

[0010] A score acquisition module, used to acquire the job score of the pending task queue; wherein the job score is used to indicate the job priority of the candidate job to be processed and the time information of inserting the candidate job into the pending task queue;

[0011] A job determination module, used to determine a target job to be processed according to the job score;

[0012] The execution machine determination module is used to determine the target execution machine of the target job according to a preset machine configuration strategy to complete the resource scheduling of the target job.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the resource scheduling method as described in any embodiment of the present invention is implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the resource scheduling method as described in any embodiment of the present invention.

[0015] The embodiment of the present invention obtains the job scores of candidate jobs in the queue of pending tasks, determines the target job from the candidate jobs according to the priority in the job scores and the time information of the insertion into the queue, configures the target execution machine for the target job according to the preset machine configuration strategy, and completes the resource scheduling of the target job. The problem of the prior art that candidate jobs are processed only according to the priority, resulting in the starvation of low-priority jobs, realizes the timely scheduling of candidate jobs, reduces the job waiting time, makes full use of machine resources according to the machine configuration strategy, and improves the resource scheduling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a resource scheduling method in Embodiment 1 of the present invention;

[0017] Figure 2 It is a flowchart of a resource scheduling method in Embodiment 2 of the present invention;

[0018] Figure 3 It is a structural block diagram of a resource scheduling device in Embodiment 3 of the present invention;

[0019] Figure 4 It is a structural diagram of a computer device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0021] Embodiment 1

[0022] Figure 1 This is a flow chart of a resource scheduling method provided in the first embodiment of the present invention. This embodiment is applicable to the case of scheduling multiple jobs. The method can be executed by a resource scheduling device. Figure 1 As shown, the method specifically comprises the following steps:

[0023] Step 110: Obtain the job score of the pending task queue; wherein the job score is used to indicate the job priority of the pending candidate job and the time information of inserting into the pending task queue.

[0024] Wherein, the candidate jobs to be processed are stored in the queue of pending tasks, and when the candidate jobs to be processed are processed, the candidate jobs are removed from the queue of pending tasks. Each candidate job is matched with a job score in the queue of pending tasks, and the job score is associated with the unique identifier of the candidate job. The unique identifier of the candidate job can be determined by information such as the application time and network card address of the candidate job. For example, UUID (Universally Unique Identifier) ​​can be obtained through Java as a unique identifier. First, a unique identifier is generated, and when the candidate job is stored in the queue of pending tasks, a job score is generated, and the job score is associated with the unique identifier and stored in the queue of pending tasks. Wherein, the job score can be used to represent the job priority of the candidate job to be processed and the time information of the candidate job being inserted into the queue of pending tasks. According to the time information of the candidate job being inserted into the queue of pending tasks, the time when the candidate job is inserted into the queue of pending tasks and the waiting time for the candidate job to be processed in the queue of pending tasks can be determined. The job priority can include 1 to 99 levels, and the larger the numerical value of the job priority, the higher the priority. In the job score, the larger the value representing the time information when the candidate job is inserted into the queue of pending tasks, the earlier the candidate job is inserted into the queue of pending tasks.

[0025] In this embodiment, optionally, the job score is a string containing 15 digits, wherein the first and second digits represent the job priority, and the third to fifteenth digits represent the time information when the candidate job is inserted into the queue of pending tasks.

[0026] Specifically, the job score can be represented by a string of numbers, and the job score can include 15 values. Among them, the first and second digits represent the job priority, which can be numbers between 1 and 99, for example, 05 or 77. The third to fifteenth digits represent the time information of the candidate job inserted into the queue of pending tasks, and the value of the time information can be the difference between the time when the candidate job is inserted into the queue of pending tasks and the preset time point. The more digits representing the value of the time information, the more accurate the calculation result of the time information, for example, nanoseconds or picoseconds can be used to calculate the time information. The beneficial effect of such a setting is that, according to the priority level, high-priority jobs can be processed first to avoid important jobs being delayed. According to the time information of the candidate job inserted into the queue of pending tasks, the time waiting for the candidate job to be processed can be obtained, which avoids the candidate job being not processed for a long time, causing the job to be starved or missed, and improves the efficiency of each job being processed, thereby improving the efficiency of resource scheduling.

[0027] In this embodiment, optionally, obtaining the job score of the pending task queue includes: determining the time difference between the time when the candidate job is inserted into the pending task queue and the preset time point based on the preset time point; generating the job score of the candidate job in the pending task queue based on the preset job priority and the time difference.

[0028] Specifically, the priority of the candidate job is predetermined, and the time information of inserting into the pending task queue is represented by the time difference between the time when the candidate job is inserted into the pending task queue and the preset time point. The preset time point can be a certain time point in the future. The larger the time difference between the time when the candidate job is inserted into the pending task queue and the preset time point, the earlier the candidate job is inserted into the pending task queue, and the more likely the candidate job is to be processed. For example, the preset time point can be 0 hours, 0 minutes, and 0 seconds on January 1, 2100. Assemble the preset job priority and the calculated time difference to obtain a complete job score. The beneficial effect of such a setting is that, according to the priority and time difference, the priority and waiting time of the job are taken into account, so that the high-priority job is scheduled as soon as possible, and the low-priority job will not be scheduled without the opportunity. The order of processing the candidate jobs is comprehensively considered, and the job priority is given priority. If the job priority is consistent, the time information of the candidate job inserted into the pending task queue is considered. In other words, the larger the job score is, the higher the priority it will be. Jobs with large job scores will be prioritized for resource scheduling, which avoids candidate jobs waiting too long or missing candidate jobs, and improves job processing efficiency during resource scheduling.

[0029] Step 120: Determine the target job to be processed according to the job score.

[0030] After obtaining the job scores, the job scores are compared and the candidate job that meets the preset requirements is selected as the target job. For example, the candidate job with the largest job score can be selected as the target job.

[0031] In this embodiment, optionally, determining the target job to be processed according to the job score includes: sorting the candidate jobs in the task queue to be processed according to the numerical values ​​of the job scores; and determining the candidate job whose sorting results meet preset requirements as the target job.

[0032] Specifically, the job score is generated when the candidate job is to be inserted into the queue of pending tasks. After the job score is generated, the numerical value of the job score is compared with the numerical value of the job score stored in the queue of pending tasks to obtain a sorting result from large to small. The job score is stored in the queue of pending tasks according to the sorting result. The larger the job score, the closer it is to the head of the queue, that is, the higher the priority and the earlier the candidate job is inserted into the queue, the closer it is to the head of the queue. The preset requirement can be to obtain the candidate job with the maximum job score. When obtaining the target job, it starts from the head of the queue, that is, the candidate job that meets the preset requirement is obtained. The beneficial effect of such a setting is that according to the numerical comparison of the job score, the job that needs to be processed first can be quickly obtained. When the job score is inserted into the queue of pending tasks, the numerical comparison is performed. When obtaining the target job, the candidate job at the head of the queue can be directly obtained, which improves the efficiency of obtaining the target job, so as to give priority to resource scheduling for the target job, which is conducive to reasonable and effective regulation and use of resources to meet the processing requirements of each job.

[0033] Step 130: Determine the target execution machine of the target job according to the preset machine configuration strategy to complete the resource scheduling of the target job.

[0034] After the target job is determined, resources are scheduled and processed for the target job, at least one target execution machine is configured for the target job, and the target job is sent to the target execution machine, which processes the target job. The target execution machine of the target job can be determined according to a preset machine configuration strategy. For example, the target job can be associated with the target execution machine to specify the target execution machine of the target job.

[0035] In this embodiment, optionally, before determining the target execution machine of the target job according to the preset machine configuration policy, it also includes: comparing the target logical resource amount of the target job and the global logical resource amount remaining in the scheduling system; if the target logical resource amount is greater than the global logical resource amount, continuing to obtain new target jobs in the pending task queue for scheduling, and storing the target job in the waiting list, and re-acquiring the target job after a preset waiting time; if the target logical resource amount is less than or equal to the global logical resource amount, determining the target execution machine of the target job according to the preset machine configuration policy.

[0036] Specifically, the target logical resource amount is the amount of resources required for the quantified target job. The target logical resource amount for each target job can be set to be consistent or inconsistent. The global logical resource is the amount of system resources obtained by quantifying the system resources. The system resources can be the CPU (Central Processing Unit) of the execution machine, memory, traditional databases, and big data type clusters. The current database and big data type cluster clients do not provide an interface for querying the current load. For these types of resources, the global logical resource amount can be used to simulate.

[0037] Compare the target logical resource amount of the target job with the remaining global logical resource amount of the scheduling system to determine whether the remaining global logical resources of the system can meet the scheduling of the target job. If the target logical resource amount is less than or equal to the global logical resource amount, it means that the scheduling system currently meets the resource scheduling conditions of the target job, and the target execution machine of the target job can be determined according to the preset machine configuration strategy. For example, the pre-simulated global logical resource amount is 5, and the pre-simulated target logical resource amount of the target job is 1. Before processing the target job, two jobs have been processed. The remaining global logical resource amount of the scheduling system is 3, which can meet the scheduling of the target job. Therefore, the target execution machine can be configured for the target job.

[0038] If the target logical resource amount is greater than the global logical resource amount, it means that the target job cannot be executed at present, and the target job is suspended. The new target job in the pending task queue is continued to be obtained for scheduling, and the original target job is stored in the waiting list. After the preset waiting time, the target job is obtained again and the resource amount is compared again. For example, if the current remaining global logical resource amount is 1 and the target logical resource amount of the target job is 2, the target job cannot be resource scheduled and the target job is stored in the waiting list. A waiting time is preset for storage in the waiting list. When the waiting time is reached, the target job is released from the waiting list, and the remaining global logical resource amount and the target logical resource amount are compared again to schedule the target job. Each time a target job is executed, the resource amount corresponding to the target logical resource in the global logical resource amount will be occupied. When the target job is processed, the occupied resource amount in the global logical resource amount is returned. Therefore, after the preset waiting time, the target job has the possibility of being executed. When the target job is stored in the waiting list, the new target job with the highest job score is obtained from the pending task queue for scheduling, and the target logical resource amount of the new target job is also compared with the remaining global logical resource amount. The beneficial effect of this setting is that before processing the target job, it is first determined whether the global logical resource amount meets the requirements of the target job. If not, the processing of the target job is directly suspended to avoid interruption after processing, fully utilize resources, save resource scheduling time, and improve resource scheduling efficiency.

[0039] In this embodiment, optionally, after comparing the target logical resource amount of the target job and the remaining global logical resource amount of the scheduling system, it also includes: if the target logical resource amount is less than or equal to the global logical resource amount, a lock mark is added to the target job to avoid the target job from being scheduled repeatedly.

[0040] Specifically, if the target logical resource amount is less than or equal to the global logical resource amount, the target job can be resource scheduled and processed. To avoid repeated processing of the target job, a lock mark is added to the target job to indicate that the target job is being executed. The Redis (Remote Dictionary Server) database can be used to lock the target job. For example, the lock mark can be associated with the unique identifier of the target job. When the target job is executed, the target job is removed from the queue of pending tasks. The beneficial effect of this setting is that if the current thread processing the target job successfully locks the target job, other threads will fail to lock the target job, ensuring that multi-threading will not make mistakes during parallel processing, avoiding repeated processing of jobs, improving the processing efficiency of the target job, and reducing resource waste.

[0041] In this embodiment, optionally, the machine configuration strategy includes at least one of the following: execution on a designated machine and execution on any machine.

[0042] Specifically, the machine configuration strategy can be pre-set. For example, for a certain target job, at least one machine can be designated as the target executor, or any machine can be selected as the target executor without specific designation. The beneficial effect of such a setting is that the flexibility of determining the target executor can be improved. By specifying the target executor, a specific executor can be allocated to the target job, avoiding the situation where the resources of the executor do not meet the execution requirements of the target job, and improving the processing efficiency of the target job.

[0043] In this embodiment, optionally, the target execution machine of the target job is determined according to a preset machine configuration policy, including: according to a pre-generated unique identifier of the target job, searching the machine configuration policy associated with the unique identifier from the machine configuration policy; and determining the target execution machine according to the target configuration policy.

[0044] Specifically, the unique identifier of the target job can be associated with the target configuration policy in the machine configuration policy. After the target job is determined, the unique identifier of the target job is read, and the target configuration policy associated with the identifier is found from the machine configuration policy. According to the target configuration policy, the target executor is determined. For example, the unique identifier of the target job is 001, and the target configuration policy is that the executor of the 001 job is any executor, then the target executor is selected arbitrarily from the alternative executors. For another example, the target configuration policy specifies that the executor of the 001 job is executor number one, then executor number one is selected from the alternative executors as the target executor. The beneficial effect of such a setting is that, by setting the machine configuration policy, when the target job is resource scheduled, it is processed according to the preset execution requirements, so as to avoid the randomly selected executor failing to meet the scheduling requirements of the job, reduce the probability of failure to execute the target job, and improve resource scheduling efficiency.

[0045] The technical solution of this embodiment obtains the job scores of candidate jobs in the pending task queue, determines the target job from the candidate jobs according to the priority in the job scores and the time information of the insertion into the queue, and configures the target execution machine for the target job according to the preset machine configuration strategy to complete the resource scheduling of the target job. This solves the problem in the prior art that candidate jobs are only processed according to priority, resulting in the starvation of low-priority jobs, realizes the timely scheduling of candidate jobs, reduces the job waiting time, and makes full use of machine resources according to the machine configuration strategy to improve the resource scheduling efficiency.

[0046] Embodiment 2

[0047] Figure 2This is a flow chart of a resource scheduling method provided by Embodiment 2 of the present invention. This embodiment is further optimized based on the above embodiment. The method can be executed by a resource scheduling device. Figure 2 As shown, the method specifically comprises the following steps:

[0048] Step 210: Obtain the job score of the pending task queue; wherein the job score is used to indicate the job priority of the pending candidate job and the time information of inserting into the pending task queue.

[0049] The job score of the candidate job in the pending task queue is obtained, and the waiting time of the candidate job inserted into the pending task queue is determined from the job score. The time when the candidate job is inserted into the pending task queue can be determined based on the last thirteen digits of the job score and a preset time point, and then the waiting time of the candidate job inserted into the pending task queue is determined based on the current time.

[0050] In this embodiment, optionally, after obtaining the job score of the pending task queue, it also includes: determining whether the pending candidate job is a timed-out and unprocessed job according to a preset timeout waiting time and the job score; if so, increasing the priority of the timed-out and unprocessed job.

[0051] Specifically, a timeout waiting time is preset, and the waiting time of the candidate job to be processed is compared with the timeout waiting time. If the waiting time exceeds the timeout waiting time, the candidate job to be processed is determined to be a timeout unprocessed job. In order to avoid the candidate job being starved to death, the candidate job can be processed with priority. The priority of the candidate job can be increased and the job score can be increased to facilitate the priority processing of the candidate job. For example, the preset timeout waiting time is 15 minutes, and 5 priorities are increased for every 15 minutes exceeding it. First, determine whether the waiting time of the candidate job exceeds 15 minutes. If it exceeds, increase the priority and obtain a new job score. Whether the waiting time of the candidate job has timed out can be determined based on the critical value of the job score of the timeout unprocessed job. The critical value of the job score of the timeout unprocessed job is the maximum job score of the unprocessed job that has not timed out. The critical value of the job score of the timeout unprocessed job can be calculated by the following formula:

[0052] y=x1×10000000000000+(α-(x2-β))

[0053] Among them, y is the critical value of the job score of the timed-out unprocessed job, x1 is the original job priority in the job score, x2 is the current time, α is the preset time point, and β is the preset timeout waiting time. Compare the calculated critical value with the job score of the candidate job. If the job score of the candidate job to be processed is greater than the calculated critical value, then the candidate job is a job that has been waiting in the queue of pending tasks for more than 15 minutes, that is, a timed-out unprocessed job. Extract the timed-out unprocessed jobs in batches, and then adjust the job scores to obtain new job scores. The new job score can be calculated using the following formula:

[0054] z=(x1+μ)×10000000000000+(α-x2)

[0055] Where z is the score of the new job, and μ is the preset priority level. The beneficial effect of this setting is that it takes into account the priority and waiting time of the job, so that high-priority jobs can be scheduled as soon as possible, and low-priority jobs will not be left without the opportunity to be scheduled. Automatically increase the priority of the job. In the case of tight resources, low-priority jobs may not have the opportunity to be scheduled. Automatically increase the priority of the job according to the waiting time of the members of the queue of pending tasks in the queue, which can prevent low-priority jobs from being starved and improve the resource scheduling efficiency of low-priority jobs.

[0056] Step 220: Determine the target job to be processed according to the job score.

[0057] Step 230: Determine the target execution machine of the target job according to the preset machine configuration strategy to complete the resource scheduling of the target job.

[0058] Step 240: Send the target job to the target execution machine, and record the first time of sending the target job.

[0059] After determining the target execution machine, an Http (Hyper Text Transfer Protocol) request is sent to the target execution machine, and the Http connection failure time threshold is preset, for example, it can be set to 5 seconds. If the scheduling end initiates an Http request and fails to connect to the target execution machine after 5 seconds, the Http request fails, and a new target execution machine is selected for retry. If the execution machine is connected, the target job is sent to the target execution machine, and the time of sending the target job to the target execution machine is recorded as the first time.

[0060] Step 250: Receive feedback information from the target execution machine, and record the second time when the feedback information is received.

[0061] Among them, after receiving the target job, the target execution machine will send feedback information, which can be used to indicate whether the target job has been received and can be processed. If the target execution machine has sufficient resources, the target execution machine will quickly send feedback information after receiving the target job, and record the time when the target execution machine sends the feedback information as the second time. After obtaining the feedback information, the feedback message can be parsed. If the feedback message indicates that the target execution machine lacks resources, a new target execution machine can be selected for retry.

[0062] Step 260: If the difference between the first time and the second time is less than a preset normal difference threshold, it is determined that the target execution machine has successfully started to execute the target job.

[0063] Among them, a normal difference threshold is preset, for example, the normal difference threshold can be set to 5 seconds. After obtaining the first time and the second time, if the feedback message indicates that the target execution machine can execute the target job normally, the difference between the first time and the second time is calculated. If the difference is less than the preset normal difference threshold, it means that the target execution machine can execute the target job normally. After the execution of the target execution machine is completed, the target job is removed from the queue of pending tasks.

[0064] In this embodiment, optionally, after sending the target job to the target executor and recording the first time of sending the target job, it also includes: if no feedback information is received from the target executor within the time of a preset normal difference threshold, it is determined that the target executor has failed to receive the target job, and the target job is sent to other executors except the target executor.

[0065] Specifically, if no feedback information is received from the target executor within the preset normal difference threshold time, it is determined that the target executor has failed to receive the target job, and a new target executor is selected to process the target job. The target executor automatically abandons the target job and will no longer send feedback information about the target job. The beneficial effect of this setting is that when the target executor cannot execute the target job, the target job can be automatically transferred to the new target executor, reducing the transmission of feedback information from the original target executor, improving job processing efficiency, making full use of the executor's resources, and improving resource scheduling efficiency.

[0066] In this embodiment, optionally, after determining the target execution machine of the target job according to the preset machine configuration strategy, it also includes: if an exception occurs in the resource scheduling process of the target job, the job score of the target job is adjusted to obtain an exception score; the exception score is stored in the exception task queue, and the target job is rescheduled after exceeding the preset retry waiting time.

[0067] Specifically, if an exception occurs in the target job during scheduling or processing, the target job cannot continue to be executed. For example, if all alternative execution machines cannot execute the target job, the target task is determined to be an abnormal job, and the abnormal job can be placed in the abnormal task queue. Before placing it in the abnormal task queue, the job score of the abnormal job can be adjusted to obtain the abnormal score. For example, the time information of inserting into the pending task queue can be modified to the time information of inserting into the abnormal task queue. A retry waiting time is preset. If the storage time of the abnormal job in the abnormal task queue exceeds the retry waiting time, the abnormal job is removed from the abnormal task queue and stored in the pending task queue for rescheduling. For example, the preset retry waiting time can be 15 minutes. The effect of such a setting is that if an exception occurs in the target job, the target task will wait for the preset retry waiting time, so that the resource scheduling of the target job will be re-performed when the resource is idle. Storing it in the abnormal task queue can avoid task loss. According to the abnormal score, the waiting time of the abnormal job can be determined to avoid the abnormal job waiting time being too long, effectively improving the resource scheduling efficiency.

[0068] In this embodiment, optionally, the abnormality score is a string containing 16-digit values, wherein the first digit is -1, the second to fourteenth digits represent the time information of the target job being inserted into the abnormal task queue, and the fifteenth to sixteenth digits represent the job priority.

[0069] Specifically, the abnormality score is different from the job score. The job score is a 15-digit string, and the abnormality score is a 16-digit string. In addition, the positions of the priority value and the time information value in the abnormality score are different from those in the job score. The first digit in the abnormality score is a fixed value of -1, and the last two digits are the preset job priority. The middle thirteen digits are the time information of the target job, that is, the abnormal job inserted into the abnormal task queue. The time information can be calculated based on the insertion time and the preset time point. The larger the time information, the earlier the insertion time. Based on the abnormality score critical value and the abnormality score, it can be calculated whether the waiting time of the abnormal job exceeds the preset retry waiting time. The abnormality score critical value is the minimum abnormality score of the abnormal job that does not exceed the preset retry waiting time. The abnormality score critical value can be determined by the following formula:

[0070]

[0071] Among them, g is the critical value of abnormal score, is the time period of the preset retry waiting time, and x2 is the current time. After obtaining the abnormal score critical value, the abnormal score critical value is compared with the abnormal score. If the abnormal score is less than the abnormal score critical value, the abnormal job to which the abnormal score belongs can be removed from the abnormal task queue. The beneficial effect of such a setting is that all jobs in the abnormal task queue that are less than the abnormal score critical value are timed out, regardless of the last two priorities. The members of the abnormal task queue can be moved back to the pending task queue in batches, making it simple to obtain the timed-out abnormal jobs, thereby improving the efficiency of resource scheduling.

[0072] The embodiment of the present invention obtains the job scores of candidate jobs in the queue of pending tasks, determines the target job from the candidate jobs according to the priority in the job scores and the time information of the queue insertion, configures the target execution machine for the target job according to the preset machine configuration strategy, and transfers the target job in time when the target execution machine cannot execute the target job, thereby completing the resource scheduling of the target job. The problem in the prior art that candidate jobs are processed only according to priority, resulting in the starvation of low-priority jobs, realizes the timely scheduling of candidate jobs, reduces the job waiting time, fully utilizes the resources of the target execution machine according to the machine configuration strategy, reduces the waiting time of resource scheduling, and improves the resource scheduling efficiency.

[0073] Embodiment 3

[0074] Figure 3 This is a structural block diagram of a resource scheduling device provided in Embodiment 3 of the present invention, which can execute a resource scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 3 As shown, the device specifically includes:

[0075] The score acquisition module 301 is used to acquire the job score of the pending task queue; wherein the job score is used to indicate the job priority of the pending candidate job and the time information of inserting into the pending task queue;

[0076] The job determination module 302 is used to determine the target job to be processed according to the job score;

[0077] The execution machine determination module 303 is used to determine the target execution machine of the target job according to a preset machine configuration strategy to complete the resource scheduling of the target job.

[0078] Optionally, the job score is a string containing 15 digits, wherein the first and second digits represent the job priority, and the third to fifteenth digits represent the time information when the candidate job is inserted into the queue of pending tasks.

[0079] Optionally, the score acquisition module 301 includes:

[0080] A time difference determination unit, used to determine the time difference between the time when the candidate job is inserted into the queue of pending tasks and the preset time point according to the preset time point;

[0081] The job score generating unit is used to generate the job score of the candidate job in the to-be-processed task queue according to the preset job priority and time difference.

[0082] Optionally, the job determination module 302 is specifically configured to:

[0083] Sort candidate jobs in the pending task queue according to the numerical values ​​of the job scores;

[0084] The candidate jobs whose sorting results meet the preset requirements are determined as target jobs.

[0085] Optionally, the device further comprises:

[0086] A resource amount comparison module is used to compare the target logical resource amount of the target job with the remaining global logical resource amount of the scheduling system before determining the target execution machine of the target job according to the preset machine configuration strategy;

[0087] The target storage module is used to continue to obtain new target jobs in the pending task queue for scheduling if the target logical resource amount is greater than the global logical resource amount, and store the target jobs in the waiting list, and re-acquire the target jobs after a preset waiting time;

[0088] The target execution module is used to determine the target execution machine of the target job according to a preset machine configuration strategy if the target logical resource amount is less than or equal to the global logical resource amount.

[0089] Optionally, the device further comprises:

[0090] The target locking module is used to add a locking mark to the target job after comparing the target logical resource amount of the target job with the remaining global logical resource amount of the scheduling system, if the target logical resource amount is less than or equal to the global logical resource amount, to avoid the target job being scheduled repeatedly.

[0091] Optionally, the machine configuration strategy includes at least one of the following: execution on a specified machine and execution on any machine.

[0092] Optionally, the execution machine determination module 303 is specifically configured to:

[0093] According to the unique identifier of the target job generated in advance, searching the target configuration policy associated with the identifier from the machine configuration policy;

[0094] Determine the target execution machine based on the target configuration strategy.

[0095] Optionally, the device further comprises:

[0096] A first time recording module, used for sending a target job to a target execution machine and recording the first time of sending the target job;

[0097] A second time recording module, used for receiving feedback information from the target execution machine and recording a second time when the feedback information is received;

[0098] The job execution module is used to determine that the target execution machine successfully starts to execute the target job if the difference between the first time and the second time is less than a preset normal difference threshold.

[0099] Optionally, the device further comprises:

[0100] The job resending module is used to determine that the target executor fails to receive the target job if no feedback information is received from the target executor within a preset normal difference threshold time, and send the target job to other executors except the target executor.

[0101] Optionally, the device further comprises:

[0102] The abnormal score determination module is used to adjust the job score of the target job to obtain the abnormal score after determining the target execution machine of the target job according to the preset machine configuration strategy, if the target job has an abnormality in the resource scheduling process;

[0103] The target rescheduling module is used to store the abnormal score in the abnormal task queue and reschedule the target job after exceeding the preset retry waiting time.

[0104] Optionally, the exception score is a string containing 16-digit values, where the first digit is -1, the second to fourteenth digits represent the time information when the target job is inserted into the exception task queue, and the fifteenth to sixteenth digits represent the job priority.

[0105] Optionally, the device further comprises:

[0106] A timeout determination module is used to determine whether a candidate job to be processed is a timeout unprocessed job according to a preset timeout waiting time and a job score;

[0107] The priority enhancement module is used to enhance the priority of the timed-out and unprocessed job if so.

[0108] The embodiment of the present invention obtains the job scores of candidate jobs in the queue of pending tasks, determines the target job from the candidate jobs according to the priority in the job scores and the time information of the insertion into the queue, and configures the target execution machine for the target job according to the preset machine configuration strategy to complete the resource scheduling of the target job. The problem of processing candidate jobs only according to priority in the prior art, resulting in the starvation of low-priority jobs, realizes the timely scheduling of candidate jobs, reduces the job waiting time, makes full use of machine resources according to the machine configuration strategy, and improves the resource scheduling efficiency.

[0109] Embodiment 4

[0110] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary computer device 400 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 4 As shown, the computer device 400 is in the form of a general computing device. The components of the computer device 400 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0112] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0113] Computer device 400 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 400, including volatile and non-volatile media, removable and non-removable media.

[0114] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Computer device 400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0115] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0116] The computer device 400 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 400, and / or communicate with any device that enables the computer device 400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 411. Furthermore, the computer device 400 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via network adapter 412. As shown, the network adapter 412 communicates with other modules of the computer device 400 via bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0117] The processing unit 401 executes various functional applications and data processing by running the program stored in the system memory 402, for example, implementing a resource scheduling method provided by an embodiment of the present invention, including:

[0118] Obtaining the job score of the pending task queue; wherein the job score is used to indicate the job priority of the candidate job to be processed and the time information of inserting into the pending task queue;

[0119] Determine the target tasks to be processed based on the task scores;

[0120] According to the preset machine configuration strategy, the target execution machine of the target job is determined to complete the resource scheduling of the target job.

[0121] Embodiment 5

[0122] Embodiment 5 of the present invention further provides a storage medium containing computer executable instructions, on which a computer program is stored. When the program is executed by a processor, a resource scheduling method provided in the embodiment of the present invention is implemented, including:

[0123] Obtaining the job score of the pending task queue; wherein the job score is used to indicate the job priority of the candidate job to be processed and the time information of inserting into the pending task queue;

[0124] Determine the target tasks to be processed based on the task scores;

[0125] According to the preset machine configuration strategy, the target execution machine of the target job is determined to complete the resource scheduling of the target job.

[0126] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0127] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0129] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A resource scheduling method, characterized in that: include: Obtaining a job score of a pending task queue; wherein the job score is used to indicate the job priority of the pending candidate job and the time information of inserting the pending task queue, and the job priority is used to indicate the importance of the pending candidate job; Determine the target job to be processed according to the job score; Determine the target execution machine of the target job according to the preset machine configuration strategy to complete the resource scheduling of the target job; The job score is a string of 15 digits, wherein the first and second digits represent the job priority, and the third to fifteenth digits represent the time information when the candidate job is inserted into the queue of pending tasks; The step of obtaining the job score of the task queue to be processed includes: According to the preset time point, determine the time difference between the time when the candidate job is inserted into the queue of pending tasks and the preset time point; Generating a job score for a candidate job in the to-be-processed task queue according to a preset job priority and the time difference; Wherein, after determining the target execution machine of the target job according to the preset machine configuration strategy, the method further includes: Sending the target job to the target execution machine, and recording the time of sending the target job to the target execution machine as the first time; receiving feedback information from the target execution machine, and recording the time when the target execution machine sends the feedback information as a second time, wherein the feedback information is used to indicate whether the target execution machine has accepted the target job; If the difference between the first time and the second time is less than a preset normal difference threshold, it is determined that the target execution machine has successfully started to execute the target job, wherein the preset normal difference threshold is 5 seconds; Wherein, after sending the target job to the target execution machine and recording the first time of sending the target job, the method further includes: If no feedback information is received from the target executor within the time of the preset normal difference threshold, it is determined that the target executor has failed to receive the target job, and the target job is sent to other executors except the target executor. When the target executor fails to receive the target job, the target executor automatically abandons the target job and no longer issues feedback information of the target job.

2. The method according to claim 1, characterized in that According to the job score, determine the target job to be processed, including: Sort candidate jobs in the pending task queue according to the numerical values ​​of the job scores; The candidate jobs whose sorting results meet the preset requirements are determined as target jobs.

3. The method according to claim 1, characterized in that: Before determining the target execution machine of the target job according to the preset machine configuration strategy, the method further includes: Compare the target logical resource amount of the target job with the remaining global logical resource amount of the scheduling system; If the target logical resource amount is greater than the global logical resource amount, then continue to obtain a new target job in the pending task queue for scheduling, store the target job in the waiting list, and re-acquire the target job after a preset waiting time; If the target logical resource amount is less than or equal to the global logical resource amount, the target execution machine of the target job is determined according to a preset machine configuration strategy.

4. The method according to claim 3, characterized in that After comparing the target logical resource amount of the target job with the remaining global logical resource amount of the scheduling system, it also includes: If the target logical resource amount is less than or equal to the global logical resource amount, a lock mark is added to the target job to prevent the target job from being repeatedly scheduled.

5. The method according to claim 1, characterized in that The machine configuration strategy includes at least one of the following: execution on a designated machine and execution on any machine.

6. The method according to claim 1, characterized in that Determining the target execution machine of the target job according to the preset machine configuration strategy includes: According to the pre-generated unique identifier of the target job, searching the target configuration policy associated with the identifier from the machine configuration policy; According to the target configuration strategy, a target execution machine is determined.

7. The method according to claim 1, characterized in that After determining the target execution machine of the target job according to the preset machine configuration strategy, the method further includes: If an exception occurs to the target job during resource scheduling, the job score of the target job is adjusted to obtain an exception score; The abnormal score is stored in an abnormal task queue, and the target job is rescheduled after exceeding a preset retry waiting time.

8. The method according to claim 7, characterized in that The abnormal score is a string containing 16 digits, wherein the first digit is -1, the second to fourteenth digits represent the time information when the target job is inserted into the abnormal task queue, and the fifteenth to sixteenth digits represent the job priority.

9. The method according to claim 1, characterized in that: After obtaining the job scores of the pending task queue, it also includes: Determine whether the candidate job to be processed is a timed-out and unprocessed job according to the preset timeout waiting time and the job score; If so, the priority of the timed-out and unprocessed job is increased.

10. A resource scheduling device, characterized in that: include: A score acquisition module, used to acquire the job score of the pending task queue; wherein the job score is used to indicate the job priority of the candidate job to be processed and the time information of inserting the candidate job into the pending task queue, and the job priority is used to indicate the importance of the candidate job to be processed; A job determination module, used to determine a target job to be processed according to the job score; An execution machine determination module is used to determine the target execution machine of the target job according to a preset machine configuration strategy to complete resource scheduling of the target job; The job score is a string of 15 digits, wherein the first and second digits represent the job priority, and the third to fifteenth digits represent the time information when the candidate job is inserted into the queue of pending tasks; The score acquisition module includes: A time difference determination unit, used to determine the time difference between the time when the candidate job is inserted into the queue of pending tasks and the preset time point according to the preset time point; A job score generating unit, used to generate a job score for a candidate job in the to-be-processed task queue according to a preset job priority and the time difference; A first time recording module, used for sending the target job to the target execution machine, and recording the time of sending the target job to the target execution machine as a first time; A second time recording module, used for receiving feedback information from the target execution machine, and recording the time when the target execution machine sends the feedback information as a second time, wherein the feedback information is used to indicate whether the target execution machine has accepted the target job; A job execution module, configured to determine that the target execution machine has successfully started to execute the target job if the difference between the first time and the second time is less than a preset normal difference threshold, wherein the preset normal difference threshold is 5 seconds; A job resending module is used to determine that the target executor has failed to receive the target job if no feedback information is received from the target executor within the time of the preset normal difference threshold, and to send the target job to other executors except the target executor, wherein when the target executor fails to receive the target job, the target executor automatically abandons the target job and no longer issues feedback information of the target job.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the resource scheduling method as described in any one of claims 1 to 9 is implemented.

12. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the resource scheduling method as described in any one of claims 1 to 9 when executed by a computer processor.