Task resource scheduling method and device, equipment and computer readable storage medium

Abstract

The application provides a task resource scheduling method, device and equipment and a computer readable storage medium. The task resource scheduling method comprises the following steps: obtaining a target task queue in a target period and task running information of a reference task queue corresponding to the target period; calculating a target initial resource quantity of the target task queue according to the task running information of the reference task queue; and performing resource allocation on the target task queue according to the target initial resource quantity. In the task resource scheduling method provided by the application, the task running information describes the historical running condition of the task queue corresponding to the target queue, so that the target initial resource quantity set based on the task running information can effectively avoid resource waste and improve the resource utilization rate of the system.

Description

Technical Field

[0001] This application relates to the field of resource scheduling technology, specifically to a task resource scheduling method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Currently, the shortage of available resources caused by the data explosion is a pressing issue that major data platforms need to address. This includes factors such as the underlying storage space within data platforms and the virtual machine resources of big data job clusters. The most common solution currently is to utilize resource management systems for resource management and scheduling.

[0003] However, while resource management systems can facilitate resource management and scheduling, they also generate a large number of "allocated but unused" resource fragments, leading to performance issues and resource waste. Summary of the Invention

[0004] This application provides a task resource scheduling method, apparatus, device, and computer-readable storage medium, aiming to solve the technical problems of serious performance and resource waste in existing resource scheduling systems.

[0005] On one hand, embodiments of this application provide a task resource scheduling method, including:

[0006] Obtain the target task queue within the target time period, and the task execution information of the reference task queue corresponding to the target time period;

[0007] Calculate the target initial resource quantity of the target task queue based on the task execution information of the reference task queue;

[0008] The target task queue is allocated resources based on the initial target resource quantity.

[0009] As a feasible implementation of this application, the task running information of the reference task queue includes the runtime segment and required resource amount of the reference task in the reference task queue;

[0010] The step of calculating the target initial resource amount of the target task queue based on the task execution information of the reference task queue includes:

[0011] Calculate the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue;

[0012] The range of values ​​for the initial resource quantity corresponding to the reference task is calculated based on the resource allocation coefficient and the required resource quantity for the reference task.

[0013] The target initial resource quantity of the target task queue is set according to the range of values ​​for the initial resource quantity corresponding to the reference task.

[0014] As a feasible implementation of this application, the step of calculating the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue includes:

[0015] The number of running tasks at each time point within the runtime segment is determined based on the runtime segment of the reference task in the reference task queue.

[0016] The resource allocation coefficient corresponding to each moment in the runtime segment is set according to the number of running tasks at each moment in the runtime segment;

[0017] By integrating the resource allocation coefficients corresponding to each moment within the runtime segment, the resource allocation coefficients corresponding to the reference task are obtained.

[0018] As a feasible implementation of this application, the step of determining the number of running tasks at each time point within the runtime segment based on the runtime segment of the reference task in the reference task queue includes:

[0019] The start and end times of the reference task are determined based on the runtime segments of the reference tasks in the reference task queue.

[0020] Arrange the start and end times of the reference task in order to obtain the start and end time sequence;

[0021] Determine the number of starting tasks and the number of ending tasks before the target time in the start-end time sequence;

[0022] The difference between the number of start tasks and the number of end tasks is set as the number of running tasks at each time between the target time and the next time in the start-end time sequence;

[0023] The number of running tasks at each moment within the runtime segment is determined based on the number of running tasks at each moment between the target moment and the next moment in the start-end time sequence.

[0024] As a feasible implementation of this application, the step of setting the target initial resource quantity of the target task queue according to the value range of the initial resource quantity corresponding to the reference task includes:

[0025] If the reference task includes multiple tasks, the intersection of the value ranges of the initial resources corresponding to each reference task is obtained to obtain the reference value range.

[0026] Set the minimum initial resource amount in the reference value range as the target initial resource amount of the target task queue.

[0027] As a feasible implementation of this application, before the step of obtaining the target task queue within the target time period and the task execution information of the reference task queue corresponding to the target time period, the method further includes:

[0028] The preset time period is divided into multiple target time periods based on the preset time interval;

[0029] The previous time period of the preset time period is divided according to the preset time interval to obtain the reference time period corresponding to each target time period.

[0030] Set the task queues within the reference time periods corresponding to each target time period as the reference task queues corresponding to each target time period.

[0031] As a feasible implementation of this application, the step of allocating resources to the target task queue based on the target initial resources includes:

[0032] If the sum of the target initial resource amounts of each target task queue is higher than the preset maximum initial resource amount, then the actual initial resource amount corresponding to each target task queue is determined according to the preset maximum initial resource amount and the priority corresponding to each target task queue.

[0033] Resource allocation is performed on each target task queue according to the actual initial resources corresponding to each target task queue.

[0034] On the other hand, embodiments of this application also provide a task resource scheduling device, including:

[0035] The acquisition module is used to acquire the target task queue within the target time period, and the task execution information of the reference task queue corresponding to the target time period;

[0036] The calculation module is used to calculate the target initial resource quantity of the target task queue based on the task execution information of the reference task queue;

[0037] The allocation module is used to allocate resources to the target task queue based on the target initial resource amount.

[0038] On the other hand, this application embodiment also provides a task resource scheduling device, which includes a processor, a memory, and a task resource scheduling program stored in the memory and executable on the processor. The processor executes the task resource scheduling program to implement the steps in the task resource scheduling method described above.

[0039] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a task resource scheduler, which is executed by a processor to implement the steps in the task resource scheduling method described above.

[0040] The task resource scheduling method provided in this application obtains the task operation information of the reference task queue corresponding to the target time period, and sets the target initial resource amount of the target task queue based on the task operation information, thereby completing the resource allocation of the target task queue. Since the task operation information describes the historical operation of the task queue corresponding to the target queue, the target initial resource amount set based on the task operation information can effectively avoid resource waste and improve the resource utilization rate of the system. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram illustrating an implementation scenario of the task resource scheduling method provided in this application embodiment;

[0043] Figure 2 This is a flowchart illustrating the steps of a task resource scheduling method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the steps for setting an initial resource quantity provided in an embodiment of this application;

[0045] Figure 4 This is a schematic flowchart illustrating the steps for calculating resource allocation coefficients according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the steps for determining the number of tasks to run, provided in an embodiment of this application;

[0047] Figure 6 This is a flowchart illustrating a step for setting a target initial resource quantity based on a value range, as provided in an embodiment of this application.

[0048] Figure 7 This is a flowchart illustrating the steps for determining a target time period and a reference task queue, provided in an embodiment of this application.

[0049] Figure 8 This is a schematic flowchart illustrating the steps of allocating resources to a target task queue, as provided in an embodiment of this application.

[0050] Figure 9 This is a complete implementation diagram of task resource scheduling provided in the embodiments of this application;

[0051] Figure 10 This is a schematic diagram of the structure of a task resource scheduling device provided in an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the structure of a task resource scheduling device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of the present invention.

[0054] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0055] To facilitate understanding of the task resource scheduling method provided in the embodiments of this application, a brief description of the relevant background technology in this application will be given first.

[0056] For data platforms to handle user demands effectively, a resource management system is typically needed for resource management and scheduling. Here, "resources" usually refers to the data platform's CPU, or data processing capacity. Taking the Hadoop distributed computing platform as an example, YARN (Yet Another Resource Negotiator), as Hadoop's resource management system, facilitates resource management through its built-in queue mechanism. However, it also generates a large number of allocated but unused resource fragments, impacting system efficiency and leading to performance issues and resource waste.

[0057] Based on the aforementioned issues, and considering the relatively small differences in the tasks and execution times of the queues to be executed daily by the data platform, this application proposes a method to calculate the resources required for each time period based on the task operation status of the corresponding reference queue in the historical records for that time period. At the same time, it sets the corresponding initial resources for that time period to ensure that the resources required by each target task within that time segment can be met under the initial resources, avoiding the generation of a large number of unused resource fragments, thereby effectively improving resource utilization and task execution efficiency.

[0058] This application provides a task resource scheduling method, apparatus, device, and computer-readable storage medium, which will be described in detail below.

[0059] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating the implementation scenario of the task resource scheduling method in this application embodiment, including a data platform 100, a resource management system 200, and a task resource scheduling device 300. Specifically, the task resource scheduling device is deployed in the resource management system 200 of the data platform 100, and by running the task resource scheduling method provided in this application embodiment, the resource management system 200 performs resource scheduling in the data platform 100.

[0060] Specifically, the data platform can be a distributed computing data platform, such as Hadoop, in which case the resource management system is YARN (Yet Another Resource Negotiator), which can provide unified resource management and scheduling for upper-layer applications. Of course, the data platform can also be other data platforms with deployed resource management systems, which will not be elaborated upon here. However, for ease of description, Hadoop will be used as the data platform and YARN as the resource management system in the following description. Those skilled in the art, based on the technical solution provided in this application, can undoubtedly obtain specific implementation schemes for applying the task resource scheduling method provided in this application to other data platforms and resource management systems.

[0061] It should be noted that, Figure 1 The schematic diagram illustrating the implementation scenario of the task resource scheduling method shown is merely an example. The task resource scheduling scenarios described in this application are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention.

[0062] Based on the above-mentioned task resource scheduling scenario, a specific implementation of the task resource scheduling method is proposed.

[0063] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the steps of a task resource scheduling method provided in an embodiment of this application. The task resource scheduling method in this embodiment includes steps 201-203:

[0064] 201. Obtain the target task queue within the target time period, and the task execution information of the reference task queue corresponding to the target time period.

[0065] In this embodiment, the target time period is typically obtained by dividing time according to certain rules. This target time period usually contains a series of target tasks to be executed. Arranging these target tasks in the required execution order yields the target task queue. In other words, the target task queue consists of a series of target tasks to be executed within the target time period in the data platform. The resource management system needs to allocate resources to each target task according to certain resource scheduling rules to ensure the normal execution of each target task, i.e., to obtain sufficient resources to meet its own resource needs. Of course, if there are no target tasks to be executed within the target time period, then obviously no resource allocation is needed; the allocated resources can be directly set to 0, which does not affect the implementation of this embodiment.

[0066] In this embodiment, the reference task queue corresponding to the target time period typically refers to the task queue within the reference time period corresponding to the target time period. Specifically, as described above in the background section, the tasks and their execution times in the queues that the data platform needs to execute each day are relatively similar. Therefore, the reference time period can be understood as a historical time period that is relatively close to the target time period in terms of task queues. In other words, the reference task queue within the reference time period is quite similar to the target task queue within the target time period, with only minor differences. Therefore, the task execution information of the reference task queue can be used as a reference for the execution of the target task queue. Typically, the task execution information includes the runtime segment of each reference task in the reference task queue and the corresponding resources obtained within that runtime segment, i.e., the required resources. The task execution information of the reference task queue can be obtained through historical log datasets from YARN.

[0067] As a feasible embodiment of this application, the specific implementation scheme for determining the target time period and the reference task queue corresponding to the target time period will be provided later. Figure 7 And its explanations and descriptions.

[0068] 202. Calculate the target initial resource quantity of the target task queue based on the task execution information of the reference task queue.

[0069] As described above, the reference task queue and the target task queue are quite similar; therefore, the task execution information of the reference task queue can be used as a reference for the target task queue. Based on this, this application proposes setting a target initial resource amount for the target task queue based on the task execution information of the reference task queue. Specifically, under this target initial resource amount, each target task in the target task queue can obtain the actual resources that just meet its resource requirements no later than the end time of the corresponding reference task in the reference task queue. This achieves the goal of improving task execution efficiency with as few resources as possible, avoiding resource waste, and improving resource utilization.

[0070] For details on the specific implementation scheme of setting the target initial resource amount of the target task queue based on the task execution information of the reference task queue, please refer to the following sections. Figure 3 And its explanations and descriptions.

[0071] 203. Allocate resources to the target task queue according to the target initial resource amount.

[0072] In this embodiment, after setting the initial resources for the target task queue, the allocation of scheduling resources in the target task queue can be achieved based on a preset strategy. For example, the most common fair strategy is to distribute the initial resources equally among the target tasks based on the number of target tasks executed simultaneously during actual operation, until the target tasks are completed, that is, the total amount of resources obtained by the target tasks reaches the required amount of resources.

[0073] As an optional embodiment of this application, the resource management system may contain multiple task queues within the same target time period. Therefore, during the actual operation of the system, the real-time status of each queue will be acquired, and the resources occupied by each task queue will be dynamically adjusted based on the target initial resource amount, thereby optimizing the task queues. Specifically, based on the preset priority of each task queue, when the sum of the initial resource amounts of all task queues exceeds the resource amount that the system can provide, resources will be allocated to the task queue with higher priority. For specific implementation steps, please refer to the following. Figure 8 And its explanations and descriptions.

[0074] The task resource scheduling method provided in this application obtains the task operation information of the reference task queue corresponding to the target time period, and sets the target initial resource amount of the target task queue based on the task operation information, thereby completing the resource allocation of the target task queue. Since the task operation information describes the historical operation of the task queue corresponding to the target queue, the target initial resource amount set based on the task operation information can effectively avoid resource waste and improve the resource utilization rate of the system.

[0075] like Figure 3 As shown, Figure 3 A flowchart illustrating the steps for setting an initial resource quantity, as provided in an embodiment of this application, is described in detail below.

[0076] In this embodiment of the application, the task execution information of the reference task queue mainly includes the runtime segment and required resource amount of the reference task in the reference task queue, specifically including steps 301 to 303:

[0077] 301. Calculate the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue.

[0078] In this embodiment of the application, the runtime segment of the reference task in the reference task queue describes the execution status of each reference task. Considering that the initial resources are allocated through a fair strategy, the ratio of the amount of resources obtained by each reference task in the runtime segment to the preset initial resource amount can be calculated by combining the runtime segments of each reference task in the reference task queue. This ratio can be understood as the resource allocation coefficient corresponding to the reference task.

[0079] Of course, it should be noted that if the runtime segments of the reference task span multiple different reference time periods, it will not affect the specific implementation of this application; it is only necessary to segment the runtime segments of the reference task according to the reference time periods. For ease of understanding, the following will combine... Figure 9 The provided task resource scheduling complete implementation diagram and corresponding complete step description provide a detailed explanation of the reference tasks spanning multiple different time periods.

[0080] As one feasible embodiment of this application, the specific steps for calculating the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task will be described later. Figure 4 And its explanations and descriptions.

[0081] 302. The range of values ​​for the initial resource quantity corresponding to the reference task is calculated based on the resource allocation coefficient and the required resource quantity of the reference task.

[0082] In this embodiment, the required resource amount corresponding to the reference task is usually a fixed value. Specifically, the resources obtained during the runtime can be set as the required resource amount corresponding to the reference task. Of course, for ease of calculation, it can be directly expressed in the form of data.

[0083] In this embodiment of the application, as described above, the resource allocation coefficient corresponding to the reference task can be understood as the ratio of the amount of resources obtained by the reference task during its runtime to a preset initial resource amount. To meet the resource allocation requirements of the reference task, the amount of resources obtained by the reference task during its runtime should be higher than its required resource amount. That is, there is a certain requirement for the initial resource amount, which is usually higher than a minimum allowed initial resource amount. Under this minimum allowed initial resource amount, the amount of resources obtained by the reference task during its runtime is exactly equal to its required resource amount. If the initial resource amount is lower than this minimum allowed initial resource amount, the amount of resources obtained by the reference task during its runtime will obviously be less than its required resource amount, meaning that the resource requirements of the reference task cannot be met. Therefore, the range of values ​​for the initial resource amount can be equivalently understood as all values ​​greater than the minimum allowed initial resource amount.

[0084] 303. Set the target initial resource quantity of the target task queue according to the value range of the initial resource quantity corresponding to the reference task.

[0085] In this embodiment of the application, as described above, if there is only one reference task in the reference task queue, the minimum initial resource amount within the range can be directly set as the target initial resource amount of the target task queue. If there are multiple reference tasks, the resource requirements of each reference task need to be guaranteed. That is, the target initial resource amount of the target task queue needs to be comprehensively set based on the range of values ​​for the initial resource amounts corresponding to each reference task. Specific setting rules can be found in the following sections. Figure 6 And its explanations and descriptions.

[0086] This application provides an implementation scheme for determining the target initial resource quantity of a target task queue based on the runtime segment and required resource quantity of a reference task in a reference task queue. The target initial resource quantity calculated by the above scheme can ensure that each target task in the target time period obtains the resource quantity that meets its needs as much as possible, that is, to minimize the waste of resources and ensure the utilization rate of resources.

[0087] like Figure 4 As shown, Figure 4 A flowchart illustrating the steps for calculating resource allocation coefficients, as provided in this application embodiment, is described in detail below.

[0088] In this embodiment of the application, a specific implementation scheme is provided for calculating the resource allocation coefficient of a reference task based on the runtime segment of the reference task in the reference task queue, specifically including steps 401 to 403:

[0089] 401. Determine the number of running tasks at each time point within the runtime segment based on the runtime segment of the reference task in the reference task queue.

[0090] In this embodiment, it is readily understood that the runtime segment of a reference task describes the time during which the reference task is in a running state. Therefore, the number of running tasks at each moment within a runtime segment can be easily determined based on the runtime segment of each reference task. For ease of understanding, a simplified embodiment is provided. For example, if the runtime segment of task 1 is 1-5 and the runtime segment of task 2 is 3-10, it can be understood that the number of running tasks is 1 during moments 1-3, 2 during moments 3-5, 1 during moments 5-10, and 0 during other moments.

[0091] Of course, considering that the number of tasks in the reference task queue is large—for example, typically dozens—it is difficult for the system to directly count the number of running tasks at each time point. As an optional embodiment of this application, a specific implementation scheme is provided to facilitate the determination of the number of running tasks at each time point; please refer to the following sections for details. Figure 5 And its explanation.

[0092] 402, Set the resource allocation coefficient corresponding to each time point in the runtime segment based on the number of running tasks at each time point in the runtime segment.

[0093] In this embodiment, the number of running tasks at each moment determines the resource allocation coefficient that each running task can be allocated. Specifically, the running tasks will share the initial resource amount equally. Therefore, the resource allocation coefficient corresponding to each moment within a runtime period can be simply understood as the reciprocal of the number of running tasks at that moment. That is, if the number of running tasks at a certain moment is 1, then the resource allocation coefficient corresponding to that moment is 1; if the number of running tasks at a certain moment is 2, then the resource allocation coefficient corresponding to that moment is 0.5; and if the number of running tasks at a certain moment is 4, then the resource allocation coefficient corresponding to that moment is 0.25.

[0094] 403. Integrate the resource allocation coefficients corresponding to each time point within the runtime segment to obtain the resource allocation coefficients corresponding to the reference task.

[0095] In this embodiment, for each reference task, the resource allocation coefficients corresponding to each time point within its runtime segment are integrated, i.e., summed, and the resulting value is the resource allocation coefficient corresponding to the reference task. Similarly, for ease of understanding, let's take the aforementioned simplified embodiment, where the runtime segments for task 1 are 1-5 and for task 2 are 3-10, for further explanation. In this case, during times 1-3, the number of running tasks is 1, and the corresponding resource allocation coefficient is 1. During times 3-5, the number of running tasks is 2, and the corresponding resource allocation coefficient is 0.5. Therefore, for task 1, its corresponding resource allocation coefficient is the integral of the resource allocation coefficients corresponding to each time point from 1 to 5, which is 1×2+0.5×2=3. That is to say, if the initial resource is y, then in the above simplified embodiment, the resource obtained by task 1 is 3y.

[0096] In this embodiment, the number of tasks running at each moment within the runtime segment is determined by the runtime segment of the reference task in the reference task queue, thereby determining the resource allocation coefficient corresponding to each moment. Then, for each reference task, the resource allocation coefficient corresponding to each moment within its runtime segment is integrated to obtain the resource allocation coefficient used to describe the ratio of the amount of resources obtained by the reference task within the runtime segment to the preset initial amount of resources.

[0097] like Figure 5 As shown, Figure 5 A flowchart illustrating the steps for determining the number of tasks to be run, as provided in this application embodiment, is described in detail below.

[0098] This application provides an implementation scheme for quickly determining the number of tasks to run when there are a large number of reference tasks, specifically including steps 501 to 505:

[0099] 501. Determine the start and end times of the reference task based on the runtime segments of the reference tasks in the reference task queue.

[0100] In this embodiment of the application, it is easy to understand that the runtime segment of the reference task can be represented by two moments, namely the start moment and the end moment, and the time period between the start moment and the end moment is the runtime segment of the reference task. Therefore, the start and end times of the reference task can be quickly determined based on its runtime segment.

[0101] 502. Arrange the start and end times of the reference task in sequence to obtain the start and end time sequence.

[0102] In this embodiment, the start and end times of each reference task are arranged sequentially to obtain the start and end time sequence. Similarly, for ease of understanding, the simplified embodiment mentioned above, in which the runtime segments of task 1 are 1 to 5 and the runtime segments of task 2 are 3 to 10, will be used for further explanation. In this case, the start and end time sequence is (1, 3, 5, 10).

[0103] 503, determine the number of starting tasks and the number of ending tasks before the target time in the start and end time sequence.

[0104] In this embodiment, the specific scheme for determining the number of start tasks and the number of end tasks is as follows: Initialize the number of start tasks and the number of end tasks to 0. Iterate through each moment in the start-end time sequence. If the moment is the start time of a reference task, increase the number of start tasks. If the moment is the end time of a reference task, increase the number of end tasks. The increase is equal to the number of start times for the corresponding reference tasks. That is, if a moment is the start time of three reference tasks, increase the number of start tasks by 3. Similarly, if the moment is the end time of four reference tasks, increase the number of end tasks by 4. Thus, after iterating through each moment in the start-end time sequence, for any target moment in the sequence, the number of start tasks and the number of end tasks before that target moment can be obtained.

[0105] 504, the difference between the number of start tasks and the number of end tasks is set as the number of running tasks at each time between the target time and the next time in the start-end time sequence.

[0106] In this embodiment of the application, the difference between the number of start tasks and the number of end tasks is calculated. This difference is the number of running tasks at each time between the target time and the next time in the start-end time sequence. This is because only the time in the start-end time sequence will affect the number of start tasks and the number of end tasks.

[0107] 505. The number of running tasks at each time point within the running segment is determined based on the number of running tasks at each time point between the target time point and the next time point in the start and end time point sequence.

[0108] In this embodiment of the application, it can be understood that the number of running tasks at each time between the target time and the next time in the start-end time sequence is the number of running tasks at the corresponding time in the running segment. Therefore, by determining the number of running tasks in each segment of the start-end time sequence, the number of running tasks at each time in the running segment can be obtained.

[0109] To facilitate understanding of the embodiments of this application, the following will be combined with Figures 3-5The content shown provides a complete implementation scheme for setting the target initial resource amount of the target task queue based on the task execution information of the reference task queue, which is detailed below:

[0110] 1) The task resource scheduling device requests the YARN historical logs from the backend and divides them into 15-minute intervals to obtain task execution information for each interval. This task execution information typically includes the task ID in the task queue, the resources required by the task, the task start time, and the task end time, etc. The task execution information is stored in a data table, as follows:

[0111] Task ID Required resources Task start time Mission End Time A 30 0 5 B 20 1 5 C 50 5 15

[0112] Taking the resource allocation coefficient of task A as an example, given that the initial resources of task A are y, the resources obtained are:

[0113]

[0114] Among them, T s Let T be the start time of task A. e Let t be the end time of task A. i To obtain the task time series after sorting all task start times and task end times, the task located at T... s and T e The time between, that is:

[0115] T s ≤t1≤...≤t k ≤T e

[0116] s i For in t i The number of tasks to start before the time, e i For in t i The number of tasks that have finished before a given time can be obtained by iterating through the task time series.

[0117] Therefore, substituting the data from the above data table into the formula provided above, we can obtain that the resources obtained by task A with an initial resource of y are: y×1+y / 2×4. Therefore, the resource allocation coefficient for task A is 3. Similarly, for task B, we can obtain that the resources obtained with an initial resource of y are: y / 2×4. Therefore, the resource allocation coefficient for task B is 2. For task C, we can obtain that the resources obtained with an initial resource of y are: y×10. Therefore, the resource allocation coefficient for task C is 10.

[0118] At this point, for each task A, B, and C, it is required that it obtains sufficient resources to meet its needs. Therefore, the initial resource y should satisfy the following requirements:

[0119]

[0120] By combining these three factors, the initial resources should satisfy: y≥10. The minimum value, y=10, can be used as the initial allocation resource for the target task queue.

[0121] like Figure 6 As shown, Figure 6 A flowchart illustrating a step for setting a target initial resource quantity based on a value range, as provided in this application embodiment, is described in detail below.

[0122] This application provides a specific implementation scheme for setting the target initial resource amount according to multiple value ranges when there are multiple reference tasks. Specifically, it includes steps 601 to 602:

[0123] 601. If the reference task includes multiple tasks, the intersection of the value ranges of the initial resources corresponding to each reference task is obtained to obtain the reference value range.

[0124] In this embodiment of the application, it can be understood that if there are multiple reference tasks, the resource requirements corresponding to each reference task need to be met. Therefore, it is necessary to find the intersection of the value ranges of the initial resources corresponding to each reference task, and the resulting new value range, that is, each initial resource within the reference value range, can ensure that the resources obtained by each reference task meet its resource requirements.

[0125] 602, set the minimum initial resource amount in the reference value range as the target initial resource amount of the target task queue.

[0126] In this embodiment of the application, as can be seen from the foregoing description, each initial resource within the reference value range can enable each reference task to obtain the resources required by it. Therefore, setting the minimum initial resource amount in the reference value range as the target initial resource amount of the target task queue can meet the needs of each task while minimizing the amount of resource allocation, thus avoiding resource waste.

[0127] The solution provided in this application embodiment, by finding the intersection of the value ranges of the initial resources corresponding to each reference task, and setting the minimum initial resource amount in the obtained reference value range as the target initial resource amount of the target task queue, can meet the needs of each task while minimizing the amount of resource allocation, thereby further avoiding resource waste and improving resource utilization.

[0128] like Figure 7 As shown, Figure 7 A flowchart illustrating the steps for determining a target time period and a reference task queue, as provided in this application embodiment, specifically includes steps 701 to 703:

[0129] 701. Divide the preset time period into multiple target time periods based on the preset time interval.

[0130] In this embodiment, the time interval can be set based on the user's actual needs, for example, every 15 minutes, every 30 minutes, or every hour. It should be noted that a shorter time interval results in a higher efficiency in resource allocation and higher resource utilization, but also increases the computational load for calculating the initial resource quantity for each target time period. Conversely, a shorter time interval results in a lower computational load for calculating the initial resource quantity for each target time period, but this will affect the efficiency of resource allocation to some extent. As an optional embodiment, a time interval of 15 minutes is selected.

[0131] In this embodiment of the application, as described above, some tasks need to be performed daily; therefore, a 24-hour time period can be used. Specifically, the 24 hours of a day are divided into 15-minute intervals to obtain multiple target time periods.

[0132] 702, the previous time period of the preset time period is divided according to the preset time interval to obtain the reference time period corresponding to each target time period.

[0133] In this embodiment of the application, taking the aforementioned division of a 24-hour day into 15-minute intervals as an example, the 24 hours of yesterday can also be divided into 15-minute intervals as an example. Furthermore, the multiple time periods obtained from the division are the reference time periods corresponding to the target time period. For example, for the target time period of 8:00 to 8:15, the corresponding reference time period is 8:00 to 8:15 of yesterday.

[0134] 703, set the task queues within the reference time periods corresponding to each target time period as the reference task queues corresponding to each target time period.

[0135] In this embodiment of the application, after obtaining the reference time period corresponding to the target time period, the task queue and its running status within the reference time period can be obtained based on the historical records of YARN. At this time, the task queue within the reference time period can be understood as the reference task queue corresponding to the target time period.

[0136] It should be noted that the technical solution provided in this application embodiment can be understood as follows: at 0:00 on each day, by statistically analyzing the task queues and their corresponding task operation information for each reference time period in the past 24 hours, the initial resource amount for each corresponding target time period of today is set accordingly. When the target time period arrives, resource scheduling is completed directly based on the pre-set initial resource amount, and then resource allocation for each task in the queue is realized based on a preset fair strategy.

[0137] like Figure 8 As shown, Figure 8 The following is a detailed flowchart illustrating the steps for allocating resources to a target task queue, as provided in an embodiment of this application.

[0138] In this embodiment of the application, considering that multiple target task queues may exist simultaneously within a certain target time period in the data platform, after specifically setting the initial resource amount of each target task queue within the target time period, it is further considered whether the data platform has implemented the allocation of the initial resource amount of each target task queue. Specifically, this includes steps 801 to 803:

[0139] 801. Calculate the sum of the target initial resource amounts for each of the target task queues, and determine whether the sum of the target initial resource amounts is higher than the preset maximum initial resource amount. If yes, proceed to step 802; otherwise, proceed to other steps.

[0140] In this embodiment of the application, the preset maximum initial resource amount is the maximum allowed resource that the data platform can provide, and this value is usually preset.

[0141] In this embodiment, for multiple target task queues within a target time period, the sum of the target initial resource amounts for each target task queue is first calculated. The result can be understood as the total amount of resources required for multiple target task queues to execute simultaneously with the predetermined initial resource amounts within the target time period. Therefore, if the sum is higher than the preset maximum initial resource amount, meaning the total amount of resources required for multiple target task queues to execute simultaneously with the predetermined initial resource amounts within the target time period exceeds the maximum allowable resources provided by the data platform, then the resources provided by the data platform cannot simultaneously meet the needs of all task queues. Therefore, the initial resource amounts of each target task queue need to be adjusted accordingly. This is specifically illustrated in subsequent step 802. Conversely, if the sum is not higher than the preset maximum initial resource amount, meaning the resources provided by the data platform can simultaneously meet the needs of all task queues, then resources can be allocated to each target task queue according to the target initial resources of each target queue.

[0142] 802. Based on the preset maximum initial resource amount and the priority corresponding to each target task queue, determine the actual initial resource corresponding to each target task queue.

[0143] In this embodiment, the priority of the target task queue is pre-set, which can be equivalently understood as the importance of the target task queue. Therefore, if the resources provided by the data platform cannot simultaneously meet the needs of all task queues, the resources provided by the data platform should prioritize the target task queues with higher priority. Thus, it is necessary to combine the resources provided by the data platform (i.e., the preset maximum resource amount) with the priority of each target task queue, and redetermine the actual initial resources corresponding to each target task queue while prioritizing the needs of the target task queues with higher priority. In this case, it can be understood that the actual initial resources of some high-priority target task queues are equal to their corresponding target initial resources, while the actual initial resources of some low-priority target task queues are 0.

[0144] 803. Allocate resources to each target task queue according to the actual initial resources corresponding to each target task queue.

[0145] In this embodiment of the application, after redetermining the actual initial resources corresponding to each target task queue based on the priority of each target task queue, similar to the above, the resources of each target task queue can be allocated according to the actual initial resources of each target queue.

[0146] In addition, it should be noted that only after some high-priority target task queues are completed and the data platform's resources are released will the task resource scheduling allocate the released resources to lower-priority target task queues.

[0147] like Figure 9 The diagram shown is a complete implementation illustration of task resource scheduling provided in an embodiment of this application. Details are as follows.

[0148] In this embodiment of the application, the complete implementation process of task resource scheduling is as follows:

[0149] 1) The task resource scheduling device requests the background to obtain the task running information in the YARN historical log, divides it into 15-minute intervals, obtains the task queue and its task running information for each time period, and stores it in a file in the form of the aforementioned data table.

[0150] 2) Based on the execution order of tasks in the data platform, rearrange the task list to obtain a task time sequence consisting of task start time and task end time;

[0151] 3) Traverse the task time series and calculate the initial resources of the target task queue within the target time period using the aforementioned scheme;

[0152] 4) The task resource scheduling device requests real-time resource status;

[0153] 5) The controller combines the real-time resource status and the initial resources of the target task queue within the target time period, and dynamically adjusts the resource allocation of each task queue in the data platform based on priority.

[0154] To better implement the task resource scheduling method in the embodiments of this application, based on the task resource scheduling method, the embodiments of this application also provide a task resource scheduling device, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of a task resource scheduling device provided in an embodiment of this application. Specifically, the task resource scheduling device includes:

[0155] The acquisition module 1001 is used to acquire the target task queue within the target time period, and the task operation information of the reference task queue corresponding to the target time period.

[0156] The calculation module 1002 is used to calculate the target initial resource quantity of the target task queue based on the task running information of the reference task queue.

[0157] The allocation module 1003 is used to allocate resources to the target task queue according to the target initial resource amount.

[0158] As a feasible embodiment of this application, the task execution information of the aforementioned reference task queue includes the runtime segment and required resource amount of the reference task in the reference task queue; the aforementioned calculation module includes:

[0159] The allocation coefficient calculation module is used to calculate the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue.

[0160] The value range calculation module is used to calculate the value range of the initial resource quantity corresponding to the reference task based on the resource allocation coefficient and the required resource quantity of the reference task.

[0161] The initial resource quantity setting module is used to set the target initial resource quantity of the target task queue according to the value range of the initial resource quantity corresponding to the reference task.

[0162] As a feasible embodiment of this application, the above-mentioned allocation coefficient calculation sub-module includes:

[0163] The task number determination unit is used to determine the number of running tasks at each time point within the runtime segment based on the runtime segment of the reference task in the reference task queue.

[0164] The time allocation coefficient setting unit is used to set the resource allocation coefficient corresponding to each time moment in the runtime segment according to the number of running tasks at each time moment in the runtime segment;

[0165] The task allocation coefficient determination unit is used to integrate the resource allocation coefficients corresponding to each time point within the runtime segment to obtain the resource allocation coefficients corresponding to the reference task.

[0166] In some embodiments of this application, the above-mentioned task quantity determination unit includes:

[0167] The start and end time determination sub-unit is used to determine the start and end times of the reference task based on the runtime segments of the reference tasks in the reference task queue.

[0168] The sorting sub-unit is used to arrange the start and end times of the reference task in order to obtain a start and end time sequence;

[0169] The task quantity statistics sub-unit is used to determine the number of starting tasks and the number of ending tasks before the target time in the start-end time sequence.

[0170] The task quantity setting sub-unit is used to set the difference between the start task quantity and the end task quantity as the number of tasks to be run at each time between the target time and the next time in the start and end time sequence.

[0171] The task quantity determination subunit is used to determine the number of tasks to run at each time point within the runtime segment based on the number of tasks to run at each time point between the target time point and the next time point in the start-end time sequence.

[0172] In some embodiments of this application, the above-mentioned initial resource quantity setting sub-module includes:

[0173] The intersection calculation unit is used to calculate the intersection of the value ranges of the initial resources corresponding to each of the reference tasks if the reference tasks include multiple ones, so as to obtain the reference value range.

[0174] An initial resource quantity setting unit is used to set the minimum initial resource quantity in the reference value range as the target initial resource quantity of the target task queue.

[0175] In some embodiments of this application, the task resource scheduling device includes a time period division module, which includes:

[0176] The target time period segmentation module is used to divide a preset time period into multiple target time periods based on a preset time interval.

[0177] The reference time period division module is used to divide the previous time period of the preset time period according to the preset time interval, so as to obtain the reference time period corresponding to each target time period.

[0178] The reference task queue determination module is used to set the task queues within the reference time period corresponding to each target time period as the reference task queues corresponding to each target time period.

[0179] In some embodiments of this application, the allocation module includes:

[0180] The actual initial resource calculation sub-module is used to determine the actual initial resources corresponding to each target task queue based on the preset maximum initial resource amount and the priority corresponding to each target task queue if the sum of the target initial resource amounts of each target task queue is higher than the preset maximum initial resource amount.

[0181] The resource allocation submodule is used to allocate resources to each of the target task queues according to the actual initial resources corresponding to each target task queue.

[0182] This invention also provides a task resource scheduling device, such as... Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of a task resource scheduling device provided in an embodiment of this application.

[0183] The task resource scheduling device includes a memory, a processor, and a task resource scheduling program stored in the memory and capable of running on the processor. When the processor executes the task resource scheduling program, it implements the steps in the task resource scheduling method of any of the above embodiments.

[0184] Specifically, the task resource scheduling device may include components such as a processor 1101 with one or more processing cores, a memory 1102 with one or more storage media, a power supply 1103, and an input unit 1104. Those skilled in the art will understand that... Figure 11 The task resource scheduling device structure shown does not constitute a limitation on the task resource scheduling device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0185] The processor 1101 is the control center of the task resource scheduling device. It connects to various parts of the device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 1102, and by calling data stored in the memory 1102, it performs various functions and processes data, thereby providing overall monitoring of the task resource scheduling device. Optionally, the processor 1101 may include one or more processing cores; preferably, the processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1101.

[0186] The memory 1102 can be used to store software programs and modules. The processor 1101 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of task resource scheduling devices, etc. In addition, the memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1102 may also include a memory controller to provide the processor 1101 with access to the memory 1102.

[0187] The task resource scheduling equipment also includes a power supply 1103 that supplies power to various components. Preferably, the power supply 1103 can be logically connected to the processor 1101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1103 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0188] The task resource scheduling device may also include an input unit 1104, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0189] Although not shown, the task resource scheduling device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 1101 in the task resource scheduling device loads the executable files corresponding to the processes of one or more applications into the memory 1102 according to the following instructions, and the processor 1101 runs the applications stored in the memory 1102, thereby implementing the steps in any of the task resource scheduling methods provided in the embodiments of the present invention.

[0190] Therefore, embodiments of the present invention provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. The computer-readable storage medium stores a task resource scheduler, which, when executed by a processor, implements the steps of any of the task resource scheduling methods provided in the embodiments of the present invention.

[0191] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0192] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0193] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0194] The above provides a detailed description of a task resource scheduling method provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for task resource scheduling, the method comprising: include: Obtain the target task queue within the target time period, and the task execution information of the reference task queue corresponding to the target time period; Calculate the target initial resource quantity of the target task queue based on the task execution information of the reference task queue; Allocate resources to the target task queue based on the initial target resource quantity; The task execution information of the reference task queue includes the runtime segment and required resource amount of the reference tasks in the reference task queue; The step of calculating the target initial resource amount of the target task queue based on the task execution information of the reference task queue includes: Calculate the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue; The range of values ​​for the initial resource quantity corresponding to the reference task is calculated based on the resource allocation coefficient and the required resource quantity for the reference task. The target initial resource quantity of the target task queue is set according to the range of values ​​for the initial resource quantity corresponding to the reference task.

2. The task resource scheduling method according to claim 1, characterized in that, The step of calculating the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue includes: The number of running tasks at each time point within the runtime segment is determined based on the runtime segment of the reference task in the reference task queue. The resource allocation coefficient corresponding to each moment in the runtime segment is set according to the number of running tasks at each moment in the runtime segment; By integrating the resource allocation coefficients corresponding to each moment within the runtime segment, the resource allocation coefficients corresponding to the reference task are obtained.

3. The task resource scheduling method according to claim 2, characterized in that, The step of determining the number of running tasks at each time point within a runtime segment based on the runtime segments of reference tasks in the reference task queue includes: The start and end times of the reference task are determined based on the runtime segments of the reference tasks in the reference task queue. Arrange the start and end times of the reference task in order to obtain the start and end time sequence; Determine the number of starting tasks and the number of ending tasks before the target time in the start-end time sequence; The difference between the number of start tasks and the number of end tasks is set as the number of running tasks at each time between the target time and the next time in the start-end time sequence; The number of running tasks at each moment within the runtime segment is determined based on the number of running tasks at each moment between the target moment and the next moment in the start-end time sequence.

4. The task resource scheduling method according to claim 1, characterized in that, The step of setting the target initial resource quantity of the target task queue according to the value range of the initial resource quantity corresponding to the reference task includes: If the reference task includes multiple tasks, the intersection of the value ranges of the initial resources corresponding to each reference task is obtained to obtain the reference value range. Set the minimum initial resource amount in the reference value range as the target initial resource amount of the target task queue.

5. The task resource scheduling method according to claim 1, characterized in that, Before the step of obtaining the target task queue within the target time period and the task execution information of the reference task queue corresponding to the target time period, the method further includes: The preset time period is divided into multiple target time periods based on the preset time interval; The previous time period of the preset time period is divided according to the preset time interval to obtain the reference time period corresponding to each target time period. Set the task queues within the reference time periods corresponding to each target time period as the reference task queues corresponding to each target time period.

6. The task resource scheduling method according to any one of claims 1 to 5, characterized in that, The step of allocating resources to the target task queue based on the target initial resources includes: If the sum of the target initial resource amounts of each target task queue is higher than the preset maximum initial resource amount, then the actual initial resource amount corresponding to each target task queue is determined according to the preset maximum initial resource amount and the priority corresponding to each target task queue. Resource allocation is performed on each target task queue according to the actual initial resources corresponding to each target task queue.

7. A task resource scheduling device, characterized in that, include: The acquisition module is used to acquire the target task queue within the target time period, and the task execution information of the reference task queue corresponding to the target time period; The calculation module is used to calculate the target initial resource quantity of the target task queue based on the task execution information of the reference task queue; The allocation module is used to allocate resources to the target task queue according to the target initial resource amount; The task execution information of the reference task queue includes the runtime segment and required resource amount of the reference tasks in the reference task queue; The step of calculating the target initial resource quantity of the target task queue based on the task execution information of the reference task queue includes: Calculate the resource allocation coefficient corresponding to the reference task based on the runtime segment of the reference task in the reference task queue; The range of values ​​for the initial resource quantity corresponding to the reference task is calculated based on the resource allocation coefficient and the required resource quantity for the reference task. The target initial resource quantity of the target task queue is set according to the range of values ​​for the initial resource quantity corresponding to the reference task.

8. A task resource scheduling device, characterized in that, The task resource scheduling device includes a processor, a memory, and a task resource scheduling program stored in the memory and executable on the processor. The processor executes the task resource scheduling program to implement the steps of the task resource scheduling method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a task resource scheduler, which is executed by a processor to implement the steps of the task resource scheduling method according to any one of claims 1 to 6.

Images