Resource scheduling method, resource scheduling device, storage medium and electronic device

Abstract

The present application provides a resource scheduling method, a resource scheduling device, a storage medium and an electronic device, the method comprising: obtaining an identification number of a target task; determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task; in the case where there is a historical memory record of the target task in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, the historical memory record of the target task including the memory value of the target task successfully executed within the historical time period; in the case where there is no historical memory record of the target task in the historical task record, determining an initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number. The method analyzes the historical task records, accurately determines the memory consumption of regularly executed tasks, continuously optimizes the memory value allocated to the tasks, and avoids the problem of job failure due to insufficient memory.

Description

Technical Field

[0001] The present application relates to the field of resource scheduling, and in particular to a resource scheduling method, a resource scheduling device, a computer-readable storage medium, and an electronic device. Background Art

[0002] The current scheduling system cannot obtain the overall resource usage of the database and the resource usage of each job. It can only passively allocate resources for job execution and cannot determine whether the allocated resources are appropriate. In other words, the existing scheduling system has the following shortcomings:

[0003] (1) A task is always assigned a fixed initial resource, which does not distinguish between different tasks. Since the resources required for specific execution will be adjusted after execution, the actual memory used will be significantly different from the initial memory.

[0004] (2) At present, the resource optimization process is generally completed manually, which takes a long time and needs to be coordinated with other optimization processes. It has great uncertainty and it is not easy to find the optimal operating resources for the task, which easily leads to low resource utilization.

[0005] (3) After suspending and waiting, it often takes a while to allocate memory, which increases the waiting time.

[0006] (4) After the error is reported, locate the cause of the problem. During the error analysis, you may encounter situations where the memory limit is exceeded or the memory is tight due to other tasks.

[0007] (5) There is no appropriate readjustment mechanism. One is to readjust in a continuous cycle without analyzing the cause of the error, resulting in errors due to resource problems that cannot be resolved; the other is to manually handle the error and distinguish whether it can be resolved by readjustment or manually allocating resources for execution.

[0008] (6) Since scheduling only uses concurrency to control the number of jobs running in the cluster, if there are multiple jobs with large resource usage, the cluster will be stuck. It is necessary to dynamically adjust concurrency and identify the cluster status in real time to avoid cluster stuck. Summary of the Invention

[0009] The main purpose of this application is to provide a resource scheduling method, a resource scheduling device, a computer-readable storage medium and an electronic device, so as to at least solve the problem that the scheduling system of the prior art cannot obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job running, resulting in low memory utilization.

[0010] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a resource scheduling method is provided, including: obtaining an identification number of a target task, wherein the identification number is used to uniquely identify the target task; determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task; in a case where there is a historical memory record of the target task in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, wherein the historical memory record of the target task includes a memory value for successfully executing the target task within a historical time period; in a case where there is no historical memory record of the target task in the historical task record, determining the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number.

[0011] Optionally, when there is a historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the historical memory record of the target task, including: when there is a historical memory record of the target task in the historical task record, the historical memory cycle of the target task is determined according to the historical memory record of the target task, the historical memory cycle is obtained according to the rule of memory consumed by successfully executing the target task within a historical time period; constructing a first memory calculation formula y(T) = avg(y(Tn)), wherein y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; according to the first memory calculation formula, the initial memory value for executing the target task is determined.

[0012] Optionally, there are multiple historical memory records of the target task. When there are historical memory records of the target task in the historical task records, the initial memory value for executing the target task is determined according to the historical memory records of the target task, including: when there are historical memory records of the target task in the historical task records, the multiple historical memory records of the target tasks are split into at least one historical memory record set according to the historical execution time corresponding to the historical memory records of each target task, one historical memory record set corresponds to one historical execution period, and the historical execution time of all the historical memory records in one historical memory record set is within the historical execution period corresponding to the historical memory record set; according to the historical execution period corresponding to each historical memory record set, the calculation weight corresponding to each historical memory record set is determined, wherein the smaller the time difference between the historical execution period and the current moment, the greater the calculation weight of the historical memory record set corresponding to the historical execution period; according to all the historical memory records of the target tasks and the calculation weight corresponding to each historical memory record set, the initial memory value for executing the target task is determined.

[0013] Optionally, there are two historical memory record sets, namely a first record set and a second record set. The initial memory value for executing the target task is determined based on all the historical memory records of the target task and the calculation weights corresponding to each historical memory record set, including: constructing a second memory calculation formula: y(T) = A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), wherein y(T) is the initial memory value, T is the start execution time of the target task, A is the calculation weight corresponding to the first record set, B is the calculation weight corresponding to the second record set, y(T-1) to (T-7) are respectively the historical memory records in the first record set, and y(T-8)+…+y(T-15) are respectively the historical memory records in the second record set; according to the second memory calculation formula, the initial memory value for executing the target task is determined.

[0014] Optionally, when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined based on the total cluster resources and the ideal concurrency number, including: when there is no historical memory record of the target task in the historical task record, the ratio of the total cluster resources to the ideal concurrency number is determined as the initial memory value for executing the target task; and / or, when there is no historical memory record of the target task in the historical task record, after determining the initial memory value for executing the target task based on the total cluster resources and the ideal concurrency number, the method also includes: when the initial memory value does not support the complete execution of the target task, a preset multiple of the initial memory value is determined as the reallocated memory value of the target task, and the reallocated memory value is used to execute the target task.

[0015] Optionally, when the initial memory value does not support the complete execution of the target task, a preset multiple of the initial memory value is determined as the reallocated memory value of the target task, and the reallocated memory value is used to execute the target task, including: when the free memory value is less than the reallocated memory value, determining the job tasks that can be completed within a first preset time period, and summing the memory occupied by the job tasks to obtain a reserve memory value; when the reserve memory value is greater than or equal to the reallocated memory value, retaining the memory allocated to the target task, and raising the priority of the target task so that the target task is executed immediately after the job task is completed; when the reserve memory value is less than the reallocated memory value, releasing the memory allocated to the target task, and placing the target task back in the task queue sequence until the free memory value is greater than or equal to the reallocated memory value, and then executing the target task.

[0016] Optionally, the method also includes: when the number of job errors within a second preset time period is greater than or equal to a preset number, determining the free memory value at the current moment; when the free memory value at the current moment is less than or equal to the preset memory value, deleting the unexecuted tasks at the current moment from the task queue sequence, and reducing the number of concurrent tasks scheduled until the free memory value is greater than the preset memory value, and then placing the unexecuted tasks back into the task queue sequence.

[0017] According to another aspect of the present application, a resource scheduling device is provided, including: an acquisition unit for acquiring an identification number of a target task, wherein the identification number is used to uniquely identify the target task; a first determination unit for determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task; a second determination unit for determining, when there is a historical memory record of the target task in the historical task record, an initial memory value for executing the target task based on the historical memory record of the target task, the historical memory record of the target task including the memory value of successfully executing the target task within a historical time period; a third determination unit for determining, when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number.

[0018] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the resource scheduling methods.

[0019] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include means for executing any one of the resource scheduling methods described.

[0020] Applying the technical solution of the present application, the above-mentioned resource scheduling method first obtains the identification number of the target task, which is used to uniquely identify the target task; secondly, based on the identification number of the target task, determines whether there is a historical memory record of the target task in the historical task record; then, if there is a historical memory record of the target task in the historical task record, determines the initial memory value for executing the target task based on the historical memory record of the target task, and the historical memory record of the target task includes the memory value of the target task successfully executed within the historical time period; finally, if there is no historical memory record of the target task in the historical task record, determines the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number. This method analyzes the historical task records to make a clearer judgment on the memory of regularly executed tasks and their time consumption, continuously optimizes the memory value allocated to the task, and makes it closer to the actual required memory value, avoiding job failures due to insufficient memory and low memory utilization due to excessive memory allocation, and solves the problem that the scheduling system of the prior art cannot obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job operation, resulting in low memory utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0022] Figure 1 A hardware structure block diagram of a mobile terminal for executing a resource scheduling method provided in an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of a resource scheduling method according to an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of a process of another resource scheduling method provided according to an embodiment of the present application is shown;

[0025] Figure 4 A structural block diagram of a resource scheduling device provided according to an embodiment of the present application is shown.

[0026] The above drawings include the following reference numerals:

[0027] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0032] Scheduling: A program that executes tasks at regular intervals;

[0033] Suspend: Temporarily release memory and wait for sufficient resources to be given to execute again.

[0034] As introduced in the background technology, the scheduling system in the prior art is unable to obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job running, and cannot determine whether the allocated resources are appropriate. In order to solve the problem that the scheduling system in the prior art is unable to obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job running, resulting in low memory utilization, the embodiments of the present application provide a resource scheduling method, a resource scheduling device, a computer-readable storage medium and an electronic device.

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG is a hardware structure block diagram of a mobile terminal of a resource scheduling method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the resource scheduling method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-mentioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of such networks may include a wireless network provided by the mobile terminal's telecommunications provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] In this embodiment, a resource scheduling method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Figure 2 Flowchart of the resource scheduling method according to the embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0040] Step S201, obtaining an identification number of a target task, the identification number being used to uniquely identify the target task;

[0041] Specifically, the target task's identification number can be used to search for historical records of the target task. For example, during actual execution, a certain number of processing cores and memory must be allocated in advance, and a certain upward floating value can be adjusted during the allocation process. Different core and memory allocations can affect job execution efficiency. For tasks that need to be executed regularly, it is necessary to estimate the approximate memory consumption of the task before execution.

[0042] By analyzing historical records, we can obtain the historical resource allocation and running time of the task, so as to analyze the resource consumption of the next task and continuously optimize the resource consumption based on previous records.

[0043] The current scheduler automatically allocates a fixed amount of resources to the task at the start of the task, with a relatively loose upward floating value. If the allocated resources exceed the required resources, the excess memory is reclaimed. If memory resources are insufficient, additional resources are allocated until the resource requirements are met. If no more resources can be allocated and the task still requests resources, the task is temporarily suspended and continues to execute after resources are allocated. If the task exceeds the resource limit or is suspended for an excessively long time, an error is reported, the task is rolled back, and resources are released.

[0044] Step S202: determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task;

[0045] Specifically, the historical task record also includes the job identification number (task identification number), start execution time, current resource status, actual resource consumption, running time and other content to ensure the continuation of subsequent operations.

[0046] Step S203: if a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, wherein the historical memory record of the target task includes memory values ​​of successful executions of the target task within a historical time period;

[0047] Specifically, when a periodic task has a history, current consumption is judged based on historical consumption. First, determine whether the periodic task has a clear pattern with respect to dates and plot a line chart of memory consumption against execution date. If no clear pattern is observed, perform a weighted calculation on the historical records. Initially, use recent historical records of the same task to infer the approximate consumption of this execution. This allows you to initially define the approximate consumption within an appropriate range, limiting subsequent adjustments. This makes resource allocation easier and prevents prolonged hangs or out-of-memory reports.

[0048] Wherein, in a case where a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task according to the historical memory record of the target task includes the following steps:

[0049] Step S2031: If a historical memory record of the target task exists in the historical task record, a historical memory period of the target task is determined based on the historical memory record of the target task. The historical memory period is obtained based on a pattern of memory consumption by successfully executing the target task within a historical time period.

[0050] Step S2032: construct a first memory calculation formula y(T)=avg(y(Tn)), where y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle.

[0051] Step S2033: Determine the initial memory value for executing the target task according to the first memory calculation formula.

[0052] Specifically, this can infer the approximate consumption of this execution, and can define the approximate consumption within a suitable range at the beginning. The range of subsequent adjustments will not be too large, which makes it easier to apply for resources and can avoid long-term suspension or memory overflow issues.

[0053] Generally, the memory consumption period of the target task is determined based on a line chart of the target task's memory consumption in historical records relative to the execution date.

[0054] There are multiple historical memory records of the target task. If a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task further includes the following steps:

[0055] Step S301: If there are historical memory records of the target task in the historical task records, split the historical memory records of the target tasks into at least one historical memory record set according to the historical execution time corresponding to each historical memory record of the target task, wherein each historical memory record set corresponds to a historical execution period, and the historical execution times of all the historical memory records in the historical memory record set are within the historical execution period corresponding to the historical memory record set;

[0056] Step S302: determining a calculation weight corresponding to each of the historical memory record sets based on the historical execution period corresponding to each of the historical memory record sets, wherein the smaller the time difference between the historical execution period and the current time, the greater the calculation weight of the historical memory record set corresponding to the historical execution period;

[0057] Step S303 : determining an initial memory value for executing the target task according to all the historical memory records of the target task and the calculation weight corresponding to each of the historical memory record sets.

[0058] Specifically, this can infer the approximate consumption of this execution, and can define the approximate consumption within a suitable range at the beginning. The range of subsequent adjustments will not be too large, which makes it easier to apply for resources and can avoid long-term suspension or memory overflow issues.

[0059] There are two historical memory record sets, namely a first record set and a second record set. Determining an initial memory value for executing the target task based on all historical memory records of the target task and the calculated weights corresponding to the historical memory record sets includes the following steps:

[0060] Step S3031, construct a second memory calculation formula:

[0061] y(T)=A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), where y(T) is the initial memory value, T is the start execution time of the target task, A is the calculation weight corresponding to the first record set, B is the calculation weight corresponding to the second record set, y(T-1) to (T-7) are the historical memory records in the first record set, and y(T-8)+…+y(T-15) are the historical memory records in the second record set.

[0062] Step S3032: Determine the initial memory value for executing the target task according to the second memory calculation formula.

[0063] Specifically, this can infer the approximate consumption of this execution, and can define the approximate consumption within a suitable range at the beginning. The range of subsequent adjustments will not be too large, which makes it easier to apply for resources and can avoid long-term suspension or memory overflow issues.

[0064] Generally, the first record set records the target task's history within the past 7 days, and the second record set records the target task's history within the past 15 to 7 days. In this embodiment, the weights are 70% for the past 7 days (i.e., A is 0.7) and 30% for the past 15 days (i.e., B is 0.3), and this is used to determine the memory consumption value. The data recorded in the first and second record sets can be adjusted according to actual conditions, and the values ​​of A and B can also be adjusted accordingly.

[0065] Step S204 : when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the total cluster resources and the ideal concurrency number.

[0066] Specifically, when a new periodic task begins executing, it is assigned a unique identification number. Since no history records exist for this task, initial resources are automatically allocated to the task. These resources are calculated by dividing the total cluster resources by the desired concurrency. If the task executes successfully, the actual memory consumption and run time are recorded in the history log. If the task fails due to insufficient memory, the pre-allocated memory value is increased to 1.1 times the pre-allocated memory value and the task continues until it succeeds.

[0067] Among them, when there is no historical memory record of the above-mentioned target task in the above-mentioned historical task record, the above-mentioned initial memory value for executing the above-mentioned target task is determined based on the total cluster resources and the ideal concurrency number, including: when there is no historical memory record of the above-mentioned target task in the above-mentioned historical task record, the ratio of the above-mentioned total cluster resources to the ideal concurrency number is determined as the initial memory value for executing the above-mentioned target task; and / or, when there is no historical memory record of the above-mentioned target task in the above-mentioned historical task record, after determining the above-mentioned initial memory value for executing the above-mentioned target task based on the total cluster resources and the ideal concurrency number, the above-mentioned method also includes: when the above-mentioned initial memory value does not support the complete execution of the above-mentioned target task, determining a preset multiple of the above-mentioned initial memory value as the reallocated memory value of the above-mentioned target task, and using the above-mentioned reallocated memory value to execute the above-mentioned target task.

[0068] Specifically, by continuously optimizing the memory value allocated to tasks to bring it closer to the actual memory value required, we can avoid job failures due to insufficient memory and low memory utilization due to excessive memory allocation. Generally, the default multiplier is 1.1.

[0069] Wherein, if the initial memory value does not support the complete execution of the target task, a preset multiple of the initial memory value is determined as the reallocated memory value of the target task, and the target task is executed using the reallocated memory value, including the following steps:

[0070] Step S401: If the free memory value is less than the reallocated memory value, determine the tasks that can be completed within a first preset time period, and sum the memory occupied by the tasks to obtain a reserve memory value.

[0071] Step S402: If the reserve memory value is greater than or equal to the reallocated memory value, retain the memory allocated to the target task and increase the priority of the target task so that the target task is executed immediately after the job task is completed.

[0072] Step S403, when the above-mentioned prepared memory value is less than the above-mentioned reallocated memory value, the memory allocated to the above-mentioned target task will be released, and the above-mentioned target task will be placed back into the task queue sequence until the above-mentioned free memory value is greater than or equal to the above-mentioned reallocated memory value, and then the above-mentioned target task will be executed.

[0073] Specifically, by using execution history records, we can have a more accurate estimate of the time of the currently executing task, determine whether there are jobs that can release memory in the near future, and control the suspension time of the task. For tasks that may be suspended for a long time, we can release or allocate memory as soon as possible to avoid long suspension time, which will lead to extended execution time.

[0074] Among them, the first preset time period can be 5 minutes. During the execution process, if there is insufficient memory, the pre-allocated memory will be increased to 1.1 times. If the memory free value is greater than the reallocated memory value, the corresponding memory will be allocated and execution will continue; if the memory free value is not enough for reallocation, the running tasks will be found, and the average running time in the historical records will be found through the task identifier, and the job tasks that can be completed within 5 minutes will be screened out, and the memory consumption of these tasks will be summed: if it is greater than the memory value to be reallocated, the task priority will be increased, the task will be suspended, and execution will continue after the memory meets the conditions; if the reallocated memory value is not met, the task will be released, the pre-allocated memory will be increased to 1.1 times, and then it will return to the queue sequence and be re-executed after the resources are met.

[0075] The above method further comprises the following steps:

[0076] Step S501: if the number of job error reports within the second preset time period is greater than or equal to a preset number, determine the free memory value at the current moment;

[0077] Step S502, when the above-mentioned free memory value at the current moment is less than or equal to the preset memory value, the unexecuted tasks at the current moment are deleted from the task queue sequence, and the number of scheduled concurrent tasks is reduced until the above-mentioned free memory value is greater than the above-mentioned preset memory value, and then the above-mentioned unexecuted tasks are placed back into the above-mentioned task queue sequence.

[0078] Specifically, if an error is reported due to exceeding memory limits, an error will be reported directly. However, tasks facing memory shortages will be returned to the queue for continued execution after memory reallocation, achieving a certain degree of automation and reducing manual work. For multiple resource-intensive jobs running simultaneously, the scheduler will identify the current remaining cluster resources, adjust the concurrency ratio in a timely manner, and control the number of subsequent jobs entering the database to reduce the possibility of database freezes.

[0079] If the pre-allocated memory exceeds a certain range after multiple re-executions, there may be a problem with the judgment data or execution statement. The task will automatically exit the queue and report an error. The execution history will also be printed for analysis of the cause of the error.

[0080] If a job reports errors multiple times within a period of time, analyze the current resource status at the time of the error. If the allocatable resources are less than 10% of all resources, suspend the unexecuted tasks and reduce all concurrency in scheduling until a task is completed and resources are released. After that, concurrency is restored and scheduling returns to normal.

[0081] The above-mentioned resource scheduling method of the present application first obtains the identification number of the target task, which is used to uniquely identify the target task; secondly, based on the identification number of the target task, determines whether there is a historical memory record of the target task in the historical task record; then, if there is a historical memory record of the target task in the historical task record, determines the initial memory value for executing the target task based on the historical memory record of the target task, and the historical memory record of the target task includes the memory value of the target task successfully executed within the historical time period; finally, if there is no historical memory record of the target task in the historical task record, determines the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number. This method analyzes the historical task records to make a clearer judgment on the memory of regularly executed tasks and their time consumption, and continuously optimizes the memory value allocated to the task to make it closer to the actual required memory value, avoiding job failures due to insufficient memory and low memory utilization due to excessive memory allocation. It solves the problem that the scheduling system of the prior art cannot obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job operation, resulting in low memory utilization.

[0082] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the resource scheduling method of the present application will be described in detail below with reference to specific embodiments.

[0083] This embodiment relates to a specific resource scheduling method, such as Figure 3 As shown, the following steps are included:

[0084] Step S1: First, determine whether the periodic task is being executed for the first time or has been executed multiple times (i.e., determine whether there is a history record of the task) based on the identification code of the periodic task;

[0085] Step S2: If the task is executed for the first time, memory is allocated according to a preset value to execute the task; if the task is executed multiple times, memory is allocated according to historical records to execute the task;

[0086] Step S3: If the task is successfully executed, the actual memory consumption and time consumed are recorded in the history record. If the task fails due to insufficient memory, the pre-allocated memory is adjusted to 1.1 times, and it is determined whether it exceeds the specified range. If so, an error is reported and the execution record is printed.

[0087] Step S4: If it does not exceed the specified range and there is remaining memory, the remaining memory is directly allocated and executed. If there is freeable memory within a certain period of time, memory is allocated and executed after the memory is released. If there is no available memory in a short period of time, an error is reported and the program re-enters the queue.

[0088] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0089] The embodiments of the present application also provide a resource scheduling device. It should be noted that the resource scheduling device of the embodiments of the present application can be used to execute the resource scheduling method provided by the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0090] The following introduces the resource scheduling device provided in the embodiment of the present application.

[0091] Figure 4 Schematic diagram of a resource scheduling device according to an embodiment of the present application. Figure 4As shown, the device includes an acquisition unit 10, a first determination unit 20, a second determination unit 30 and a third determination unit 40, the acquisition unit 10 is used to obtain the identification number of the target task, and the above identification number is used to uniquely identify the above target task; the first determination unit 20 is used to determine whether there is a historical memory record of the above target task in the historical task record according to the identification number of the above target task; the second determination unit 30 is used to determine the initial memory value for executing the above target task according to the historical memory record of the above target task when there is a historical memory record of the above target task in the above historical task record, and the historical memory record of the above target task includes the memory value of successfully executing the above target task within the historical time period; the third determination unit 40 is used to determine the above initial memory value for executing the above target task according to the total resources of the cluster and the ideal concurrency number when there is no historical memory record of the above target task in the above historical task record.

[0092] The above-mentioned resource scheduling device of the present application includes an acquisition unit, a first determination unit, a second determination unit and a third determination unit. The acquisition unit is used to obtain the identification number of the target task, and the identification number is used to uniquely identify the target task; the first determination unit is used to determine whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task; the second determination unit is used to determine the initial memory value for executing the target task based on the historical memory record of the target task when there is a historical memory record of the target task in the historical task record, and the historical memory record of the target task includes the memory value of the target task successfully executed within the historical time period; the third determination unit is used to determine the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number when there is no historical memory record of the target task in the historical task record. The device analyzes historical task records to make a clearer judgment on the memory and time consumption of regularly executed tasks, and continuously optimizes the memory value allocated to the task to make it closer to the actual memory value required, avoiding job failures due to insufficient memory and low memory utilization due to excessive memory allocation. It solves the problem that the scheduling system of the existing technology cannot obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job operation, resulting in low memory utilization.

[0093] In some optional instances, the second determination unit includes a first determination module, a first construction module, and a second determination module. The first determination module is used to determine the historical memory cycle of the target task based on the historical memory record of the target task when there is a historical memory record of the target task in the historical task record. The historical memory cycle is obtained based on the law of memory consumed by successfully executing the target task within the historical time period; the first construction module is used to construct a first memory calculation formula y(T) = avg(y(Tn)), where y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; the second determination module is used to determine the initial memory value used to execute the target task based on the first memory calculation formula. According to the first memory calculation formula, the initial memory value used to execute the target task is determined.

[0094] In some optional instances, there are multiple historical memory records of the above-mentioned target task, and the second determination unit includes a first splitting module, a third determination module and a fourth determination module. The first splitting module is used to split the historical memory records of the multiple target tasks into at least one historical memory record set according to the historical execution time corresponding to the historical memory records of each target task when there are historical memory records of the above-mentioned target task in the above-mentioned historical task records. One historical memory record set corresponds to a historical execution period, and the historical execution time of all the historical memory records in one historical memory record set is within the historical execution period corresponding to the historical memory record set; the third determination module is used to determine the calculation weight corresponding to each historical memory record set according to the historical execution period corresponding to each historical memory record set, wherein the smaller the time difference between the historical execution period and the current moment, the greater the calculation weight of the historical memory record set corresponding to the historical execution period; the fourth determination module is used to determine the initial memory value used to execute the above-mentioned target task based on the historical memory records of all the above-mentioned target tasks and the calculation weight corresponding to each historical memory record set. This way, we can infer the approximate consumption of this execution and define the approximate consumption within a suitable range at the beginning. The range of subsequent adjustments will not be too large, which makes it easier to apply for resources and avoids long-term suspension or memory overflow issues.

[0095] In some optional instances, there are two historical memory record sets, namely the first record set and the second record set. The fourth determination module includes a second construction module and a fifth determination module. The second construction module is used to construct a second memory calculation formula: y(T) = A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), wherein y(T) is the initial memory value, T is the start execution time of the target task, A is the calculation weight corresponding to the first record set, B is the calculation weight corresponding to the second record set, y(T-1) to (T-7) are respectively the historical memory records in the first record set, and y(T-8)+…+y(T-15) are respectively the historical memory records in the second record set; the fifth determination module is used to determine the initial memory value used to execute the target task according to the second memory calculation formula. This way, we can infer the approximate consumption of this execution and define the approximate consumption within a suitable range at the beginning. The range of subsequent adjustments will not be too large, which makes it easier to apply for resources and avoids long-term suspension or memory overflow issues.

[0096] In this embodiment, the third determination unit includes a determination subunit, which is used to determine the ratio of the total cluster resources to the ideal concurrency as the initial memory value for executing the target task when there is no historical memory record of the target task in the historical task record; the device also includes a sixth determination module, which is used to determine the initial memory value for executing the target task based on the total cluster resources and the ideal concurrency when there is no historical memory record of the target task in the historical task record, and then, if the initial memory value does not support the complete execution of the target task, determine the preset multiple of the initial memory value as the reallocated memory value of the target task, and use the reallocated memory value to execute the target task. By continuously optimizing the memory value allocated to the task and making it closer to the actually required memory value, job failures due to insufficient memory and low memory utilization due to excessive memory allocation are avoided.

[0097] An optional solution, the sixth determination module includes a first determination submodule, a first processing submodule and a second processing submodule, the first determination submodule is used to determine the job tasks that can be executed and completed within a first preset time period when the free memory value is less than the above-mentioned reallocated memory value, and sum the memory occupied by the above-mentioned job tasks to obtain a reserve memory value; the first processing submodule is used to retain the memory allocated to the above-mentioned target task when the above-mentioned reserve memory value is greater than or equal to the above-mentioned reallocated memory value, and increase the priority of the above-mentioned target task so that the above-mentioned target task is executed immediately after the above-mentioned job task is completed; the second processing submodule is used to release the memory allocated to the above-mentioned target task when the above-mentioned reserve memory value is less than the above-mentioned reallocated memory value, and place the above-mentioned target task back into the task queue sequence until the above-mentioned free memory value is greater than or equal to the above-mentioned reallocated memory value, and then execute the above-mentioned target task. By executing historical records, a more accurate estimate of the time of the currently executing task can be obtained, and it can be determined whether there is a job that can release memory in the near future, control the suspension time of the task, and release or allocate memory as soon as possible for tasks that may be suspended for a long time to avoid excessive suspension time, which leads to extended execution time.

[0098] As an optional solution, the above-mentioned device also includes a seventh determination module and a processing module. The seventh determination module is used to determine the free memory value at the current moment when the number of job error reports within the second preset time period is greater than or equal to the preset number; the processing module is used to delete the unexecuted tasks at the current moment from the task queue sequence when the above-mentioned free memory value at the current moment is less than or equal to the preset memory value, and reduce the number of concurrent tasks scheduled until the above-mentioned free memory value is greater than the above-mentioned preset memory value, and then place the above-mentioned unexecuted tasks back into the above-mentioned task queue sequence. In the case of an error, if it is due to exceeding the memory limit, an error will be reported directly, and due to memory shortage, the task will return to the queue to continue execution after the memory is reallocated, thereby achieving a certain degree of automation and reducing personnel work. For multiple jobs with large resource usage running at the same time, the scheduler will identify the current cluster resource surplus, adjust the number of concurrency in time, and control the jobs that enter the database after reduction to reduce the possibility of database jamming.

[0099] The resource scheduling device includes a processor and memory. The acquisition unit and other components are stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0100] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, this solves the problem of low memory utilization, which occurs when existing scheduling systems cannot obtain the overall database resource usage or the resource usage of each job, and can only passively allocate resources to run jobs.

[0101] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0102] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the resource scheduling method.

[0103] Specifically, the resource scheduling method includes:

[0104] Step S201, obtaining an identification number of a target task, the identification number being used to uniquely identify the target task;

[0105] Specifically, the target task's identification number can be used to search for records of the target task in the history. For example, during actual execution, a certain number of processing cores and memory must be allocated in advance, and a certain upward floating value can be adjusted during the allocation process. Different core and memory allocations can affect job execution efficiency. For tasks that need to be executed regularly, it is necessary to estimate the approximate memory consumption of the task before execution.

[0106] Step S202: determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task;

[0107] Specifically, the historical task record also includes the job identification number (task identification number), start execution time, current resource status, actual resource consumption, running time and other content to ensure the continuation of subsequent operations.

[0108] Step S203: if a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, wherein the historical memory record of the target task includes memory values ​​of successful executions of the target task within a historical time period;

[0109] Specifically, when a periodic task has a history, current consumption is judged based on historical consumption. First, determine whether the periodic task has a clear pattern with respect to dates and plot a line chart of memory consumption against execution date. If no clear pattern is observed, perform a weighted calculation on the historical records. Initially, use recent historical records of the same task to infer the approximate consumption of this execution. This allows you to initially define the approximate consumption within an appropriate range, limiting subsequent adjustments. This makes resource allocation easier and prevents prolonged hangs or out-of-memory reports.

[0110] Step S204 : when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the total cluster resources and the ideal concurrency number.

[0111] Specifically, when a new periodic task begins executing, it is assigned a unique identification number. Since no history records exist for this task, initial resources are automatically allocated to the task. These resources are calculated by dividing the total cluster resources by the desired concurrency. If the task executes successfully, the actual memory consumption and run time are recorded in the history log. If the task fails due to insufficient memory, the pre-allocated memory value is increased to 1.1 times the pre-allocated memory value and the task continues until it succeeds.

[0112] Optionally, when there is a historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the historical memory record of the target task, including: when there is a historical memory record of the target task in the historical task record, the historical memory cycle of the target task is determined according to the historical memory record of the target task, the historical memory cycle is obtained according to the law of memory consumed by successfully executing the target task within the historical time period; constructing a first memory calculation formula y(T) = avg(y(Tn)), wherein y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; according to the first memory calculation formula, the initial memory value for executing the target task is determined.

[0113] Optionally, there are multiple historical memory records of the above-mentioned target task. When there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, the initial memory value for executing the above-mentioned target task is determined according to the historical memory records of the above-mentioned target task, including: when there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, according to the historical execution time corresponding to the historical memory records of each of the above-mentioned target tasks, the multiple historical memory records of the above-mentioned target tasks are split into at least one historical memory record set, one historical memory record set corresponds to one historical execution period, and the historical execution time of all the above-mentioned historical memory records in one historical memory record set is within the above-mentioned historical execution period corresponding to the above-mentioned historical memory record set; according to the above-mentioned historical execution period corresponding to each of the above-mentioned historical memory record sets, the calculation weight corresponding to each of the above-mentioned historical memory record sets is determined, wherein the smaller the time difference between the above-mentioned historical execution period and the current moment, the greater the calculation weight of the above-mentioned historical memory record set corresponding to the above-mentioned historical execution period; according to the historical memory records of all the above-mentioned target tasks and the calculation weight corresponding to each of the above-mentioned historical memory record sets, the initial memory value for executing the above-mentioned target task is determined.

[0114] Optionally, there are two historical memory record sets, namely the first record set and the second record set. According to all the historical memory records of the above-mentioned target tasks and the calculation weights corresponding to each of the above-mentioned historical memory record sets, the initial memory value used to execute the above-mentioned target task is determined, including: constructing a second memory calculation formula: y(T) = A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), wherein y(T) is the above-mentioned initial memory value, T is the start execution time of the above-mentioned target task, A is the calculation weight corresponding to the above-mentioned first record set, B is the calculation weight corresponding to the above-mentioned second record set, y(T-1) to (T-7) are respectively the above-mentioned historical memory records in the above-mentioned first record set, and y(T-8)+…+y(T-15) are respectively the above-mentioned historical memory records in the above-mentioned second record set; according to the above-mentioned second memory calculation formula, the initial memory value used to execute the above-mentioned target task is determined.

[0115] Optionally, when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined based on the total cluster resources and the ideal concurrency number, including: when there is no historical memory record of the target task in the historical task record, the ratio of the total cluster resources to the ideal concurrency number is determined as the initial memory value for executing the target task; and / or, when there is no historical memory record of the target task in the historical task record, after determining the initial memory value for executing the target task based on the total cluster resources and the ideal concurrency number, the method further includes: when the initial memory value does not support the complete execution of the target task, determining a preset multiple of the initial memory value as the reallocated memory value of the target task, and using the reallocated memory value to execute the target task.

[0116] Optionally, when the above-mentioned initial memory value does not support the complete execution of the above-mentioned target task, the preset multiple of the above-mentioned initial memory value is determined as the reallocated memory value of the above-mentioned target task, and the above-mentioned reallocated memory value is used to execute the above-mentioned target task, including: when the free memory value is less than the above-mentioned reallocated memory value, determine the job tasks that can be completed within the first preset time period, and sum up the memory occupied by the above-mentioned job tasks to obtain the reserve memory value; when the above-mentioned reserve memory value is greater than or equal to the above-mentioned reallocated memory value, retain the memory allocated to the above-mentioned target task, and increase the priority of the above-mentioned target task so that the above-mentioned target task is executed immediately after the above-mentioned job task is completed; when the above-mentioned reserve memory value is less than the above-mentioned reallocated memory value, release the memory allocated to the above-mentioned target task, and place the above-mentioned target task back in the task queue sequence until the above-mentioned free memory value is greater than or equal to the above-mentioned reallocated memory value, and then execute the above-mentioned target task.

[0117] Optionally, the above method also includes: when the number of job errors within the second preset time period is greater than or equal to the preset number, determining the free memory value at the current moment; when the above free memory value at the current moment is less than or equal to the preset memory value, deleting the unexecuted tasks at the current moment from the task queue sequence, and reducing the number of concurrent tasks scheduled until the above free memory value is greater than the above preset memory value, and then placing the above unexecuted tasks back into the above task queue sequence.

[0118] An embodiment of the present invention provides a processor, which is used to run a program, wherein the resource scheduling method is executed when the program is running.

[0119] Specifically, the resource scheduling method includes:

[0120] Step S201, obtaining an identification number of a target task, the identification number being used to uniquely identify the target task;

[0121] Specifically, the target task's identification number can be used to search for records of the target task in the history. For example, during actual execution, a certain number of processing cores and memory must be allocated in advance, and a certain upward floating value can be adjusted during the allocation process. Different core and memory allocations can affect job execution efficiency. For tasks that need to be executed regularly, it is necessary to estimate the approximate memory consumption of the task before execution.

[0122] Step S202: determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task;

[0123] Specifically, the historical task record also includes the job identification number (task identification number), start execution time, current resource status, actual resource consumption, running time and other content to ensure the continuation of subsequent operations.

[0124] Step S203: if a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, wherein the historical memory record of the target task includes memory values ​​of successful executions of the target task within a historical time period;

[0125] Specifically, when a periodic task has a history, current consumption is judged based on historical consumption. First, determine whether the periodic task has a clear pattern with respect to dates and plot a line chart of memory consumption against execution date. If no clear pattern is observed, perform a weighted calculation on the historical records. Initially, use recent historical records of the same task to infer the approximate consumption of this execution. This allows you to initially define the approximate consumption within an appropriate range, limiting subsequent adjustments. This makes resource allocation easier and prevents prolonged hangs or out-of-memory reports.

[0126] Step S204 : when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the total cluster resources and the ideal concurrency number.

[0127] Specifically, when a new periodic task begins executing, it is assigned a unique identification number. Since no history records exist for this task, initial resources are automatically allocated to the task. These resources are calculated by dividing the total cluster resources by the desired concurrency. If the task executes successfully, the actual memory consumption and run time are recorded in the history log. If the task fails due to insufficient memory, the pre-allocated memory value is increased to 1.1 times the pre-allocated memory value and the task continues until it succeeds.

[0128] Optionally, when there is a historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the historical memory record of the target task, including: when there is a historical memory record of the target task in the historical task record, the historical memory cycle of the target task is determined according to the historical memory record of the target task, the historical memory cycle is obtained according to the law of memory consumed by successfully executing the target task within the historical time period; constructing a first memory calculation formula y(T) = avg(y(Tn)), wherein y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; according to the first memory calculation formula, the initial memory value for executing the target task is determined.

[0129] Optionally, there are multiple historical memory records of the above-mentioned target task. When there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, the initial memory value for executing the above-mentioned target task is determined according to the historical memory records of the above-mentioned target task, including: when there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, according to the historical execution time corresponding to the historical memory records of each of the above-mentioned target tasks, the multiple historical memory records of the above-mentioned target tasks are split into at least one historical memory record set, one historical memory record set corresponds to one historical execution period, and the historical execution time of all the above-mentioned historical memory records in one historical memory record set is within the above-mentioned historical execution period corresponding to the above-mentioned historical memory record set; according to the above-mentioned historical execution period corresponding to each of the above-mentioned historical memory record sets, the calculation weight corresponding to each of the above-mentioned historical memory record sets is determined, wherein the smaller the time difference between the above-mentioned historical execution period and the current moment, the greater the calculation weight of the above-mentioned historical memory record set corresponding to the above-mentioned historical execution period; according to the historical memory records of all the above-mentioned target tasks and the calculation weight corresponding to each of the above-mentioned historical memory record sets, the initial memory value for executing the above-mentioned target task is determined.

[0130] Optionally, there are two historical memory record sets, namely the first record set and the second record set. According to all the historical memory records of the above-mentioned target tasks and the calculation weights corresponding to each of the above-mentioned historical memory record sets, the initial memory value used to execute the above-mentioned target task is determined, including: constructing a second memory calculation formula: y(T) = A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), wherein y(T) is the above-mentioned initial memory value, T is the start execution time of the above-mentioned target task, A is the calculation weight corresponding to the above-mentioned first record set, B is the calculation weight corresponding to the above-mentioned second record set, y(T-1) to (T-7) are respectively the above-mentioned historical memory records in the above-mentioned first record set, and y(T-8)+…+y(T-15) are respectively the above-mentioned historical memory records in the above-mentioned second record set; according to the above-mentioned second memory calculation formula, the initial memory value used to execute the above-mentioned target task is determined.

[0131] Optionally, when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined based on the total cluster resources and the ideal concurrency number, including: when there is no historical memory record of the target task in the historical task record, the ratio of the total cluster resources to the ideal concurrency number is determined as the initial memory value for executing the target task; and / or, when there is no historical memory record of the target task in the historical task record, after determining the initial memory value for executing the target task based on the total cluster resources and the ideal concurrency number, the method further includes: when the initial memory value does not support the complete execution of the target task, determining a preset multiple of the initial memory value as the reallocated memory value of the target task, and using the reallocated memory value to execute the target task.

[0132] Optionally, when the above-mentioned initial memory value does not support the complete execution of the above-mentioned target task, the preset multiple of the above-mentioned initial memory value is determined as the reallocated memory value of the above-mentioned target task, and the above-mentioned reallocated memory value is used to execute the above-mentioned target task, including: when the free memory value is less than the above-mentioned reallocated memory value, determine the job tasks that can be completed within the first preset time period, and sum up the memory occupied by the above-mentioned job tasks to obtain the reserve memory value; when the above-mentioned reserve memory value is greater than or equal to the above-mentioned reallocated memory value, retain the memory allocated to the above-mentioned target task, and increase the priority of the above-mentioned target task so that the above-mentioned target task is executed immediately after the above-mentioned job task is completed; when the above-mentioned reserve memory value is less than the above-mentioned reallocated memory value, release the memory allocated to the above-mentioned target task, and place the above-mentioned target task back in the task queue sequence until the above-mentioned free memory value is greater than or equal to the above-mentioned reallocated memory value, and then execute the above-mentioned target task.

[0133] Optionally, the above method also includes: when the number of job errors within the second preset time period is greater than or equal to the preset number, determining the free memory value at the current moment; when the above free memory value at the current moment is less than or equal to the preset memory value, deleting the unexecuted tasks at the current moment from the task queue sequence, and reducing the number of concurrent tasks scheduled until the above free memory value is greater than the above preset memory value, and then placing the above unexecuted tasks back into the above task queue sequence.

[0134] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0135] Step S201, obtaining an identification number of a target task, the identification number being used to uniquely identify the target task;

[0136] Step S202: determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task;

[0137] Step S203: if a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, wherein the historical memory record of the target task includes memory values ​​of successful executions of the target task within a historical time period;

[0138] Step S204 : when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the total cluster resources and the ideal concurrency number.

[0139] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0140] Optionally, when there is a historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the historical memory record of the target task, including: when there is a historical memory record of the target task in the historical task record, the historical memory cycle of the target task is determined according to the historical memory record of the target task, the historical memory cycle is obtained according to the law of memory consumed by successfully executing the target task within the historical time period; constructing a first memory calculation formula y(T) = avg(y(Tn)), wherein y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; according to the first memory calculation formula, the initial memory value for executing the target task is determined.

[0141] Optionally, there are multiple historical memory records of the above-mentioned target task. When there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, the initial memory value for executing the above-mentioned target task is determined according to the historical memory records of the above-mentioned target task, including: when there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, according to the historical execution time corresponding to the historical memory records of each of the above-mentioned target tasks, the multiple historical memory records of the above-mentioned target tasks are split into at least one historical memory record set, one historical memory record set corresponds to one historical execution period, and the historical execution time of all the above-mentioned historical memory records in one historical memory record set is within the above-mentioned historical execution period corresponding to the above-mentioned historical memory record set; according to the above-mentioned historical execution period corresponding to each of the above-mentioned historical memory record sets, the calculation weight corresponding to each of the above-mentioned historical memory record sets is determined, wherein the smaller the time difference between the above-mentioned historical execution period and the current moment, the greater the calculation weight of the above-mentioned historical memory record set corresponding to the above-mentioned historical execution period; according to the historical memory records of all the above-mentioned target tasks and the calculation weight corresponding to each of the above-mentioned historical memory record sets, the initial memory value for executing the above-mentioned target task is determined.

[0142] Optionally, there are two historical memory record sets, namely the first record set and the second record set. According to all the historical memory records of the above-mentioned target tasks and the calculation weights corresponding to each of the above-mentioned historical memory record sets, the initial memory value used to execute the above-mentioned target task is determined, including: constructing a second memory calculation formula: y(T) = A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), wherein y(T) is the above-mentioned initial memory value, T is the start execution time of the above-mentioned target task, A is the calculation weight corresponding to the above-mentioned first record set, B is the calculation weight corresponding to the above-mentioned second record set, y(T-1) to (T-7) are respectively the above-mentioned historical memory records in the above-mentioned first record set, and y(T-8)+…+y(T-15) are respectively the above-mentioned historical memory records in the above-mentioned second record set; according to the above-mentioned second memory calculation formula, the initial memory value used to execute the above-mentioned target task is determined.

[0143] Optionally, when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined based on the total cluster resources and the ideal concurrency number, including: when there is no historical memory record of the target task in the historical task record, the ratio of the total cluster resources to the ideal concurrency number is determined as the initial memory value for executing the target task; and / or, when there is no historical memory record of the target task in the historical task record, after determining the initial memory value for executing the target task based on the total cluster resources and the ideal concurrency number, the method further includes: when the initial memory value does not support the complete execution of the target task, determining a preset multiple of the initial memory value as the reallocated memory value of the target task, and using the reallocated memory value to execute the target task.

[0144] Optionally, when the above-mentioned initial memory value does not support the complete execution of the above-mentioned target task, the preset multiple of the above-mentioned initial memory value is determined as the reallocated memory value of the above-mentioned target task, and the above-mentioned reallocated memory value is used to execute the above-mentioned target task, including: when the free memory value is less than the above-mentioned reallocated memory value, determine the job tasks that can be completed within the first preset time period, and sum up the memory occupied by the above-mentioned job tasks to obtain the reserve memory value; when the above-mentioned reserve memory value is greater than or equal to the above-mentioned reallocated memory value, retain the memory allocated to the above-mentioned target task, and increase the priority of the above-mentioned target task so that the above-mentioned target task is executed immediately after the above-mentioned job task is completed; when the above-mentioned reserve memory value is less than the above-mentioned reallocated memory value, release the memory allocated to the above-mentioned target task, and place the above-mentioned target task back in the task queue sequence until the above-mentioned free memory value is greater than or equal to the above-mentioned reallocated memory value, and then execute the above-mentioned target task.

[0145] Optionally, the above method also includes: when the number of job errors within the second preset time period is greater than or equal to the preset number, determining the free memory value at the current moment; when the above free memory value at the current moment is less than or equal to the preset memory value, deleting the unexecuted tasks at the current moment from the task queue sequence, and reducing the number of concurrent tasks scheduled until the above free memory value is greater than the above preset memory value, and then placing the above unexecuted tasks back into the above task queue sequence.

[0146] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0147] Step S201, obtaining an identification number of a target task, the identification number being used to uniquely identify the target task;

[0148] Step S202: determining whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task;

[0149] Step S203: if a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task based on the historical memory record of the target task, wherein the historical memory record of the target task includes memory values ​​of successful executions of the target task within a historical time period;

[0150] Step S204 : when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the total cluster resources and the ideal concurrency number.

[0151] Optionally, when there is a historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined according to the historical memory record of the target task, including: when there is a historical memory record of the target task in the historical task record, the historical memory cycle of the target task is determined according to the historical memory record of the target task, the historical memory cycle is obtained according to the law of memory consumed by successfully executing the target task within the historical time period; constructing a first memory calculation formula y(T) = avg(y(Tn)), wherein y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; according to the first memory calculation formula, the initial memory value for executing the target task is determined.

[0152] Optionally, there are multiple historical memory records of the above-mentioned target task. When there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, the initial memory value for executing the above-mentioned target task is determined according to the historical memory records of the above-mentioned target task, including: when there are historical memory records of the above-mentioned target task in the above-mentioned historical task records, according to the historical execution time corresponding to the historical memory records of each of the above-mentioned target tasks, the multiple historical memory records of the above-mentioned target tasks are split into at least one historical memory record set, one historical memory record set corresponds to one historical execution period, and the historical execution time of all the above-mentioned historical memory records in one historical memory record set is within the above-mentioned historical execution period corresponding to the above-mentioned historical memory record set; according to the above-mentioned historical execution period corresponding to each of the above-mentioned historical memory record sets, the calculation weight corresponding to each of the above-mentioned historical memory record sets is determined, wherein the smaller the time difference between the above-mentioned historical execution period and the current moment, the greater the calculation weight of the above-mentioned historical memory record set corresponding to the above-mentioned historical execution period; according to the historical memory records of all the above-mentioned target tasks and the calculation weight corresponding to each of the above-mentioned historical memory record sets, the initial memory value for executing the above-mentioned target task is determined.

[0153] Optionally, there are two historical memory record sets, namely the first record set and the second record set. According to all the historical memory records of the above-mentioned target tasks and the calculation weights corresponding to each of the above-mentioned historical memory record sets, the initial memory value used to execute the above-mentioned target task is determined, including: constructing a second memory calculation formula: y(T) = A×avg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), wherein y(T) is the above-mentioned initial memory value, T is the start execution time of the above-mentioned target task, A is the calculation weight corresponding to the above-mentioned first record set, B is the calculation weight corresponding to the above-mentioned second record set, y(T-1) to (T-7) are respectively the above-mentioned historical memory records in the above-mentioned first record set, and y(T-8)+…+y(T-15) are respectively the above-mentioned historical memory records in the above-mentioned second record set; according to the above-mentioned second memory calculation formula, the initial memory value used to execute the above-mentioned target task is determined.

[0154] Optionally, when there is no historical memory record of the target task in the historical task record, the initial memory value for executing the target task is determined based on the total cluster resources and the ideal concurrency number, including: when there is no historical memory record of the target task in the historical task record, the ratio of the total cluster resources to the ideal concurrency number is determined as the initial memory value for executing the target task; and / or, when there is no historical memory record of the target task in the historical task record, after determining the initial memory value for executing the target task based on the total cluster resources and the ideal concurrency number, the method further includes: when the initial memory value does not support the complete execution of the target task, determining a preset multiple of the initial memory value as the reallocated memory value of the target task, and using the reallocated memory value to execute the target task.

[0155] Optionally, when the above-mentioned initial memory value does not support the complete execution of the above-mentioned target task, the preset multiple of the above-mentioned initial memory value is determined as the reallocated memory value of the above-mentioned target task, and the above-mentioned reallocated memory value is used to execute the above-mentioned target task, including: when the free memory value is less than the above-mentioned reallocated memory value, determine the job tasks that can be completed within the first preset time period, and sum up the memory occupied by the above-mentioned job tasks to obtain the reserve memory value; when the above-mentioned reserve memory value is greater than or equal to the above-mentioned reallocated memory value, retain the memory allocated to the above-mentioned target task, and increase the priority of the above-mentioned target task so that the above-mentioned target task is executed immediately after the above-mentioned job task is completed; when the above-mentioned reserve memory value is less than the above-mentioned reallocated memory value, release the memory allocated to the above-mentioned target task, and place the above-mentioned target task back in the task queue sequence until the above-mentioned free memory value is greater than or equal to the above-mentioned reallocated memory value, and then execute the above-mentioned target task.

[0156] Optionally, the above method also includes: when the number of job errors within the second preset time period is greater than or equal to the preset number, determining the free memory value at the current moment; when the above free memory value at the current moment is less than or equal to the preset memory value, deleting the unexecuted tasks at the current moment from the task queue sequence, and reducing the number of concurrent tasks scheduled until the above free memory value is greater than the above preset memory value, and then placing the above unexecuted tasks back into the above task queue sequence.

[0157] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0158] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0162] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0163] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0164] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0165] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0166] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0167] 1) The above-mentioned resource scheduling method of the present application first obtains the identification number of the target task, which is used to uniquely identify the target task; secondly, based on the identification number of the target task, determines whether there is a historical memory record of the target task in the historical task record; then, if there is a historical memory record of the target task in the historical task record, determines the initial memory value for executing the target task based on the historical memory record of the target task, and the historical memory record of the target task includes the memory value of the target task successfully executed within the historical time period; finally, if there is no historical memory record of the target task in the historical task record, determines the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number. This method analyzes the historical task records to make a clearer judgment on the memory of regularly executed tasks and their time consumption, and continuously optimizes the memory value allocated to the task to make it closer to the actual required memory value, avoiding job failures due to insufficient memory and low memory utilization due to excessive memory allocation. It solves the problem that the scheduling system of the prior art cannot obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job operation, resulting in low memory utilization.

[0168] 2) The above-mentioned resource scheduling device of the present application includes an acquisition unit, a first determination unit, a second determination unit and a third determination unit. The acquisition unit is used to obtain the identification number of the target task, and the identification number is used to uniquely identify the target task; the first determination unit is used to determine whether there is a historical memory record of the target task in the historical task record based on the identification number of the target task; the second determination unit is used to determine the initial memory value for executing the target task based on the historical memory record of the target task when there is a historical memory record of the target task in the historical task record, and the historical memory record of the target task includes the memory value of the target task successfully executed within the historical time period; the third determination unit is used to determine the initial memory value for executing the target task based on the total resources of the cluster and the ideal concurrency number when there is no historical memory record of the target task in the historical task record. The device analyzes historical task records to make a clearer judgment on the memory and time consumption of regularly executed tasks, and continuously optimizes the memory value allocated to the task to make it closer to the actual memory value required, avoiding job failures due to insufficient memory and low memory utilization due to excessive memory allocation. It solves the problem that the scheduling system of the existing technology cannot obtain the usage of the entire database resources and the resource occupancy of each job, and can only passively allocate resources for job operation, resulting in low memory utilization.

[0169] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A resource scheduling method, characterized in that: include: Obtaining an identification number of a target task, wherein the identification number is used to uniquely identify the target task; Determining, based on the identification number of the target task, whether a historical memory record of the target task exists in the historical task record; In a case where a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task according to the historical memory record of the target task, wherein the historical memory record of the target task includes a memory value of a successful execution of the target task within a historical time period; If there is no historical memory record of the target task in the historical task record, determining a ratio of total cluster resources to an ideal concurrency number as an initial memory value for executing the target task; When the initial memory value does not support the complete execution of the target task, a preset multiple of the initial memory value is determined as the reallocated memory value of the target task, and the reallocated memory value is used to execute the target task, including: when the free memory value is less than the reallocated memory value, determining the job tasks that can be completed within a first preset time period, and summing the memory occupied by the job tasks to obtain a reserve memory value; when the reserve memory value is greater than or equal to the reallocated memory value, retaining the memory allocated to the target task, and raising the priority of the target task so that the target task is executed immediately after the job task is completed; when the reserve memory value is less than the reallocated memory value, releasing the memory allocated to the target task, and placing the target task back into the task queue sequence until the free memory value is greater than or equal to the reallocated memory value, and then executing the target task.

2. The resource scheduling method according to claim 1, characterized in that: In a case where a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task according to the historical memory record of the target task includes: If a historical memory record of the target task exists in the historical task record, determining a historical memory period of the target task based on the historical memory record of the target task, where the historical memory period is obtained based on a pattern of memory consumed by successfully executing the target task within a historical time period; Constructing a first memory calculation formula y(T)=avg(y(Tn)), where y(T) is the initial memory value of the target task, T is the start execution time of the target task, y(Tn) is the memory value required to execute the target task within one historical memory cycle, and n is the historical memory cycle; The initial memory value for executing the target task is determined according to the first memory calculation formula.

3. The resource scheduling method according to claim 1, characterized in that: There are multiple historical memory records of the target task. If a historical memory record of the target task exists in the historical task record, determining an initial memory value for executing the target task according to the historical memory record of the target task includes: In a case where a historical memory record of the target task exists in the historical task record, splitting the historical memory records of the multiple target tasks into at least one historical memory record set according to the historical execution time corresponding to the historical memory record of each target task, one historical memory record set corresponding to one historical execution period, and the historical execution times of all the historical memory records in one historical memory record set falling within the historical execution period corresponding to the historical memory record set; Determining a calculation weight corresponding to each historical memory record set according to the historical execution period corresponding to each historical memory record set, wherein the smaller the time difference between the historical execution period and the current moment, the greater the calculation weight of the historical memory record set corresponding to the historical execution period; An initial memory value for executing the target task is determined based on all historical memory records of the target task and the calculation weight corresponding to each historical memory record set.

4. The resource scheduling method according to claim 3, characterized in that: There are two historical memory record sets, namely a first record set and a second record set. Determining an initial memory value for executing the target task based on all historical memory records of the target task and calculation weights corresponding to the historical memory record sets includes: Construct the second memory calculation formula: y(T)=Aavg(y(T-1)+…+y(T-7))+Bavg(y(T-8)+…+y(T-15)), where y(T) is the initial memory value, T is the start execution time of the target task, A is the calculation weight corresponding to the first record set, B is the calculation weight corresponding to the second record set, y(T-1) to (T-7) are respectively the historical memory records in the first record set, and y(T-8)+…+y(T-15) are respectively the historical memory records in the second record set; An initial memory value for executing the target task is determined according to the second memory calculation formula.

5. The resource scheduling method according to claim 1, characterized in that: The method further comprises: If the number of job error reports within the second preset time period is greater than or equal to the preset number, determining the free memory value at the current moment; When the free memory value at the current moment is less than or equal to the preset memory value, the unexecuted tasks at the current moment are deleted from the task queue sequence, and the number of scheduled concurrent tasks is reduced until the free memory value is greater than the preset memory value, and then the unexecuted tasks are placed back into the task queue sequence.

6. A resource scheduling device, characterized in that: include: an acquiring unit, configured to acquire an identification number of a target task, wherein the identification number is used to uniquely identify the target task; a first determining unit, configured to determine, based on the identification number of the target task, whether a historical memory record of the target task exists in the historical task record; a second determining unit, configured to determine, when a historical memory record of the target task exists in the historical task record, an initial memory value for executing the target task based on the historical memory record of the target task, the historical memory record of the target task including a memory value for successfully executing the target task within a historical time period; a third determining unit, configured to determine, when no historical memory record of the target task exists in the historical task record, a ratio of total cluster resources to an ideal concurrency number as an initial memory value for executing the target task; The apparatus further includes a sixth determining module configured to, if the initial memory value does not support complete execution of the target task, determine a preset multiple of the initial memory value as a reallocated memory value for the target task, and execute the target task using the reallocated memory value. The sixth determining module further includes a first determining submodule, a first processing submodule, and a second processing submodule. The first determining submodule is configured to, if the free memory value is less than the reallocated memory value, determine the job tasks that can be executed and completed within a first preset time period, and sum the memory occupied by the job tasks to obtain a reserve memory value. The first processing submodule is configured to, when the reserve memory value is greater than or equal to the reallocated memory value, reserve the memory allocated to the target task and increase the priority of the target task so that the target task is executed immediately after the job task is completed; The second processing submodule is used to release the memory allocated to the target task when the prepared memory value is less than the reallocated memory value, and place the target task back into the task queue sequence until the free memory value is greater than or equal to the reallocated memory value, and then execute the target task.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the resource scheduling method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the resource scheduling method described in any one of claims 1 to 5.

Images