Resource allocation method and allocation device, and computer-readable storage medium
By calculating the weight coefficients and initial parameters of the task indicators and automatically configuring resource parameters, the problems of waste and inefficiency in Spark on YARN batch task processing are solved, intelligent dynamic allocation of resources is realized, and the utilization efficiency of computing clusters is improved.
Patent Information
- Application Number
- CN202110291028.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-03-18
AI Technical Summary
In Spark on YARN batch task processing, manual configuration of parameters leads to waste of resources and inefficient task execution, especially for tasks with different resource requirements. The lack of automated resource allocation methods leads to high professional capabilities and low efficiency.
By receiving resource allocation tasks and initial parameters, the weight coefficients of the task indicators are calculated, and the resource configuration parameters required for the task are determined based on these coefficients and initial parameters, and intelligent dynamic allocation of resources is realized to avoid resource waste and inefficiency caused by manual configuration.
It realizes intelligent dynamic allocation of resources, avoids the problems of waste of resources and inefficient task execution, and improves the overall utilization efficiency of the computing cluster.
Smart Images

Figure CN113010315B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of resource allocation, and in particular to a resource allocation method and allocation device, and a computer-readable storage medium. Background Art
[0002] Spark is a fast general computing engine for large-scale data processing. In related technologies, when using traditional spark on yarn to submit tasks, the yarn queue can manage the resource allocation between tasks. For example, it can control the allocation of resources according to different queue types and priorities. However, manual debugging is still required for each task to specify the configuration parameters of the task to determine its allocated resources. If the system needs to automatically initiate a large number of tasks in batches, there will be many problems with manually setting the task parameters of spark. For example, problem 1: if a unified default parameter is set, redundant resources will be allocated to tasks with low resource requirements, resulting in a waste of resources; problem 2: tasks with large resource requirements will affect the execution efficiency or even fail to execute due to insufficient resource allocation; problem 3: these parameters are usually manually set and adjusted by developers, which requires high professional ability of users, making it difficult for some non-professionals to use.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present invention provide a resource allocation method and allocation device, and a computer-readable storage medium, so as to at least solve the technical problem in the related art that manual configuration of parameters during batch task processing will lead to resource waste and low task execution efficiency.
[0005] According to one aspect of an embodiment of the present invention, there is provided a resource allocation method, comprising: receiving a resource allocation task and resource initial parameters, wherein the resource allocation task includes a plurality of task indicators and a weight coefficient of each of the task indicators; calculating an impact factor of resources required for the task according to the weight coefficient of each of the task indicators; determining configuration parameters of each task resource factor based on the impact factor and the resource initial parameters; and allocating corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor.
[0006] Optionally, before receiving the resource allocation task and the resource initial parameters, the resource allocation method further includes: obtaining a resource request task cluster in a historical time period, wherein the resource request task cluster includes the resource allocation amount corresponding to each resource request task; and calculating the resource initial parameters based on all resource allocation amounts.
[0007] Optionally, the resource initial parameters include: number of CPU cores and memory usage.
[0008] Optionally, after receiving the resource allocation task and the initial resource parameters, the resource allocation method further includes: determining a computing task abstract interface, wherein the computing task abstract interface is used to define the resource allocation task so as to perform resource configuration for the resource allocation task; and determining various task indicators and indicator attributes of the resource allocation task.
[0009] Optionally, the indicator attributes include at least one of the following: the number of tables involved in the task, the task size, the database complexity level, and historical log analysis results of tasks of the same type.
[0010] Optionally, after determining the task indicators and indicator attributes of the resource allocation task, the resource allocation method also includes: obtaining task data of all resource allocation tasks within a historical preset time period; based on the task data of all resource allocation tasks within the historical preset time period and a multivariate linear regression equation, calculating the weight coefficient of each task indicator and the indicator value of each task indicator.
[0011] Optionally, the step of calculating the influencing factor of the resources required for the task according to the weight coefficient of each of the task indicators includes: inputting the weight coefficient and indicator value of each task indicator into the task performance prediction model, and outputting the predicted execution time of the resource allocation task under the resource parameter list; based on the ratio between the predicted execution time and the average execution time of historical tasks, determining the influencing factor of the resources required for the resource allocation task.
[0012] Optionally, the step of determining the configuration parameters of each task resource factor based on the influencing factor and the resource initial parameters includes: determining the value range of each task indicator; determining the configuration parameters of each task resource factor based on the influencing factor, the resource initial parameters and the value range of each task indicator.
[0013] According to another aspect of an embodiment of the present invention, a resource allocation device is also provided, including: a receiving unit, used to receive a resource allocation task and resource initial parameters, wherein the resource allocation task includes multiple task indicators and a weight coefficient of each of the task indicators; a calculating unit, used to calculate the influence factor of the resources required for the task according to the weight coefficient of each of the task indicators; a determining unit, used to determine the configuration parameters of each task resource factor based on the influence factor and the resource initial parameters; and an allocating unit, used to allocate corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor.
[0014] Optionally, the resource allocation device also includes: a first acquisition module, used to obtain a resource request task cluster in a historical time period before receiving the resource allocation task and the resource initial parameters, wherein the resource request task cluster includes the resource allocation amount corresponding to each resource request task; a first calculation module, used to calculate the resource initial parameters based on all resource allocation amounts.
[0015] Optionally, the resource initial parameters include: number of CPU cores and memory usage.
[0016] Optionally, the resource allocation device also includes: a first determination module, used to determine a computing task abstract interface after receiving the resource allocation task and resource initial parameters, wherein the computing task abstract interface is used to define the resource allocation task so as to perform resource configuration for the resource allocation task; a second determination module, used to determine various task indicators and indicator attributes of the resource allocation task.
[0017] Optionally, the indicator attributes include at least one of the following: the number of tables involved in the task, the task size, the database complexity level, and historical log analysis results of tasks of the same type.
[0018] Optionally, the resource allocation device also includes: a second acquisition module, used to obtain the task data of all resource allocation tasks within a historical preset time period after determining the various task indicators and indicator attributes of the resource allocation task; a second calculation module, used to calculate the weight coefficient of each task indicator and the indicator value of each task indicator based on the task data of all resource allocation tasks within the historical preset time period and a multivariate linear regression equation.
[0019] Optionally, the calculation unit includes: a first output submodule, used to input the weight coefficient and indicator value of each task indicator into a task performance prediction model, and output the predicted execution time of the resource allocation task under the resource parameter list; a first determination submodule, used to determine the influencing factor of the resources required for the resource allocation task based on the ratio between the predicted execution time and the average execution time of historical tasks.
[0020] Optionally, the determination unit includes: a first determination submodule, used to determine the value range of each of the task indicators; and a second determination submodule, used to determine the configuration parameters of each task resource factor based on the influencing factor, the initial resource parameters and the value range of each of the task indicators.
[0021] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium is used to store a computer program, wherein when the computer program is executed by a processor, the device where the computer-readable storage medium is located controls the device to execute the above-mentioned resource allocation method.
[0022] In an embodiment of the present invention, a resource allocation task and resource initial parameters are received, wherein the resource allocation task includes multiple task indicators and a weight coefficient of each task indicator, and then the influence factor of the resource required for the task is calculated according to the weight coefficient of each task indicator, and then the configuration parameters of each task resource factor are determined based on the influence factor and the resource initial parameters, and finally the corresponding resources are allocated to the resource allocation task based on the configuration parameters of each task resource factor. In this embodiment, the intelligent dynamic allocation of resources is realized by the method of automatically configuring the computing resource parameters and the multi-indicator weight calculation method of the task parameter influence factor, avoiding the tedious work generated when manually configuring the parameters, and at the same time, avoiding the waste of resources or low efficiency of the computing cluster caused by different computing tasks using the same resources in batch tasks, thereby solving the technical problem in the related technology that manual configuration of parameters will lead to waste of resources and low efficiency of task execution when batch tasks are processed. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0024] Figure 1 is a flow chart of an optional resource allocation method according to an embodiment of the present invention;
[0025] Figure 2 is a schematic diagram of another optional intelligent dynamic resource allocation according to an embodiment of the present invention;
[0026] Figure 3 is a schematic diagram of an optional resource allocation device according to an embodiment of the present invention; DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0029] To facilitate those skilled in the art to understand the present invention, some terms or nouns involved in the embodiments of the present invention are explained below:
[0030] Hadoop is a distributed system infrastructure that allows users to develop distributed programs without understanding the underlying details of the distributed system, making full use of the power of the cluster for high-speed computing and storage. The core design of the Hadoop framework is: HDFS and MapReduce. HDFS provides storage for massive data, while MapReduce provides computing for massive data.
[0031] Spark is a fast and versatile computing engine designed for large-scale data processing. It is an open source cluster computing environment similar to Hadoop. Spark enables in-memory distributed data sets. In addition to providing interactive queries, it can also optimize iterative workloads. It can perform a variety of operations, including SQL queries, text processing, machine learning, etc.
[0032] YARN is a general resource management system that provides unified resource management and scheduling for upper-layer applications.
[0033] The batch tasks involved in the following embodiments of the present invention include but are not limited to: spark tasks, hadoop tasks and other tasks involving resource allocation, etc., and an intelligent dynamic resource allocation method is used to solve the resource configuration problem faced by batch tasks when working.
[0034] The present invention is described in detail below in conjunction with various embodiments.
[0035] Embodiment 1
[0036] According to an embodiment of the present invention, a resource allocation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0037] Figure 1 is a flow chart of an optional resource allocation method according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:
[0038] Step S102, receiving a resource allocation task and resource initial parameters, wherein the resource allocation task includes a plurality of task indicators and a weight coefficient of each task indicator;
[0039] Step S104, calculating the impact factor of the resources required for the task according to the weight coefficient of each task indicator;
[0040] Step S106, determining the configuration parameters of each task resource factor based on the influencing factors and the initial resource parameters;
[0041] Step S108: allocating corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor.
[0042] Through the above steps, a resource allocation task and resource initial parameters can be received, wherein the resource allocation task includes multiple task indicators and a weight coefficient of each task indicator, and then the influence factor of the resources required for the task is calculated according to the weight coefficient of each task indicator, and then the configuration parameters of each task resource factor are determined based on the influence factor and the initial resource parameters, and finally the corresponding resources are allocated to the resource allocation task based on the configuration parameters of each task resource factor. In this embodiment, the intelligent dynamic allocation of resources is realized by the method of automatically configuring the computing resource parameters and the multi-indicator weight calculation method of the task parameter influencing factor, avoiding the tedious work generated when manually configuring the parameters, and also avoiding the waste of resources or low efficiency of the computing cluster caused by different computing tasks using the same resources in batch tasks, thereby solving the technical problem that manual configuration of parameters will lead to waste of resources and low efficiency of task execution when performing batch task processing in the related technology.
[0043] The embodiments of the present invention are described in detail below in combination with the above-mentioned implementation steps.
[0044] Step S102: receiving a resource allocation task and initial resource parameters, wherein the resource allocation task includes a plurality of task indicators and a weight coefficient of each task indicator.
[0045] In this embodiment, multiple task indicators in the resource allocation task include but are not limited to: the number and size of tables involved in the task, as well as sql complexity, historical log analysis results of similar tasks and other indicator attributes, wherein the relevant information of the number and size of tables involved in the task can be obtained by sql string processing or through the hadoop file system; sql complexity can be obtained through a series of secondary indicators in sql, wherein the secondary indicators include: the number and size of join targets, and the physical execution plan can be analyzed through the relevant API (application programming interface) provided by spark_sql, so as to obtain information such as the number of stages involved in the task.
[0046] The initial resource parameters include but are not limited to: the number of CPU cores of the executor, the memory usage of the executor process, the number of CPU cores of the driver, the memory usage of the driver process, and other public parameter lists. The initial resource parameters can be initialized according to the cluster scale and daily task volume, that is, the average resource parameters can be calculated.
[0047] Optionally, before receiving the resource allocation task and the resource initial parameters, the resource allocation method also includes: obtaining a resource request task cluster in a historical time period, wherein the resource request task cluster includes the resource allocation amount corresponding to each resource request task; and calculating the resource initial parameters based on all resource allocation amounts.
[0048] Alternatively, the resource initial parameters include: the number of CPU cores and memory usage.
[0049] The number of CPU cores includes but is not limited to: the number of CPU cores of the executor, the number of CPU cores of the driver, etc. The memory usage includes but is not limited to: the memory usage of the executor process, the memory usage of the driver process, etc.
[0050] Optionally, after receiving the resource allocation task and the initial resource parameters, the resource allocation method also includes: determining a computing task abstract interface, wherein the computing task abstract interface is used to define the resource allocation task to perform resource allocation for the resource allocation task; and determining various task indicators and indicator attributes of the resource allocation task.
[0051] Alternatively, the indicator attributes include at least one of the following: the number of tables involved in the task, the task size, the database complexity level, and historical log analysis results of tasks of the same type.
[0052] In this embodiment, the information about the number and size of tables involved in the task can be obtained by processing the SQL string or by obtaining it through the Hadoop file system. The database complexity level can be obtained from a series of secondary indicators in the database. The above-mentioned SQL complexity has the same meaning as the database complexity.
[0053] Optionally, after determining the task indicators and indicator attributes of the resource allocation task, the resource allocation method also includes: obtaining task data of all resource allocation tasks within a historical preset time period; based on the task data of all resource allocation tasks within the historical preset time period and a multivariate linear regression equation, calculating the weight coefficient of each task indicator and the indicator value of each task indicator.
[0054] In this embodiment, the weight coefficient constants of multiple indicators can be calculated based on the acquired historical task data and the multiple linear regression equation. Because some tasks have more parameter indicators, the following multiple linear regression model is used to calculate the weights of multiple indicators:
[0055]
[0056] Among them, y is the task running time, x is the indicator value, w is the weight, and b is the bias term.
[0057] After analyzing and calculating a large amount of historical log data, curve fitting is performed to calculate the weight coefficient constant of the indicator.
[0058] Step S104, calculating the impact factor of the resources required for the task according to the weight coefficient of each task indicator.
[0059] Optionally, the step of calculating the influencing factor of the resources required for the task according to the weight coefficient of each task indicator includes: inputting the weight coefficient and indicator value of each task indicator into the task performance prediction model, and outputting the predicted execution time of the resource allocation task under the resource parameter list; based on the ratio between the predicted execution time and the average execution time of historical tasks, determining the influencing factor of the resources required for the resource allocation task.
[0060] For example, when calculating the resource impact factor of a task, the regression model obtained from the above historical log is used as the performance prediction model of the spark task, and the indicator values and weights of the specific instance task are input to obtain the predicted execution time t of the task under the default average resource parameter list, and then the ratio r of the t value to the average duration of historical tasks is used as the impact factor of the required resources.
[0061] Step S106, determining the configuration parameters of each task resource factor based on the influencing factors and the initial resource parameters.
[0062] Optionally, the step of determining the configuration parameters of each task resource factor based on the influencing factors and the initial resource parameters includes: determining the value range of each task indicator; determining the configuration parameters of each task resource factor based on the influencing factors, the initial resource parameters and the value range of each task indicator.
[0063] In this embodiment, the value range of each task indicator includes but is not limited to: the CPU core, memory, etc. of the queue. According to the influencing factors of each task, the initial resource parameters and the value range of the indicator, different calculation methods are used to obtain the configuration parameters of each task resource factor.
[0064] Step S108: allocating corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor.
[0065] In this embodiment, tasks can be submitted according to different parameter configurations obtained from different tasks, thereby dynamically allocating different resources to each submitted task. This not only ensures sufficient resource allocation, but also avoids wasting unnecessary resources on tasks with low resource requirements, thereby improving the utilization efficiency of the entire cluster.
[0066] Figure 2 is a schematic diagram of another optional intelligent dynamic resource allocation according to an embodiment of the present invention, such as Figure 2 As shown in the figure, when performing intelligent dynamic resource allocation, first, the weight coefficient constants of multiple indicators are calculated based on historical log data and multivariate linear regression equations, and then the task indicators are calculated, where the task indicators include the number of tables involved in the task, the size of the table file, the SQL complexity, the historical task analysis results, other indicators and other indicator attributes, and then the computing task abstract interface is defined, and the indicator values of the specific task are calculated and the computing task instance is initialized. Then, the average resource parameters are calculated according to the actual cluster size and daily task volume, so as to obtain the average resource parameters of different task types, and then the influence factor of the resources required for the instance task is obtained according to the weight coefficient of each indicator. Finally, according to the calculated influence factor of each instance task and the average resource parameter of the instance task and referring to the actual size of the cluster and the value range of each indicator (for example, the CPU core, memory, etc. of the queue), different calculation methods are used to obtain each parameter configuration of the task, and then the task is executed, so as to achieve dynamic allocation of different resources to each submitted task, which not only ensures sufficient allocation of resources, but also ensures a small amount of idle resources, thereby improving the utilization efficiency of the entire cluster.
[0067] The present invention is described below in conjunction with another optional embodiment.
[0068] Embodiment 2
[0069] The embodiment of the present invention provides a resource allocation device, and each implementation unit included in the allocation device corresponds to each implementation step in the above-mentioned embodiment 1.
[0070] Figure 3 is a schematic diagram of an optional resource allocation device according to an embodiment of the present invention, such as Figure 3 As shown, the information processing device includes: a receiving unit 30, a calculating unit 32, a determining unit 34, and an allocating unit 36, wherein:
[0071] The receiving unit 30 is used to receive a resource allocation task and resource initial parameters, wherein the resource allocation task includes a plurality of task indicators and a weight coefficient of each of the task indicators;
[0072] A calculation unit 32, used to calculate the impact factor of the resources required for the task according to the weight coefficient of each of the task indicators;
[0073] A determination unit 34, configured to determine configuration parameters of each task resource factor based on the influencing factor and the initial resource parameter;
[0074] The allocation unit 36 is used to allocate corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor.
[0075] The above-mentioned resource allocation device can receive resource allocation tasks and resource initial parameters through the receiving unit 30, wherein the resource allocation task includes multiple task indicators and a weight coefficient of each task indicator, and then the calculation unit 32 calculates the influence factor of the resources required for the task according to the weight coefficient of each task indicator, and then the determination unit 34 determines the configuration parameters of each task resource factor based on the influence factor and the resource initial parameters, and finally the allocation unit 36 allocates the corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor. In this embodiment, the intelligent dynamic allocation of resources is realized by the method of automatically configuring the computing resource parameters and the multi-indicator weight calculation method of the task parameter influencing factor, avoiding the tedious work generated when manually configuring the parameters, and also avoiding the waste of resources or low efficiency of the computing cluster caused by different computing tasks using the same resources in batch tasks, thereby solving the technical problem in the related technology that manual configuration of parameters will lead to waste of resources and low efficiency of task execution when batch tasks are processed.
[0076] Optionally, the resource allocation device also includes: a first acquisition module, used to obtain a resource request task cluster in a historical time period before receiving the resource allocation task and the resource initial parameters, wherein the resource request task cluster includes the resource allocation amount corresponding to each resource request task; a first calculation module, used to calculate the resource initial parameters based on all resource allocation amounts.
[0077] Optionally, the resource initial parameters include: the number of CPU cores and memory usage.
[0078] Optionally, the resource allocation device also includes: a first determination module, used to determine the computing task abstract interface after receiving the resource allocation task and the resource initial parameters, wherein the computing task abstract interface is used to define the resource allocation task so as to perform resource configuration for the resource allocation task; a second determination module, used to determine the various task indicators and indicator attributes of the resource allocation task.
[0079] Optionally, the indicator attributes include at least one of the following: the number of tables involved in the task, the task size, the database complexity level, and historical log analysis results of tasks of the same type.
[0080] Optionally, the resource allocation device also includes: a second acquisition module, used to obtain the task data of all resource allocation tasks within a historical preset time period after determining the various task indicators and indicator attributes of the resource allocation task; a second calculation module, used to calculate the weight coefficient of each task indicator and the indicator value of each task indicator based on the task data of all resource allocation tasks within the historical preset time period and a multivariate linear regression equation.
[0081] Optionally, the calculation unit includes: a first output submodule, used to input the weight coefficient and indicator value of each task indicator into a task performance prediction model, and output the predicted execution time of the resource allocation task under the resource parameter list; a first determination submodule, used to determine the influencing factor of the resources required for the resource allocation task based on the ratio between the predicted execution time and the average execution time of historical tasks.
[0082] Optionally, the determination unit includes: a first determination submodule, used to determine the value range of each task indicator; and a second determination submodule, used to determine the configuration parameters of each task resource factor based on the influencing factor, the initial resource parameter and the value range of each task indicator.
[0083] The resource allocation device may further include a processor and a memory. The receiving unit 30, the calculating unit 32, the determining unit 34, the allocating unit 36, etc. are all stored in the memory as program units. The processor executes the program units stored in the memory to implement corresponding functions.
[0084] The above processor includes a kernel, which retrieves the corresponding program unit from the memory. The kernel can be set to one or more, and the kernel parameters are adjusted to calculate the impact factor of the resources required for the task, and then the configuration parameters of each task resource factor are determined based on the impact factor and the initial resource parameters, and finally the corresponding resources are allocated to the resource allocation task based on the configuration parameters of each task resource factor.
[0085] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.
[0086] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the storage medium being used to store a computer program, wherein when the computer program is executed by a processor, the device where the computer-readable storage medium is located controls the device to execute the above-mentioned resource allocation method.
[0087] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: receiving a resource allocation task and initial resource parameters, then calculating the impact factor of the resources required for the task based on the weight coefficient of each task indicator, and then determining the configuration parameters of each task resource factor based on the impact factor and the initial resource parameters, and finally allocating corresponding resources to the resource allocation task based on the configuration parameters of each task resource factor.
[0088] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0089] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0092] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0094] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A resource allocation method, It is characterized in that include: Receiving a resource allocation task and resource initial parameters, wherein the resource allocation task includes a plurality of task indicators and a weight coefficient of each of the task indicators; After receiving the resource allocation task and the initial resource parameters, the resource allocation method further includes: determining a computing task abstract interface, wherein the computing task abstract interface is used to define the resource allocation task so as to perform resource allocation for the resource allocation task; determining various task indicators and indicator attributes of the resource allocation task, wherein the indicator attributes include at least one of the following: the number of tables involved in the task, the task size, the database complexity level, and the historical log analysis results of the same type of tasks. After determining various task indicators and indicator attributes of the resource allocation task, the resource allocation method further includes: obtaining task data of all resource allocation tasks within a historical preset time period; calculating the weight coefficient of each of the task indicators and the indicator value of each of the task indicators based on the task data of all resource allocation tasks within the historical preset time period and a multivariate linear regression equation; Calculate the impact factor of the resources required for the task according to the weight coefficient of each task indicator; Determine the configuration parameters of each task resource factor based on the influencing factors and the initial resource parameters; Based on the configuration parameters of each task resource factor, corresponding resources are allocated to the resource allocation task.
2. The method according to claim 1, It is characterized in that Before receiving the resource allocation task and the resource initial parameters, the resource allocation method further includes: Acquire a resource request task cluster in a historical time period, wherein the resource request task cluster includes a resource allocation amount corresponding to each resource request task; Based on all resource allocations, calculate resource initial parameters.
3. The method according to any one of claims 1 to 2, It is characterized in that The resource initial parameters include: number of CPU cores and memory usage.
4. The method according to claim 1, It is characterized in that The step of calculating the impact factor of the resources required for the task according to the weight coefficient of each of the task indicators comprises: Input the weight coefficient and indicator value of each task indicator into the task performance prediction model, and output the predicted execution time of the resource allocation task under the resource parameter list; Based on the ratio of the predicted execution time to the average execution time of historical tasks, an influencing factor of the resources required for the resource allocation task is determined.
5. The method according to claim 1, It is characterized in that The step of determining the configuration parameters of each task resource factor based on the influencing factors and the initial resource parameters includes: Determining the value range of each of the task indicators; Based on the influencing factors, the initial resource parameters and the value range of each task indicator, the configuration parameters of each task resource factor are determined.
6. A resource allocation device, It is characterized in that include: A receiving unit, configured to receive a resource allocation task and resource initial parameters, wherein the resource allocation task includes a plurality of task indicators and a weight coefficient of each of the task indicators; A calculation unit, used to calculate the impact factor of the resources required for the task according to the weight coefficient of each of the task indicators; A determination unit, configured to determine configuration parameters of each task resource factor based on the influencing factor and the initial resource parameter; an allocation unit, configured to allocate corresponding resources to the resource allocation task based on configuration parameters of each task resource factor; The resource allocation device also includes: a first determination module, which is used to determine a computing task abstract interface after receiving a resource allocation task and initial resource parameters, wherein the computing task abstract interface is used to define the resource allocation task so as to perform resource allocation on the resource allocation task; a second determination module, which determines various task indicators and indicator attributes of the resource allocation task, wherein the indicator attributes include at least one of the following: the number of tables involved in the task, the task size, the database complexity level, and historical log analysis results of tasks of the same type. The resource allocation device also includes: a second acquisition module, which is used to acquire task data of all resource allocation tasks within a historical preset time period after determining various task indicators and indicator attributes of the resource allocation task; a second calculation module, which is used to calculate the weight coefficient of each task indicator and the indicator value of each task indicator based on the task data of all resource allocation tasks within the historical preset time period and a multivariate linear regression equation.
7. A computer-readable storage medium, It is characterized in that The storage medium is used to store a computer program, wherein the computer program, when executed by a processor, controls the device where the computer-readable storage medium is located to execute the resource allocation method according to any one of claims 1 to 5.
