A method and device for dynamic scheduling of computing resources

Abstract

The present invention proposes a method for dynamic scheduling of computing resources, which includes the following steps: in response to two algorithms for resource allocation and dynamic adjustment, automatic analysis of the characteristics of each task and automatic matching of the resource scheduling engine are realized; the underlying component layer responds to and executes the resource adjustment instructions issued. By optimizing the characteristics of different engines and combining the task rule library, a refined resource allocation system is automatically constructed with a computer program, thereby forming a resource scheduling solution for different types of tasks, meeting the dynamic scheduling needs of computing resources in various business scenarios, solving the long-standing problem of poor operational stability of business models in the era of big data, realizing a multi-task fusion scheduling mode, and dynamically scheduling the operational resource requirements of business models in various business scenarios in real time, greatly improving the operational stability of business models, increasing the coverage of big data empowerment of business, and reducing the construction cost of big data projects.

Description

Technical Field

[0001] The present invention belongs to the technical field of computing resource scheduling, and in particular relates to a method and device for dynamic scheduling of computing resources. Background Art

[0002] With the popularization of smart terminals, society has entered the era of big data, and various data are growing explosively. Business models need to be based on massive data while meeting complex application scenarios to support rapidly changing business operations. The computing resources required for business model operations are increasing, and the requirements for operational stability are becoming increasingly higher. How to reasonably schedule and allocate the computing resources required for business model operations to enable each business model to run faster and more stably has become the focus of work to improve business analysis capabilities and analysis efficiency.

[0003] Due to the large data volumes and complex business scenarios of big data projects, traditional computing resource allocation methods all use preset fixed resource pool quotas and running thread numbers. That is, specific computing resources are fixedly allocated to a single task or task set to run the task. These technologies have the following shortcomings:

[0004] 1) Poor adaptability to business changes. The data sources accessed by the big data platform are diverse and change frequently. The operating environment of the business model will also change frequently. Fixed allocation of specific computing resources often makes it difficult to timely match the resource requirements of the changing operating environment. This leads to abnormal situations such as interruptions or errors in the operation of business model tasks, seriously affecting the big data platform's ability to support business.

[0005] 2) Low resource utilization efficiency. In the era of big data, the amount of data in various domains is huge. In order to ensure the normal operation of various business models as much as possible, a fixed allocation method often allocates a large amount of computing resources. The accumulation of redundant computing resources will result in low computing resource utilization efficiency and exponentially increase project construction costs.

[0006] In view of this, it is very meaningful to propose a method and device for dynamic scheduling of computing resources. Summary of the Invention

[0007] In order to solve the problems of poor applicability to business changes and low resource utilization efficiency in traditional technical resource allocation methods, the present invention provides a method and device for dynamic scheduling of computing resources to solve the above-mentioned technical defects.

[0008] In a first aspect, the present invention provides a method for dynamically scheduling computing resources, the method comprising the following steps:

[0009] In response to the two algorithms of resource allocation and dynamic adjustment, the characteristics of each task are automatically analyzed and the resource scheduling engine is automatically matched;

[0010] The underlying component layer responds to and executes the resource adjustment instructions.

[0011] Preferably, the resource allocation algorithm includes:

[0012] A multi-task fusion scheduling pool is preset to analyze the characteristics of various computing tasks, calculate the initial resources required for the tasks, and allocate corresponding resource pools to finely manage resource request queues and task execution queues.

[0013] Preferably, the dynamic adjustment algorithm includes:

[0014] Continuously track and analyze the running status of each task, obtain and record the changing patterns of task operation;

[0015] When the preset critical value that requires resource adjustment is reached, adjustment instructions are automatically issued to the underlying resource pool to achieve dynamic adjustment.

[0016] Further preferably, the resource allocation algorithm specifically includes:

[0017] Initialize the resource pool and divide the resource pool of each region into three types of units: public, combined, and exclusive. Each unit describes a set of computing and storage resources.

[0018] Based on the associated resource requirements of the new task and the resource allocation of existing tasks with similar resource requirements, determine whether the initial resource size requested by the new task is sufficient to conduct an initial analysis of the running task;

[0019] By default, new tasks are assigned to public units for execution. After the task runs for a period of time and the operation status stabilizes, they are adjusted to exclusive or combined units to perform resource initialization.

[0020] The new task calls the preset corresponding engine to perform the analysis task.

[0021] Further preferably, the exclusive type unit is for tasks that run in real time and have clear running resource requirements; the combination type unit is for tasks that run on a scheduled basis and have clear running resource requirements; and the public type unit is for tasks that have no clear running resource requirements.

[0022] Further preferably, the dynamic adjustment algorithm specifically further includes: dynamically adjusting resource allocation of three types of units, namely public, exclusive and combined, in different zones.

[0023] Further preferably, the dynamic adjustment algorithm specifically includes:

[0024] Regularly analyze the changing patterns of actual task resource consumption to find the running patterns of new tasks, and then adjust the resource initialization values ​​of new tasks to complete the scheduling of new tasks;

[0025] Regularly analyze the changing patterns of the number of associated resources and the actual resource consumption of tasks to find the running patterns of new tasks, dynamically adjust the resource allocation of old tasks in advance, ensure that tasks will not be abnormal or interrupted, and complete the scheduling of the old tasks;

[0026] The dynamically adjusted parameters Ug, Ud, and Uz of each zone in the resource pool are sent to the underlying component layer as instructions to execute the re-arrangement and application of the resource pool.

[0027] In a second aspect, the present invention further provides a device for dynamically scheduling computing resources, comprising:

[0028] Analysis module: Two algorithms for responding to resource allocation and dynamic adjustment, enabling automatic analysis of the characteristics of each task;

[0029] Matching module: Two algorithms for responding to resource allocation and dynamic adjustment, enabling automatic matching of resource scheduling engines;

[0030] Execution module: used to enable the underlying component layer to respond to and execute the resource adjustment instructions.

[0031] In a third aspect, an embodiment of the present invention provides an electronic device comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0032] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) By optimizing the characteristics of different engines and combining them with the task rule library, a refined resource allocation system is automatically constructed using computer programs, thereby forming resource scheduling solutions for different types of tasks, meeting the dynamic scheduling needs of computing resources in various business scenarios, and solving the long-standing problem of poor operational stability of business models in the big data era.

[0035] (2) The present invention designs a multi-task fusion scheduling pool and proposes two algorithms: resource allocation and dynamic adjustment. It comprehensively analyzes the priority, resource consumption, fault tolerance, business attributes and other information of each task to obtain the characteristics of the task. Combined with the resource situation, a refined resource allocation system is constructed; the running status of each task is continuously tracked and analyzed, the changing rules of task operation are analyzed and predicted, and adjustment instructions are automatically issued to the underlying resource pool. It is highly adaptable to flexible and changeable business scenarios, ensuring the stable and efficient operation of the business model, and improving the support capabilities of big data for business analysis.

[0036] (3) Currently, there is no similar optimization algorithm on the market. This method has been implemented and integrated into our products. Based on massive data scenarios, this algorithm can adapt to the resource scheduling requirements of different types of business models and maximize resource utilization. According to actual calculations, the construction cost can be reduced by more than 50% under the data volume of hundreds of billions, and it can adapt to complex business scenario changes.

[0037] (4) The method of the present invention realizes a multi-task fusion scheduling mode, dynamically scheduling the operating resource requirements of the business model under each business scenario in real time, greatly improving the operating stability of the business model, increasing the coverage of big data empowerment of the business, and reducing the construction cost of big data projects.

[0038] (5) Through the resource allocation algorithm, the characteristics of each task are comprehensively analyzed. Combined with the resource situation, a refined resource allocation system is constructed. Through the dynamic adjustment algorithm, the changing rules of task operation are analyzed and predicted. When the critical value that requires resource adjustment is reached, the adjustment instruction is automatically issued to the underlying resource pool. It is highly adaptable to flexible and changeable business scenarios, ensuring the stable and efficient operation of the business model and improving the support capabilities of big data for business analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the present invention. Other embodiments and many of the expected advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with respect to each other. Like reference numerals designate corresponding similar parts.

[0040] Figure 1 is a diagram of an exemplary device architecture in which an embodiment of the present invention may be applied;

[0041] Figure 2 A flowchart of a method for dynamic scheduling of computing resources according to an embodiment of the present invention;

[0042] Figure 3A schematic diagram of a flow chart of a resource allocation algorithm in a method for dynamic scheduling of computing resources according to an embodiment of the present invention;

[0043] Figure 4 A schematic flow chart of a dynamic adjustment algorithm in a method for dynamic scheduling of computing resources according to an embodiment of the present invention;

[0044] Figure 5 This is a flowchart of the overall analysis of the method for dynamic scheduling of computing resources according to an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of a flow chart of a system for dynamic scheduling of computing resources according to an embodiment of the present invention;

[0046] Figure 7 It is a schematic structural diagram of a computer device suitable for implementing an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description and are shown by illustrative specific embodiments in which the present invention can be put into practice. To this end, directional terms, such as "top", "bottom", "left", "right", "up", "down", etc., are used with reference to the orientation of the figures being described. Because the components of the embodiments can be positioned in several different orientations, directional terms are used for the purpose of illustration and are in no way limiting. It should be understood that other embodiments can be utilized or logical changes can be made without departing from the scope of the present invention. Therefore, the following detailed description should not be adopted in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0049] Figure 1 An exemplary system architecture 100 is shown to which a method for processing information or an apparatus for processing information according to an embodiment of the present invention can be applied.

[0050] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0051] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0052] The terminal devices 101 , 102 , and 103 may be various electronic devices with communication functions, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0053] The server 105 may be a server that provides various services, such as a background information processing server that processes verification request information sent by the terminal devices 101, 102, and 103. The background information processing server may analyze and perform other processing on the received verification request information and obtain a processing result (e.g., verification success information indicating that the verification request is a legitimate request).

[0054] It should be noted that the method for processing information provided in the embodiment of the present invention is generally executed by the server 105, and accordingly, the device for processing information is generally set in the server 105. In addition, the method for sending information provided in the embodiment of the present invention is generally executed by the terminal devices 101, 102, and 103, and accordingly, the device for sending information is generally set in the terminal devices 101, 102, and 103.

[0055] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or multiple software modules, without specific limitation herein.

[0056] Due to the large data volumes and complex business scenarios of big data projects, traditional computing resource allocation methods all use preset fixed resource pool quotas and running thread numbers. That is, specific computing resources are fixedly allocated to a single task or task set to run the task. These technologies have the following shortcomings:

[0057] Poor adaptability to business changes. The data sources accessed by the big data platform are diverse and frequently change, and the operating environment of the business model will frequently change accordingly. Fixed allocation of specific computing resources often makes it difficult to timely match the resource requirements of the changing operating environment. This leads to abnormal situations such as interruptions or errors in the operation of business model tasks, seriously affecting the big data platform's ability to support business.

[0058] Low resource utilization efficiency. In the era of big data, the amount of data in various domains is very large. In order to ensure the normal operation of various business models as much as possible, the fixed allocation method often allocates a large amount of computing resources. The accumulation of redundant computing resources will result in low computing resource utilization efficiency and the project construction cost will increase exponentially.

[0059] To address the above issues, the present invention proposes a method and device for dynamic scheduling of computing resources. This method primarily utilizes two algorithms: resource allocation and dynamic adjustment. This method supports efficient operation of business models for massive amounts of data, improves the stability and cost-effectiveness of business model operation, and enhances the ability of big data to support business analysis. Compared to existing technologies, this invention has the following features:

[0060] 1) Different resource allocation methods. This invention innovatively obtains the characteristics of each task by comprehensively analyzing information such as task priority, resource consumption, fault tolerance, and business attributes. Combined with resource conditions, a refined resource allocation system is constructed, making the resource scheduling pool the optimal model for resource allocation in different business scenarios.

[0061] 2) The resource adjustment method is different. The present invention continuously tracks and analyzes the running status of each task to obtain the changing rules of task operation. When the critical value that requires resource adjustment is reached, the adjustment instruction can be automatically issued to the underlying resource pool. In this way, it can adapt to flexible and changeable business scenarios and ensure the stable and efficient operation of the business model.

[0062] Figure 2 The embodiment of the present invention discloses a method for dynamic scheduling of computing resources, such as Figure 2 As shown, the method includes the following steps:

[0063] S1. In response to two algorithms for resource allocation and dynamic adjustment, automatic analysis of the characteristics of each task and automatic matching of the resource scheduling engine are achieved;

[0064] S2. The underlying component layer responds to and executes the resource adjustment instructions.

[0065] Specifically, the resource allocation algorithm includes: presetting a multi-task fusion scheduling pool, analyzing the characteristics of various computing tasks, calculating the initial resources required for the tasks, and allocating corresponding resource pools to finely manage resource request queues and task execution queues.

[0066] Further, such as Figure 3 As shown, the resource allocation algorithm specifically includes:

[0067] S101. Initialize the resource pool and divide the resource pool of each region into three types of units: public, combined, and exclusive. Each unit describes a set of computing and storage resources.

[0068] S102: Based on the associated resource requirements of the new task and the resource allocation of existing tasks with similar resource requirements, determine whether the initial resource size requested by the new task is sufficient to perform an initial analysis of the running task;

[0069] S103. By default, the new task is assigned to a public unit for execution. After the task runs for a period of time and the running status is stable, it is adjusted to an exclusive or combined unit to initialize the running resources.

[0070] S104: The new task calls the preset corresponding engine to execute the analysis task.

[0071] Preferably, the exclusive type unit is for tasks that run in real time and have clear running resource requirements; the combination type unit is for tasks that run on a scheduled basis and have clear running resource requirements; and the public type unit is for tasks that have no clear running resource requirements.

[0072] Furthermore, the dynamic adjustment algorithm includes continuously tracking and analyzing the operating status of each task, capturing and recording the changing patterns of task operation. When a preset threshold is reached requiring resource adjustment, an adjustment instruction is automatically issued to the underlying resource pool, enabling dynamic adjustment. This dynamically adjusts resource allocation across three types of units: public, exclusive, and combined, across different zones.

[0073] Specifically, such as Figure 4 As shown, the dynamic adjustment algorithm specifically includes:

[0074] S201. Regularly analyze the changing patterns of actual task resource consumption to find the running patterns of new tasks, and then adjust the resource initialization values ​​of the new tasks to complete the scheduling of the new tasks.

[0075] S202. Regularly analyzing the changing patterns of the quantity of associated resources and the actual resource consumption of tasks to find the running patterns of new tasks, dynamically adjusting the resource allocation of old tasks in advance, ensuring that tasks will not be abnormal or interrupted, and completing the scheduling of the old tasks;

[0076] S203: Send the dynamically adjusted parameters Ug, Ud, and Uz of each zone in the resource pool as instructions to the underlying component layer to execute the re-arrangement and application of the resource pool.

[0077] Specifically, in this embodiment, the goal of the present invention is to analyze the characteristics of each task through program analysis, automatically match the resource scheduling engine, and issue resource adjustment instructions to the underlying resource pool to ensure the stable and efficient operation of the business model, improve resource utilization efficiency, and reduce the construction cost of the big data platform.

[0078] The resource scheduling process is mainly based on the following two core libraries:

[0079] 1. Task rule base: Obtains the characteristics of different types of tasks and analysis information on the required resources, as defined in Table 1 below.

[0080]

[0081] Table 1: Task rules Table 2. Resource allocation table: Get the resource scheduling information of each task, defined as follows Table 2.

[0082]

[0083]

[0084] Table 2: Resource Allocation Table

[0085] Specifically, the overall analysis process of the present invention is as follows Figure 5 As shown. Aiming at the efficient operation of business models for massive data, the present invention designs a multi-task fusion scheduling pool. Through two algorithms, resource allocation and dynamic adjustment, it comprehensively analyzes each task's priority, resource consumption, fault tolerance, business attributes, and other information to obtain task characteristics. Combined with resource availability, it builds a refined resource allocation system. This system continuously tracks and analyzes the operating status of each task, analyzes and predicts the changing patterns of task operation, and automatically issues adjustment instructions to the underlying resource pool. This system is highly adaptable to flexible and changing business scenarios, ensures the stable and efficient operation of business models, and enhances the support capabilities of big data for business analysis.

[0086] The present invention mainly uses two algorithms, resource allocation and dynamic adjustment, to automatically analyze the characteristics of each task, automatically match the resource scheduling engine, and issue resource adjustment instructions to the underlying component layer for execution, ensuring that the resources required by the business model are sufficient and run stably and efficiently. The main analysis model is as follows:

[0087] 1. Resource allocation algorithm: Due to the large data volume and complex business scenarios of big data projects, traditional computing resource allocation methods use a fixed preset resource pool quota and running thread number, which cannot meet the resource scheduling requirements of various running tasks.

[0088] This algorithm designs a multi-task fusion scheduling pool, analyzes the characteristics of various computing tasks, calculates the initial resources required by the computing tasks, allocates corresponding resource pools, and meticulously manages resource request queues and task execution queues. It builds an efficient and reasonable resource allocation mechanism, implements more flexible resource request queuing and task execution allocation logic, and ensures the stability and economy of the business model operation. Based on this analysis concept, the following algorithm illustrates how to allocate resources to a new task by receiving the input parameters Tn [task ID LWId, main resource ZSJY, associated node GLZD, requested resource CZJY, task type LWLC, start time KSSZ, interval time JGSZ, application identifier EYBZ, execution engine ZSYC].

[0089] a. Resource pool initialization: To ensure computing resource allocation for different types of tasks, in this embodiment, the resource allocation algorithm innovatively designs a new resource pool structure. The resource pool of each region is divided into three types of units: public, combined, and exclusive. Each unit describes a set of computing and storage resources. Exclusive units are for tasks that run in real time and have clear running resource requirements; combined units are for tasks that run on a scheduled basis and have clear running resource requirements; and public units are for tasks that have no clear running resource requirements.

[0090] Specifically, for a 16C256G resource pool, the default initialization size of the public type unit is 16C256G, the size of the exclusive type unit is 0C0G, and the size of the combined type unit is 0C0G, that is, Ug = 16C256G, Ud = 0C0G, and Uz = 0C0G.

[0091] b. Initial analysis of running tasks: The size of task-associated resources plays a decisive role in analyzing the resources required to run tasks. Therefore, this algorithm mainly uses the associated resource requirements of the new task, combined with the resource allocation of existing tasks with similar resource requirements, to determine whether the initial resource size requested by the new task is sufficient, thereby avoiding abnormalities in the initial startup of the task.

[0092] The specific algorithm is as follows: 1) obtain the task rule set Yn, and obtain the task rule set Yt with similar resource requirements from Yn, with the filtering condition "Yn.JYSL.node = Tn.JYSL.node and Yn.JYSL.resource = Tn.JYSL.resource", and the sorting condition Yn.JYSL.node.quantity;

[0093] 2) If there are running tasks with similar resources in the task rule library, and the current resources of the task exceed 20% of the resources requested by the new task, the initial resource size of the new task is set to the current resources of the task in the library. Otherwise, the resources requested by the new task are used as the initial resource size, where 20% is an adjustable threshold set according to engineering practice.

[0094] if (Yt.count>0and Yt.CSDS.resourcesize>Tn.CZJY.resourcesize*(1+20%)){Tn.CSDS.resourcesize=Yt.CSDS.resourcesize}

[0095] else{Tn.CSDS.resource size = Tn.CZJY.resource size}.

[0096] c. Initialization of running resources: The running resources required for each new task are related to many factors such as the size of the associated resources and the task processing efficiency. Therefore, the running status of the new task in the initial stage is unstable, and the initialized resource requirements are not 100% accurate and reasonable. Therefore, the present invention will assign the new task to the public type unit by default to run. After running for a period of time, the task running status stabilizes and then adjusts it to an exclusive or combined type unit to ensure the scientific and reasonable use of resource management.

[0097] The specific algorithm is as follows: obtain the resource allocation set Jn, filter and obtain the units of common types in each zone whose idle size exceeds 100% of the initial resource size of the new task;

[0098] 100% is an adjustable threshold set based on engineering practice. If there are multiple units that meet the criteria, the unit with the most identical application identifiers is found based on the application identifier of the new task. This way, similar application types are placed in the same area of ​​the resource pool as much as possible to facilitate management.

[0099] The filtering conditions are: Jn.TYPE = 'Public' and Jn.AVAILABLE>Tn.CSDS.Resource size*(1+100%); the sorting condition is count(Jn.TASK.EYBZ=Tn.EYBZ), the running resources of the new task are allocated to J1, and the relevant parameters of J1 are updated at the same time, that is, Tn.CSDS=[J1.ZONE, J1.UNITNUM, Tn.CSDS.Resource size]; J1.AVAILABLE=J1.AVAILABLE-Tn.CSDS.Resource size, J1.TASK=J1.TASK+[Tn.LWId, Tn.CSDS.Resource size], J1.TASKLIST=J1.TASKLIST+1.

[0100] d. Run task execution: The new task calls the corresponding engine Tn.ZSYC to execute the analysis task.

[0101] 2. Dynamic Adjustment Algorithm: Traditional task operation resource allocation is fixed and static, and cannot be automatically adjusted according to changes in the operating characteristics of the big data platform. This leads to abnormal situations such as interruptions or errors in the business model operation tasks, which seriously affects the big data platform's ability to support the business. The present invention continuously tracks and analyzes the operating status of each task to obtain the changing rules of task operation. When the critical value that requires resource adjustment is reached, the adjustment instruction can be automatically issued to the underlying resource pool, dynamically adjusting the resource allocation of the three types of units: public, exclusive, and combined in different zones, ensuring the stable and efficient operation of the business model, while also maximizing resource utilization and reducing construction costs. Based on this analysis concept, the algorithm includes two scenarios: new task adjustment and old task adjustment.

[0102] a. New task scheduling: Regularly analyze the changing patterns of actual task resource consumption to identify the operating patterns of new tasks and adjust the resource initialization values ​​for the new tasks.

[0103] The specific algorithm is as follows: 1) traverse and obtain the task rule set Yn every day, and obtain the new task rule set Yt running in the public type unit Ug from Yn, with the filtering condition "Yn.CSDS.ZONEID.TYPE = 'public'";

[0104] 2) Use the variance formula to analyze the changing pattern of the actual resource consumption of the task, that is, the fluctuation rate of actual resource usage S = ∑i = 1..n(Yi.ZSZG.consumed resource size-average value) / n; if S is within a certain range, it means that the operation is basically stable, such as the variable threshold [-20%, +20%] set according to engineering experience; if it exceeds the set range, it means that the fluctuation is large and the operation is not stable; when the analysis period exceeds the variable threshold set according to engineering experience, such as 15 days, that is, Day(SystemDate-Yt.KSSZ)>15 days, the new task with stable operation will be transferred from the public unit.

[0105] The transfer rules are as follows: 1) If the interval execution time of the task minus the average execution time is less than 1 hour, an exclusive unit is cut from the public unit and given to the new task, with a certain redundancy ratio, such as 50%. This ratio is a variable threshold based on engineering practice, i.e., Udi size = MAX(Yn.ZSZG.consumed resource size) * (1+Ratio), Udi.TYPE = 'exclusive'; Yn.CSDS = [Udi.ZONEID, Udi.ZONENUM, Udi.UNITNUM, Udi.STYLE]; Ug = Ug-Udi;

[0106] 2) If the task's interval execution time - average execution time > 1 hour, it should be transferred to a composite unit. This algorithm allocates composite units according to different resource usage ranges, such as 0-100M, 100M-200M, etc., to avoid the situation where the composite area uses the maximum resource usage to allocate space and the usage rate is low most of the time. If a unit with the same usage range already exists and the task's time does not conflict with other tasks, the task will be assigned to the existing unit. Otherwise, a new unit will be created.

[0107] Judgment condition: if (MAX(Yn.ZSZG.consumed resource size)*(1+Ratio)∈Jn.SCALE andjn.TYPE='combination' and Yt.KSSZ not in(jn.TASK.KSSZ~jn.TASK.KSSZ+JGSZ)){Yn.TASK=Yn.TASK+Yt}

[0108] else{Udi size = MAX(Yn.ZSZG.consumed resource size)*(1+Ratio), Udi.TYPE = 'combination'; Yn.CSDS = [Udi.ZONEID, Udi.ZONENUM, Udi.UNITNUM, Udi.STYLE], Ug = Ug-Udi}.

[0109] b. Scheduling old tasks: Regularly analyze the changing patterns of the number of associated resources and the actual resource consumption of tasks to find the operating patterns of new tasks, and dynamically adjust the resource allocation of old tasks in advance to ensure that tasks will not experience abnormalities or interruptions.

[0110] The specific algorithm is as follows: 1) traverse and obtain the task rule set Yn every week, and obtain the task rule set Yt running on the exclusive type unit Ud and the combination type unit Uz from Yn, with the filtering condition "Yn.CSDS.ZONEID.TYPE = 'exclusive' or Yn.CSDS.ZONEID.TYPE = 'combination'";

[0111] 2) When the actual resource consumption of a task exceeds a certain proportion of the current allocated resource size (this proportion is a variable threshold set based on engineering practice experience, such as 80%), the size of the resource block is automatically increased based on factors such as the data volume change range. When the actual resource consumption of a task is lower than a certain proportion of the current allocated resource size (this proportion is a variable threshold set based on engineering practice experience, such as 50%), the size of the resource block is automatically reduced based on factors such as the data volume change range. When the actual resource consumption of a task exceeds a certain period (this period is a variable threshold set based on engineering practice experience, such as 90 days) and is 0, the allocated resource block is automatically cancelled.

[0112] If (Yn.ZSZG. Consumed resource size / Yn.CSDS>80%) {Yn.CSDS=Yn.CSDS*(1+Ratio-20%)} else if (Yn.ZSZG. Consumed resource size / Yn.CSDS<50%) {Yn.CSDS=Yn.ZSZG. Consumed resource size*(1+Ratio)}

[0113] else if (Yn.ZSZ.consumed resource size=0and Day(SystemDate-Yn.ZSZ.start time)>90 days) {Yn.CSDS=0}.

[0114] c. Resource pool orchestration: The dynamically adjusted parameters Ug, Ud, and Uz of each zone in the resource pool are sent as instructions to the underlying component layer to execute the re-orchestration and application of the resource pool.

[0115] In a second aspect, an embodiment of the present invention further discloses a device for dynamically scheduling computing resources, such as Figure 6 As shown, the system includes: an analysis module 61 , a matching module 62 and an execution module 63 .

[0116] In this embodiment, the analysis module 61 is used to respond to two algorithms of resource allocation and dynamic adjustment to realize automatic analysis of the characteristics of each task; the matching module 62 is used to respond to two algorithms of resource allocation and dynamic adjustment to realize automatic matching of the resource scheduling engine; the execution module 63 is used to enable the underlying component layer to respond to and execute the issued resource adjustment instructions.

[0117] By optimizing the characteristics of different engines and combining them with a task rule library, the present invention automatically constructs a refined resource allocation system with a computer program, thereby forming a resource scheduling solution for different types of tasks, meeting the dynamic scheduling needs of computing resources in various business scenarios, and solving the long-standing problem of poor operational stability of business models in the big data era.

[0118] The present invention designs a multi-task fusion scheduling pool and proposes two algorithms: resource allocation and dynamic adjustment. It comprehensively analyzes the priority, resource consumption, fault tolerance, business attributes and other information of each task to obtain the characteristics of the task. Combined with the resource situation, a refined resource allocation system is constructed. The system continuously tracks and analyzes the running status of each task, analyzes and predicts the changing laws of task operation, and automatically issues adjustment instructions to the underlying resource pool. It is highly adaptable to flexible and changeable business scenarios, ensures the stable and efficient operation of the business model, and enhances the support capabilities of big data for business analysis.

[0119] Currently, there are no similar optimization algorithms on the market. This method has been implemented and integrated into our products. Based on massive data scenarios, this algorithm can adapt to the resource scheduling needs of different business models and maximize resource utilization. Actual measurements have shown that, with data volumes in the hundreds of billions, it can reduce construction costs by over 50%, and it can adapt to complex business scenarios.

[0120] Reference below Figure 7 , which shows an electronic device (eg Figure 1 A structural diagram of a computer device 700 (server or terminal device shown). Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0121] like Figure 7 As shown, the computer device 700 includes a central processing unit (CPU) 701 and a graphics processing unit (GPU) 702, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 703 or the program loaded from the storage part 709 to the random access memory (RAM) 706. Various programs and data required for the operation of the device 700 are also stored in the RAM 704. The CPU 701, GPU 702, ROM 703 and RAM 704 are connected to each other via a bus 705. An input / output (I / O) interface 706 is also connected to the bus 705.

[0122] The following components are connected to the I / O interface 706: an input section 707 including a keyboard, a mouse, and the like; an output section 708 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 709 including a hard disk and the like; and a communication section 710 including a network interface card such as a LAN card or a modem. The communication section 710 performs communication processing via a network such as the Internet. A drive 711 may also be connected to the I / O interface 706 as needed. A removable medium 712, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 711 as needed, so that a computer program read therefrom can be installed into the storage section 709 as needed.

[0123] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication part 710, and / or installed from a removable medium 712. When the computer program is executed by the central processing unit (CPU) 701 and the graphics processing unit (GPU) 702, the above-mentioned functions defined in the method of the present invention are performed.

[0124] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable medium, or any combination thereof. Computer-readable media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, apparatuses, or components, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0125] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The modules involved in the embodiments of the present invention may be implemented in software or hardware, and may also be set in a processor.

[0128] As another aspect, the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently and not incorporated into the electronic device. The computer-readable medium carries one or more programs. When executed by the electronic device, the one or more programs enable the electronic device to: automatically analyze the characteristics of each task and automatically match the resource scheduling engine in response to two algorithms for resource allocation and dynamic adjustment; and have the underlying component layer respond to and execute resource adjustment instructions.

[0129] The above description is merely an illustration of the preferred embodiments of the present invention and the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

Claims

1. A method for dynamic scheduling of computing resources, characterized in that: The method comprises the following steps: Obtain analysis information on the characteristics of different types of tasks and required resources based on the task rule library; obtain resource scheduling information for each task based on the resource allocation table; allocate resources to the new task by receiving the input parameter Tn of a new task; In response to two algorithms for resource allocation and dynamic adjustment, automatic analysis of the characteristics of each task and automatic matching of the resource scheduling engine are achieved; wherein: The resource allocation algorithm includes: presetting a multi-task fusion scheduling pool, initializing the resource pool, and dividing the resource pool in each area into three types of units: public, combined and exclusive, each unit describing a group of computing and storage resources; judging whether the initial resource size requested by the new task is sufficient based on the associated resource requirements of the new task and the resource allocation of existing tasks with similar resource requirements, so as to perform an initial analysis of the running task; by default, the new task is assigned to the public type unit to run, and after running for a period of time and the task running situation is stable, it is adjusted to the exclusive or combined type unit to initialize the running resources; the new task calls the preset corresponding engine to execute the analysis task; the exclusive type unit is for tasks that run in real time and have clear running resource requirements; the combined type unit is for tasks that run on a scheduled basis and have clear running resource requirements; the public type unit is for tasks that have no clear running resource requirements; The dynamic adjustment algorithm includes: regularly analyzing the change pattern of actual resource consumption of tasks to find the operation pattern of new tasks, and then adjusting the resource initialization value of the new tasks to complete the scheduling of new tasks; regularly analyzing the change pattern of the number of associated resources and the change pattern of actual resource consumption of tasks to find the operation pattern of new tasks, and dynamically adjusting the resource allocation of old tasks in advance to ensure that tasks will not be abnormal or interrupted, and completing the scheduling of the old tasks; the dynamically adjusted parameters Ug, Ud, and Uz of each zone in the resource pool are sent as instructions to the underlying component layer; The underlying component layer responds to and executes the resource adjustment instruction, thereby performing the rearrangement of the resource pool.

2. The method for dynamic scheduling of computing resources according to claim 1, characterized in that: The resource allocation algorithm includes: Analyze the characteristics of various computing tasks, calculate the initial resources required for the tasks, and allocate corresponding resource pools to finely manage resource request queues and task execution queues.

3. The method for dynamic scheduling of computing resources according to claim 1, characterized in that: The dynamic adjustment algorithm includes: Continuously track and analyze the running status of each task, obtain and record the changing patterns of task operation; When the preset critical value that requires resource adjustment is reached, adjustment instructions are automatically issued to the underlying resource pool to achieve dynamic adjustment.

4. The method for dynamic scheduling of computing resources according to claim 1, characterized in that: The dynamic adjustment algorithm specifically includes: dynamically adjusting the resource allocation of three types of units: public, exclusive, and combined in different zones.

5. The method for dynamic scheduling of computing resources according to claim 1, characterized in that: The dynamic adjustment algorithm includes performing rearrangement and application of the resource pool.

6. A device for dynamic scheduling of computing resources, characterized in that: The method according to any one of claims 1 to 5, further comprising: Analysis module: Two algorithms for responding to resource allocation and dynamic adjustment, enabling automatic analysis of the characteristics of each task; Matching module: Two algorithms for responding to resource allocation and dynamic adjustment, enabling automatic matching of resource scheduling engines; Execution module: used to enable the underlying component layer to respond to and execute the resource adjustment instructions.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images