Computing power resource configuration scheduling method and system based on cloud computing platform

Abstract

The invention relates to the technical field of resource scheduling, and particularly discloses a computing power resource configuration scheduling method and system based on a cloud computing platform, and the method comprises the following steps: S1, obtaining a cloud computing platform composed of a plurality of same computing nodes, and obtaining historical task records; dividing a task code of the historical task into a plurality of code blocks, setting a standard operation rate, and obtaining consumption time and obtaining time of each code block; s2, acquiring a distributed task, and dividing the distributed task into a plurality of sub-tasks; obtaining a target database and target obtaining time of the subtasks; grouping all the sub-tasks to obtain a plurality of task groups; finally obtaining a plurality of grouping results; s3, obtaining the predicted consumption time of the subtasks, and predicting the predicted total consumption time of the cloud computing platform under each grouping result to obtain a total time set; the optimal group is obtained according to the total time set, and the cloud computing platform distributes the distributed tasks to the computing nodes according to the optimal group.

Description

Technical Field

[0001] The present invention relates to the technical field of resource scheduling, and particularly to a computing power resource allocation and scheduling method and system based on a cloud computing platform. Background Art

[0002] Computing power resource allocation and scheduling is a process of allocating and managing computing resources in a computer cluster, data center or cloud computing environment; by reasonably allocating computing power resources, idle and waste of resources can be avoided, ensuring that computing tasks can be efficiently executed and improving the overall resource utilization efficiency.

[0003] In the prior art, tasks are often allocated to computing nodes based on the priority of tasks or the consumption of computing power resources by tasks. However, this method has certain limitations. It mainly focuses on trying to improve the computing power utilization efficiency by adjusting the task allocation order. For example, strategies such as giving priority to processing tasks with high priority or tasks with less consumption of computing power resources. But this means only optimizes the queuing and allocation links of tasks, and does not deeply optimize the task execution process itself. That is to say, there are no further improvement and optimization measures for the execution method and execution steps of a single task on a specific computing node, and it is impossible to explore the potential of improving computing power utilization efficiency and enhancing the overall system performance from the micro level of task execution, making it difficult to meet the increasingly complex and diverse computing demand scenarios. Summary of the Invention

[0004] The purpose of the present invention is to provide a computing power resource allocation and scheduling method and system based on a cloud computing platform to solve the above technical problems.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A computing power resource allocation and scheduling method based on a cloud computing platform includes the following steps:

[0007] Step S1: Obtain a cloud computing platform composed of several identical computing nodes, obtain historical task records, where the historical task records include the task codes of all historical tasks, divide the task codes into several code blocks; set a standard operating rate, obtain the consumption time and acquisition time of each code block;

[0008] Step S2: Obtain distributed tasks and divide them into several subtasks, obtain the target database and target acquisition time of the subtasks; group all subtasks, and preferentially record several subtasks with the same target database as the same group, denoted as a task group, and finally obtain several grouping results;

[0009] Step S3: Obtain the estimated consumption time of the subtask, predict the estimated total consumption time of the cloud computing platform under each grouping result, and obtain a total time set according to the estimated total consumption time of each grouping result; obtain the optimal grouping according to the total time set, and the cloud computing platform distributes distributed tasks to each computing node according to the optimal grouping.

[0010] As a further solution of the present invention: The process of dividing the task code into several code blocks includes:

[0011] Obtain all functions called in the code block, where the functions include built-in functions, custom functions, and library functions; determine the boundaries of each function, where the boundaries include loading data and returning, processing data and returning, analyzing data and returning, and saving results; record the boundaries of each function as a code node, and form a code block by every two adjacent code nodes.

[0012] As a further solution of the present invention: The process of setting the standard running rate includes:

[0013] Set the highest load L max , where the highest load is the load value at which the cloud computing platform will experience lag, and the load value includes CPU utilization and memory usage rate; set several rate nodes, obtain the load values when the cloud computing platform runs at each rate node, and obtain a load value set {L1, L2,..., L M}, where L M represents the load value corresponding to the Mth rate node, and M is the total number of rate nodes; select the load values that satisfy L r < L max from the load value set, and record them as candidate load values, where L r represents any element in the load value set; obtain the rate nodes corresponding to each candidate load value, record them as candidate rate nodes, and obtain the maximum value among all candidate rate nodes, which is recorded as the standard running rate.

[0014] As a further solution of the present invention: The process of obtaining the consumption time and acquisition time of the code block includes:

[0015] Record the start time of executing the code block, and obtain the end time when the code block execution ends. Obtain the difference between the start time and the end time, and record it as the consumption time;

[0016] Record the start moment of obtaining data from the database, and record the end moment when the data acquisition ends. Obtain the difference between the start moment and the end moment, and record it as the acquisition difference. Record the acquisition difference as the acquisition time.

[0017] As a further solution of the present invention: Obtain the input data of the subtask, and obtain the database where the input data is located, denoted as the target database, and obtain the acquisition time of the target database, denoted as the target acquisition time.

[0018] As a further solution of the present invention: The process of obtaining the total expected consumption time of the cloud computing platform includes:

[0019] The expected consumption time of the subtask is the sum of the consumption times of all code blocks included in the subtask; according to the expected consumption time, predict the total expected consumption time of the cloud computing platform under each grouping result where N is the total number of task groups in the grouping result, k represents the kth task group, T i represents the consumption time of the ith code block in the task group, μ k represents the total acquisition time within the kth task group, t i represents the acquisition time of the ith code block, m i represents the optimization coefficient of the ith code block. If there is a code block in the task group with the same target database as the ith code block, then m i = 0; otherwise, m i = 1.

[0020] As a further solution of the present invention: The process of obtaining the total acquisition time μ k within the kth task group includes:

[0021] Obtain the acquisition times of all code blocks included in the task group to obtain an acquisition time sequence, eliminate duplicate elements in the acquisition time sequence to obtain an acquisition time set; add up all the acquisition times in the acquisition time set to obtain the total acquisition time of the task group.

[0022] As a further solution of the present invention: A computing power resource configuration and scheduling system based on a cloud computing platform includes:

[0023] Initial module: Obtain a cloud computing platform composed of several identical computing nodes, obtain historical task records, where the historical task records include the task codes of all historical tasks, divide the task codes into several code blocks; set a standard operating rate, and obtain the consumption time and acquisition time of each code block;

[0024] Grouping module: Obtain distributed tasks and divide them into several subtasks, obtain the target database and target acquisition time of the subtasks; group all subtasks, and preferentially record several subtasks with the same target database as the same group, denoted as a task group, and finally obtain several grouping results;

[0025] Scheduling module: Obtain the estimated consumption time of the subtasks, predict the total estimated consumption time of the cloud computing platform under each grouping result, and obtain a total time set according to the total estimated consumption time of each grouping result; obtain the optimal grouping according to the total time set, and the cloud computing platform distributes distributed tasks to each computing node in the optimal grouping.

[0026] Advantages of the present invention:

[0027] The present invention optimizes the task code in detail and deeply, delving into the microscopic level of task execution for optimization, and tapping the potential to improve the utilization efficiency of computing power; by dividing the task code into several code blocks and analyzing the consumption time and acquisition time of each code block, the task execution process can be optimized more precisely; grouping according to the target database of the subtasks and dynamically adjusting the resource allocation of the computing nodes enables more flexible and efficient utilization of computing power resources; through grouping optimization, duplicate operations for data acquisition are reduced, and data access efficiency is improved; and by optimizing the task execution process and flexible scheduling of computing power resources, the overall performance of the system is significantly improved, meeting the increasingly complex and diverse computing demand scenarios; by predicting the total estimated consumption time of each grouping result and selecting the optimal grouping, it is ensured that the task can be completed in the shortest time; this method and system can adapt to different types of computing tasks and cloud computing platforms, with strong versatility and adaptability; by setting a standard operating rate and an optimization coefficient, it can adapt to different load conditions and task types; by reasonably allocating computing power resources, idle and waste of resources are avoided, ensuring that computing tasks can be efficiently executed; by preferentially grouping subtasks with the same target database, the time for data acquisition is reduced, and the task execution efficiency is improved; the present invention significantly improves the utilization efficiency of computing power and the overall system performance by deeply optimizing the task execution process and flexible scheduling of computing power resources, meets the complex and diverse computing demand scenarios, and has strong versatility and adaptability. Description of the Drawings

[0028] The present invention will be further described below with reference to the accompanying drawings.

[0029] Figure 1 It is a schematic flowchart of the method for configuring and scheduling computing power resources based on a cloud computing platform according to the present invention. Detailed Embodiments

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] Please refer to Figure 1 As shown, the present invention is a computing power resource allocation and scheduling method based on a cloud computing platform, including the following steps:

[0032] Step S1: Obtain a cloud computing platform composed of a number of identical computing nodes, obtain the historical task records of the cloud computing platform, the historical task records include the task codes of all historical tasks, and divide the task codes into several code blocks according to the called functions;

[0033] Set the standard operating rate of the computing node, obtain the time consumed by the computing node to run each code block at the standard operating rate, denoted as the consumption time; obtain all databases communicatively connected to the cloud computing platform, and obtain the time consumed by the cloud computing platform to obtain data in each database, denoted as the acquisition time;

[0034] It can be understood that the task codes of historical tasks are divided into several code blocks according to the called functions, and each code block represents an independent logical unit, usually corresponding to a function or a group of related functions; by dividing the code blocks, the execution time and resource consumption of each code block can be analyzed more carefully; setting the standard operating rate of the computing node, that is, the operating speed of the computing node under the optimal load, and determining a highest operating rate that will not cause system jamming by testing the load values under different rate nodes;

[0035] It should be noted that by dividing the code blocks and recording the consumption time, the bottlenecks and inefficient parts in task execution can be identified, so as to carry out targeted optimization; by setting the standard operating rate, ensure that the computing node runs under the optimal load, avoid resource waste and system jamming; recording the acquisition time helps to identify latency problems in data access, optimize the data acquisition strategy, and reduce the data access time; the records of the consumption time and the acquisition time provide data support for subsequent task scheduling and resource allocation, ensuring that tasks can be executed efficiently;

[0036] As a preferred embodiment of the present invention, the process of dividing the code blocks includes:

[0037] Obtain all the functions called in the code block, the functions include built-in functions, custom functions, and library functions; determine the boundaries of each function, the boundaries include loading data and returning, processing data and returning, analyzing data and returning, and saving results; record each function boundary as a code node, and form a code block by every two adjacent code nodes;

[0038] It can be understood that all the functions called in the code block are obtained, including built-in functions, custom functions, and library functions. These functions represent the main operations and logic in the code block. Determine the boundaries of each function, that is, the input, output, and internal logic of the function, specifically including:

[0039] Load data and return: The function loads data from an external source (such as a database) and returns the result;

[0040] Process data and return: The function processes the data and returns the processed result;

[0041] Analyze data and return: The function analyzes the data and returns the analysis result;

[0042] Save the result: The function saves the result to external storage (such as a database);

[0043] Define the boundary of each function as a code node, and the code node represents an independent operation unit of the function. Each adjacent pair of code nodes constitutes a code block. A code block is a logical unit composed of multiple related operations and usually represents a complete subtask or operation step;

[0044] It should be noted that by dividing the code block, the complex task code is decomposed into multiple independent modules, which is convenient for management and optimization. The division of the code block makes the code structure clearer, facilitating understanding and maintenance. By analyzing the execution time and resource consumption of each code block, the bottlenecks and inefficient parts in the task execution are identified, so as to carry out targeted optimization. The division of the code block provides a basis for subsequent task scheduling and resource allocation, ensuring that the task can be executed efficiently. By optimizing the execution order and resource allocation of the code block, the overall performance and response speed of the cloud computing platform are improved to meet complex and diverse computing requirements. By analyzing function calls, determining function boundaries, defining code nodes, and forming code blocks, the complex task code is decomposed into multiple independent modules. The purpose is to achieve modular design, improve the readability and maintainability of the code, optimize task execution, support dynamic scheduling, and thus improve the overall system performance;

[0045] In a preferred embodiment of the present invention, the process of setting the standard running rate includes:

[0046] Set the highest load L max , where the highest load is the load value at which the cloud computing platform will experience lag, and the load value includes CPU utilization and memory usage. Set a number of rate nodes, and obtain the load values when the cloud computing platform runs at each rate node to obtain a load value set {L1, L2,..., L M}, where L Mdenotes the load value corresponding to the Mth rate node, where M is the total number of rate nodes; select from the load value set the load values that satisfy L r <L max , and denote them as candidate load values, where L r represents any element in the load value set; obtain the rate nodes corresponding to each candidate load value, denote them as candidate rate nodes, and obtain the maximum value among all candidate rate nodes, which is denoted as the standard operating rate;

[0047] It can be understood that the highest load is the maximum load value at which the cloud computing platform does not experience jamming during operation; set a number of rate nodes, and each rate node represents the load condition of the cloud computing platform at different operating rates; screen out from the load value set the load values that satisfy L r <L max , and denote them as candidate load values, and the rate nodes corresponding to these load values will not cause system jamming;

[0048] It should be noted that by setting the highest load and screening candidate load values, it is ensured that the cloud computing platform does not experience jamming at the standard operating rate and the system stability is maintained;

[0049] As a preferred embodiment of the present invention, the process of obtaining the consumption time includes:

[0050] Record the start time of executing the code block, and obtain the end time when the code block execution ends, and obtain the difference between the start time and the end time, which is denoted as the consumption time; by testing the load values at different rate nodes, find the optimal operating rate to ensure the efficient utilization of resources (such as CPU and memory) and avoid resource waste;

[0051] As a preferred embodiment of the present invention, the process of obtaining the acquisition time includes:

[0052] Record the start moment of obtaining data from the database, and record the end moment when the data acquisition ends, obtain the difference between the start moment and the end moment, which is denoted as the acquisition difference, and denote the acquisition difference as the acquisition time; determine the standard operating rate to enable the cloud computing platform to operate at the optimal load, improve the overall system performance and response speed;

[0053] Step S2: Obtain the distributed task, divide the distributed task into several subtasks, obtain the input data of the subtasks, and obtain the database where the input data is located, denoted as the target database, and obtain the acquisition time of the target database, denoted as the target acquisition time; group all the subtasks, and preferentially mark several subtasks with the same target database as the same group, denoted as the task group, and finally obtain several grouping results; setting the standard running rate provides a basis for subsequent task scheduling and resource allocation, and supports dynamic adjustment and optimization; by setting and testing different rate nodes, it can adapt to different load conditions and ensure efficient operation under various task requirements.

[0054] As a preferred embodiment of the present invention, during the process of grouping all the subtasks, after preferentially marking several subtasks with the same target database as the same group, if there are remaining subtasks, directly mark the remaining several subtasks as a task group.

[0055] Step S3: Obtain the task code of the subtask, and obtain the estimated consumption time of each subtask according to the code blocks included in the task code and the consumption time of the code blocks; each subtask belonging to the same task group is run by the same computing node, and according to the estimated consumption time, predict the estimated total consumption time of the cloud computing platform under each grouping result. Where N is the total number of task groups within the grouping result, k represents the kth task group, T i represents the consumption time of the ith code block within the task group, μ k represents the total acquisition time within the kth task group, t i represents the acquisition time of the ith code block, m i represents the optimization coefficient of the ith code block. If there is a code block with the same target database as the ith code block within the task group, then m i = 0; otherwise, m i = 1.

[0056] According to the estimated total consumption time of each grouping result, obtain the total time set K, obtain the minimum value in the total time set K, denoted as Min(K); obtain the grouping result corresponding to the minimum value Min(K), denoted as the optimal grouping, and the cloud computing platform allocates the distributed task to each computing node according to the optimal grouping. The computing node preferentially obtains the input data of each code block and runs each code block in sequence.

[0057] As a preferred embodiment of the present invention, the estimated consumption time of the subtask is the sum of the consumption times of all the code blocks included in the subtask.

[0058] As a preferred embodiment of the present invention, the total acquisition time μ within the kth task groupk The obtaining process includes:

[0059] Obtain the acquisition time of all code blocks included in the task group to obtain an acquisition time series, eliminate duplicate elements in the acquisition time series to obtain an acquisition time set; add up all the acquisition times in the acquisition time set to obtain the total acquisition time of the task group;

[0060] It can be understood that the acquisition time of all code blocks in the task group is obtained to form an acquisition time series; duplicate elements in the acquisition time series are eliminated to obtain an acquisition time set. This is because multiple code blocks may obtain data from the same data source, and duplicate acquisition times only need to be calculated once; add up all the times in the acquisition time set to obtain the total acquisition time of the task group;

[0061] It should be noted that by calculating the estimated consumption time of subtasks, the execution time of each subtask can be accurately estimated, providing a basis for task scheduling and resource allocation; by eliminating duplicate acquisition times, duplicate calculations are avoided, the data acquisition time is optimized, and unnecessary waiting time is reduced; by calculating the total acquisition time of the task group, the execution time of the task group can be predicted more accurately, thereby optimizing task scheduling and improving the overall efficiency of the system; the calculation of the estimated consumption time and the total acquisition time provides data support for selecting the best grouping to ensure that tasks can be completed in the shortest time; by accurately estimating and optimizing the task execution time and data acquisition time, the overall performance and response speed of the cloud computing platform are improved.

[0062] A computing power resource configuration and scheduling system based on a cloud computing platform includes:

[0063] Initial module: Obtain a cloud computing platform composed of several identical computing nodes, obtain historical task records, where the historical task records include the task codes of all historical tasks, divide the task codes into several code blocks; set a standard running rate, and obtain the consumption time and acquisition time of each code block;

[0064] Grouping module: Obtain distributed tasks and divide them into several subtasks, obtain the target database and target acquisition time of the subtasks; group all subtasks, and preferentially mark several subtasks with the same target database as the same group, denoted as a task group, and finally obtain several grouping results;

[0065] Scheduling module: Obtain the estimated consumption time of the subtasks, and predict the estimated total consumption time of the cloud computing platform under each grouping result. According to the estimated total consumption time of each grouping result, obtain a total time set; obtain the best grouping according to the total time set, and the cloud computing platform distributes distributed tasks to each computing node according to the best grouping.

[0066] The above has described in detail an embodiment of the present invention, but the content described above is only a preferred embodiment of the present invention and cannot be considered as defining the scope of implementation of the present invention. All equivalent changes and improvements made in accordance with the scope of the application of the present invention should still fall within the scope covered by the patent of the present invention.

Claims

1. A computing power resource allocation and scheduling method based on a cloud computing platform, characterized in that Including the following steps: Step S1: Obtain a cloud computing platform composed of several identical computing nodes, obtain historical task records, where the historical task records include the task codes of all historical tasks, divide the task codes into several code blocks; set a standard running rate, obtain the consumption time and acquisition time of each code block; Step S2: Obtain a distributed task and divide it into several subtasks, obtain the target database and target acquisition time of the subtasks; Group all the subtasks. First, mark several subtasks with the same target database as the same group, denoted as a task group, and finally obtain several grouping results; Step S3: Obtain the estimated consumption time of the subtasks, and predict the estimated total consumption time of the cloud computing platform under each grouping result. According to the estimated total consumption time of each grouping result, obtain a total time set; obtain the best grouping based on the total time set, and the cloud computing platform distributes the distributed tasks to each computing node according to the best grouping.

2. The computing power resource allocation and scheduling method based on a cloud computing platform according to claim 1, wherein In step S1, the process of dividing the task codes into several code blocks includes: Obtain all the functions called in the code block, where the functions include built-in functions, custom functions, and library functions; determine the boundaries of each function, where the boundaries include loading data and returning, processing data and returning, analyzing data and returning, and saving results; mark the boundaries of each function as a code node, and form a code block by every two adjacent code nodes.

3. The computing power resource allocation and scheduling method based on a cloud computing platform according to claim 1, wherein In step S1, the process of setting the standard running rate includes: Set the maximum load L max , where the maximum load is the load value at which the cloud computing platform will experience lag, and the load value includes CPU utilization and memory usage; set a number of rate nodes, obtain the load values when the cloud computing platform runs at each rate node, and obtain a load value set {L1, L2,..., L M}], where L M represents the load value corresponding to the Mth rate node, and M is the total number of rate nodes; select the load values that satisfy L r < L max from the load value set, and denote them as candidate load values, where L r represents any element in the load value set; obtain the rate nodes corresponding to each candidate load value, denote them as candidate rate nodes, and obtain the maximum value among all candidate rate nodes, which is denoted as the standard operating rate.

4. The computing power resource allocation and scheduling method based on a cloud computing platform according to claim 1, wherein, In step S1, the process of obtaining the consumption time and acquisition time of the code block includes: Record the start time of executing the code block, and obtain the end time when the code block execution ends. Obtain the difference between the start time and the end time, denoted as the consumption time; Record the start moment of obtaining data from the database, and record the end moment when the data acquisition ends. Obtain the difference between the start moment and the end moment, denoted as the acquisition difference, and denote the acquisition difference as the acquisition time.

5. The computing power resource allocation and scheduling method based on a cloud computing platform according to claim 1, wherein In step S2, obtain the input data of the subtask, and obtain the database where the input data is located, denoted as the target database, and obtain the acquisition time of the target database, denoted as the target acquisition time.

6. The computing power resource configuration and scheduling method based on a cloud computing platform according to claim 1, characterized in that In step S3, the process of obtaining the estimated total consumption time of the cloud computing platform includes: The estimated consumption time of the subtask is the sum of the consumption times of all code blocks included in the subtask; according to the estimated consumption time, predict the total estimated consumption time of the cloud computing platform under each grouping result where N is the total number of task groups in the grouping result, k represents the kth task group, T i represents the consumption time of the ith code block in the task group, μ k represents the total acquisition time within the kth task group, t i represents the acquisition time of the ith code block, m i represents the optimization coefficient of the ith code block. If there is a code block in the task group with the same target database as the ith code block, then m i = 0; otherwise, m i = 1.

7. The computing power resource allocation and scheduling method based on a cloud computing platform according to claim 6, wherein In step S3, the acquisition total time μ within the k-th task group k is obtained through the following process: Obtain the acquisition times of all the code blocks included in the task group, obtain an acquisition time sequence, eliminate the duplicate elements in the acquisition time sequence, and obtain an acquisition time set; add up all the acquisition times in the acquisition time set to obtain the total acquisition time of the task group.

8. A computing power resource allocation and scheduling system based on a cloud computing platform, characterized in that, Including: Initial module: Obtain a cloud computing platform composed of several identical computing nodes, obtain historical task records, where the historical task records include the task codes of all historical tasks, divide the task codes into several code blocks; set a standard running rate, obtain the consumption time and acquisition time of each code block; Grouping module: Obtain a distributed task and divide it into several subtasks, obtain the target database and target acquisition time of the subtasks; Group all subtasks. First, mark several subtasks with the same target database as the same group, denoted as a task group, and finally obtain several grouping results; Scheduling module: Obtain the estimated consumption time of the subtasks, and predict the estimated total consumption time of the cloud computing platform under each grouping result. Obtain a total time set according to the estimated total consumption time of each grouping result; Obtain the best grouping according to the total time set, and the cloud computing platform distributes distributed tasks to each computing node according to the best grouping.

Images