Self-correcting computing power measurement method, system and apparatus

Abstract

The present application relates to the technical field of computing power networks, in particular to a self-correcting computing power measurement method, system and apparatus. The method comprises: monitoring performance data of an application, analyzing per-transaction resource consumption metrics in the temporal dimension, determining trough periods of metric values as idle hours, and determining peak periods of the metric values as busy hours; randomly selecting one idle hour and one busy hour every day / per week, respectively triggering a measurement task manager to carry out a test, and outputting an idle period, a busy period, an idle hour measurement value and a busy hour measurement value of the system every day / per week; and if a difference of the per-transaction resource consumption metric values of the same period in two consecutive weeks is greater than a preset warning value, re-triggering the measurement task manager for carrying out a test, and outputting a new computing power measurement value. The present invention outputs a temporal-dimension computing power measurement result, comprising the idle period, the busy period, the idle hour measurement value and the busy hour measurement value of the system, thereby more accurately reflecting changes of the computing power along with time.

Description

A method, system and device for automatically correcting computing power measurement

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 2023117182398, filed on December 14, 2023, entitled “A method, system and device for automatically correcting computational power measurement,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present invention relates to the field of computing power network technology, and in particular to a method, system and device for automatically correcting computing power measurement. Background Art

[0004] In the computing network sector, accurate computing power measurement can improve system efficiency and reduce application costs. Computing power measurement is typically determined using a benchmarking method. A load test is run once, or periodically, on a specified computing resource to obtain one or more application-specific metrics as the computing power metric.

[0005] With the rapid development of cloud computing technology, the computing resources provided by data centers to applications are typically virtual machines or containers of certain specifications. These resources are virtualized and over-allocated on top of physical resources. The actual computing power provided varies over time, with significant differences between idle and busy system times. Current computing power measurement methods fail to reflect this change over time, leading to inaccurate measurement results and excessive resource consumption.

[0006] Single-time testing: Measurement results are inaccurate, depending on the system's busyness and over-allocation policy at the time of testing, and they also fail to reflect changes over time. Periodic testing: Measurement results are inaccurate and consume excessive computing resources. For example, if testing is performed once a day in the morning, but the system is idle in the morning and busy in the afternoon, the measurement results will not reflect the busy period. Changing the test to hourly and taking the average still fails to reflect changes over time and consumes excessive computing resources.

[0007] Current computing power measurement methods are generally based on the assumption that computing power is continuously stable, which cannot reflect the changes in computing power over time. There are problems such as inaccurate measurement results and excessive resource consumption. Summary of the Invention

[0008] In view of this, the present invention provides an automatically corrected computing power measurement method, system and device, which can output computing power measurement results related to the time dimension and automatically correct the measurement results when the system changes without increasing excessive resource consumption.

[0009] On the one hand, the present invention proposes an automatic correction method for computing power measurement, including:

[0010] Monitor application performance data, analyze unit business volume resource consumption indicators in the time dimension, determine the low time period of the indicator value as idle time, and determine the peak time period of the indicator value as busy time;

[0011] Randomly select one idle time and one busy time every day / week, trigger the measurement task manager to perform a test, and output the system's daily / weekly idle period, busy period, idle time measurement value, and busy time measurement value;

[0012] If the difference in the unit business volume resource consumption index value between the current and next two weeks is greater than the preset warning value, the measurement task manager is re-triggered to perform the test and output a new computing power measurement value.

[0013] Furthermore, the CPU utilization and GPU utilization of computing resources are collected, and the business concurrency data of the application is collected and reported to the monitoring data management module.

[0014] Furthermore, the unit business volume resource consumption indicator includes: CPU consumption per 100 business concurrency and / or the increase in CPU consumption per 100 additional business concurrency and / or GPU consumption per 100 business concurrency and / or the increase in GPU consumption per 100 additional business concurrency.

[0015] Furthermore, it also includes:

[0016] The monitoring agent module collects CPU utilization and GPU utilization of computing resources, collects application business concurrency data, and reports it to the monitoring data management module.

[0017] Furthermore, the method further includes: receiving monitoring data reported by the monitoring Agent module, and saving the monitoring data into a database using the generation time of the monitoring data as an index.

[0018] Furthermore, the resource consumption index per unit business volume can also be obtained from the computing power resource base.

[0019] Furthermore, the unit business volume resource consumption indicator in the calculation time dimension of the cycle is saved in the database with the monitoring data generation time as the index.

[0020] Furthermore, the business concurrency data is reported through the interface of the reporting SDK module;

[0021] Or business concurrency data can also be obtained from the application's operation and maintenance system.

[0022] Compared with the prior art, the automatic correction method for computing power measurement in the embodiment of the present invention has the following advantages:

[0023] Output time-dimensional computing power measurement results, including system idle periods, busy periods, idle time measurement values, and busy time measurement values, to more accurately reflect the changes in computing power over time; by monitoring application performance data to determine system changes, automatically trigger re-measurement, and output corrected computing power measurement values, improving the accuracy of measurement results and avoiding excessive resource consumption caused by periodic testing.

[0024] On the other hand, the present invention also provides an automatically corrected arithmetic power measurement system, which applies the above-mentioned automatically corrected arithmetic power measurement method, including:

[0025] The computing resource base, the IAAS layer of the data center, manages physical machine clusters and provides virtual machines and containers of various specifications based on virtualization technology;

[0026] Test computing resources, creating virtual machines or containers of specified specifications on demand for computing power measurement testing tasks;

[0027] Application computing resources, which are virtual machines or containers of specified specifications for long-term use, and in which applications run;

[0028] Applications, which are applications that undertake actual business, including websites, databases, and game service programs;

[0029] Benchmark programs, typical applications, including: database and web server programs;

[0030] The measurement task manager is used to manage the execution process of computing power measurement tasks, including creating computing power resources to be tested, deploying benchmark programs, executing tests, and collecting test results.

[0031] The metric scheduler calculates and monitors the CPU consumption per unit of business volume over time to identify idle and busy periods, as well as significant changes in system performance. It generates computing power measurement tasks based on policies and calls the computing power measurement task manager to execute the tasks.

[0032] The data manager receives performance data reported by the monitoring agent and saves it to the time series database;

[0033] The database supports the time dimension and supports query and analysis statements based on time conditions.

[0034] Monitoring Agent, collects performance data of computing resources and reports it to the monitoring data manager;

[0035] Upload SDK to provide applications with an interface for uploading business concurrency data.

[0036] On the other hand, the present invention also provides an automatic correction arithmetic power measurement device, which, when in operation, executes the above-mentioned automatic correction arithmetic power measurement method.

[0037] It is understandable that the above-mentioned automatic correction of the calculation power measurement system and device have the same beneficial effects as the automatic correction of the calculation power measurement method, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0039] FIG1 is a schematic diagram of the structure of an automatic correction calculation power measurement system according to an embodiment of the present invention;

[0040] 2 is a statistical diagram of CPU consumption indicators per unit of business volume - day according to an embodiment of the present invention;

[0041] 3 is a statistical diagram of CPU consumption indicators per unit of business volume - week embodiment of the present invention;

[0042] FIG4 is a comparison chart of CPU consumption indicators per unit of business volume according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] In the description of this application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0045] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout this application, unless otherwise specified, "plurality" means two or more.

[0046] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0047] The preferred embodiment of the present invention provides a method and apparatus for automatically correcting hashrate measurement, and sets a hashrate measurement scheduler module. The main functions of the module include:

[0048] A. Identification of busy and off-peak hours: Monitor application performance data and analyze resource consumption indicators per unit of business volume over time (e.g., CPU consumption per 100 concurrent businesses, or the increase in CPU consumption per 100 concurrent businesses, GPU consumption per 100 concurrent businesses, or the increase in GPU consumption per 100 concurrent businesses). The off-peak hours are defined as the time period when the indicator values ​​are low, and the busy hours are defined as the time period when the indicator values ​​are high.

[0049] B. Time dimension computing power measurement: Select one idle time and one busy time of the day / week, trigger the measurement task manager to perform a test respectively, and output the system's daily / weekly idle period, busy period, idle time measurement value, and busy time measurement value.

[0050] C. Automatically correct computing power metrics: Continuously monitor application performance data. When significant changes are found in the resource consumption indicator values ​​per unit of business volume during the same period of the previous and next two weeks (e.g., a difference greater than 30%), re-trigger the measurement task manager for testing and output a new computing power metric value.

[0051] 2. Added data manager module, the main functions include:

[0052] Receive monitoring data reported by the monitoring agent and save it to the database with the generation time of the monitoring data as the index.

[0053] The resource consumption index of unit business volume in the time dimension is calculated periodically (such as every 5 minutes) and saved in the database with the time when the monitoring data is generated as the index.

[0054] 3. Add a monitoring agent module to collect CPU utilization and GPU utilization of computing resources (VM / container), collect application business concurrency data, and report it to the monitoring data management module.

[0055] 4. Add a reporting SDK module to provide applications with an interface for reporting business concurrency data.

[0056] 1-4 , some examples and measurement data of the present invention are listed below to illustrate the significant advancements of the present invention:

[0057] Example 1: Collecting application performance data

[0058] 1. The application calls the SDK interface to report real-time business concurrency data, such as 100 concurrent transactions per second at 10:10:10 on April 16, 2023.

[0059] 2. The monitoring agent receives reported business concurrency data;

[0060] 3. The monitoring agent collects real-time performance data of used computing resources, including CPU utilization and GPU utilization. For example, at 2023-04-16 10:10:10, CPU utilization was 80%, and at 2023-04-16 10:10:10, GPU utilization was 28%.

[0061] 4. The monitoring agent reports real-time performance data and real-time business concurrency data of computing resources;

[0062] 5. The data manager receives real-time performance data and real-time business concurrency data of computing resources and saves the data to the time series database according to the time dimension, as shown in Table 1.

[0063] Table 1

[0064] Example 2: Generating time-based computing power measurement results

[0065] 1. Calculate the resource consumption indicators per unit of business volume in the time dimension (such as CPU consumption per 100 concurrent businesses, or the increase in CPU consumption per 100 additional concurrent businesses, GPU consumption per 100 concurrent businesses, or the increase in GPU consumption per 100 additional concurrent businesses);

[0066] 2. Count the peak and valley time periods of the average unit business volume resource consumption index every hour of the day, and use them as the high-probability busy and high-probability idle periods of the day. As shown in Figure 2, 1:00-14:00 is the idle period, and 20:00-23:00 is the busy period.

[0067] 3. Count the peak and valley periods of the average unit business volume resource consumption indicator every hour of the week, and use these as the most likely busy and idle periods of the week. As shown in Figure 3, Tuesday 1:00-14:00 is the idle period, and Saturday 20:00-23:00 is the busy period.

[0068] 4. Call the measurement task manager to run a test during the daily off-peak and busy hours. Call the measurement task manager to run a test during the weekly off-peak and busy hours.

[0069] 5. Output the system's daily high-probability idle time, high-probability busy time, idle time metrics, and busy time metrics. Output the system's weekly high-probability idle time, high-probability busy time, idle time metrics, and busy time metrics.

[0070] As you can see, by calculating resource consumption metrics per unit of business volume over time, such as CPU consumption per 100 concurrent businesses or the increase in CPU consumption per 100 concurrent businesses, we can more accurately measure the system's processing capacity over different time periods, enabling better understanding and monitoring of system performance. By calculating peak and valley periods for the average per unit of business volume resource consumption metrics on a daily and weekly basis, we can identify busy and idle periods within the system, which is crucial for optimizing resource allocation and improving system efficiency and responsiveness. Based on daily and weekly per unit of business volume resource consumption metrics, we can promptly identify changes in system resource requirements over different time periods, allowing us to adjust resource allocation to meet system needs and improve system performance and responsiveness. By running the measurement task manager during daily and busy hours, as well as weekly and off-peak hours, we can obtain more accurate and reliable measurement results, enabling a better understanding and monitoring of system performance and providing more precise decision support for optimizing resource allocation. Outputting system data such as the daily high-probability idle period, high-probability busy period, idle time measurement value, busy time measurement value, as well as the weekly high-probability idle period, high-probability busy period, idle time measurement value, busy time measurement value, etc., can provide powerful decision-making support for system administrators and decision makers, helping them better understand the system's operating status, optimize resource allocation, and improve system performance.

[0071] Example 3: Automatically correcting the calculation result

[0072] 1. Calculate the average resource consumption index per unit of business volume for each hour of the day and compare it with the statistical data for the same period last week. Periods with a statistical change of more than 30% are considered periods of significant change. If there are more than 12 periods of significant change, it is considered that the system performance has changed significantly, as shown in Figure 4.

[0073] In this embodiment, if the number of significant change periods exceeds 12, it is considered that the system performance has changed significantly. However, it should be noted that this number is a preset warning value and can be adjusted based on experience and actual conditions;

[0074] 2. Recalculate the daily and weekly idle and busy hours;

[0075] 3. Re-invoke the measurement task manager and run a test during the daily off-peak and busy hours. Re-invoke the measurement task manager and run a test during the weekly off-peak and busy hours.

[0076] 4. Output new measurement results, including the daily high-probability idle period, high-probability busy period, idle time measurement value, and busy time measurement value, including the weekly high-probability idle period, high-probability busy period, idle time measurement value, and busy time measurement value.

[0077] It should be noted that regarding monitoring system performance, this embodiment compares the average value of the resource consumption indicator per unit of business volume with the data from the same time period last week to identify periods of significant change, thereby monitoring whether system performance has undergone significant changes. If the number of periods of significant change exceeds a preset warning value (e.g., 12), system performance is considered to have undergone significant changes.

[0078] Adjusting resource allocation: After identifying significant changes in system performance, this embodiment recalculates the idle and busy hours by day and week, which helps to more accurately identify the busy and idle periods of the system in a day or week, thereby more effectively adjusting resource allocation and improving resource utilization.

[0079] Regarding automatic correction of measurement results: After recalculating idle and busy hours, this embodiment re-invokes the measurement task manager for testing during both daily and weekly idle and busy hours, thereby obtaining more accurate measurement results. This automatic correction method can promptly detect and correct potential measurement errors, improving measurement accuracy and reliability.

[0080] The Measurement Task Manager primarily measures and monitors system performance. It calculates the average resource consumption per unit of business volume and compares it with data from the same time period last week to identify periods of significant change. Furthermore, the Measurement Task Manager recalculates daily and weekly idle and busy hours to more accurately identify the system's busy and idle periods, enabling more effective resource allocation adjustments and improving resource utilization. To output new measurement results, the Measurement Task Manager re-runs tests based on daily and weekly idle and busy hours to obtain more accurate measurement results. These new measurement results, including daily and weekly high-probability idle and high-probability busy hours, idle and busy hour metrics, can provide powerful decision support for system administrators and decision makers, helping them better understand system health, optimize resource allocation, and improve system performance.

[0081] In summary, the embodiments of the present invention provide a method, system, and device for automatically correcting computing power measurement. Upon practical application, the method, system, and device monitor and collect the actual operating performance data of applications, identify the system's idle and busy times, select the idle and busy times for testing, and output the system's high-probability idle period, high-probability busy period, idle-time computing power measurement values, and busy-time computing power measurement values. When significant changes are detected in the actual operating performance data of an application, the method determines that the system's virtualization and over-allocation policies have changed, retests, and outputs new computing power measurement values.

[0082] The above is only an embodiment of the present invention, but it cannot limit the scope of the present invention. Any structural changes made according to the present invention should be deemed to fall within the scope of protection of the present invention and be subject to restrictions as long as they do not lose the essence of the present invention.

[0083] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not to be regarded as improper limitations of the present invention.

[0084] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0085] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

[0087] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An automatic correction method for measuring computing power, characterized in that: include: Monitor the performance data of the application, analyze the resource consumption index of the unit business volume in the time dimension, determine the low time period of the index value as the idle time, and determine the peak time period of the index value as the busy time; Randomly select one idle time and one busy time every day / week, trigger the measurement task manager to perform a test respectively, and output the idle time period, busy time period, idle time measurement value and busy time measurement value of the system every day / week; If the difference in the resource consumption index value per unit business volume in the same period of the current and next two weeks is greater than the preset warning value, the measurement task manager is re-triggered for testing and a new computing power measurement value is output.

2. The automatic correction calculation power measurement method according to claim 1 is characterized in that: Collect the CPU utilization and GPU utilization of computing resources, collect the business concurrency data of applications, and report them to the monitoring data management module.

3. The automatic correction calculation power measurement method according to claim 1 is characterized in that: The resource consumption index per unit business volume includes: CPU consumption per 100 business concurrency and / or the increase in CPU consumption per 100 additional business concurrency and / or GPU consumption per 100 business concurrency and / or the increase in GPU consumption per 100 additional business concurrency.

4. The automatic correction calculation power measurement method according to claim 1 is characterized in that: Also includes: The monitoring Agent module collects the CPU utilization and GPU utilization of computing resources, collects the application's business concurrency data, and reports it to the monitoring data management module.

5. The automatic correction calculation power measurement method according to claim 4 is characterized in that: Also includes: Receive the monitoring data reported by the monitoring agent module and save it to the database with the generation time of the monitoring data as the index.

6. The automatic correction calculation power measurement method according to claim 4 is characterized in that: The resource consumption index per unit business volume can also be obtained from the computing resource base.

7. The automatic correction calculation power measurement method according to claim 5, characterized in that: The unit business volume resource consumption indicator in the calculation time dimension of the cycle is saved in the database with the monitoring data generation time as the index.

8. The automatic correction calculation power measurement method according to claim 5, characterized in that: The business concurrency data is reported through the interface of the reporting SDK module; Or the business concurrency data is obtained from the application's operation and maintenance system.

9. An automatic correction arithmetic power measurement system, applied to the automatic correction arithmetic power measurement method according to any one of claims 1 to 8, characterized in that: include: The computing resource base, the IAAS layer of the data center, manages physical machine clusters and provides virtual machines and containers of various specifications based on virtualization technology; Test computing resources, virtual machines or containers of specified specifications created on demand, for computing power measurement test tasks; Application computing resources are virtual machines or containers of specified specifications used for long-term use, in which applications are running; Application programs, which are used to perform actual business, include websites, databases, and game service programs; Benchmark programs, typical applications, including: database and Web server programs; The measurement task manager is used to manage the execution process of computing power measurement tasks, including the steps of creating computing power resources to be tested, deploying benchmark programs, executing tests, and collecting test results; The metric scheduler calculates and monitors the CPU consumption index of unit business volume in the time dimension, which is used to identify the idle and busy times of the system and major changes in system performance, generates computing power measurement tasks according to the strategy, and calls the computing power measurement task manager to execute the tasks; Data manager, receives performance data reported by monitoring agent and saves it to time series database; The database is a database that supports the time dimension and supports query and analysis statements based on time conditions; Monitoring Agent, collects performance data of computing resources and reports it to the monitoring data manager; Upload SDK to provide an interface for applications to upload business concurrency data.

10. An automatic correction arithmetic power measurement device, characterized in that: When it is running, it executes the automatically corrected computing power measurement method as described in any one of claims 1-8.

Images