System performance evaluation methods and devices, storage media and electronic devices

Abstract

This invention provides a system performance evaluation method, apparatus, storage medium, and electronic device, comprising: determining a data sampling time and collecting various data requests from the system within the data sampling time; determining the transaction type to which each data request belongs; determining the request proportion of each transaction type based on the number of each data request for each transaction type, and applying the request proportions to determine the system's response time coefficient; applying the response time coefficient to determine the system's online processing evaluation value for each transaction type; calculating the online processing fluctuation range based on each online processing evaluation value, and using the online processing fluctuation range to evaluate the system's performance. This invention introduces the concept of a response time coefficient, which can accurately calculate the system's online processing fluctuation range. The entire process reduces manual intervention, decreases the workload of staff, and reduces the time cost of evaluation, effectively improving the efficiency of system performance evaluation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for evaluating system performance, a storage medium, and an electronic device. Background Technology

[0002] Big data systems are applied in all aspects of life, and enterprises in various fields use them to facilitate business operations and work. Data processing is the core function of a big data system. Currently, data processing is usually done online, requiring collaboration with other systems. The entire process involves large amounts of data and high complexity.

[0003] Processing large volumes and highly complex data puts a heavy strain on system performance. To ensure stable operation, system performance needs to be evaluated so that maintenance personnel can perform timely maintenance. Currently, the common method for evaluating system performance is for maintenance personnel to process various performance data to assess the system's performance. This method is time-consuming and inefficient. Summary of the Invention

[0004] In view of this, the present invention provides a system performance evaluation method and apparatus, storage medium and electronic device. The present invention reduces the involvement of personnel, shortens the time required to evaluate system performance, and improves evaluation efficiency during the system performance evaluation process.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention discloses a method for evaluating system performance, comprising:

[0007] Determine the data sampling time and collect each data request from the system within the data sampling time.

[0008] Determine the transaction type to which each of the data requests belongs;

[0009] Based on the number of each data request for each transaction type, determine the request percentage for each transaction type, and apply the respective request percentages to determine the response time coefficient of the system.

[0010] The response time coefficient is applied to determine the online processing evaluation value of the system for each transaction type;

[0011] The online processing evaluation values ​​are calculated to obtain the online processing fluctuation range, and the online processing fluctuation range is used to evaluate the performance of the system.

[0012] Optionally, in the above method, determining the system's response time coefficient based on the proportion of each request applied includes:

[0013] Get the complexity of each transaction type;

[0014] The complexity and request ratio of each transaction type are calculated to obtain the response coefficient for each transaction type;

[0015] The response coefficients are summed to obtain the system's response time coefficients.

[0016] Optionally, in the above method, applying the response time coefficient to determine the online processing evaluation value of the system for each transaction type includes:

[0017] Determine the response time for each data request;

[0018] For each transaction type, the arithmetic mean of the query rate per second for that transaction type is determined based on the response time of each data request belonging to that transaction type;

[0019] The response time coefficient, the arithmetic mean of each transaction type, and the preset response time index value are processed to obtain the online processing evaluation value for each transaction type.

[0020] Optionally, in the above method, the step of calculating the online processing fluctuation range for each of the online processing evaluation values ​​includes:

[0021] The average value of each of the online processing evaluation values ​​is obtained by averaging the online processing evaluation values.

[0022] Based on a preset calculation method, each of the online processing evaluation values ​​and the average online processing evaluation value are processed to obtain the online processing fluctuation range.

[0023] Optionally, in the above method, evaluating the system performance using the online processing fluctuation range includes:

[0024] Among the preset performance dimensions, the performance dimension to which the online processing fluctuation range belongs is marked as the first target performance dimension, and the remaining performance indicators are all marked as the second target performance dimension.

[0025] The evaluation indicators in the first target performance dimension that correspond to the online processing fluctuation range are marked, and the index values ​​of each unmarked evaluation indicator in the first target performance dimension are obtained.

[0026] Based on the online processing fluctuation range and the values ​​of each indicator, an evaluation result of the system in the first target performance dimension is generated.

[0027] The above methods may also include:

[0028] Determine the evaluation metrics for each of the second target performance dimensions;

[0029] Obtain the value of each evaluation metric for each of the second target performance dimensions;

[0030] For each of the second target performance dimensions, an evaluation result of the system in the second target performance dimension is generated based on the index values ​​of each evaluation index of the second target performance dimension.

[0031] A second aspect of the present invention discloses a system performance evaluation apparatus, comprising:

[0032] The acquisition unit is used to determine the data sampling time and acquire various data requests from the system within the data sampling time.

[0033] The first determining unit is used to determine the transaction type to which each data request belongs;

[0034] The second determining unit is used to determine the request proportion of each transaction type based on the number of each data request for each transaction type, and to determine the response time coefficient of the system by applying each of the request proportions.

[0035] The third determining unit is used to apply the response time coefficient to determine the online processing evaluation value of the system for each type of transaction;

[0036] An evaluation unit is used to calculate each of the online processing evaluation values ​​to obtain the online processing fluctuation range, and to use the online processing fluctuation range to evaluate the performance of the system.

[0037] Optionally, the second determining unit in the aforementioned apparatus includes:

[0038] Get the sub-unit, used to obtain the complexity of each transaction type;

[0039] The calculation subunit is used to calculate the complexity and request ratio of each transaction type to obtain the response coefficient of each transaction type;

[0040] The summation processing subunit is used to sum the various response coefficients to obtain the response time coefficient of the system.

[0041] Optionally, the third determining unit in the aforementioned apparatus includes:

[0042] The first determining subunit is used to determine the response time for each data request;

[0043] The second determining subunit is used to determine the arithmetic average of the query rate per second for each transaction type based on the response time of each data request belonging to that transaction type.

[0044] The first obtaining subunit is used to process the response time coefficient, the arithmetic mean of each transaction type, and the preset response time index value to obtain the online processing evaluation value of each transaction type.

[0045] Optionally, the evaluation unit in the aforementioned apparatus includes:

[0046] The averaging subunit is used to average the online processing evaluation values ​​to obtain the average online processing evaluation value.

[0047] The second obtaining subunit is used to process each of the online processing evaluation values ​​and the average value of the online processing evaluation based on a preset calculation method to obtain the online processing fluctuation range.

[0048] Optionally, the evaluation unit in the aforementioned apparatus includes:

[0049] The first marking subunit is used to mark the performance dimension to which the online processing fluctuation range belongs as the first target performance dimension among the preset performance dimensions, and to mark the remaining performance indicators as the second target performance dimension.

[0050] The second marking subunit is used to mark the evaluation indicators in the first target performance dimension that correspond to the online processing fluctuation range, and to obtain the index values ​​of each unmarked evaluation indicator in the first target performance dimension.

[0051] A generation subunit is used to generate an evaluation result of the system in the first target performance dimension based on the online processing fluctuation range and each of the indicator values.

[0052] The aforementioned apparatus may optionally further include:

[0053] The fourth determining unit is used to determine the evaluation indicators for each of the second target performance dimensions;

[0054] The acquisition unit is used to acquire the index value of each evaluation index for each of the second target performance dimensions;

[0055] The generation unit is used to generate an evaluation result of the system in the second target performance dimension based on the index values ​​of each evaluation index of the second target performance dimension for each second target performance dimension.

[0056] A third aspect of the present invention discloses a storage medium comprising stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides executes the system performance evaluation method described above.

[0057] The fourth aspect of the present invention discloses an electronic device, including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors using the system performance evaluation method described above.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] This invention provides a system performance evaluation method, apparatus, storage medium, and electronic device. The method includes: determining a data sampling time and collecting various data requests from the system within the data sampling time; determining the transaction type of each data request; determining the request percentage of each transaction type based on the number of data requests for each transaction type, and applying the request percentages to determine the system's response time coefficient; applying the response time coefficient to determine the system's online processing evaluation value for each transaction type; calculating the online processing fluctuation range based on the online processing evaluation values, and using the online processing fluctuation range to evaluate the system's performance. This invention introduces the concept of a response time coefficient, which can accurately calculate the system's online processing fluctuation range. The entire process reduces manual intervention, the workload of staff, and the time cost of evaluation, effectively improving the efficiency of system performance evaluation. Furthermore, using the online processing fluctuation range to evaluate system performance can improve the accuracy of system performance evaluation. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a system performance evaluation method provided in an embodiment of the present invention;

[0062] Figure 2 A flowchart of a method for determining the system response time coefficient based on the proportion of each request in an application, as provided in an embodiment of the present invention;

[0063] Figure 3 The flowchart of the method for determining the online processing evaluation value of each transaction type by means of the application response time coefficient provided in the embodiments of the present invention is as follows:

[0064] Figure 4 A flowchart of a method for evaluating system performance using online processing fluctuation range provided in an embodiment of the present invention;

[0065] Figure 5 A schematic diagram of the structure of a system performance evaluation device provided in an embodiment of the present invention;

[0066] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] Terminology Explanation:

[0070] Online Transaction Processing (OLTP), also known as Real-Time System, supports fast transaction response and high concurrency. It is a database application that reflects the current operational status of an enterprise and completes the daily tasks included in enterprise management. It generally does not involve complex query and analysis processing.

[0071] Online Analytical Processing (OLAP), also known as Decision Support System (DSS), is a major application of data warehouse systems. It is specifically designed to support complex analytical operations, with a focus on decision support for decision-makers and senior management.

[0072] Big data: Big data is characterized by its massive size, diverse sources, rapid generation, and variability, and is difficult to process effectively using traditional data architectures.

[0073] Performance metrics: A set of parameters that can be used to evaluate the performance of an application system.

[0074] Processing capacity: The time taken and the number of transactions completed are calculated from the time the client sends a request to the server until the server's response is received.

[0075] Response time coefficient: An indicator that reflects the characteristics of big data, defined based on the complexity of online analytical processing.

[0076] Fluctuation range: refers to the distance between peaks and troughs formed by repeated fluctuations in processing capacity within a certain period. It is calculated as the ratio of standard deviation to mean.

[0077] Traditional relational database applications utilize online transaction processing (OLTP), which involves basic, routine transaction processing, recording immediate add, delete, update, and query operations. These transactions are composed of short, atomic transactions; for example, depositing or withdrawing money at a bank constitutes a transaction. A key performance requirement for online transaction processing systems is performance metrics, specifically real-time response time (Response Time), which is the time required for the server to respond to a request after the user submits data at the terminal. Another key metric is processing capacity, specifically the number of transactions processed per unit of time, TPS (transactions per second).

[0078] Big data processing, a core application of data warehousing, is an online analytical processing (OLAP) transaction. It supports complex analytical operations and is characterized by large data volumes, requiring users to statistically analyze massive amounts of data to obtain the desired information. Real-time requirements are not high; it focuses on decision support and typically involves dynamic queries, providing intuitive and easy-to-understand results. A typical application is a complex dynamic reporting system. The long and unstable online transaction processing time of this system can lead to performance instability, affecting its data processing capabilities. Therefore, it is necessary to evaluate system performance and perform timely maintenance.

[0079] Traditional methods for evaluating system performance require significant investment and are inefficient. Furthermore, traditional evaluation methods struggle to determine a system's online analytical processing capabilities and fluctuation range, which is a problem that urgently needs to be addressed.

[0080] This invention can be used in a wide range of general-purpose or special-purpose computing environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, and distributed computing environments including any of the above devices. This invention can be applied to online systems or data systems capable of processing various types of data, both of which are constructed using computer terminals or data processing equipment.

[0081] Reference Figure 1 The following is a flowchart of a system performance evaluation method provided by an embodiment of the present invention, which is described in detail below:

[0082] S101. Determine the data sampling time and collect each data request from the system within the data sampling time.

[0083] Receive system evaluation instructions, parse the system evaluation instructions, and obtain the data sampling time in the system evaluation instructions.

[0084] System evaluation instructions are instructions generated by staff when they need to evaluate the performance of the system. System evaluation instructions include, but are not limited to, information such as data sampling time, system identification, and instruction generation time.

[0085] Staff can set the data sampling time according to actual needs. For example, the data sampling time can be the previous 3 days, the previous 5 days, or a specific date.

[0086] Collect all data requests from the system within the data sampling period from the system's historical database. It should be noted that the data requests are those that the system has processed within the data sampling period.

[0087] S102. Determine the transaction type to which each data request belongs.

[0088] It should be noted that the system can handle requests of various transaction types, including but not limited to online processing types and business processing types.

[0089] Each data request has a corresponding transaction type.

[0090] S103. Based on the number of each data request for each transaction type, determine the request percentage for each transaction type, and apply the request percentages to determine the system's response time coefficient.

[0091] Furthermore, the request percentage for a transaction type is the proportion of the number of data requests of that transaction type in the total number of data requests; for example, the total number of each type of data request is counted; the number of data requests for each transaction type is determined; for each transaction type, the request percentage for that transaction type is obtained by dividing the number of data requests for that transaction type by the total number of data requests.

[0092] It should be noted that the request percentage of a transaction type can also be called the online analytical processing (OLAP) transaction percentage, which is the percentage of the number of requests for that transaction in the total number of requests in the entire scope of the analysis to be examined.

[0093] By using the proportion of each request, the system's response time coefficient within that data sampling period is determined. It should be noted that the response time system is related to the number of transaction types involved within that data sampling period.

[0094] S104. Apply the response time coefficient to determine the system's online processing evaluation value for each transaction type.

[0095] The online processing evaluation value is an assessment of the system's online processing performance when handling requests of this type of transaction. This value can then be used to calculate the system's online processing fluctuation range.

[0096] S105. Calculate the online processing evaluation values ​​to obtain the online processing fluctuation range, and use the online processing fluctuation range to evaluate the system performance.

[0097] The method provided in this invention involves determining a data sampling time and collecting various data requests from the system within that time; determining the transaction type of each data request; determining the request percentage of each transaction type based on the number of data requests for each transaction type, and applying the request percentages to determine the system's response time coefficient; applying the response time coefficient to determine the system's online processing evaluation value for each transaction type; calculating the online processing fluctuation range based on the online processing evaluation values; and using the online processing fluctuation range to evaluate the system's performance. This invention introduces the concept of a response time coefficient, which allows for precise calculation of the system's online processing fluctuation range. The entire process reduces manual intervention, decreases the workload of staff, and reduces the time cost of evaluation, effectively improving the efficiency of system performance evaluation. Furthermore, using the online processing fluctuation range to evaluate system performance improves the accuracy of the evaluation.

[0098] Reference Figure 2 The flowchart of the method for determining the system response time coefficient based on the proportion of each request provided in the embodiments of the present invention is described in detail below:

[0099] S201, Obtain the complexity of each transaction type.

[0100] Based on a pre-defined transaction complexity data table, the complexity of each transaction type is determined.

[0101] The transaction complexity data table contains the complexity of each type of transaction that the system can handle. It should be noted that the complexity of a transaction type can be understood as the complexity of the processing logic of the transaction type. The complexity can be set according to various complexity indicators such as the amount of data involved in the processing logic of that transaction type, the number of database tables, data dimensions, the number of data aggregations, and complex derived data.

[0102] Data volume can be categorized into hundreds of thousands, millions, tens of millions, and hundreds of millions or more; the number of database tables can be categorized into 1, 3, 5, and more than 5; data dimensions can be categorized into two-dimensional, three-dimensional, four-dimensional, five-dimensional, and more than 5; the number of data aggregations can be categorized into 1, 2, 3, 4, and more than 4; complex derived data can be categorized as yes or no (such as period comparison, share calculation, equal mean square deviation, etc.); furthermore, different values ​​of complexity indicators correspond to different complexities, as shown in Table 1, which illustrates the complexity of various complexity indicators at different values.

[0103] It should be noted that the data volume, number of database tables, data dimensions, number of summaries, and derived data in Table 1 are all complexity indicators. The level definition indicates the specific value of the complexity indicator.

[0104] Furthermore, the complexity of a transaction type is the sum of the complexity corresponding to the amount of data, the complexity corresponding to the number of database tables, the complexity corresponding to the data dimensions, the complexity corresponding to the number of aggregations, and the complexity corresponding to the derived data. For example, if the amount of data for transaction type 1 is in the hundreds of thousands, the number of database tables is 1, the data dimension is two-dimensional, the number of aggregations is 1, and there is no derived data, then the complexity of transaction type 1 is 5.

[0105]

[0106] Table 1

[0107] S202. Calculate the complexity and request ratio of each transaction type to obtain the response coefficient of each transaction type.

[0108] It should be noted that for each transaction type, the response coefficient for that transaction type can be obtained by multiplying the complexity of that transaction type by the proportion of requests; for example, the response coefficient for transaction type 1 is obtained by multiplying the proportion of requests 1 by the complexity 1.

[0109] S203. Sum the various response coefficients to obtain the system response time coefficients.

[0110] For example, the response time coefficient = response coefficient 1 + response coefficient 2 + ... + response coefficient n = request percentage 1 * complexity 1 + request percentage 2 * complexity 2 + ... + request percentage n * complexity n; thus, the system's response time coefficient can be obtained.

[0111] In the method provided by this invention, the system response time coefficient can be calculated by applying the complexity and request ratio of each transaction type. The response time coefficient is determined based on the various transaction types involved in the system during the data sampling time. The response time coefficient is dynamic, thereby allowing for dynamic evaluation of the system performance.

[0112] Reference Figure 3 The flowchart of the method for determining the online processing evaluation value of each transaction type by means of the application response time coefficient provided in the embodiments of the present invention is described in detail below:

[0113] S301. Determine the response time for each data request.

[0114] Response time is the total time taken from the start of executing the data request to receiving the response data.

[0115] S302. For each transaction type, based on the response time of each data request belonging to that transaction type, determine the arithmetic mean of the query rate per second for that transaction type.

[0116] It should be noted that queries per second (QPS) can be used to represent the query rate. When calculating the arithmetic mean of the QPS for a transaction type, it is necessary to first calculate the QPS for each data request of the transaction type. Furthermore, the QPS of the various data requests of the transaction type can be formed into a set q(q1,q2,...,q...). n ), where q1 is the QPS of data request 1 of transaction type, q2 is the QPS of data request 2 of transaction type, and so on, which will not be elaborated here.

[0117] Furthermore, for each data request QPS of each transaction type, the requests are grouped according to the response time coefficient and the response time metric value, and then the arithmetic mean of the query rate per second for that transaction type is calculated. It should be noted that the response time metric value is a value set in advance for the transaction type, and the response time metric value is different for different transaction types.

[0118] Furthermore, the QPS of various transaction types can be grouped into a set Q(Q1, Q2, ..., Q...). n), where Q1 represents the arithmetic mean of transaction type 1, Q2 represents the arithmetic mean of transaction type 2, and so on, which will not be elaborated here.

[0119] S303. Process the response time coefficient, the arithmetic mean of each transaction type, and the preset response time index value to obtain the online processing evaluation value for each transaction type.

[0120] For each transaction type, the response time coefficient, the arithmetic mean of that transaction type, and the response time metric value are processed to obtain the online processing evaluation value for that transaction type. For example, the online processing evaluation value = arithmetic mean / (response time coefficient * response time metric value). Further, the online processing evaluation value can be a value characterizing the system's online analytical processing capability (APT, Analytical Per Time). APT is the ability to complete a certain number of online analytical processing operations within a certain time. APT is related to QPS, response time system, and response time metric value. Preferably, the online processing evaluation values ​​for various transaction types can be grouped into an array A, where A = number of concurrent requests / (response time system * response time metric value) = arithmetic mean of QPS for each group / (response time coefficient * response time metric value) = Q / (response time coefficient * response time metric value), where the number of concurrent requests can be the total number of data requests. For example, A(A1, A2, ..., A3) = ... n ); where A1 represents the online processing evaluation value of transaction type 1, A2 represents the online processing evaluation value of transaction type 2, and so on, which will not be elaborated here.

[0121] The response time metric is an expected value set based on the complexity of the transaction type, ensuring that requests belonging to that transaction type can receive a response within a given range.

[0122] Referring to Table 2, this is a summary table of the complexity, request ratio, and preset response time index values ​​for each type of transaction provided by the present invention.

[0123] Transaction type Complexity Request percentage Response time metric 1 Complexity 1 1% Indicator value 1 2 Complexity 2 2% Indicator value 2 ... … … … n Complexity n percentage n index value n

[0124] Table 2

[0125] Furthermore, after determining the online processing evaluation value for each transaction type, the online processing fluctuation range can be calculated using this value. It should be noted that the online processing fluctuation range is the distance from the mean caused by repeated fluctuations in the system's online processing and analysis capabilities over a certain period. Furthermore, the online processing fluctuation range is related to APT (Advanced Performance Testing), and the calculation can be performed as APT standard deviation / APT mean.

[0126] In determining the fluctuation range of online processing, the average value of each online processing evaluation is first calculated to obtain the average value of the online processing evaluation. Based on a preset calculation method, the online processing evaluation values ​​and the average value of the online processing evaluation are processed to obtain the fluctuation range of online processing. Furthermore, the fluctuation range of online processing can be the discrete coefficient of the above array A. Furthermore, the formula applied by the preset calculation method is: Where c represents the online processing fluctuation range. A1 is the online processing fluctuation assessment value for transaction type 1, A2 is the online processing fluctuation assessment value for transaction type 2, and so on. Further details will not be provided here.

[0127] In the process of evaluating system performance, this invention introduces the concept of response time coefficient. Based on this, the system's online analysis capability and fluctuation range are calculated. The calculation results can be used to quickly and accurately evaluate the system's performance.

[0128] Reference Figure 4 The flowchart below illustrates a method for evaluating system performance using online processing fluctuation range, as provided in an embodiment of the present invention.

[0129] S401. Among the preset performance dimensions, mark the performance dimension to which the online processing fluctuation range belongs as the first target performance dimension, and mark the remaining performance indicators as the second target performance dimension.

[0130] The preset performance dimensions include, but are not limited to, single transaction dimension, mixed transaction processing capability dimension, and stability dimension. Optionally, performance dimensions can also be referred to as performance indicators.

[0131] Furthermore, the performance dimension to which the online processing fluctuation range belongs is the stability dimension, and thus the stability dimension is determined as the first target performance dimension, while the single transaction dimension and the mixed transaction processing capability dimension are both determined as the second target performance dimensions.

[0132] S402. Mark the evaluation indicators in the first target performance dimension that correspond to the online processing fluctuation range, and obtain the index values ​​of each unmarked evaluation indicator in the first target performance dimension.

[0133] Each performance dimension has multiple evaluation metrics. For example, the evaluation metrics corresponding to the stability dimension include, but are not limited to, the total number of transactions within the system's stable runtime, the transaction response time within the stable runtime, the transaction response time compliance rate within the stable runtime, the fluctuation range of stable transaction processing capacity, and resource utilization - memory usage. Furthermore, the fluctuation range of stable transaction processing capacity here refers to the online processing fluctuation range.

[0134] It should be noted that when obtaining the values ​​of each unidentified and evaluated metric in the first target performance dimension, they can be obtained from the data corresponding to the evaluation metric. For example, resource utilization rate - memory usage can be obtained from the system's memory data, and the total transaction volume during the system's stable operation period can be statistically obtained from the system's stable operation data.

[0135] S403. Based on the fluctuation range of online processing and the values ​​of various indicators, generate the evaluation results of the system in the first target performance dimension.

[0136] It should be noted that each evaluation indicator has a corresponding judgment criterion. For each evaluation indicator, it can be determined whether the evaluation indicator meets the judgment criterion, thereby obtaining the judgment result of the evaluation indicator. Furthermore, the evaluation result of the system in the first target performance dimension can be generated based on each judgment result.

[0137] S404. Determine the evaluation metrics for each second objective performance dimension.

[0138] S405. Obtain the value of each evaluation metric for each second target performance dimension.

[0139] The value of each evaluation metric can be extracted from the data corresponding to the evaluation metric in the system during the acquisition process. For example, the evaluation metric of resource utilization rate - CPU utilization rate in hybrid processing capability can be extracted from the system's CPU usage data.

[0140] S406. For each second target performance dimension, based on the index values ​​of each evaluation index of the second target performance dimension, generate the evaluation result of the system in the second target performance dimension.

[0141] Furthermore, each evaluation indicator in each second target performance dimension has a judgment criterion. For each evaluation indicator in each second target performance dimension, it can be determined whether the indicator value meets the judgment criterion of the evaluation indicator, and a judgment result is generated. For each second target performance dimension, an evaluation result is generated based on the judgment results of each performance dimension.

[0142] Referring to Table 3, this document summarizes the evaluation indicators for each performance dimension provided in the embodiments of the present invention, the definition of the indicator value for each evaluation indicator, and the judgment criteria for the value of each evaluation indicator.

[0143]

[0144] Table 3

[0145] Preferably, the requirements in the table can be the requirements for evaluating the performance of the system. The judgment criteria in the table are that the evaluation indicators can be determined to meet the judgment requirements after the indicator values ​​meet the criteria. Furthermore, when the proportion of evaluation indicators that meet the requirements is less than the preset proportion, an evaluation result of unstable system performance can be generated. When the proportion of evaluation indicators that meet the requirements is greater than or equal to the preset proportion, an evaluation result of stable system performance can be generated.

[0146] Preferably, the stability of the system can also be evaluated using the fluctuation range alone. For example, when the fluctuation range is not within a preset range, an evaluation result of system instability is generated, and when the fluctuation range is within the preset range, an evaluation result of system stability is generated.

[0147] Furthermore, in practical applications, the present invention can be implemented using four modules, as follows:

[0148] Module 1 defines performance requirements. The performance requirements of a big data system are measured by performance metrics (performance dimensions). By defining performance requirements, the performance metrics of the big data system are determined, and the response time system is defined. The content of this module is set in advance.

[0149] Module 2 establishes the relationships between requirement items. Furthermore, the relationships between requirement items can be referenced from the evaluation indicators in Table 3. The content of this module is pre-set.

[0150] Module 3 collects basic data such as the number of concurrent requests and response time; furthermore, it can collect the required data based on the requirements established in Module 2.

[0151] Module 4 calculates QPS, APT, and fluctuation range based on the data collected in Module 3.

[0152] This invention calculates the processing capacity and fluctuation range based on the relationship between response time, response time coefficient, and processing capacity, and obtains the evaluation results of the system's online analytical processing capacity and stability.

[0153] Currently, the online transaction processing in the system relies on traditional relational databases. This aims to allow applications to immediately send raw data to the computing center for processing, writing or updating only the necessary data, and responding with results within a very short time to process individual transactions quickly. Performance requirements are mature, with clear performance metrics for measurement, including response time, processing capacity, and well-defined computational rules.

[0154] For online analytical processing of big data, application processing relies on data warehouses, handling data in the petabyte (P) range. User query requirements are complex, involving not only querying or manipulating one or a few records in a single table, but also data analysis and information synthesis of tens of millions of records across multiple tables. The performance requirements for these queries have both similarities and differences to traditional online transaction processing.

[0155] With the rapid development of database technology and the typical application of big data systems, the performance requirements of online analytical processing (OLAP) have become increasingly prominent. From being ignored or weakened, they have now become a focus of attention for practitioners and management. Performance requirements are urgent, but there is no clear statistical method or rule for judging the stability of OLAP.

[0156] Based on the aforementioned characteristics of online transaction processing (OLTP) and online analytical processing (OLAP), this invention derives the performance requirements of OLAP from the performance requirements of OLTP. Building upon the performance metrics of OLTP, such as response time and processing capacity, it further expands upon these metrics by proposing the concept of a response time coefficient. This coefficient is used to calculate the processing capacity and fluctuation range of OLAP transactions, thereby assessing the operational stability of big data systems.

[0157] and Figure 1 Corresponding to the method shown, the present invention provides a system performance evaluation device for supporting... Figure 1 The method shown is implemented using a device located in a big data system or an online processing system.

[0158] Reference Figure 5 The following is a schematic diagram of the structure of a system performance evaluation device provided in an embodiment of the present invention, and is described in detail below:

[0159] The acquisition unit 501 is used to determine the data sampling time and acquire various data requests of the system within the data sampling time.

[0160] The first determining unit 502 is used to determine the transaction type to which each data request belongs;

[0161] The second determining unit 503 is used to determine the request ratio of each transaction type based on the number of each data request for each transaction type, and to determine the response time coefficient of the system by applying each request ratio.

[0162] The third determining unit 504 is used to apply the response time coefficient to determine the online processing evaluation value of the system for each type of transaction;

[0163] Evaluation unit 505 is used to calculate each of the online processing evaluation values ​​to obtain the online processing fluctuation range, and to use the online processing fluctuation range to evaluate the performance of the system.

[0164] The apparatus provided in this invention determines a data sampling time and collects various data requests from the system within that time. It then determines the transaction type of each data request; based on the number of data requests for each transaction type, it determines the proportion of requests for each transaction type and applies this proportion to determine the system's response time coefficient; applying the response time coefficient, it determines the system's online processing evaluation value for each transaction type; it calculates the online processing fluctuation range for each online processing evaluation value and uses this fluctuation range to evaluate the system's performance. This invention introduces the concept of a response time coefficient, which allows for precise calculation of the system's online processing fluctuation range. The entire process reduces manual intervention, decreases the workload of staff, and reduces the time cost of evaluation, effectively improving the efficiency of system performance evaluation. Furthermore, using the online processing fluctuation range to evaluate system performance improves the accuracy of the evaluation.

[0165] In another embodiment provided by the present invention, the second determining unit 503 of the device may be configured as follows:

[0166] Get the sub-unit, used to obtain the complexity of each transaction type;

[0167] The calculation subunit is used to calculate the complexity and request ratio of each transaction type to obtain the response coefficient of each transaction type;

[0168] The summation processing subunit is used to sum the various response coefficients to obtain the response time coefficient of the system.

[0169] In another embodiment provided by the present invention, the third determining unit 504 of the device may be configured as follows:

[0170] The first determining subunit is used to determine the response time for each data request;

[0171] The second determining subunit is used to determine the arithmetic average of the query rate per second for each transaction type based on the response time of each data request belonging to that transaction type.

[0172] The first obtaining subunit is used to process the response time coefficient, the arithmetic mean of each transaction type, and the preset response time index value to obtain the online processing evaluation value of each transaction type.

[0173] In another embodiment provided by the present invention, the evaluation unit 505 of the device may be configured as follows:

[0174] The averaging subunit is used to average the online processing evaluation values ​​to obtain the average online processing evaluation value.

[0175] The second obtaining subunit is used to process each of the online processing evaluation values ​​and the average value of the online processing evaluation based on a preset calculation method to obtain the online processing fluctuation range.

[0176] In another embodiment provided by the present invention, the evaluation unit 505 of the device may be configured as follows:

[0177] The first marking subunit is used to mark the performance dimension to which the online processing fluctuation range belongs as the first target performance dimension among the preset performance dimensions, and to mark the remaining performance indicators as the second target performance dimension.

[0178] The second marking subunit is used to mark the evaluation indicators in the first target performance dimension that correspond to the online processing fluctuation range, and to obtain the index values ​​of each unmarked evaluation indicator in the first target performance dimension.

[0179] A generation subunit is used to generate an evaluation result of the system in the first target performance dimension based on the online processing fluctuation range and each of the indicator values.

[0180] In another embodiment of the present invention, the device further includes:

[0181] The fourth determining unit is used to determine the evaluation indicators for each of the second target performance dimensions;

[0182] The acquisition unit is used to acquire the index value of each evaluation index for each of the second target performance dimensions;

[0183] The generation unit is used to generate an evaluation result of the system in the second target performance dimension based on the index values ​​of each evaluation index of the second target performance dimension for each second target performance dimension.

[0184] This invention also provides a storage medium that includes stored instructions, wherein the execution of the instructions controls the device containing the storage medium to perform the aforementioned system performance evaluation method.

[0185] This invention also provides an electronic device, the structural schematic of which is shown below. Figure 6 As shown, it specifically includes a memory 601 and one or more instructions 602, wherein one or more instructions 602 are stored in the memory 601 and configured to be executed by one or more processors 603 to perform the above-mentioned system performance evaluation method.

[0186] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of this invention.

[0187] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0188] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0189] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating system performance, characterized in that, include: Determine the data sampling time and collect each data request from the system within the data sampling time. Determine the transaction type to which each of the data requests belongs; Based on the number of each data request for each transaction type, determine the request percentage for each transaction type, and apply the respective request percentages to determine the response time coefficient of the system. The method for determining the system's response time coefficient by calculating the proportion of each request in the application includes: obtaining the complexity of each transaction type; calculating the complexity of each transaction type and the request proportion to obtain the response coefficient of each transaction type; and summing the response coefficients to obtain the system's response time coefficient. The online processing evaluation value of the system for each transaction type is determined by applying the response time coefficient, including: determining the response time of each data request; for each transaction type, determining the arithmetic mean of the query rate per second for that transaction type based on the response time of each data request belonging to that transaction type; and processing the response time coefficient, the arithmetic mean of each transaction type, and a preset response time index value to obtain the online processing evaluation value for each transaction type. The online processing evaluation values ​​are calculated to obtain the online processing fluctuation range, and the online processing fluctuation range is used to evaluate the performance of the system.

2. The method according to claim 1, characterized in that, The calculation of each of the online processing evaluation values ​​to obtain the online processing fluctuation range includes: The average value of each of the online processing evaluation values ​​is obtained by averaging the online processing evaluation values. Based on a preset calculation method, each of the online processing evaluation values ​​and the average online processing evaluation value are processed to obtain the online processing fluctuation range.

3. The method according to claim 1, characterized in that, The evaluation of the system's performance using the online processing fluctuation range includes: Among the preset performance dimensions, the performance dimension to which the online processing fluctuation range belongs is marked as the first target performance dimension, and the remaining performance indicators are all marked as the second target performance dimension. The evaluation indicators in the first target performance dimension that correspond to the online processing fluctuation range are marked, and the index values ​​of each unmarked evaluation indicator in the first target performance dimension are obtained. Based on the online processing fluctuation range and the values ​​of each indicator, an evaluation result of the system in the first target performance dimension is generated.

4. The method according to claim 3, characterized in that, Also includes: Determine the evaluation metrics for each of the second target performance dimensions; Obtain the value of each evaluation metric for each of the second target performance dimensions; For each of the second target performance dimensions, an evaluation result of the system in the second target performance dimension is generated based on the index values ​​of each evaluation index of the second target performance dimension.

5. A system performance evaluation device, characterized in that, include: The acquisition unit is used to determine the data sampling time and acquire various data requests from the system within the data sampling time. The first determining unit is used to determine the transaction type to which each data request belongs; The second determining unit is used to determine the request proportion of each transaction type based on the number of each data request for each transaction type, and to determine the response time coefficient of the system by applying each of the request proportions. The second determining unit includes: an acquisition subunit, a calculation subunit, and a summation processing subunit; The acquisition subunit is used to acquire the complexity of each transaction type; The computation subunit is used to calculate the complexity and request ratio of each transaction type to obtain the response coefficient of each transaction type; The summation processing subunit is used to sum up each of the response coefficients to obtain the response time coefficient of the system; The third determining unit is used to apply the response time coefficient to determine the online processing evaluation value of the system for each type of transaction; The third determining unit includes: a first determining subunit, a second determining subunit, and a first obtaining subunit; The first determining subunit is used to determine the response time for each data request; The second determining subunit is used to determine, for each transaction type, the arithmetic mean of the query rate per second for that transaction type based on the response time of each data request belonging to that transaction type; The first obtaining subunit is used to process the response time coefficient, the arithmetic mean of each transaction type and the preset response time index value to obtain the online processing evaluation value of each transaction type. An evaluation unit is used to calculate each of the online processing evaluation values ​​to obtain the online processing fluctuation range, and to use the online processing fluctuation range to evaluate the performance of the system.

6. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides is controlled to perform the system performance evaluation method as described in any one of claims 1-4.

7. An electronic device, characterized in that, It includes memory, and one or more instructions, wherein one or more instructions are stored in memory and configured to be executed by one or more processors using the system performance evaluation method as described in any one of claims 1-4.

Images