Performance test-based monitoring index determination method and device, and electronic equipment

Abstract

The application provides a performance test-based monitoring index determination method and device and electronic equipment, which can be used in the field of cloud computing. The method comprises the following steps: obtaining a first data set of performance index data and a second data set of monitoring index data of a measured system; determining the union set of the first data set and the second data set as a to-be-analyzed set; determining the performance index data in the to-be-analyzed set as an initial clustering center, performing clustering processing on the to-be-analyzed set, and obtaining a clustering cluster; if the clustering cluster comprises performance index data and monitoring index data, determining the distance between the performance index data and the monitoring index data in the clustering cluster; according to the distance between the performance index data and the monitoring index data in the clustering cluster, determining a target monitoring index of a performance index corresponding to the performance index data from monitoring indexes corresponding to the monitoring index data; and outputting the monitoring index data of the target monitoring index to a user for viewing.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a method, apparatus, and electronic device for determining monitoring indicators based on performance testing. Background Technology

[0002] With the rapid development of financial services, information systems are becoming increasingly large-scale, and performance testing is an important means of ensuring the quality of information systems.

[0003] Because the systems under test are often very large, involving dozens or even hundreds of servers, each with multiple network cards and disks, the number of server resource monitoring metrics is enormous. Therefore, it is necessary to identify the monitoring metrics that have the greatest impact on the overall performance of the system under test from this large amount of data, so that developers and operations personnel can focus on them. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for determining monitoring indicators based on performance testing, in order to improve the efficiency and accuracy of determining monitoring indicators.

[0005] Firstly, this application provides a method for determining monitoring metrics based on performance testing, including:

[0006] Obtain the first set of performance metrics and the second set of monitoring metrics of the system under test;

[0007] The union of the first data set and the second data set is determined as the set to be analyzed;

[0008] The performance indicators in the set to be analyzed are determined as the initial cluster centers, and the set to be analyzed is subjected to clustering to obtain the clusters of the set to be analyzed.

[0009] If the cluster includes both the performance metric and the monitoring metric, then the distance between the performance metric in the cluster and each monitoring metric in the cluster is determined; wherein the distance characterizes the importance of the monitoring metric to the performance metric.

[0010] Based on the distance between the performance index in the cluster and each monitoring index in the cluster, the target monitoring index of the performance index is determined from the monitoring indexes of the cluster.

[0011] The target monitoring metrics are output to the user for viewing.

[0012] The performance metrics characterize the processing status of the system under test for transaction requests; the monitoring metrics characterize the resource usage of the server of the system under test; each performance metric includes sampled data from at least two sampling points within a preset first sampling period, and each monitoring metric includes sampled data from at least two sampling points within a preset second sampling period; the first data set includes at least two performance metrics, and the second data set includes at least two monitoring metrics.

[0013] Secondly, this application provides a monitoring indicator determination device based on performance testing, comprising:

[0014] The data acquisition module is used to acquire a first set of performance metrics and a second set of monitoring metrics of the system under test.

[0015] The data merging module is used to determine the union of the first data set and the second data set, which is the set to be analyzed;

[0016] The clustering module is used to determine the performance indicators in the set to be analyzed as the initial cluster centers, perform clustering processing on the set to be analyzed, and obtain the clusters of the set to be analyzed.

[0017] A distance determination module is used to determine the distance between the performance index in the cluster and each monitoring index in the cluster if the cluster includes both the performance index and the monitoring index; wherein the distance characterizes the importance of the monitoring index to the performance index.

[0018] The indicator determination module is used to determine the target monitoring indicator of the performance indicator from the monitoring indicators of the cluster based on the distance between the performance indicator in the cluster and each monitoring indicator in the cluster.

[0019] The data output module is used to output the target monitoring indicators to the user for viewing;

[0020] The performance metrics characterize the processing status of the system under test for transaction requests; the monitoring metrics characterize the resource usage of the server of the system under test; each performance metric includes sampled data from at least two sampling points within a preset first sampling period, and each monitoring metric includes sampled data from at least two sampling points within a preset second sampling period; the first data set includes at least two performance metrics, and the second data set includes at least two monitoring metrics.

[0021] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0022] The memory stores computer-executed instructions;

[0023] The processor executes computer execution instructions stored in the memory to implement the performance test-based monitoring metric determination method as described in the first aspect of this application.

[0024] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the performance testing-based monitoring metric determination method as described in the first aspect of this application.

[0025] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the performance testing-based monitoring metric determination method described in the first aspect of this application.

[0026] This application provides a method, apparatus, and electronic device for determining monitoring indicators based on performance testing. It obtains a set of data containing both performance and monitoring indicators by acquiring a first set of performance indicators and a second set of data containing monitoring indicators. The performance indicators in the set to be analyzed are used as initial cluster centers to cluster the set, resulting in clusters. It is determined whether each cluster contains both performance and monitoring indicators. If so, the distance between the performance and monitoring indicators is calculated for each cluster. This distance represents the importance of the monitoring indicator to the performance indicator; for example, the closer the distance, the more important the monitoring indicator. Based on the distance between the performance and monitoring indicators in the clusters, the target monitoring indicator is determined from the monitoring indicators and then output to the user for review. This solves the problem of omissions or inaccuracies in target monitoring indicators caused by relying on manual methods and historical results to determine them in existing technologies. It effectively saves manpower and time, reduces labor costs, and improves the accuracy and efficiency of monitoring indicator selection. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] Figure 1 A flowchart illustrating a method for determining monitoring metrics based on performance testing, provided in an embodiment of this application;

[0029] Figure 2 A flowchart illustrating a method for determining monitoring metrics based on performance testing, provided in an embodiment of this application;

[0030] Figure 3 A structural block diagram of a monitoring index determination device based on performance testing provided in an embodiment of this application;

[0031] Figure 4 A structural block diagram of a monitoring index determination device based on performance testing provided in an embodiment of this application;

[0032] Figure 5 A structural block diagram of an electronic device provided in an embodiment of this application;

[0033] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0034] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0037] It should be noted that the method, apparatus and electronic device for determining monitoring indicators based on performance testing of this application can be used in the field of cloud computing, or in any field other than cloud computing. The application field of the method, apparatus and electronic device for determining monitoring indicators based on performance testing of this application is not limited.

[0038] With the rapid development of financial services, the information systems of banks and other financial institutions are growing in scale. Performance testing is a crucial means of ensuring the quality of these systems. Performance metrics such as TPS (Transactions per Second), average response time, and success rate are commonly used to measure the performance of the system under test. These systems are often large-scale, involving dozens or even hundreds of servers. Server resource monitoring metrics typically include CPU, memory, network interface card (NIC) traffic, disk space, page load, and more. Each server often has multiple NICs and disks, resulting in a massive number of server resource monitoring metrics. Therefore, it is necessary to identify the monitoring metrics that have the greatest impact on the overall performance of the system under test—the high-importance metrics—from this large amount of data. These high-importance metrics are then included in the test report for developers and operations personnel to focus on.

[0039] However, currently, performance testers often select important monitoring metrics based on their personal experience, or they may choose metrics that have exceeded server resource thresholds during actual testing. This selection method is inaccurate, relies too heavily on manual intervention and historical results, and is prone to overlooking some important monitoring metrics. The low efficiency and precision of metric selection lead to inaccurate test result analysis and inaccurate risk warnings in test reports.

[0040] This application provides a method, apparatus, and electronic device for determining monitoring indicators based on performance testing, which aims to solve the above-mentioned technical problems in the prior art.

[0041] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0042] Figure 1 This is a flowchart illustrating a method for determining monitoring metrics based on performance testing, according to an embodiment of this application. The method is executed by a device for determining monitoring metrics based on performance testing. Figure 1 As shown, the method includes the following steps:

[0043] S101. Obtain the first set of performance indicators and the second set of monitoring indicators of the system under test.

[0044] For example, with the rapid development of financial business, the scale of bank information systems is growing larger and larger. Performance testing of these systems is a crucial means of ensuring their quality; the system under test can be a financial institution's information system, etc. Performance metrics for the system under test can include TPS (Transactions Per Second), average response time, and success rate. TPS refers to the number of transaction requests processed per second, average response time is the average response time of the system to transaction requests, and success rate is the success rate of processing transaction requests. These performance metrics can be obtained after testing the system under test. Performance metrics can represent the system's handling of transaction requests. By using these metrics, it can be determined whether the system's performance meets actual operational requirements, thus achieving the testing objective. For example, if the success rate obtained after performance testing is 98%, and the preset operational requirement is a success rate of 95%, then it can be determined that the system under test was successfully tested and meets the actual operational requirements.

[0045] After performance testing is completed, performance metrics generated by the system under test in processing transaction requests during this test can be obtained. Specifically, this includes TPS, average response time, and success rate. The obtained performance metrics are combined into a set of performance metrics, serving as the first dataset. This first dataset can include multiple performance metrics. For each performance metric, multiple sampling points can be obtained. Sampling points represent the time of sampling, i.e., obtaining data for the performance metric at multiple times. A preset sampling period for the performance metric is used as the first sampling period. Sampling data corresponding to multiple sampling points are collected from the first sampling period to obtain the performance metric data at those sampling points. For example, if the performance metric is success rate and the first sampling period is one hour, the success rate can be determined every minute within one hour of the performance test, resulting in 60 sample data points for the success rate.

[0046] The first dataset can include multiple performance metrics, each corresponding to sampled data from multiple sampling points. Sampled data can be represented as a string, for example: sampling time, metric name, and sampled value. Each performance metric can correspond to multiple strings of sampled data. These multiple strings are concatenated into a single string, resulting in one string corresponding to one performance metric. For example, to obtain sampled data for a performance metric at two sampling points, concatenating the two strings yields the following string: sampling time 1, metric name 1, sampled value 1, sampling time 2, metric name 2, sampled value 2. Here, metric name 1 and metric name 2 are identical. Each performance metric corresponds to a concatenated string. The first dataset can include multiple concatenated strings, and each concatenated string can be a set element within the first dataset.

[0047] Monitoring metrics can represent the resource usage of the servers in the system under test. For example, monitoring metrics may include CPU, memory, network traffic, disk space, page swapping, etc. The system under test is large-scale, associated with one or more servers, each with multiple monitoring metrics. All monitoring metrics corresponding to all servers in the system under test form a dataset, serving as a second dataset. That is, the second dataset can include multiple monitoring metrics. The monitoring metrics of the servers are obtained during performance testing. For each monitoring metric, multiple sampling points can be obtained within a preset sampling period, which can be the second sampling period. The first and second sampling periods can be the same or different. For example, if the monitoring metric is CPU utilization and the second sampling period is one hour, CPU utilization can be determined every half minute within one hour of performance testing, obtaining 120 sample data points of CPU utilization.

[0048] The second data set can include multiple monitoring metrics, each corresponding to sampled data from multiple sampling points. Sampled data can also be represented as a string, for example: sampling time, metric name, and sampled value. Each monitoring metric on each server can correspond to multiple strings of sampled data. These multiple strings are concatenated to form a single string, ensuring that each monitoring metric on each server corresponds to a single string. For example, to obtain sampled data from two sampling points for a monitoring metric on a server, concatenating the two data points results in the following string: sampling time 1, metric name 1, sampled value 1, sampling time 2, metric name 2, sampled value 2. Here, metric name 1 and metric name 2 are identical. Each monitoring metric corresponds to a concatenated string. The second data set can include multiple concatenated strings, each representing a monitoring metric within the second data set.

[0049] S102. Determine the union of the first data set and the second data set as the set to be analyzed.

[0050] For example, the first data set and the second data set are merged to obtain the union of the first data set and the second data set. The union of the first data set and the second data set is determined as the set to be analyzed. That is, the set to be analyzed includes both performance indicator data and monitoring indicator data.

[0051] S103. Determine the performance indicators in the set to be analyzed as the initial cluster centers, perform clustering on the set to be analyzed, and obtain the clusters of the set to be analyzed.

[0052] For example, all performance metrics in the set to be analyzed are determined, and these performance metrics are used as initial cluster centers. Based on the initial cluster centers, the set to be analyzed is clustered. Through continuous clustering iterations, multiple clusters are obtained. The number of clusters can be the same as the number of performance metrics in the set to be analyzed. For example, if the set to be analyzed includes three performance metrics—TPS, average response time, and success rate—three clusters can be generated. As another example, if the set to be analyzed includes three performance metrics and five monitoring metrics, the string data corresponding to these three performance metrics are used as the initial cluster centers. Multiple sample values ​​can be obtained from the string corresponding to each performance metric, and these sample values ​​are combined into coordinates to serve as the initial cluster centers. For example, if each performance metric corresponds to sample values ​​at two sampling points, namely sample value 1 and sample value 2, then the coordinates of the initial cluster center corresponding to that performance metric can be represented as (sample value 1, sample value 2). In this embodiment, the clustering algorithm is not specifically limited.

[0053] In this embodiment, clustering is performed on the set to be analyzed to obtain clusters of the set to be analyzed, including: determining the distance between the initial cluster center and each element in the set to be analyzed; and based on the distance between the initial cluster center and each element in the set to be analyzed, clustering is performed on the set to be analyzed using a preset K-means clustering algorithm to obtain clusters of the set to be analyzed.

[0054] Specifically, the distance between the initial cluster centers and each element in the set to be analyzed is determined. The elements in the set include the initial cluster centers, which are the data of the performance indicators. Therefore, the distance between the initial cluster centers and each monitoring indicator in the set can be determined. Euclidean distance can be used to measure the distance between the elements of the set; that is, the Euclidean distance between the performance indicator and each monitoring indicator is determined. Distance calculation can be performed using sampled values ​​of each indicator. Each indicator has multiple sampled values, which constitute the coordinate data of the indicator. Distance calculation is then performed based on the coordinate data of the performance indicator and the coordinate data of the monitoring indicator.

[0055] The preset clustering algorithm can be K-means clustering. Based on the distance between the initial cluster centers and each monitoring indicator, the preset K-means clustering algorithm is used to cluster the elements in the set to be analyzed, resulting in clusters of the set to be analyzed. For example, if there are K system performance indicators, K clusters will be obtained in the end.

[0056] The advantage of this setup is that, through a preset clustering algorithm, elements in the set to be analyzed are automatically grouped, and monitoring indicators that have a significant impact on performance metrics are identified. This enables the automatic determination of monitoring indicators, reducing manpower and time, and improving the efficiency and accuracy of indicator determination.

[0057] S104. If the cluster includes performance indicators and monitoring indicators, then determine the distance between the performance indicators in the cluster and each monitoring indicator in the cluster.

[0058] For example, after obtaining multiple clusters, the metrics in each cluster are determined, and it is determined whether a cluster contains both performance metrics and monitoring metrics. If a cluster contains only performance metrics or only monitoring metrics, no further calculations are performed on that cluster. If a cluster contains both performance metrics and monitoring metrics, the performance metrics in the cluster are determined. For each performance metric, the distance between that performance metric and each monitoring metric in that cluster is calculated. This distance characterizes the importance of the monitoring metric to the performance metric; the closer the distance, the greater the influence of the monitoring metric on the performance metric; the farther the distance, the smaller the influence of the monitoring metric on the performance metric. For example, if a cluster has two performance metrics and five monitoring metrics, then five distances can be calculated for each performance metric.

[0059] In this embodiment, determining the distance between the performance index in the cluster and each monitoring index in the cluster includes: determining the Euclidean distance between the performance index in the cluster and each monitoring index in the cluster according to a preset Euclidean distance calculation method.

[0060] Specifically, a preset Euclidean distance calculation method can be used to calculate the distance between the performance metric in a cluster and each monitoring metric in that cluster. The calculated distance is the Euclidean distance. Each metric corresponds to a coordinate system composed of multiple sampled values. Based on the coordinates corresponding to the performance metric and the monitoring metric, the Euclidean distance between the performance metric and the monitoring metric is obtained.

[0061] The advantage of this setup is that by calculating the Euclidean distance, the degree of influence of the monitoring indicators on the performance indicators can be determined without manual measurement, which greatly reduces labor costs and improves the accuracy of the selection of monitoring indicators.

[0062] S105. Based on the distance between the performance indicators in the cluster and each monitoring indicator in the cluster, determine the target monitoring indicator for the performance indicators from the monitoring indicators of the cluster.

[0063] For example, after determining the distance between the performance metric in the cluster and each monitoring metric in the cluster, a monitoring metric can be selected from the monitoring metrics of the cluster based on the distance between the performance metric and each monitoring metric in the cluster. The selected monitoring metric is then determined as the target monitoring metric for that performance metric. Pre-defined rules for selecting monitoring metrics can be used; for example, the rule could be to determine the monitoring metric corresponding to the smallest distance as the target monitoring metric.

[0064] If a cluster contains multiple performance metrics, then for each performance metric, there is a corresponding distance to each monitoring metric. This allows us to determine the target monitoring metric for each performance metric. Different performance metrics can have different target monitoring metric. If a cluster contains only performance metrics, then for each performance metric in that cluster, it is not necessary to determine a target monitoring metric. That is, when monitoring a performance metric of the system under test, the user needs to obtain data from all monitoring metrics.

[0065] In this embodiment, the target monitoring index for the performance index is determined from the monitoring indexes of the cluster based on the distance between the performance index in the cluster and each monitoring index in the cluster. This includes: sorting the distance between the performance index in the cluster and each monitoring index in the cluster; and determining the monitoring index located at the preset sorting position as the target monitoring index for the performance index.

[0066] Specifically, for each performance metric within a cluster, the distances between that performance metric and every other monitoring metric in the cluster can be sorted. For example, the distances can be sorted from smallest to largest. One or more sorting positions are pre-defined, and the monitoring metrics located at these pre-defined positions are identified as the target monitoring metrics for that performance metric. For example, the top two monitoring metrics can be identified as target monitoring metrics. That is, the monitoring metrics that appear earlier in the order are important resource monitoring metrics for the system's performance and need to be output to the user for their focused attention. If a cluster contains both performance metrics and monitoring metrics, then a corresponding target monitoring metric can be identified for each performance metric within that cluster.

[0067] The advantage of this setup is that it sorts the relevant monitoring metrics according to Euclidean distance, with the metrics appearing first in the order being the most important resource monitoring metrics for that performance indicator, and these are then output to the user. This metric selection method is highly accurate, has low manual costs, and can select important monitoring metrics based on current test scenario data, without relying on historical data or experience, thus improving the efficiency and accuracy of metric determination.

[0068] S106. Output the target monitoring indicators to the user for viewing.

[0069] For example, after obtaining the target monitoring indicators, these indicators are output to the user, informing them that these indicators have a significant impact on system performance and that the user can focus on monitoring them. Alternatively, sampled data of the target monitoring indicators can be sent to the user, allowing them to improve the system under test based on this data, thereby ensuring that the system's performance meets actual operational requirements.

[0070] This application provides a method for determining monitoring metrics based on performance testing. By acquiring a first set of performance metrics and a second set of monitoring metrics, an analysis set containing both performance and monitoring metrics is obtained. The performance metrics in the analysis set are used as initial cluster centers to cluster the analysis set, resulting in clusters. It is determined whether each cluster contains both performance and monitoring metrics. If so, the distance between the performance and monitoring metrics is calculated for each cluster. This distance represents the importance of the monitoring metrics to the performance metrics; for example, the closer the distance, the more important the monitoring metrics. Based on the distance between the performance and monitoring metrics in the clusters, the target monitoring metrics are determined from the monitoring metrics, and then output to the user for viewing. This method solves the problem of omissions or inaccuracies in target monitoring metrics caused by relying on manual methods and historical results to determine them in existing technologies. It effectively saves manpower and time, reduces labor costs, and improves the accuracy and efficiency of monitoring metric selection.

[0071] Figure 2 This is a flowchart illustrating a method for determining monitoring metrics based on performance testing, which is an optional embodiment based on the above embodiments.

[0072] In this embodiment, the union of the first data set and the second data set is determined as the set to be analyzed. This can be further refined as follows: normalize the first data set and the second data set to obtain a normalized set of performance indicators and a normalized set of monitoring indicators; and determine the union of the normalized set of performance indicators and the normalized set of monitoring indicators as the set to be analyzed.

[0073] like Figure 2 The method includes the following steps:

[0074] S201. Obtain the first set of performance indicators and the second set of monitoring indicators of the system under test.

[0075] For example, this step can refer to step S101 above, and will not be repeated here.

[0076] S202. Normalize the first data set and the second data set to obtain the normalized set of performance indicators and the normalized set of monitoring indicators.

[0077] For example, in the first dataset, the units for different performance metrics may differ. For instance, success rate might be a percentage, and average response time might be time-based. Similarly, in the second dataset, the units for different monitoring metrics may also differ. To facilitate subsequent calculations, the first and second datasets can be normalized to ensure that the data in both datasets falls within the range of 0 to 1. Normalizing the first dataset yields a normalized set of performance metrics, and normalizing the second dataset yields a normalized set of monitoring metrics.

[0078] In this embodiment, the normalization process can be performed using the Min-max (minimum value minus maximum value) standardization method. This transforms the elements in the first and second data sets into dimensionless data, resulting in a normalized set of performance indicators and a normalized set of monitoring indicators. The elements in the normalized set of performance indicators are the normalized performance indicators, and the elements in the normalized set of monitoring indicators are the normalized monitoring indicators.

[0079] In this embodiment, the first and second data sets are normalized to obtain a normalized set of performance indicators and a normalized set of monitoring indicators. This includes: performing data filtering on the second data set to obtain a third data set of monitoring indicators; adjusting the number of sampling points for the performance indicators in the first data set and the number of sampling points for the monitoring indicators in the third data set to obtain a fourth data set of performance indicators and a fifth data set of monitoring indicators; and performing normalization on the fourth and fifth data sets to obtain the normalized set of performance indicators and the normalized set of monitoring indicators.

[0080] Specifically, the second dataset includes all monitoring metrics from multiple servers, resulting in a large data volume. Therefore, data filtering can be performed on the second dataset to reduce its size. The filtered second dataset will then be designated as the third dataset for monitoring metrics. Data filtering rules can be pre-set, and the second dataset can be filtered according to these rules. For example, the filtering rule could be to identify each monitoring metric corresponding to each server and remove duplicate data for each metric.

[0081] Each monitoring metric in the third dataset corresponds to multiple sampling points, and each performance metric in the first dataset corresponds to multiple sampling points. The number of sampling points for each monitoring metric in the third dataset may be the same as or different from the number of sampling points for each performance metric in the first dataset. After obtaining the third dataset, the number of sampling points in both the first and third datasets can be adjusted. For example, the number of sampling points for each metric in both datasets can be reduced so that the number of sampling points for each monitoring metric in the third dataset is the same as the number of sampling points for each performance metric in the first dataset. After adjusting the number of sampling points, the first dataset becomes the fourth dataset, and the third dataset becomes the fifth dataset. For example, if each performance metric in the first dataset has two sampling points, and each monitoring metric in the third dataset has three sampling points, then the number of sampling points for each monitoring metric in the third dataset can be reduced by one, making the number of sampling points for each monitoring metric in the third dataset two, while the first dataset remains unchanged. The original first data set can be identified as the fourth data set, and the third data set after deletion becomes the fifth data set.

[0082] Normalization is performed on both the fourth and fifth datasets. For example, the Min-max normalization method can be used to standardize both datasets, converting the data in both datasets into dimensionless data. Normalization of the fourth dataset yields a normalized set of performance metrics, and normalization of the fifth dataset yields a normalized set of monitoring metrics.

[0083] The advantage of this setup is that, before clustering, the first and second datasets are preprocessed to unify the data format and quantity, which facilitates subsequent calculations and improves calculation accuracy and efficiency.

[0084] In this embodiment, the second data set is subjected to data filtering processing to obtain a third data set of monitoring indicators, including: determining the fluctuation data of the sampling data to which the monitoring indicators belong in the second data set; if the fluctuation data is within a preset fluctuation range, then the monitoring indicators are filtered out from the second data set; and the third data set is obtained based on the monitoring indicators retained in the second data set.

[0085] Specifically, the second dataset includes multiple monitoring metrics, each of which can identify its associated server. By identifying the server of each metric, the same monitoring metrics across different servers can be distinguished. For example, if the system under test is associated with five servers, and each server corresponds to five monitoring metrics, then the second dataset contains a total of 25 monitoring metrics. For each monitoring metric in the second dataset, sampled data is obtained. The sampled data can include multiple sampling points and the corresponding sample values ​​for each sampling point. Based on the sampled data of the monitoring metrics, the fluctuation of the sampled data is determined, and the fluctuation of the monitoring metrics is represented by fluctuation change data. The fluctuation change data can be in the form of a curve, representing the ups and downs of the monitoring metrics at different sampling points. The maximum and minimum values ​​of the sampled data of the monitoring metrics can be determined, and the difference between the maximum and minimum values ​​is determined as the fluctuation difference, which can be included in the fluctuation change data.

[0086] A pre-defined fluctuation range can be set for different monitoring indicators. The fluctuation range can be the minimum value of the fluctuation difference. It checks if the fluctuating data falls within the preset fluctuation range. If so, the monitoring indicator is removed from the second dataset, i.e., the sampled data under that indicator is deleted. For example, it checks if the fluctuation difference of a monitoring indicator is less than the preset minimum value; if so, the sampled data under that indicator is deleted. For each server, the fluctuation range is checked for each monitoring indicator. The same monitoring indicator on different servers does not affect each other. For example, given servers A and B, the memory indicator for server A is deleted, but the disk space indicator is retained; the memory indicator for server B is retained, but the disk space indicator is deleted. The remaining monitoring indicators in the second dataset are then used to define the third dataset.

[0087] The advantage of this setting is that monitoring indicators with small fluctuation ranges can be considered to have little impact on performance indicators. Deleting monitoring indicators with small fluctuation ranges can reduce the amount of calculation and improve the efficiency of selecting monitoring indicators.

[0088] In this embodiment, adjusting the number of sampling points for performance indicators in the first data set and the number of sampling points for monitoring indicators in the third data set to obtain a fourth data set for performance indicators and a fifth data set for monitoring indicators includes: adjusting the number of sampling points for performance indicators in the first data set and the number of sampling points for monitoring indicators in the third data set to be consistent based on the time interval between sampling points for performance indicators in the first data set and the time interval between sampling points for monitoring indicators in the third data set, thereby obtaining a fourth data set for performance indicators and a fifth data set for monitoring indicators; wherein, in the fourth data set, the time interval between two consecutive sampling points for performance indicators is consistent with the time interval between two consecutive sampling points for monitoring indicators in the fifth data set.

[0089] Specifically, the number of sampling points for performance metrics in the first dataset and the number of sampling points for monitoring metrics in the third dataset may differ. For example, if the first and second sampling periods are the same (one hour), the time interval between sampling points for performance metrics in the first dataset is one minute, while the time interval between sampling points for monitoring metrics in the third dataset is half a minute, then each performance metric in the first dataset will have 60 sampled values, and each monitoring metric in the third dataset will have 120 sampled values. Filtering the second dataset simply removes the entire sampled data of the monitoring metrics; it does not affect the number of sampled values ​​in the monitoring metrics.

[0090] Based on the time intervals of the sampling points for performance metrics in the first dataset and the time intervals of the sampling points for monitoring metrics in the third dataset, the number of sampling points for performance metrics in the first dataset and the number of sampling points for monitoring metrics in the third dataset are adjusted to be consistent, resulting in a fourth dataset corresponding to the first dataset and a fifth dataset corresponding to the third dataset. This ensures that the time intervals of the sampling points for performance metrics in the fourth dataset are consistent with the time intervals of the sampling points for monitoring metrics in the fifth dataset. That is, the time interval between two consecutive sampling points for performance metrics in the fourth dataset is consistent with the time interval between two consecutive sampling points for monitoring metrics in the fifth dataset. For example, if the time interval for the sampling points for performance metrics in the first dataset is 1 minute and the time interval for the sampling points for monitoring metrics in the third dataset is half a minute, then the first dataset can be directly used as the fourth dataset, and the number of sampling points for monitoring metrics in the third dataset can be halved, meaning one sampling point can be deleted every other sampling point. Alternatively, the sampling time of the sampling points in the first dataset can be determined, and the sampling points corresponding to that sampling time in the third dataset can be retained, while the other sampling points can be deleted. Each sampling point corresponds to its own sampled data, that is, it corresponds to a sampled value at the sampling time. Deleting a sampling point means deleting the sampling data corresponding to that sampling point.

[0091] The advantage of this setup is that by adjusting the number of sampling points in the dataset, the sampling points of performance indicators and monitoring indicators can be matched, making it easier to determine the impact of monitoring indicators on performance indicators at the same time, improving the accuracy of monitoring indicator selection, reducing subsequent calculations, and thus improving the efficiency of monitoring indicator selection.

[0092] S203. Determine the union of the normalized set of performance indicators and the normalized set of monitoring indicators, which is the set to be analyzed.

[0093] For example, the normalized sets of performance metrics and monitoring metrics are merged to obtain the union of the normalized sets of performance metrics and monitoring metrics. This union of the normalized sets of performance metrics and monitoring metrics is then used as the set to be analyzed. That is, the data in the set to be analyzed are all normalized metric data.

[0094] S204. Determine the performance indicators in the set to be analyzed as the initial cluster centers, perform clustering on the set to be analyzed, and obtain the clusters of the set to be analyzed.

[0095] For example, this step can refer to step S103 above, and will not be repeated here.

[0096] S205. If the cluster includes performance indicators and monitoring indicators, then determine the distance between the performance indicators in the cluster and each monitoring indicator in the cluster; where the distance characterizes the importance of the monitoring indicator to the performance indicator.

[0097] For example, this step can refer to step S104 above, and will not be repeated here.

[0098] S206. Based on the distance between the performance indicators in the cluster and each monitoring indicator in the cluster, determine the target monitoring indicator for the performance indicators from the monitoring indicators of the cluster.

[0099] For example, this step can refer to step S105 above, and will not be repeated here.

[0100] S207. Output the target monitoring indicators to the user for viewing.

[0101] For example, this step can refer to step S106 above, and will not be repeated here.

[0102] This application provides a method for determining monitoring metrics based on performance testing. By acquiring a first set of performance metrics and a second set of monitoring metrics, an analysis set containing both performance and monitoring metrics is obtained. The performance metrics in the analysis set are used as initial cluster centers to cluster the analysis set, resulting in clusters. It is determined whether each cluster contains both performance and monitoring metrics. If so, the distance between the performance and monitoring metrics is calculated for each cluster. This distance represents the importance of the monitoring metrics to the performance metrics; for example, the closer the distance, the more important the monitoring metrics. Based on the distance between the performance and monitoring metrics in the clusters, the target monitoring metrics are determined from the monitoring metrics, and then output to the user for viewing. This method solves the problem of omissions or inaccuracies in target monitoring metrics caused by relying on manual methods and historical results to determine them in existing technologies. It effectively saves manpower and time, reduces labor costs, and improves the accuracy and efficiency of monitoring metric selection.

[0103] Figure 3 This is a structural block diagram of a monitoring indicator determination device based on performance testing, provided in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 3 The device includes: a data acquisition module 301, a data merging module 302, a clustering module 303, a distance determination module 304, an index determination module 305, and a data output module 306.

[0104] Data acquisition module 301 is used to acquire a first set of performance indicators and a second set of monitoring indicators of the system under test;

[0105] The data merging module 302 is used to determine the union of the first data set and the second data set, which is the set to be analyzed;

[0106] Clustering module 303 is used to determine the performance indicators in the set to be analyzed as the initial cluster centers, perform clustering processing on the set to be analyzed, and obtain the clusters of the set to be analyzed.

[0107] The distance determination module 304 is used to determine the distance between the performance index in the cluster and each monitoring index in the cluster if the cluster includes the performance index and the monitoring index; wherein the distance characterizes the importance of the monitoring index to the performance index;

[0108] The indicator determination module 305 is used to determine the target monitoring indicator of the performance indicator from the monitoring indicators of the cluster based on the distance between the performance indicators in the cluster and each monitoring indicator in the cluster.

[0109] The data output module 306 is used to output the target monitoring indicators to the user for viewing;

[0110] The performance metrics characterize the processing status of the system under test for transaction requests; the monitoring metrics characterize the resource usage of the server of the system under test; each performance metric includes sampled data from at least two sampling points within a preset first sampling period, and each monitoring metric includes sampled data from at least two sampling points within a preset second sampling period; the first data set includes at least two performance metrics, and the second data set includes at least two monitoring metrics.

[0111] Figure 4 This application provides a structural block diagram of a monitoring index determination device based on performance testing, in which... Figure 3 Based on the illustrated embodiments, as Figure 4 As shown, the data merging module 302 includes a normalization unit 3021 and a merging unit 3022.

[0112] Normalization unit 3021 is used to normalize the first data set and the second data set to obtain the normalized set of the performance indicators and the normalized set of the monitoring indicators.

[0113] The merging unit 3022 is used to determine the union of the normalized set of the performance indicators and the normalized set of the monitoring indicators, which is the set to be analyzed.

[0114] In one example, normalization unit 3021 includes:

[0115] A filtering subunit is used to perform data filtering processing on the second data set to obtain a third data set of the monitoring indicators;

[0116] An adjustment subunit is used to adjust the number of sampling points for performance indicators in the first data set and the number of sampling points for monitoring indicators in the third data set, so as to obtain a fourth data set for performance indicators and a fifth data set for monitoring indicators.

[0117] The processing subunit is used to normalize the fourth data set and the fifth data set to obtain the normalized set of the performance indicators and the normalized set of the monitoring indicators.

[0118] In one example, filtering sub-units are specifically used for:

[0119] Determine the fluctuation data of the sampling data to which the monitoring indicator belongs in the second data set;

[0120] If the fluctuation data is within a preset fluctuation range, the monitoring indicator is removed from the second data set;

[0121] The third data set is obtained based on the monitoring metrics retained in the second data set.

[0122] In one example, adjusting the sub-unit is specifically used for:

[0123] Based on the time interval between sampling points of performance indicators in the first data set and the time interval between sampling points of monitoring indicators in the third data set, the number of sampling points of performance indicators in the first data set and the number of sampling points of monitoring indicators in the third data set are adjusted to be consistent, resulting in a fourth data set of performance indicators and a fifth data set of monitoring indicators; wherein, in the fourth data set, the time interval between two consecutive sampling points of performance indicators is consistent with the time interval between two consecutive sampling points of monitoring indicators in the fifth data set.

[0124] In one example, clustering module 303 is specifically used for:

[0125] Determine the distance between the initial cluster centers and each element in the set to be analyzed;

[0126] Based on the distance between the initial cluster centers and each element in the set to be analyzed, the set to be analyzed is clustered using a preset K-means clustering algorithm to obtain the clusters of the set to be analyzed.

[0127] In one example, the distance determination module 304 is specifically used for:

[0128] Based on a preset Euclidean distance calculation method, the Euclidean distance between the performance index in the cluster and each monitoring index in the cluster is determined.

[0129] In one example, the indicator determination module 305 is specifically used for:

[0130] The performance metrics in the clusters are sorted according to their distances to each monitoring metric in the clusters.

[0131] The monitoring indicators located at the preset sorting positions are determined as the target monitoring indicators for the performance indicators.

[0132] Figure 5 A structural block diagram of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device includes: a memory 51 and a processor 52; the memory 51 is a memory used to store instructions executable by the processor 52.

[0133] The processor 52 is configured to perform the methods provided in the above embodiments.

[0134] The electronic device also includes a receiver 53 and a transmitter 54. The receiver 53 is used to receive instructions and data sent by other devices, and the transmitter 54 is used to send instructions and data to external devices.

[0135] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0136] Device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output (I / O) interface 612, sensor component 614, and communication component 616.

[0137] Processing component 602 typically controls the overall operation of device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0138] Memory 604 is configured to store various types of data to support the operation of device 600. Examples of this data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0139] Power supply component 606 provides power to various components of device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 600.

[0140] Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0141] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0142] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0143] Sensor assembly 614 includes one or more sensors for providing status assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in the position of device 600 or a component of device 600, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0144] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0145] In an exemplary embodiment, device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0146] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0147] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a terminal device, the terminal device is able to execute the aforementioned method for determining monitoring metrics based on performance testing.

[0148] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0149] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for determining monitoring metrics based on performance testing, characterized in that, include: Obtain the first set of performance metrics and the second set of monitoring metrics of the system under test; The union of the first data set and the second data set is determined as the set to be analyzed; The performance indicators in the set to be analyzed are determined as the initial cluster centers, and the set to be analyzed is subjected to clustering to obtain the clusters of the set to be analyzed. If the cluster includes both the performance metric and the monitoring metric, then the distance between the performance metric in the cluster and each monitoring metric in the cluster is determined; wherein the distance characterizes the importance of the monitoring metric to the performance metric. Based on the distance between the performance index in the cluster and each monitoring index in the cluster, the target monitoring index of the performance index is determined from the monitoring indexes of the cluster. The target monitoring metrics are output to the user for viewing. The performance metrics characterize the processing status of the system under test for transaction requests; the monitoring metrics characterize the resource usage of the server of the system under test; each performance metric includes sampled data from at least two sampling points within a preset first sampling period, and each monitoring metric includes sampled data from at least two sampling points within a preset second sampling period; the first data set includes at least two performance metrics, and the second data set includes at least two monitoring metrics.

2. The method according to claim 1, characterized in that, The union of the first data set and the second data set is determined as the set to be analyzed, including: The first data set and the second data set are normalized to obtain the normalized set of the performance indicators and the normalized set of the monitoring indicators. The union of the normalized set of the performance indicators and the normalized set of the monitoring indicators is determined as the set to be analyzed.

3. The method according to claim 2, characterized in that, Normalize the first data set and the second data set to obtain a normalized set of the performance metrics and a normalized set of the monitoring metrics, including: The second dataset is filtered to obtain a third dataset of the monitoring indicators. The number of sampling points for the performance indicators in the first data set and the number of sampling points for the monitoring indicators in the third data set are adjusted to obtain a fourth data set for the performance indicators and a fifth data set for the monitoring indicators. The fourth and fifth data sets are normalized to obtain the normalized sets of the performance indicators and the monitoring indicators.

4. The method according to claim 3, characterized in that, The second dataset is filtered to obtain a third dataset of the monitoring indicators, including: Determine the fluctuation data of the sampling data to which the monitoring indicator belongs in the second data set; If the fluctuation data is within a preset fluctuation range, the monitoring indicator is removed from the second data set; The third data set is obtained based on the monitoring metrics retained in the second data set.

5. The method according to claim 3, characterized in that, The number of sampling points for performance metrics in the first data set and the number of sampling points for monitoring metrics in the third data set are adjusted to obtain a fourth data set for performance metrics and a fifth data set for monitoring metrics, including: Based on the time interval between sampling points of performance indicators in the first data set and the time interval between sampling points of monitoring indicators in the third data set, the number of sampling points of performance indicators in the first data set and the number of sampling points of monitoring indicators in the third data set are adjusted to be consistent, resulting in a fourth data set of performance indicators and a fifth data set of monitoring indicators; wherein, in the fourth data set, the time interval between two consecutive sampling points of performance indicators is consistent with the time interval between two consecutive sampling points of monitoring indicators in the fifth data set.

6. The method according to claim 1, characterized in that, Clustering is performed on the set to be analyzed to obtain clusters of the set to be analyzed, including: Determine the distance between the initial cluster centers and each element in the set to be analyzed; Based on the distance between the initial cluster centers and each element in the set to be analyzed, the set to be analyzed is clustered using a preset K-means clustering algorithm to obtain the clusters of the set to be analyzed.

7. The method according to claim 1, characterized in that, Determining the distance between the performance metrics in the cluster and each monitoring metric in the cluster includes: Based on a preset Euclidean distance calculation method, the Euclidean distance between the performance index in the cluster and each monitoring index in the cluster is determined.

8. The method according to any one of claims 1-7, characterized in that, Based on the distance between the performance metrics in the cluster and each monitoring metric in the cluster, the target monitoring metric for the performance metrics is determined from the monitoring metric of the cluster, including: The performance metrics in the clusters are sorted according to their distances to each monitoring metric in the clusters. The monitoring indicators located at the preset sorting positions are determined as the target monitoring indicators for the performance indicators.

9. A device for determining monitoring indicators based on performance testing, characterized in that, include: The data acquisition module is used to acquire a first set of performance metrics and a second set of monitoring metrics of the system under test. The data merging module is used to determine the union of the first data set and the second data set, which is the set to be analyzed; The clustering module is used to determine the performance indicators in the set to be analyzed as the initial cluster centers, perform clustering processing on the set to be analyzed, and obtain the clusters of the set to be analyzed. A distance determination module is used to determine the distance between the performance index in the cluster and each monitoring index in the cluster if the cluster includes both the performance index and the monitoring index; wherein the distance characterizes the importance of the monitoring index to the performance index. The indicator determination module is used to determine the target monitoring indicator of the performance indicator from the monitoring indicators of the cluster based on the distance between the performance indicator in the cluster and each monitoring indicator in the cluster. The data output module is used to output the target monitoring indicators to the user for viewing; The performance metrics characterize the processing status of the system under test for transaction requests; the monitoring metrics characterize the resource usage of the server of the system under test; each performance metric includes sampled data from at least two sampling points within a preset first sampling period, and each monitoring metric includes sampled data from at least two sampling points within a preset second sampling period; the first data set includes at least two performance metrics, and the second data set includes at least two monitoring metrics.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

Images