Performance test method and device, equipment, storage medium and program product
By correlating performance test data and production operation data, quantitatively calculating deviations, and generating improvement strategies, the problem of low accuracy in performance testing in existing technologies is solved, achieving higher accuracy and lower production operation risks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2025-06-26
- Publication Date
- 2026-05-05
AI Technical Summary
The low accuracy of performance test results from existing technologies leads to high risks in production operations.
By acquiring performance test data and production operation performance data of the system under test, performing correlation processing, quantifying and calculating the deviation, and generating improvement strategies to improve the accuracy of performance test results.
This improved the accuracy of performance test results and reduced production operation risks.
Smart Images
Figure CN121979769A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed systems, and more particularly to a performance testing method, apparatus, device, storage medium, and program product. Background Technology
[0002] As banking services become increasingly diverse and are transformed using distributed and other technological architectures, system performance stability faces greater challenges. Performance testing, as a crucial support for ensuring stable production operations, is therefore of paramount importance.
[0003] Currently, performance testing of distributed services mainly involves testing multiple key performance indicators before and after deployment, and then evaluating the performance of the distributed service based on the test results of each performance indicator.
[0004] However, due to the high complexity and the constantly changing nature of production operations, the accuracy of performance test results for existing technologies is low, leading to high risks in production operations. Summary of the Invention
[0005] This application provides a performance testing method, apparatus, equipment, storage medium, and program product to solve the technical problem that the low accuracy of performance test results in the prior art leads to high risks in production operations.
[0006] Firstly, this application provides a performance testing method, including:
[0007] Obtain performance test data of the system under test;
[0008] Obtain the production operation performance data of the system under test;
[0009] The performance test data and the production operation performance data are correlated to obtain at least one set of correlated data; wherein each set of correlated data includes performance test correlated data and production performance test correlated data corresponding to the same performance type;
[0010] The correlation data is subjected to quantitative calculation to obtain the deviation of the correlation data;
[0011] Based on the deviation, a performance testing improvement strategy for the system under test is determined.
[0012] In one possible implementation, acquiring the performance test data of the system under test includes: acquiring at least one of performance test requirements, performance test scenarios, and performance types of performance tests; performing performance tests on the system under test according to at least one of the performance test requirements, the performance test scenarios, and the performance types of performance tests; and collecting the performance test data of the system under test during the performance test.
[0013] In one possible implementation, acquiring the production operation performance data of the system under test includes: acquiring the acquisition parameters of the performance test data; and acquiring the production operation performance data of the system under test based on the acquisition parameters.
[0014] In one possible implementation, the performance type includes a service performance metric type and a system performance metric type; wherein the service performance metric type includes at least one of throughput, response time, and success rate, and the system performance metric type includes at least one of CPU utilization, memory, and network traffic.
[0015] In one possible implementation, the service performance metric type includes throughput; correspondingly, the quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: determining the maximum test throughput based on the performance test associated data; determining the maximum production throughput based on the production performance test associated data; calculating the ratio of the maximum production throughput to the maximum test throughput; calculating a first difference between the ratio of the maximum production throughput to the maximum test throughput and 1; and determining the deviation of the associated data corresponding to the throughput based on the first difference.
[0016] In one possible implementation, the service performance metric type includes response time; correspondingly, the quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: determining a response time test benchmark value based on the performance test associated data; determining a response time production benchmark value based on the production performance test associated data; calculating a second difference between the response time production benchmark value and the response time test benchmark value; and determining the deviation of the associated data corresponding to the response time based on the ratio of the second difference to the response time test benchmark value.
[0017] In one possible implementation, the service performance metric type includes success rate; correspondingly, the quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: determining the performance test success rate based on the performance test associated data; determining the production operation success rate based on the production performance test associated data; calculating a third difference between the production operation success rate and the performance test success rate; and determining the deviation of the associated data corresponding to the success rate based on the third difference.
[0018] In one possible implementation, the step of quantifying the associated data to obtain the deviation of the associated data includes: when the performance type corresponding to the associated data is a system performance indicator type, determining the estimated peak change data and the actual peak change data corresponding to the system performance indicator type based on the performance test associated data and the production performance test associated data; and determining the deviation of the associated data corresponding to the system performance indicator type based on the fourth difference between the actual peak change data and the estimated peak change data.
[0019] In one possible implementation, determining the performance testing improvement strategy for the system under test based on the deviation includes: obtaining the deviation improvement threshold corresponding to the performance type; and determining the improvement priority and direction of the performance type based on the deviation improvement threshold and the deviation corresponding to the performance type.
[0020] In one possible implementation, after performing quantitative calculations on the associated data to obtain the deviation of the associated data, the method further includes: obtaining a deviation alarm threshold corresponding to the performance type; and determining whether to execute an alarm strategy and emergency response strategy for the performance type based on the deviation alarm threshold and the deviation of the performance type.
[0021] Secondly, this application provides a performance testing apparatus, comprising:
[0022] The first acquisition module is used to acquire performance test data of the system under test.
[0023] The second acquisition module is used to acquire the production operation performance data of the system under test.
[0024] The first processing module is used to perform correlation processing on the performance test data and the production operation performance data to obtain at least one set of correlation data; wherein, each set of correlation data includes performance test correlation data and production performance test correlation data corresponding to the same performance type;
[0025] The second processing module is used to perform quantitative calculations on the associated data to obtain the deviation of the associated data;
[0026] A determination module is used to determine a performance testing improvement strategy for the system under test based on the deviation.
[0027] Thirdly, this application provides an electronic device, including: a memory and a processor;
[0028] The memory stores computer-executed instructions;
[0029] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0030] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.
[0031] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0032] The performance testing method, apparatus, equipment, storage medium, and program products provided in this application, by combining the performance test data of the system under test with the production operation performance data for correlation and quantitative processing, obtain the deviation of the correlation data, so as to quantitatively measure the quality and effectiveness of performance testing, generate improvement strategies, improve the performance testing process, thereby improving the accuracy of performance test results and reducing production operation risks. Attached Figure Description
[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0034] Figure 1 This is a schematic diagram of the architecture of the performance testing system provided in the embodiments of this application;
[0035] Figure 2 A flowchart illustrating a performance testing method provided in one embodiment of this application;
[0036] Figure 3 A flowchart illustrating a performance testing method provided in another embodiment of this application;
[0037] Figure 4 This is a schematic diagram of the performance testing device provided in the embodiments of this application;
[0038] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0039] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0041] It should be noted that the performance testing methods, apparatus, devices, storage media, and program products provided in this application can be used in the distributed field, or in any field other than distributed fields. The application fields of the performance testing methods, apparatus, devices, storage media, and program products in this application are not limited.
[0042] Figure 1 This is a schematic diagram of the architecture of a performance testing system provided in an embodiment of this application. Figure 1 As shown, the performance testing system includes a performance testing management module 101, a performance data acquisition module 102, a performance testing result analysis and evaluation module 103, a performance testing module 104, and a production operation monitoring module 105. The performance testing management module 101, performance data acquisition module 102, performance testing result analysis and evaluation module 103, performance testing module 104, and production operation monitoring module 105 are capable of communicating with each other.
[0043] The performance testing module 104 includes a pressure machine 1041, which includes multiple pressure recording nodes.
[0044] Among them, the performance test management module 101 is used to manage performance test requirements, performance test scenarios, performance test types, and performance test data.
[0045] The performance data acquisition module 102 is used to acquire performance test data and production operation performance data of the system under test using a preset acquisition algorithm and acquisition frequency, so as to ensure that the acquisition methods of performance test data and production operation performance data are consistent.
[0046] The performance test result analysis and evaluation module 103 is used to correlate the performance test data and production operation performance data of the system under test to obtain correlated data; to perform quantitative calculations on the correlated data to obtain the deviation of the correlated data; and to determine the performance test improvement strategy for the system under test based on the deviation.
[0047] The performance testing module 104 is used to perform performance testing on the system under test, including stress testing the system under test using a pressure machine 1041.
[0048] Among them, the production operation monitoring module 105 is used to manage production operation performance data.
[0049] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0050] Figure 2 This is a flowchart illustrating a performance testing method provided in one embodiment of this application. The execution entity of this embodiment can be... Figure 1 The performance testing system shown can also be other computer equipment, and this embodiment does not impose any particular limitations on it. Figure 2 As shown, the method includes:
[0051] S201: Obtain performance test data for the system under test.
[0052] Specifically, S201 includes S2011 to S2013:
[0053] S2011: Obtain at least one of the following: performance test requirements, performance test scenarios, and performance types of performance tests.
[0054] Performance testing requirements include one or more of the following: capability verification requirements, capacity planning requirements, defect discovery and optimization requirements, stability and reliability requirements, and security and data consistency requirements.
[0055] The performance testing scenarios include one or more of the following: concurrent performance scenarios, scalability scenarios, network latency and failure scenarios, data-intensive scenarios, and mixed business scenarios.
[0056] The performance types of performance testing can include one or more of the following: load testing, stress testing, stability testing, benchmark testing, and fault tolerance testing.
[0057] S2012: Perform performance testing on the system to be tested based on at least one of the following: performance testing requirements, performance testing scenarios, and performance types of performance testing.
[0058] Specifically, performance testing is performed on the system under test based on performance testing requirements; or, based on performance testing scenarios; or, based on the performance type of the performance test; or, based on both performance testing requirements and performance testing scenarios; or, based on both performance testing requirements and performance type of the performance test; or, based on both performance testing scenarios and performance type of the performance test; or, based on all three of the performance testing requirements, scenarios, and performance type of the performance test.
[0059] The performance types include service performance metrics and system performance metrics. Service performance metrics include at least one of throughput, response time, and success rate, while system performance metrics include at least one of CPU utilization, memory, and network traffic.
[0060] S2013: During the performance testing process, collect performance test data of the system under test.
[0061] Specifically, the performance test data of the system under test is collected using a preset acquisition algorithm and acquisition frequency.
[0062] The preset data acquisition algorithms can include sliding window averaging, exponentially weighted moving average, percentile calculation, and peak detection, among others.
[0063] S202: Obtain production operation performance data of the system under test.
[0064] Specifically, S202 includes S2021 to S2022:
[0065] S2021: Acquisition parameters for obtaining performance test data.
[0066] The acquisition parameters include the acquisition algorithm and the acquisition frequency.
[0067] S2022: Collect production operation performance data of the system under test based on the collected parameters.
[0068] S203: Perform correlation processing on performance test data and production operation performance data to obtain at least one set of correlation data; wherein each set of correlation data includes performance test correlation data and production performance test correlation data corresponding to the same performance type.
[0069] Specifically, preset filtering and standardization algorithms are used to filter and standardize performance test data and production operation performance data; the processed performance test data and production operation performance data are correlated by time dimension, mapped by load level, and paired by key indicators to obtain at least one set of correlated data.
[0070] Among them, performance types include service performance index types and system performance index types.
[0071] S204: Perform quantitative calculations on the associated data to obtain the deviation of the associated data.
[0072] Specifically, the relative deviation between the performance test correlation data and the production performance test correlation data is calculated based on the correlation data to obtain the degree of deviation of the correlation data.
[0073] Optionally, the service performance metric type includes throughput. Accordingly, step S204 also provides another implementation method, detailed below:
[0074] S204-a1: Determine the maximum throughput of the test based on the performance test correlation data.
[0075] Specifically, throughput data in the performance test correlation data is compared to determine the maximum throughput of the test.
[0076] S204-a2: Determine the maximum production throughput based on production performance test correlation data.
[0077] Specifically, throughput data in the production performance test correlation data is compared to determine the maximum production throughput.
[0078] S204-a3: Calculate the ratio of maximum production throughput to maximum test throughput.
[0079] S204-a4: Calculate the first difference between the ratio of the maximum production throughput to the maximum test throughput and 1.
[0080] S204-a5: Based on the first difference, determine the deviation of the associated data corresponding to the throughput.
[0081] Specifically, the absolute value of the first difference is taken and converted into a percentage value; the percentage value is multiplied by a preset throughput weight to determine the deviation of the associated data from the throughput.
[0082] By quantifying the throughput of associated data to obtain the deviation, the quality and effectiveness of throughput performance testing can be intuitively represented, providing data support for optimizing performance testing, thereby improving the accuracy of performance test results and supporting cost reduction and efficiency improvement in production operations.
[0083] Optionally, the formula for calculating the deviation of the associated data from the throughput is: Deviation = |(Maximum production throughput / Maximum test throughput)*100% - 100%|*Preset throughput weight.
[0084] Optionally, the service performance metric type includes response time. Accordingly, step S204 also provides another implementation method, detailed below:
[0085] S204-b1: Determine the baseline value for response time testing based on performance test correlation data.
[0086] S204-b2: Determine the production baseline value for response time based on production performance test correlation data.
[0087] S204-b3: Calculate the second difference between the production baseline response time and the test baseline response time.
[0088] S204-b4: Determine the deviation of the associated data from the response time based on the ratio of the second difference to the response time test baseline.
[0089] Specifically, the absolute value of the ratio of the second difference to the response time test benchmark is taken, and the absolute value is converted into a percentage value; the percentage value is multiplied by a preset response time weight to determine the deviation of the associated data from the response time.
[0090] By quantifying the response time of related data to obtain the deviation, the performance difference between testing and production can be accurately located, providing data support for optimizing performance testing, thereby improving the accuracy of performance test results and supporting cost reduction and efficiency improvement in production operations.
[0091] Optionally, the service performance metric type includes success rate. Accordingly, step S204 also provides another implementation method, detailed below:
[0092] S204-c1: Determine the performance test success rate based on the performance test correlation data.
[0093] S204-c2: Determine the production operation success rate based on production performance test correlation data.
[0094] S204-c3: Calculate the third difference between the performance test success rate and the production operation success rate.
[0095] S204-c4: Determine the deviation of the associated data from the success rate based on the third difference.
[0096] Specifically, the third difference is multiplied by a preset success rate weight to determine the deviation of the associated data from the success rate.
[0097] By quantifying the success rate of correlated data to obtain the deviation, performance testing optimization can be promoted, ultimately achieving the dual goals of improving the accuracy of performance test results and enhancing resource utilization.
[0098] Optionally, when the performance type corresponding to the associated data is a system performance metric type, wherein the system performance metric type is at least one of CPU utilization, memory, and network traffic, step S204 also provides another implementation method, detailed below:
[0099] S204-d1: Based on the performance test correlation data and the production performance test correlation data, determine the estimated peak change data and the actual peak change data corresponding to the system performance index type.
[0100] For example, when the system performance metric type is CPU utilization, the performance test associated data includes the linear scaling factor of the throughput of multiple business services in the distributed system with CPU utilization and the peak utilization of CPU before the system under test is put into production. The production performance test associated data includes the peak utilization of CPU after the system under test is put into production.
[0101] For example, when the system performance metric is CPU utilization, the difference between the peak CPU utilization after the system under test is put into production and the peak CPU utilization before the system under test is put into production is calculated to determine the actual peak change data; the estimated peak change data is calculated based on the linear proportionality coefficient between the throughput of multiple services and CPU utilization and the throughput of multiple services.
[0102] For example, if there are 2 business services, the formula for calculating the estimated peak change data is: Estimated peak change in CPU utilization = (throughput of business service 1 * linear proportionality coefficient of throughput of business service 1 and CPU utilization) + (throughput of business service 2 * linear proportionality coefficient of throughput of business service 2 and CPU utilization).
[0103] S204-d2: Determine the deviation of the correlated data from the system performance index type based on the fourth difference between the actual peak change data and the estimated peak change data.
[0104] Specifically, the fourth difference is multiplied by the weight corresponding to the preset system performance index type to determine the deviation of the associated data from the system performance index type.
[0105] By quantifying the deviation of system performance indicator types through changes in the associated data, performance testing optimization can be promoted, ultimately achieving the dual goals of improving the accuracy of performance test results and enhancing resource utilization.
[0106] S205: Based on the deviation, determine the performance testing improvement strategy for the system under test.
[0107] Specifically, S205 includes S2051 to S2052:
[0108] S2051: Obtain the deviation improvement threshold corresponding to the performance type.
[0109] S2052: Determine the improvement priority and direction of the performance type based on the deviation improvement threshold and the deviation corresponding to the performance type.
[0110] Specifically, the improvement priority is determined based on the difference between the deviation improvement threshold corresponding to the performance type and the deviation corresponding to the performance type; the improvement direction is determined based on the improvement priority.
[0111] Specifically, if the deviation corresponding to the performance type is greater than or equal to the deviation improvement threshold corresponding to the performance type, the improvement priority is determined to be high priority; if the deviation corresponding to the performance type is less than the deviation improvement threshold corresponding to the performance type, the improvement priority is determined to be low priority.
[0112] Specifically, if the improvement priority is high, the corresponding improvement direction will be matched according to the characteristics of the performance type; if the improvement priority is low, the improvement direction will be determined to be included in the daily optimization plan.
[0113] Specifically, if the performance type is throughput and the improvement priority is high, then the improvement directions are determined as follows: Hardware: increase server nodes and upgrade network bandwidth to improve overall processing capacity; Software: adjust load balancing strategies.
[0114] Specifically, if the performance type is response time and the improvement priority is high, the improvement direction is determined to be to increase CPU or memory resources for high-load components, or to split microservices to reduce the pressure on a single node.
[0115] Specifically, if the performance type is success rate and the improvement priority is high priority, the improvement direction is to add a retry mechanism or analyze the failure reasons through logs and fix them accordingly. For example, if the failure reason is parameter error, then validate the input parameters; if the failure reason is abnormal dependent service, then add a circuit breaker policy for dependent service.
[0116] Specifically, if the performance type is CPU utilization and the improvement priority is high, the improvement direction is determined to be to locate processes with high CPU usage, adjust process resource quotas, or migrate computationally intensive tasks to dedicated servers or containers.
[0117] Specifically, if the performance type is memory and the improvement priority is high, then the improvement direction is determined to be to detect application memory leaks, clean up expired sessions and temporary files, or use distributed caching to reduce the memory pressure on a single machine.
[0118] Specifically, if the performance type is network traffic and the improvement priority is high, then the improvement direction is determined to be limiting non-core business bandwidth, reducing the number of network connections, or compressing transmitted data.
[0119] The performance testing method provided in this embodiment combines the performance test data of the system under test with the production operation performance data for correlation and quantitative processing to obtain the deviation of the correlation data. This allows for the quantitative measurement of the quality and effectiveness of the performance test, the generation of improvement strategies, and the improvement of the performance testing process, thereby increasing the accuracy of the performance test results and reducing production operation risks.
[0120] Optionally, based on the above embodiments, after step S204, a threshold alarm process is also included, which is detailed below:
[0121] S206: Obtain the deviation alarm threshold corresponding to the performance type.
[0122] S207: Based on the deviation alarm threshold and deviation corresponding to the performance type, determine whether to execute the alarm policy and emergency response policy for the performance type.
[0123] Specifically, if the deviation corresponding to a performance type is greater than or equal to the deviation alarm threshold corresponding to the performance type, the alarm policy for that performance type is executed; and the emergency response policy is activated based on the urgency of the performance type.
[0124] Specifically, if the urgency level of the performance type is high priority, an emergency response strategy will be activated; if the urgency level of the performance type is low priority, an alarm strategy will be executed and a manual assessment will be awaited, and a decision on whether to activate the emergency response strategy will be made based on the results of the manual assessment.
[0125] If the performance type is throughput, the alarm strategy is to send alarm information to the operations and development team terminals via SMS, email, or instant messaging tools. The alarm information includes the actual current throughput value, target value, deviation, and duration. The emergency response strategy is as follows: 1. Trigger load balancing adjustments to divert some traffic to other available server nodes or service instances; 2. Automatically increase the number of server instances or upgrade server specifications according to preset rules; 3. Investigate for malicious attacks.
[0126] If the performance type is response time, the alarm strategy is to send alarm information to the operations and development team terminals via SMS, email, and the internal alarm platform. The alarm information includes data such as average response time, the specific interface or service of the slow-response request, and deviation. The emergency response strategy is: 1. Enable caching to cache hot data; 2. Suspend non-core business processes or tasks and prioritize allocating system resources to core business processes.
[0127] If the performance type is success rate, the alarm strategy is to send alarm information to the operations and development team terminals via SMS and email, including a detailed report of failed requests. The emergency response strategy is as follows: 1. Automatically retry failed requests according to the preset number of retries and intervals; 2. If the failure is due to a third-party service anomaly, switch to a backup service; 3. Call the fallback interface to return default data.
[0128] If the performance type is CPU utilization, the alarm strategy is to push alarm information to the terminals of the operations and maintenance and development teams through the operation and maintenance monitoring system. The alarm message includes CPU utilization, the name and process identifier of the process whose utilization has reached the first preset threshold, and the duration. The emergency response strategy is: 1. Automatically terminate or suspend non-critical processes that occupy CPU resources to the second preset threshold; 2. Increase the CPU resource quota of the server.
[0129] If the performance type is memory, the alarm strategy is to send alarm information to the operations and maintenance and development team terminals through the operations and maintenance monitoring system. The alarm information includes memory usage, remaining available memory, and process information where memory usage has reached the third preset threshold. The emergency response strategy is as follows: 1. Trigger the memory garbage collection mechanism to clear objects and temporary data that have not been used for a preset time; 2. Shut down unnecessary services or applications that have reached the fourth preset threshold of memory usage.
[0130] If the performance type is network traffic, the alarm policy is to send alarm information to the operations and development team terminals through the network monitoring system. The alarm information includes the current traffic value, bandwidth utilization, the time of the traffic peak, and the source Internet Protocol address. The emergency response policy is to automatically enable traffic limiting policies to limit the network traffic of non-critical services.
[0131] By determining whether to implement alarm policies and emergency response policies for performance types based on the deviation alarm thresholds and deviations corresponding to the performance types, performance anomalies can be handled in a timely manner, the performance testing process can be improved, and production operation risks can be reduced.
[0132] Figure 3 This is a flowchart illustrating a performance testing method provided in another embodiment of this application. The execution entity in this embodiment can be... Figure 1 The performance testing system shown can also be other computer equipment, and this embodiment does not impose any particular limitations on it. Figure 3 As shown, the method includes:
[0133] S301: The performance testing module performs performance testing on the system under test according to the performance testing plan.
[0134] The performance testing scheme is either improved through an improvement strategy or configured according to performance testing parameters, wherein the performance testing parameters include at least one of performance testing requirements, performance testing scenarios, and performance types of performance testing.
[0135] S302: The performance data acquisition module collects the performance test data of the system under test and sends the performance test data to the performance test management module.
[0136] S303: The performance test management module determines whether the performance test data meets the production requirements. If the performance test data meets the production requirements, it generates a production-ready suggestion and sends the production-ready suggestion to the operation and maintenance personnel's terminal so that the operation and maintenance personnel can put the system to be tested into production.
[0137] S304: After the system under test is put into production, the performance data acquisition module collects production operation performance data of the system under test for a complete preset business cycle and sends the production operation performance data to the production operation monitoring module.
[0138] S305: The production operation monitoring module sends production operation performance data to the performance test result analysis and evaluation module.
[0139] S306: The performance test management module sends performance test data to the performance test result analysis and evaluation module.
[0140] S307: The performance test result analysis and evaluation module performs correlation and quantitative calculations based on production operation performance data and performance test data to obtain the deviation of multiple performance indicators of the performance test.
[0141] S308: The performance test result analysis and evaluation module determines the improvement strategies for multiple performance indicators based on the deviation of multiple performance indicators, and sends the improvement strategies to the performance test module.
[0142] S309: The performance testing module improves the performance testing plan for the system under test according to the improvement strategy, and repeats step S301.
[0143] The performance testing method provided in this embodiment quantitatively evaluates the quality and effectiveness of performance testing by correlating production operation performance data and performance test data, generating improvement strategies to improve the performance testing process and support cost reduction and efficiency improvement in production operations. The effectiveness of performance testing is continuously monitored based on the dynamic changes in the production operation of the system under test.
[0144] Figure 4 This is a schematic diagram of the performance testing device provided in an embodiment of this application. Figure 4As shown, the performance testing device 40 provided in this embodiment includes: a first acquisition module 401, a second acquisition module 402, a first processing module 403, a second processing module 404, and a determination module 405.
[0145] The first acquisition module 401 is used to acquire performance test data of the system under test;
[0146] The second acquisition module 402 is used to acquire the production operation performance data of the system under test;
[0147] The first processing module 403 is used to perform correlation processing on the performance test data and the production operation performance data to obtain at least one set of correlation data; wherein, each set of correlation data includes performance test correlation data and production performance test correlation data corresponding to the same performance type;
[0148] The second processing module 404 is used to perform quantitative calculation processing on the associated data to obtain the deviation of the associated data;
[0149] The determination module 405 is used to determine a performance testing improvement strategy for the system under test based on the deviation.
[0150] In one possible implementation, the first acquisition module 401 is specifically used to: acquire at least one of performance test requirements, performance test scenarios, and performance types of performance tests; perform performance tests on the system under test according to at least one of the performance test requirements, the performance test scenarios, and the performance types of performance tests; and collect performance test data of the system under test during the performance test.
[0151] In one possible implementation, the second acquisition module 402 is specifically used to: acquire the acquisition parameters of the performance test data; and acquire the production operation performance data of the system under test according to the acquisition parameters.
[0152] In one possible implementation, the performance type includes a service performance metric type and a system performance metric type; wherein the service performance metric type includes at least one of throughput, response time, and success rate, and the system performance metric type includes at least one of CPU utilization, memory, and network traffic.
[0153] In one possible implementation, the service performance metric type includes throughput; accordingly, the second processing module 404 is specifically configured to: determine the maximum test throughput based on the performance test correlation data; determine the maximum production throughput based on the production performance test correlation data; calculate the ratio of the maximum production throughput to the maximum test throughput; calculate a first difference between the ratio of the maximum production throughput to the maximum test throughput and 1; and determine the deviation of the correlation data from the throughput based on the first difference.
[0154] In one possible implementation, the service performance metric type includes response time; accordingly, the second processing module 404 is specifically configured to: determine a response time test benchmark value based on the performance test correlation data; determine a response time production benchmark value based on the production performance test correlation data; calculate a second difference between the response time production benchmark value and the response time test benchmark value; and determine the deviation of the correlation data from the response time based on the ratio of the second difference to the response time test benchmark value.
[0155] In one possible implementation, the service performance metric type includes success rate; correspondingly, the second processing module 404 is specifically used to: determine the performance test success rate based on the performance test associated data; determine the production operation success rate based on the production performance test associated data; calculate a third difference between the production operation success rate and the performance test success rate; and determine the deviation of the associated data from the success rate based on the third difference.
[0156] In one possible implementation, the second processing module 404 is specifically used to: when the performance type corresponding to the associated data is a system performance indicator type, determine the estimated peak change data and the actual peak change data corresponding to the system performance indicator type based on the performance test associated data and the production performance test associated data; and determine the deviation of the associated data corresponding to the system performance indicator type based on the fourth difference between the actual peak change data and the estimated peak change data.
[0157] In one possible implementation, the determining module 405 is specifically used to: obtain the deviation improvement threshold corresponding to the performance type; and determine the improvement priority and improvement direction of the performance type based on the deviation improvement threshold and the deviation corresponding to the performance type.
[0158] In one possible implementation, the determining module 405 is further configured to: obtain the deviation alarm threshold corresponding to the performance type; and determine whether to execute the alarm strategy and emergency response strategy for the performance type based on the deviation alarm threshold and the deviation corresponding to the performance type.
[0159] The performance testing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0160] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0161] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.
[0162] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0163] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0164] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0165] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0166] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0167] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0168] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0169] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0170] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0171] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0172] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0173] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0174] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0175] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0176] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0177] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0178] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A performance testing method, characterized in that, include: Obtain performance test data of the system under test; Obtain the production operation performance data of the system under test; The performance test data and the production operation performance data are correlated to obtain at least one set of correlated data; wherein each set of correlated data includes performance test correlated data and production performance test correlated data corresponding to the same performance type; The correlation data is subjected to quantitative calculation to obtain the deviation of the correlation data; Based on the deviation, a performance testing improvement strategy for the system under test is determined.
2. The method according to claim 1, characterized in that, The acquisition of performance test data for the system under test includes: Obtain at least one of the following: performance testing requirements, performance testing scenarios, and performance types of performance testing. Perform performance testing on the system under test according to at least one of the performance testing requirements, the performance testing scenario, and the performance type of the performance test. During the performance testing process, performance test data of the system under test is collected.
3. The method according to claim 1, characterized in that, The acquisition of the production operation performance data of the system under test includes: Obtain the acquisition parameters for the performance test data; Based on the acquisition parameters, collect the production operation performance data of the system under test.
4. The method according to any one of claims 1 to 3, characterized in that, The performance types include service performance index types and system performance index types; The service performance metric types include at least one of throughput, response time, and success rate, and the system performance metric types include at least one of CPU utilization, memory, and network traffic.
5. The method according to claim 4, characterized in that, The service performance metric types include throughput; Accordingly, the quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: Based on the performance test correlation data, determine the maximum throughput of the test; Based on the production performance test correlation data, determine the maximum production throughput; Calculate the ratio of the maximum production throughput to the maximum test throughput; Calculate the first difference between the ratio of the maximum production throughput to the maximum test throughput and 1; Based on the first difference, the deviation of the associated data from the throughput is determined.
6. The method according to claim 4, characterized in that, The service performance metrics include response time; Accordingly, the quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: Based on the performance test correlation data, determine the response time test baseline value; Based on the production performance test correlation data, determine the production baseline value for response time; Calculate the second difference between the production baseline value of the response time and the test baseline value of the response time; The deviation of the associated data from the response time is determined based on the ratio of the second difference to the response time benchmark value.
7. The method according to claim 4, characterized in that, The service performance metrics include success rate; Accordingly, the quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: Based on the performance test correlation data, determine the performance test success rate; Based on the production performance test correlation data, the production operation success rate is determined; Calculate the third difference between the production operation success rate and the performance test success rate; Based on the third difference, the deviation of the associated data from the success rate is determined.
8. The method according to claim 4, characterized in that, The quantitative calculation processing of the associated data to obtain the deviation of the associated data includes: When the performance type corresponding to the associated data is a system performance indicator type, the estimated peak change data and the actual peak change data corresponding to the system performance indicator type are determined based on the performance test associated data and the production performance test associated data. Based on the fourth difference between the actual peak change data and the estimated peak change data, the deviation of the associated data corresponding to the system performance index type is determined.
9. The method according to any one of claims 5 to 8, characterized in that, The step of determining a performance testing improvement strategy for the system under test based on the deviation includes: Obtain the deviation improvement threshold corresponding to the performance type; Based on the deviation improvement threshold corresponding to the performance type and the deviation corresponding to the performance type, the improvement priority and direction of the performance type are determined.
10. The method according to claim 9, characterized in that, After performing quantitative calculations on the correlated data to obtain the deviation of the correlated data, the method further includes: Obtain the deviation alarm threshold corresponding to the performance type; Based on the deviation alarm threshold corresponding to the performance type and the deviation corresponding to the performance type, determine whether to execute the alarm policy and emergency response policy for the performance type.
11. A performance testing device, characterized in that, include: The first acquisition module is used to acquire performance test data of the system under test. The second acquisition module is used to acquire the production operation performance data of the system under test. The first processing module is used to perform correlation processing on the performance test data and the production operation performance data to obtain at least one set of correlation data; wherein, each set of correlation data includes performance test correlation data and production performance test correlation data corresponding to the same performance type; The second processing module is used to perform quantitative calculations on the associated data to obtain the deviation of the associated data; A determination module is used to determine a performance testing improvement strategy for the system under test based on the deviation.
12. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 10.
14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.