Software system performance testing methods, apparatus and electronic equipment
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BAIDU (CHINA) CO LTD
- Filing Date
- 2026-01-21
- Publication Date
- 2026-06-02
Smart Images
Figure CN122132260A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to the fields of data processing, large models, and natural language processing. Background Technology
[0002] In related technologies, when performing performance testing on software systems, the entire testing process requires manual intervention and confirmation from the user at each stage to configure the various parameters needed for performance testing. The generation, execution, and analysis of test scripts are independent of each other, making continuous intelligent optimization and real-time adjustment impossible. Furthermore, the intelligent generation of performance test scripts relies on the accuracy of large models, which may lead to incompletely accurate script generation and require manual verification in certain complex business scenarios. Summary of the Invention
[0003] This disclosure provides a performance testing method, apparatus, and electronic device for a software system.
[0004] According to one aspect of this disclosure, a performance testing method for a software system is provided, comprising: Obtain test information for the software system, including test requirements, scenario descriptions, and test instructions; Based on the test requirements and scenario descriptions, obtain the test script JMX file; Generate load test parameters based on the JMX file and test instructions; Multiple rounds of performance testing are conducted based on the load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained. The load test parameters are then updated based on the real-time performance indicators. The updated load test parameters are used to conduct the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and test data for all rounds of performance testing are obtained. The test data is analyzed and processed to obtain a test report.
[0005] According to another aspect of this disclosure, a performance testing apparatus for a software system is provided, comprising: The information acquisition module is used to acquire test information of the software system, including test requirement information, scenario description information, and test instructions. The script generation module is used to obtain test script JMX files based on test requirement information and scenario description information; The parameter generation module is used to generate load test parameters based on the JMX file and test instructions; The iterative testing module is used to perform multiple rounds of performance testing based on load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and test data for all rounds of performance testing are obtained. The report generation module is used to analyze and process test data to generate test reports.
[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the performance testing method of the software system described above.
[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the performance testing method of the software system described above.
[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the performance testing method of the software system described above.
[0009] In this embodiment, a test script JMX file is obtained based on test requirement information and scenario description information, enabling automatic conversion from natural language to test scripts and improving script generation efficiency. Load test parameters are generated based on the JMX file and test instructions, and multiple rounds of performance testing are performed. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on these indicators. The updated load test parameters are then used for the next round of performance testing, and the real-time performance indicators from the next round are obtained. This process continues until the real-time performance indicators meet the termination conditions, resulting in test data for all rounds of performance testing. Adaptive execution of performance testing improves processing efficiency, enables continuous intelligent optimization and real-time adjustment, and enhances the accuracy and flexibility of software system performance testing.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system according to an exemplary embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system according to an exemplary embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system according to an exemplary embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system according to an exemplary embodiment of the present disclosure; Figure 5 This is a schematic diagram of a performance testing apparatus for a software system according to an exemplary embodiment of the present disclosure; Figure 6 This is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation
[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0013] The following is an introduction to the terminology used in this disclosure.
[0014] Data processing is a fundamental component of systems engineering and automatic control. It permeates all areas of social production and life. The development of data processing technology and the breadth and depth of its applications have profoundly influenced the progress of human society. Data is a form of expression of facts, concepts, or instructions, which can be processed manually or automatically. After being interpreted and given meaning, data becomes information. Data processing involves the collection, storage, retrieval, processing, transformation, and transmission of data. The fundamental purpose of data processing is to extract and derive valuable and meaningful data from large amounts of potentially chaotic and difficult-to-understand data.
[0015] Computer technology encompasses a wide range of topics, which can be broadly categorized into computer system technology, computer device technology, computer component technology, and computer assembly technology. Specifically, computer technology includes: the fundamental principles of arithmetic methods and arithmetic unit design, instruction sets, central processing unit (CPU) design, pipelining principles and their application in CPU design, memory architecture, and buses and input / output systems.
[0016] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. The core idea is to learn patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent.
[0017] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Its main applications include machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, and Chinese OCR.
[0018] JMX is a script file used by the JMeter performance testing tool. It stores the entire test plan in XML format. XML stands for Extensible Markup Language.
[0019] JSON represents a structured format for data exchange and configuration.
[0020] Ramp-Up represents startup time / climbing time, the total time it takes for all virtual users to start up, used to simulate a gradually increasing load.
[0021] The Document Object Model (DOM) is a cross-platform, programming language-independent interface that parses and maps HTML or XML documents into a tree structure (object model). In this tree, each tag, attribute, and text becomes a node object that can be manipulated and modified by a programming language (such as JavaScript).
[0022] ThreadGroup represents a thread group, the starting point of the test, and defines the number of simulated users (threads), the number of loops, and the startup method.
[0023] lxml is a powerful Python XML / HTML processing library that can be used in conjunction with JMeter for complex response data extraction. HTML stands for Hypertext Markup Language, a markup language.
[0024] JMeter is an open-source performance and load testing tool primarily used to simulate high-concurrency users putting pressure on servers, networks, or objects to test their performance, load capacity, and stability.
[0025] HTTPSampler stands for HTTP Request Sampler, a core component used to send HTTP / HTTPS requests to web servers.
[0026] HeaderManager stands for Header Manager, which is used to add or modify request headers for Hypertext Transfer Protocol (HTTP) requests.
[0027] CSVDataSet represents a CSV data file configuration that reads test data from an external CSV file to achieve parameterization.
[0028] ConstantTimer is a fixed timer that adds a fixed pause time after each request to control the request frequency.
[0029] hashTree stands for hash tree, a data structure used internally by JMeter to store test plan structure and component relationships.
[0030] ResponseAssertion represents a response assertion, used to verify whether the server response meets expectations, such as checking text or code.
[0031] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0032] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system as shown in this application, such as... Figure 1 As shown, the method includes the following steps: S101, Obtain test information for the software system, including test requirements, scenario descriptions, and test instructions.
[0033] In this embodiment of the disclosure, the performance testing of the software system can be applied to multiple scenarios. For example, in the enterprise application performance testing scenario, the performance testing of the software system can realize the performance verification of web applications, mobile applications, and API interfaces; in the e-commerce platform load testing scenario, the performance testing of the software system can realize the system carrying capacity verification before promotional activities; in the game server performance testing scenario, the performance testing of the software system can realize the concurrent user carrying capacity test of online game servers.
[0034] In this embodiment of the disclosure, the test information of the software system can be information triggered by a preset terminal device or information obtained from a database. Specifically, the test requirement information describes performance test requirements and includes at least one or more of the following: business type, interface protocol, and concurrency mode; the test instructions include at least one or more of the following: test case identifier, target performance metric, and constraints; and the scenario description information includes at least one or more of the following: number of threads, ramp-up time, request for uniform resource locator (URL), timeout, and assertion rules.
[0035] S102, based on test requirement information and scenario description information, obtain the test script JMX file.
[0036] In some implementations, test requirement information and scenario description information are input into a large language model (LLM), which then performs reasoning and prediction on the test requirement information and scenario description information to obtain a test script JMX file.
[0037] In some implementations, a preset data exchange and configuration JSON template is populated based on test requirement information and scenario description information to obtain a JSON file; based on the JSON file, a preset processing library is called to perform document object model (DOM) parsing and dynamic modification to obtain a JMX file.
[0038] S103 generates load test parameters based on the JMX file and test instructions.
[0039] In some implementations, the JMX file and test instructions are input into a large language model, which then performs inference and prediction on the JMX file and test instructions to generate load test parameters.
[0040] In some implementations, test instructions are parsed to obtain test case identifiers, target performance metrics, and constraints; test cases associated with the test case identifiers are obtained; historical test databases are queried to obtain historical execution data similar to the test cases; and load test parameters are generated based on the historical execution data, target performance metrics, and constraints.
[0041] S104. Perform multiple rounds of performance testing based on load test parameters. After each round of performance testing, obtain multi-dimensional real-time performance indicators, update the load test parameters based on the real-time performance indicators, use the updated load test parameters to perform the next round of performance testing, obtain the real-time performance indicators obtained from the next round of performance testing, until the real-time performance indicators meet the termination conditions, and obtain the test data of all rounds of performance testing.
[0042] Optionally, load test parameters include the number of threads, startup time, and continuous runtime.
[0043] In some implementations, multi-dimensional real-time performance metrics are obtained after each round of performance testing, including service throughput (TPS / QPS), transaction response time (RT), system resource utilization (CPU, memory, disk I / O), and error rate. Load test parameters are updated based on the real-time performance metrics, prioritizing a conservative strategy of reducing concurrent users to avoid system overload. The updated load test parameters are used for the next round of performance testing, and the real-time performance metrics from the next round are obtained until the real-time performance metrics meet the termination conditions, thus obtaining test data for all rounds of performance testing.
[0044] S105, analyze and process the test data to obtain a test report.
[0045] This disclosure integrates test data from all rounds to generate a detailed test report that includes performance and capacity conclusions.
[0046] In some implementations, test data is input into a large language model, which then analyzes and infers from the test data to generate a test report.
[0047] In some implementations, multi-dimensional data analysis, trend prediction, and anomaly detection are performed on the test data to generate a multi-dimensional test report.
[0048] In this embodiment, a test script JMX file is obtained based on test requirement information and scenario description information, enabling automatic conversion from natural language to test scripts and improving script generation efficiency. Load test parameters are generated based on the JMX file and test instructions, and multiple rounds of performance testing are performed. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on these indicators. The updated load test parameters are then used for the next round of performance testing, and the real-time performance indicators from the next round are obtained. This process continues until the real-time performance indicators meet the termination conditions, resulting in test data for all rounds of performance testing. Adaptive execution of performance testing improves processing efficiency, enables continuous intelligent optimization and real-time adjustment, and enhances the accuracy and flexibility of software system performance testing.
[0049] This disclosure is primarily applied to performance testing, performance monitoring, and performance optimization of software systems, and is particularly suitable for performance testing scenarios that require large-scale, multi-scenario, and complex business logic.
[0050] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system as shown in this application, such as... Figure 2 As shown, the method includes the following steps: S201, Obtain test information for the software system. Test information includes test requirements, scenario descriptions, and test instructions.
[0051] For details regarding step S201, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.
[0052] S202: Fill in the preset data exchange and configuration JSON template according to the test requirements information and scenario description information, and obtain the JSON file.
[0053] In some implementations, the target JMX template path is determined from multiple candidate JMX template paths based on the test requirement information; the scenario description information is parsed to obtain candidate test parameters; and a JSON template is populated based on the candidate test parameters and the target JMX template path to obtain a JSON file.
[0054] In some implementations, the target JMX template path is determined from multiple candidate JMX template paths based on the test requirement information, including: obtaining the tags of each candidate JMX template path; extracting keywords from the performance test requirements; matching the keywords with the tags of each candidate JMX template path, and taking the candidate JMX template path corresponding to the tag with the highest matching degree as the target JMX template path.
[0055] The following section details the automatic generation and conversion mechanism for performance test scripts based on large models disclosed in this publication.
[0056] In this embodiment, the test requirement information is a performance test requirement described in natural language. This information is input into a large language model. Upon receiving the description, the large language model first performs keyword extraction to identify key elements such as business type, interface protocol, and concurrency mode. Using a predefined tag-template mapping, the extracted keywords are matched with existing candidate JMX template paths to obtain the corresponding target JMX template path.
[0057] After identifying the business scenario, the LLM further parses the input scenario description information, extracts test parameters (such as the number of threads, ramp-up time, request URL, timeout, assertion rules, etc.), and outputs a JSON file according to a predefined JSON template. The JSON template structure includes a `params` field to store candidate test parameters and a `tag` field to record the path of the target JMX template. In other words, the value of `tag(key)` in the JSON template is the storage path of the JMX template. Therefore, this disclosure achieves decoupling between "semantic parameters" and "execution template".
[0058] S203: Based on the JSON file, call the preset processing library to perform DOM parsing and dynamic modification, and obtain the JMX file.
[0059] In some implementations, the JMX template is obtained based on the target JMX template path in the JSON file; based on the candidate test parameters in the JSON file, a preset processing library is called to perform DOM parsing and dynamic modification on the JMX template to obtain an Extensible Markup Language Tree; the Extensible Markup Language Tree is serialized to obtain the JMX file.
[0060] In some implementations, after obtaining the JMX file, the process further includes: performing data validation on the JMX file and obtaining the data validation result; in response to the data validation result indicating that the JMX file has structural errors or missing parameters, re-calling a preset processing library to perform DOM parsing and dynamic modification based on the JSON file to update the JMX file.
[0061] In this embodiment, the corresponding JMX template is loaded based on the tag field in the JSON file, and a preset processing library (lxml library) is called to parse the candidate parameters in the JSON file, performing DOM parsing and dynamic modification on the JMX template. The parameter backfilling process includes: For ThreadGroup elements, dynamically update attributes such as the number of concurrent users (numThreads), Ramp-Up time (rampTime), and number of loops; For the HTTPSampler element, populate fields such as target URL, request method, and parameter content; For optional components such as HeaderManager, CSVDataSet, and ConstantTimer, XML nodes are dynamically created and inserted into the corresponding hashTree based on the JSON configuration. For assertions (such as ResponseAssertion), generate assertion nodes based on the validation conditions in the JSON and attach them.
[0062] In this process, the hierarchical operation interface provided by lxml is used to ensure that node insertion, attribute updates and indentation format all conform to the JMeter JMX file specification, thereby improving data accuracy.
[0063] In this embodiment of the disclosure, after the parameter backfilling is completed, the modified XML tree is serialized and output as a JMX file, and syntax validation is performed. For example, the format of the XML is validated, missing fields in components are checked, and controller hierarchy structure errors are verified to ensure that the file can be loaded and executed normally by JMeter. If a format error or missing parameters are found, the system automatically falls back to the JSON intermediate layer and re-triggers the parameter backfilling logic. That is, step S203 is re-executed to ensure that the generated JMX file is complete and usable and to avoid data errors.
[0064] S204 generates load test parameters based on the JMX file and test instructions.
[0065] S205 performs multiple rounds of performance testing based on load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and the test data for all rounds of performance testing are obtained.
[0066] S206, Analyze and process the test data to obtain a test report.
[0067] For details regarding steps S204 to S206, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.
[0068] In this disclosure, a preset data exchange and configuration JSON template is populated based on test requirement information and scenario description information to obtain a JSON file. Based on the JSON file, a preset processing library is called to perform document object model (DOM) parsing and dynamic modification to obtain a JMX file. This decouples the "semantic parameters" from the "execution template," improves data accuracy, and enables automatic conversion from natural language to test scripts. This can improve script generation efficiency, avoid data errors, support subsequent iterative performance testing processes, and improve the accuracy of software system performance testing.
[0069] Figure 3 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system as shown in this application, such as... Figure 3 As shown, load test parameters are generated based on the JMX file and test instructions, including: S301, Obtain test information for the software system, including test requirements, scenario descriptions, and test instructions.
[0070] S302: Based on test requirement information and scenario description information, obtain the test script JMX file.
[0071] For details regarding steps S301 to S302, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.
[0072] S303, parse the test instructions to obtain the test case identifier, target performance indicators and constraints.
[0073] In this embodiment of the disclosure, during the intelligent parameter initialization phase, test instructions input in natural language are acquired. The test instructions are then parsed using a Large Language Model (LLM) to identify test case identifiers, target performance metrics, and constraints. For example, the maximum query rate per second (QPS) is 50, and the duration is 10ms. Execution stops when the response time exceeds 10ms.
[0074] S304, retrieve the test case associated with the test case identifier.
[0075] In some implementations, there is an association between test case identifiers and test cases, and the test cases associated with the test case identifier can be obtained based on the association.
[0076] S305, query the historical test database to obtain historical execution data similar to the test cases.
[0077] In some implementations, historical execution data similar to the current test case is obtained by querying the historical test database using Reinforced Generation (RAG) technology.
[0078] Optionally, historical execution data can include the number of threads set for each round of execution, duration, average response time, throughput, CPU usage, and memory utilization.
[0079] S306 generates load test parameters based on historical execution data, target performance indicators, and constraints.
[0080] In some implementations, a set of initial load test parameters is generated by combining historical execution data, target performance indicators, and constraints.
[0081] In some implementations, multiple sets of initial load test parameters are generated by combining historical execution data, target performance indicators, and constraints. The load test parameters that are triggered among the multiple sets of load test parameters are used as the load test parameters for the first round of performance testing.
[0082] Optionally, load test parameters may include the number of threads, startup time, and continuous runtime.
[0083] This disclosure can automatically generate load test parameters, improve the efficiency of software system performance testing, and increase data accuracy.
[0084] S307 performs multiple rounds of performance testing based on load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and test data for all rounds of performance testing are obtained.
[0085] In some implementations, based on a preset multi-objective decision tree algorithm, each real-time performance indicator and its corresponding reference performance indicator range are compared and analyzed to calculate the optimized load parameters for the next round of performance testing; and the load test parameters are updated according to the optimized load parameters.
[0086] In some implementations, optimizing load parameters includes the number of concurrent users, and real-time performance metrics include multiple metrics. The number of concurrent users is adjusted in response to at least one real-time performance metric falling within a corresponding reference performance metric range, and at least one real-time performance metric not falling within the corresponding reference performance metric range. Optionally, the reference performance metric range can be determined based on a target performance metric and a preset error threshold. For example, the difference between the target performance metric and the error threshold, and the sum of the target performance metric and the error threshold, can be used as the upper and lower limits of the reference performance metric range, respectively.
[0087] In some implementations, the real-time performance metric is determined to meet the termination condition in response to each real-time performance metric falling within the corresponding reference performance metric range; or, the real-time performance metric is determined to meet the termination condition in response to the error rate being greater than a preset error rate threshold.
[0088] In this embodiment of the disclosure, during the multi-round iterative execution and monitoring phase, the first round of performance testing is initiated using the initialized load test parameters; during the test execution, multi-dimensional real-time performance indicators of the software system under test are collected in real time, including service throughput (TPS / QPS), transaction response time (RT), system resource utilization (CPU, memory, disk I / O) and error rate.
[0089] In this embodiment, the target performance index represents the target reference value of the real-time performance index. During the multi-objective decision-making and dynamic optimization phase, the collected real-time performance index is compared and analyzed with the preset target reference value. Based on a preset multi-objective decision tree algorithm, the optimized load parameters for the next round of testing are calculated. The core of the decision tree algorithm includes: Termination criteria: If the error rate reaches 100%, the test is terminated; or, if the error between all real-time performance metrics and the target performance metrics is less than the predetermined error threshold, the test is terminated.
[0090] Conflict resolution strategy: When some real-time performance indicators have reached the target while others have not, a conservative strategy of reducing the number of concurrent users is preferred to avoid system overload. In other words, if the error between some real-time performance indicators and the target performance indicators is less than the predetermined error threshold, and the error between some real-time performance indicators and the target performance indicators is greater than or equal to the predetermined error threshold, the strategy of reducing the number of concurrent users is preferred.
[0091] Binary search convergence: When it is necessary to reduce or increase the load, the binary search algorithm is used to quickly and convergently adjust the number of concurrent users. The new formula for calculating the number of concurrent users is:
[0092] Where VUcurrent represents the current number of virtual concurrent users, VUlower_bound represents the lower bound of virtual concurrent users, and VUnew represents the new number of concurrent users. The lower bound is updated iteratively.
[0093] In this embodiment, the running results are compared with a set target reference value to determine whether to reduce or increase the number of concurrent users. For example, if a certain indicator is close to the reference value, but other indicators exceed the reference value, and the concurrency needs to be reduced, then the concurrency test is reduced. If there is a lower boundary, a binary search method is used. If there is no lower boundary, half the number of concurrent users is taken. If a certain indicator is close to the reference value, but other indicators have not reached the reference value and the error rate is within a reasonable range, then the number of concurrent users is increased.
[0094] In this embodiment of the disclosure, the new parameters determined during the closed-loop feedback and result generation stage are input into the next round of performance testing, and the testing is repeated iteratively to form a closed-loop feedback loop of "execution, monitoring, decision-making, and adjustment" until the termination condition is met.
[0095] S308 analyzes and processes the test data to obtain a test report.
[0096] For details regarding step S308, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.
[0097] This disclosure improves processing efficiency through adaptive performance testing, enabling continuous intelligent optimization and real-time adjustment. It achieves a closed-loop cycle of "execution, monitoring, decision-making, and adjustment" during performance testing, thereby improving the efficiency and accuracy of performance limit exploration and capacity boundary identification. It realizes intelligent performance testing that combines real-time monitoring and dynamic tuning, improving the accuracy and flexibility of software system performance testing.
[0098] Figure 4 This is a schematic diagram illustrating an exemplary implementation of a performance testing method for a software system as shown in this application, such as... Figure 4 As shown, the method includes the following steps: S401: Obtain test information for the software system. Test information includes test requirements, scenario descriptions, and test instructions.
[0099] S402: Based on test requirement information and scenario description information, obtain the test script JMX file.
[0100] S403 generates load test parameters based on the JMX file and test instructions.
[0101] S404 performs multiple rounds of performance testing based on load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and the test data for all rounds of performance testing are obtained.
[0102] For details regarding steps S401 to S404, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.
[0103] It should be noted that the test data includes at least the task execution status information, test case data, and real-time performance metrics during the performance testing process.
[0104] S405 analyzes the task execution status information in the test data and generates a task execution report.
[0105] The following describes the intelligent analysis and report generation mechanism based on a three-branch parallel architecture.
[0106] In this embodiment, the test data is preprocessed, and the preprocessed test data simultaneously enters three parallel analysis branches: the analysis task branch, the analysis use case branch, and the intelligent analysis branch. Each branch uses independent processing threads and computing resources to ensure the efficiency and stability of parallel processing.
[0107] The analysis task branch is specifically responsible for monitoring information collection, including system resource monitoring (CPU, memory, IO, network), execution process tracking, intelligent log parsing, and code repository status management. This branch obtains execution status information through a real-time data acquisition interface and generates task execution reports.
[0108] In performance testing scenarios, codebase state management is mainly used to trace the root cause of performance problems (associating performance test results with code changes), trend analysis (performance comparison of different versions), performance regression detection (automatically discovering whether a new version introduces performance degradation), and complete test context (recording the code state during testing).
[0109] S406 analyzes the test case data in the test data and generates a test case analysis report.
[0110] In this embodiment, the test case analysis branch is responsible for managing and matching historical test case data. It identifies the best matching pattern using a test case similarity algorithm, maintains a test case knowledge base, and provides intelligent recommendation services. This branch performs comparative analysis on test case data from historical test data to generate a test case analysis report.
[0111] In this embodiment, different matching modes may include, for example, exact matching (all key features, such as interface path, HTTP method, concurrent users, test duration, and test scenario are completely identical), scenario matching (matching based on business scenario characteristics; for example, all data are from previous years' holiday promotions), and technology stack matching (technology stack identification, such as front-end, back-end, database, caching, message queue, and containerization). The intelligent recommendation service is used to perform matching based on the above matching modes, recommend monitoring strategies, collect and compare data results of the same type, and automatically identify risk points.
[0112] S407 performs multi-dimensional data analysis, trend prediction, and anomaly detection on test data, generating a multi-dimensional analysis report.
[0113] In this embodiment, the intelligent analysis branch performs core algorithm processing, including advanced analysis functions such as multi-dimensional data analysis, trend prediction, and anomaly detection on the test data. This branch uses machine learning algorithms and statistical analysis methods to generate a deep intelligent analysis report; the analysis goal is to find the performance patterns of the system under different concurrency levels and correlate performance indicators with resource consumption.
[0114] Test data may also include time series data (collected per second), timestamps, number of concurrent users, current QPS, average or median response time, percentile metrics describing system performance or response time (e.g., TP50, TP90, TP95, TP99, etc.), error rate, number of successful requests, number of failed requests, network throughput, and monitoring data of the machine under test (CPU, memory, disk, network), etc.
[0115] S408 integrates the task execution report, use case analysis report, and multidimensional analysis report to obtain the test report.
[0116] In this embodiment, the analysis results from the three branches are integrated using an intelligent fusion algorithm to automatically generate differentiated analysis reports based on test requirements and application scenarios. This disclosure supports functions such as real-time report push, historical report retrieval, and report template customization, enabling visualized display and intelligent application of analysis results.
[0117] In this embodiment, the three-branch parallel analysis architecture can overcome the bottleneck of serial processing, improve the accuracy of anomaly detection, shorten the response time from hours to minutes, reduce manual intervention, and provide a fully intelligent solution for the performance testing platform from generation to execution to analysis.
[0118] Figure 5 This is a schematic diagram of a performance testing apparatus for a software system according to an exemplary embodiment of the present disclosure, such as... Figure 5 As shown, the performance testing device 500 for this software system includes: The information acquisition module 510 is used to acquire test information of the software system, including test requirement information, scenario description information and test instructions. The script generation module 520 is used to obtain test script JMX files based on test requirement information and scenario description information; The parameter generation module 530 is used to generate load test parameters based on the JMX file and test instructions; The iterative testing module 540 is used to perform multiple rounds of performance testing based on load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and test data for all rounds of performance testing are obtained. The report generation module 550 is used to analyze and process the test data to obtain a test report.
[0119] In some implementations, the script generation module 520 is further configured to: Fill the preset data exchange and configuration JSON template according to the test requirements and scenario description information, and obtain the JSON file; Based on the JSON file, a preset processing library is called to perform DOM parsing and dynamic modification to obtain the JMX file.
[0120] In some implementations, the script generation module 520 is further configured to: Based on the test requirements, the target JMX template path is determined from multiple candidate JMX template paths; Parse the scene description information to obtain candidate test parameters; The JSON template is populated based on the candidate test parameters and the target JMX template path to obtain a JSON file.
[0121] In some implementations, the script generation module 520 is further configured to: Get the tag for each candidate JMX template path; Extract keywords from performance testing requirements; The keywords are matched with the tags of each candidate JMX template path, and the candidate JMX template path corresponding to the tag with the highest matching degree is taken as the target JMX template path.
[0122] In some implementations, the script generation module 520 is further configured to: Retrieve the JMX template based on the target JMX template path in the JSON file; Based on the candidate test parameters in the JSON file, the preset processing library is called to perform DOM parsing and dynamic modification of the JMX template to obtain the Extensible Markup Language tree; The Extensible Markup Language tree is serialized to obtain a JMX file.
[0123] In some implementations, the script generation module 520 is further configured to: Perform data validation on the JMX file and obtain the data validation results; In response to data validation results indicating that the JMX file has structural errors or missing parameters, the default processing library is called again to perform DOM parsing and dynamic modification based on the JSON file in order to update the JMX file.
[0124] In some implementations, the parameter generation module 530 is further configured to: Parse test instructions to obtain test case identifiers, target performance metrics, and constraints; Retrieve the test case associated with the test case identifier; Query the historical test database to obtain historical execution data similar to the test cases; Based on historical execution data, target performance indicators, and constraints, load test parameters are generated.
[0125] In some implementations, the iterative testing module 540 is also used for: Based on the preset multi-objective decision tree algorithm, each real-time performance indicator and its corresponding reference performance indicator range are compared and analyzed to calculate the optimized load parameters for the next round of performance testing. Update the load test parameters based on the optimized load parameters.
[0126] In some implementations, optimizing load parameters includes the number of concurrent users, and real-time performance metrics include multiple metrics. The iterative testing module 540 is also used for: If at least one real-time performance metric falls within the corresponding reference performance metric range and at least one real-time performance metric does not fall within the corresponding reference performance metric range, adjust the number of concurrent users.
[0127] In some implementations, real-time performance metrics include service throughput, transaction response time, system resource utilization, and error rate. The iterative testing module 540 is also used for: If each real-time performance metric falls within the corresponding reference performance metric range, the termination condition is determined to be met; or, If the error rate exceeds a preset error rate threshold, the real-time performance metrics are determined to meet the termination conditions.
[0128] In some implementations, the test data includes at least task execution status information, test case data, and real-time performance metrics during the performance testing process. The report generation module 550 is also used for: Analyze the task execution status information in the test data and generate a task execution report; Analyze the test case data in the test data and generate a test case analysis report; Perform multi-dimensional data analysis, trend prediction, and anomaly detection on test data, and generate multi-dimensional analysis reports; The task execution report, use case analysis report, and multidimensional analysis report are merged to obtain the test report.
[0129] In this embodiment, a test script JMX file is obtained based on test requirement information and scenario description information, enabling automatic conversion from natural language to test scripts and improving script generation efficiency. Load test parameters are generated based on the JMX file and test instructions, and multiple rounds of performance testing are performed. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on these indicators. The updated load test parameters are then used for the next round of performance testing, and the real-time performance indicators from the next round are obtained. This process continues until the real-time performance indicators meet the termination conditions, resulting in test data for all rounds of performance testing. Adaptive execution of performance testing improves processing efficiency, enables continuous intelligent optimization and real-time adjustment, and enhances the accuracy and flexibility of software system performance testing.
[0130] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0131] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0132] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.
[0133] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0134] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a performance testing method for a software system. For example, in some embodiments, the performance testing method for a software system may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the performance testing method for the software system described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the performance testing method for a software system by any other suitable means (e.g., by means of firmware).
[0135] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0136] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0140] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0141] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A performance testing method for a software system, wherein, include: Obtain test information for the software system, including test requirement information, scenario description information, and test instructions; Based on the test requirement information and the scenario description information, obtain the test script JMX file; Based on the JMX file and the test instructions, generate load test parameters; Multiple rounds of performance testing are performed based on the load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and test data for all rounds of performance testing are obtained. The test data is analyzed and processed to obtain a test report.
2. The method according to claim 1, wherein, The step of obtaining the test script JMX file based on the test requirement information and the scenario description information includes: Fill the preset data exchange and configuration JSON template according to the test requirement information and the scenario description information, and obtain the JSON file; Based on the JSON file, a preset processing library is invoked to perform Document Object Model (DOM) parsing and dynamic modification to obtain the JMX file.
3. The method according to claim 2, wherein, The step of filling a preset data exchange and configuration JSON template according to the test requirement information and the scenario description information, and obtaining a JSON file, includes: Based on the test requirement information, the target JMX template path is determined from multiple candidate JMX template paths; The scene description information is parsed to obtain candidate test parameters; The JSON template is populated based on the candidate test parameters and the target JMX template path to obtain the JSON file.
4. The method according to claim 3, wherein, The step of determining the target JMX template path from multiple candidate JMX template paths based on the test requirement information includes: Obtain the tag for each of the candidate JMX template paths; Extract keywords from the performance testing requirements; The keyword is matched with the tag of each candidate JMX template path, and the candidate JMX template path corresponding to the tag with the highest matching degree is taken as the target JMX template path.
5. The method according to claim 3 or 4, wherein, The step of calling a preset processing library to perform Document Object Model (DOM) parsing and dynamic modification based on the JSON file, and obtaining the JMX file, includes: Obtain the JMX template based on the target JMX template path in the JSON file; Based on the candidate test parameters in the JSON file, a preset processing library is called to perform DOM parsing and dynamic modification of the JMX template to obtain an extensible markup language tree; The Extensible Markup Language tree is serialized to obtain the JMX file.
6. The method according to any one of claims 2-4, wherein, After obtaining the JMX file, the process further includes: Perform data verification on the JMX file and obtain the data verification result; In response to the data verification result indicating that the JMX file has a structural error or missing parameters, the preset processing library is called again to perform DOM parsing and dynamic modification based on the JSON file in order to update the JMX file.
7. The method according to any one of claims 1-4, wherein, The step of generating load test parameters based on the JMX file and the test instructions includes: Parse the test instructions to obtain the test case identifier, target performance indicators, and constraints; Obtain the test case associated with the test case identifier; Query the historical test database to obtain historical execution data similar to the test case. The load test parameters are generated based on historical execution data, the target performance indicators, and the constraints.
8. The method according to any one of claims 1-4, wherein, The step of updating the load test parameters according to the real-time performance metrics includes: Based on the preset multi-objective decision tree algorithm, each real-time performance indicator and its corresponding reference performance indicator range are compared and analyzed to calculate the optimized load parameters for the next round of performance testing. Update the load test parameters based on the optimized load parameters.
9. The method according to claim 8, wherein, The optimized load parameters include the number of concurrent users, and the real-time performance metrics include multiple metrics. The process of comparing and analyzing the real-time performance metrics with preset reference performance metrics to calculate the optimized load parameters for the next round of performance testing includes: In response to at least one of the real-time performance metrics falling within the corresponding reference performance metric range, and at least one of the real-time performance metrics not falling within the corresponding reference performance metric range, the number of concurrent users is adjusted.
10. The method according to claim 1, wherein, The real-time performance metrics include service throughput, transaction response time, system resource utilization, and error rate. The method further includes: In response to the fact that each real-time performance metric falls within the corresponding reference performance metric range, it is determined that the real-time performance metric meets the termination condition; or, In response to the error rate being greater than a preset error rate threshold, the real-time performance metric is determined to meet the termination condition.
11. The method according to any one of claims 1-4, wherein, The test data includes at least task execution status information, test case data, and real-time performance metrics during the performance testing process. The analysis and processing of the test data to obtain a test report includes: The task execution status information in the test data is analyzed to generate a task execution report; The test case data in the test data is analyzed to generate a test case analysis report; Perform multi-dimensional data analysis, trend prediction, and anomaly detection on the test data to generate a multi-dimensional analysis report; The test report is obtained by fusing the task execution report, the use case analysis report, and the multidimensional analysis report.
12. A performance testing device for a software system, wherein, include: The information acquisition module is used to acquire test information of the software system, including test requirement information, scenario description information, and test instructions. The script generation module is used to obtain test script JMX files based on the test requirement information and the scenario description information; The parameter generation module is used to generate load test parameters based on the JMX file and the test instructions; The iterative testing module is used to perform multiple rounds of performance testing based on the load test parameters. After each round of performance testing, multi-dimensional real-time performance indicators are obtained, and the load test parameters are updated based on the real-time performance indicators. The updated load test parameters are then used to perform the next round of performance testing, and the real-time performance indicators obtained from the next round of performance testing are obtained. This process continues until the real-time performance indicators meet the termination conditions, and test data for all rounds of performance testing are obtained. The report generation module is used to analyze and process the test data to obtain a test report.
13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.
15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-11.