Performance pressure testing system and method applied to large model streaming interface

Abstract

The invention discloses a performance pressure test system and method applied to a large model streaming interface, belongs to the technical field of artificial intelligence, and aims to solve the technical problem of how to comprehensively evaluate the performance of the large model streaming interface in a high-concurrency scene. Comprising a test data management module used for constructing a test data set; the concurrent access simulation module is used for dynamically configuring concurrent access parameters and simulating and generating a concurrent access scene of a plurality of users; the performance monitoring module is used for monitoring and recording the large-model streaming interface in the concurrent access scene; the fault-tolerant processing module is used for analyzing and processing the abnormal condition of the large model interface in the concurrent access scene based on a predefined abnormal processing strategy and forming an error log file; the post-processing module is used for calculating a performance index based on the monitoring data; and the result output module is used for constructing a test result file based on the error log file and the performance index and outputting the test result file.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a performance stress testing system and method applied to a large model streaming interface. Background Art

[0002] With the rapid development of artificial intelligence technology, large model streaming interfaces play an increasingly important role in data processing and analysis. However, in the current performance testing technology and traditional open source tools (such as jmeter, etc.) for large model streaming interfaces, it is difficult to obtain performance indicators such as the first token time and the number of tokens per second, and it is impossible to fully evaluate the performance of large model streaming interfaces in high-concurrency scenarios.

[0003] How to comprehensively evaluate the performance of large model streaming interfaces in high-concurrency scenarios is a technical problem that needs to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a performance stress testing system and method applied to large model streaming interfaces to solve the technical problem of how to comprehensively evaluate the performance of large model streaming interfaces in high concurrency scenarios.

[0005] In a first aspect, the present invention provides a performance stress testing system for a large model streaming interface, including a test data management module, a concurrent access simulation module, a performance monitoring module, a fault-tolerant processing module, a post-processor, and a result output module;

[0006] The test data management module is used to construct a test data set based on the data type and business scenario that the streaming interface needs to process, and to manage the test data set;

[0007] The concurrent access simulation module is used to dynamically configure concurrent access parameters and simulate and generate concurrent access scenarios for multiple users;

[0008] The performance monitoring module is used to monitor and record the large model streaming interface in the concurrent access scenario based on the customized monitoring indicators and monitoring items to obtain monitoring data;

[0009] The fault-tolerant processing module analyzes and processes the abnormal conditions of the large model interface in the concurrent access scenario based on the predefined exception handling strategy, and forms an error log file, which records the abnormal conditions and the handling process and results of the abnormal conditions through the error log file;

[0010] The post-processing module is used to calculate performance indicators based on monitoring data;

[0011] The result output module is used to construct a test result file based on the error log file and the performance index, and output the test result file.

[0012] Preferably, the test data management module interacts with the user through a test data management interface to provide the following services:

[0013] Supports users to import predefined text files through the data management interface or import text files from authorized third-party interfaces, and supports the import of test data in multiple formats, including JSON and CSV;

[0014] Supports users to filter the test data in the imported text file to obtain test data that matches the quantity type and business scenario to be processed, and randomly select test data from the matched test data to build a test data set;

[0015] Provides a data caching mechanism for imported text files and constructed test data sets.

[0016] Preferably, the concurrent access control module interacts with the user through a concurrent access control interface to provide the following services:

[0017] Support users to dynamically configure concurrent access parameters through the concurrent access control interface, including the number of user requests, the IP address and port of each user request, the user generation rate, the test duration, the user request interval time, and the interface URL of the large model interface to be tested;

[0018] Simulate concurrent access scenarios based on configured concurrent access parameters, and perform concurrency control on user requests in concurrent access scenarios through multi-threading or multi-process technology;

[0019] Based on the user request retry mechanism, ensure that each user request can be responded to;

[0020] Record the sending and receiving time of each user request.

[0021] The performance monitoring module interacts with the user through the performance monitoring interface to provide the following services:

[0022] Custom monitoring indicators, which include at least first token time, average number of tokens per second, average number of characters per second, response time, and throughput;

[0023] Based on custom monitoring indicators and custom monitoring items, add new monitoring indicators as needed;

[0024] When performing performance stress testing on large model streaming interfaces in concurrent scenarios, monitor data is collected and stored in real time;

[0025] The monitoring data is displayed in real time through the performance monitoring interface.

[0026] Preferably, the fault-tolerant processing module interacts with the user through a fault-tolerant processing interface to provide the following services:

[0027] Users can customize the abnormal situations that may occur in the large model streaming interface during the user's access request through the fault-tolerant processing interface, and configure the exception handling strategy;

[0028] During the performance stress test of the large model streaming interface in a concurrent scenario, the exceptions that occur during the user access request process of the large model streaming interface are captured and recorded, and the exception type and occurrence time corresponding to the exception are recorded;

[0029] Handle the abnormal situations based on the exception handling strategy, retry the user request for recoverable abnormal situations based on the automatic retry mechanism, and record the handling process and results of the abnormal situations;

[0030] If the error caused by the abnormal situation can be recovered based on the retry mechanism, the processing result is a test success. If the error caused by the abnormal situation exceeds expectations, the processing result is a test failure, and the performance stress test of the large model streaming interface in the current concurrent scenario is interrupted;

[0031] Generate an error log file based on the exception type, occurrence time, resulting error, processing process, and processing results corresponding to the exception situation.

[0032] Preferably, the post-processing module stores the calculated key performance indicators in a memory, the key performance indicators including first token time, average number of tokens per second, and average number of characters per second;

[0033] When calculating the first token time, calculate the time from when the user request is sent to when the first token is received;

[0034] When calculating the average number of tokens per second, calculate the total number of tokens divided by the test time;

[0035] To calculate the average characters per second, take the total number of characters and divide it by the test time.

[0036] Preferably, the result output module interacts with the user through a result output interface to provide the following services:

[0037] The result output interface supports users to customize the output format, path and name of the test result file;

[0038] For a successful performance stress test, output all performance indicators and save them to the sqlite3 database;

[0039] For failed performance stress tests, output error log files;

[0040] Read the data in sqlite3 and store it in a predefined data file. The format of the data file includes EXCEL.

[0041] Output the data file and error log file as test result files according to the predefined name and path.

[0042] In a second aspect, the present invention provides a performance stress testing method applied to a large model streaming interface. The performance stress testing of a large model streaming interface is implemented by a performance stress testing system applied to a large model streaming interface as described in any one of the first aspects. When executing the performance stress testing, concurrent access parameters are configured to simulate concurrent access scenarios of multiple users, and performance stress testing is performed on concurrent user access requests of a specified large model streaming interface under the concurrent access scenario.

[0043] The performance stress testing system method applied to a large model streaming interface of the present invention has the following advantages:

[0044] 1. Comprehensiveness: Ability to comprehensively evaluate the performance of large model streaming interfaces in high-concurrency scenarios;

[0045] 2. Automation: The testing process is highly automated, reducing manual intervention and improving testing efficiency;

[0046] 3. Scalability: It supports testing requirements of different scales and types and has good scalability.

[0047] 4. Real-time: Ability to monitor and record performance indicators in real time and detect performance problems in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] The present invention is further described below in conjunction with the accompanying drawings.

[0050] Figure 1 This is a flowchart of the workflow of a performance stress testing system applied to a large model streaming interface in Example 1. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0052] The embodiments of the present invention provide a performance stress testing system and method applied to a large model streaming interface, which are used to solve the problem of how to comprehensively evaluate the performance of a large model streaming interface in a high concurrency scenario.

[0053] Embodiment 1:

[0054] The present invention discloses a performance stress testing system applied to a large model streaming interface, comprising a test data management module, a concurrent access simulation module, a performance monitoring module, a fault-tolerant processing module, a post-processor and a result output module.

[0055] The test data management module is used to build test data sets based on the data types and business scenarios that the streaming interface needs to process, and to manage the test data sets.

[0056] As a specific implementation of the test data management module, this module interacts with users through the test data management interface to provide the following services:

[0057] (1) Supports users to import predefined text files through the data management interface or import text files from authorized third-party interfaces, and supports the import of test data in multiple formats, including JSON and CSV;

[0058] (2) Support users to filter the test data in the imported text file to obtain test data that matches the quantity type and business scenario to be processed, and randomly select test data from the matched test data to construct a test data set, ensuring that the data used in each test is random and avoiding result deviation;

[0059] (3) Provide a data caching mechanism for imported text files and constructed test data sets to improve data reading efficiency.

[0060] The concurrent access simulation module is used to dynamically configure concurrent access parameters and simulate the generation of concurrent access scenarios for multiple users.

[0061] As a specific implementation of the concurrent access control module, this module simulates multiple clients sending requests to the streaming interface at the same time to test its performance under high concurrency conditions. Specifically, it provides the following services:

[0062] (1) Support users to dynamically configure concurrent access parameters through the concurrent access control interface, including the number of user requests, the IP address and port of each user request, the user generation rate, the test duration, the user request interval time, and the interface URL of the large model interface to be tested;

[0063] (2) Simulate concurrent access scenarios based on configured concurrent access parameters and perform concurrency control on user requests in concurrent access scenarios through multi-threading or multi-process technology;

[0064] (3) Based on the user request retry mechanism, ensure that each user request can be responded to;

[0065] (4) Record the sending and receiving time of each user request to facilitate subsequent analysis.

[0066] The performance monitoring module is used to monitor and record the large model streaming interface in concurrent access scenarios based on customized monitoring indicators and monitoring items to obtain monitoring data.

[0067] As a specific implementation of the performance monitoring module, this module monitors and records the performance indicators of the streaming interface in real time, supports custom monitoring items, and provides the following services:

[0068] (1) Custom monitoring indicators, which include at least first token time, average number of tokens per second, average number of characters per second, response time, and throughput;

[0069] (2) Based on customized monitoring indicators and customized monitoring items, new monitoring indicators can be added as needed;

[0070] (3) When performing performance stress testing of large model streaming interfaces in concurrent scenarios, real-time monitoring data is collected and stored to ensure data integrity and accuracy;

[0071] (4) The monitoring data is displayed in real time through the performance monitoring interface, making it easier to observe and analyze the monitoring data.

[0072] The fault-tolerant processing module analyzes and processes the abnormal situations of the large model interface in the concurrent access scenario based on the predefined exception handling strategy, and forms an error log file, which records the abnormal situations and the handling process and results of the abnormal situations.

[0073] As a specific implementation of the fault-tolerant processing module, this module handles various abnormal situations that may occur during interface access to ensure the stability and reliability of the test process. Specifically, it provides the following services:

[0074] (1) Users customize the abnormal situations that may occur in the large model streaming interface during the user's access request process through the fault-tolerant processing interface, and configure the exception handling strategy;

[0075] (2) During the performance stress test of the large model streaming interface in a concurrent scenario, capture and record the abnormal situations that occur in the large model streaming interface during the user access request process, and record the corresponding abnormal type and occurrence time of the abnormal situation;

[0076] (3) Handle the abnormal situations that occur based on the exception handling strategy, and retry the user request for recoverable abnormal situations based on the automatic retry mechanism, and record the handling process and results of the abnormal situations;

[0077] (4) If the error caused by the abnormal situation can be recovered based on the retry mechanism, the processing result is a test success. If the error caused by the abnormal situation exceeds expectations, the processing result is a test failure, and the performance stress test of the large model streaming interface in the current concurrent scenario is interrupted;

[0078] (5) Generate an error log file based on the exception type, occurrence time, resulting error, processing process, and processing result corresponding to the exception situation. The error log file can be used to locate the problem for easy file resolution.

[0079] The post-processing module is used to calculate performance indicators based on monitoring data.

[0080] As a specific implementation of the post-processing module, this module calculates self-defined performance indicators, especially key performance indicators, based on the monitoring data collected during the performance stress test, and stores the calculated performance indicators in the memory for subsequent output.

[0081] Among them, the key performance indicators include the first token time, the average number of tokens per second, and the average number of characters per second. When calculating the first token time, calculate the time from the user request being sent to the first token being received; when calculating the average number of tokens per second, calculate the total number of tokens divided by the test time; when calculating the average number of characters per second, calculate the total number of characters divided by the test time.

[0082] The result output module is used to construct a test result file based on the error log file and performance indicators, and output the test result file.

[0083] As a specific implementation of the result output module, this module outputs the test results, including performance indicators and error information, and saves the results to a file. The specific operations are as follows:

[0084] (1) Support users to customize the output format, path and name of test result files through the result output interface;

[0085] (2) For successful performance stress tests, output all performance indicators and save them in the sqlite3 database;

[0086] (3) For failed performance stress tests, output error log files;

[0087] (4) Read the data in sqlite3 and store it in a predefined data file, the format of the data file includes EXCEL;

[0088] (5) Output the data file and error log file as test result files according to the predefined name and path.

[0089] This system solves the problem that traditional test tools cannot obtain the first token time and the average number of tokens per second in the performance stress test of large-scale streaming interfaces. When the system is working, the functions of the test data management module, concurrent access simulation module, performance monitoring module, fault-tolerant processing module, post-processor and result output module in the system are coded or scripted. Manually, you only need to fill in the basic information (IP address, PORT port, interface URL, test data) and execution information (planned execution time, number of concurrent users) and click to execute. The execution results can directly view the average response time, first token time, average number of tokens per second, and average number of characters per second.

[0090] Based on this system, the performance stress test process of the large model streaming interface is as follows: Figure 1 As shown:

[0091] (1) Setting concurrent access parameters through the concurrent access simulation interface, including the number of user requests, the IP address and port of each user request, the user generation rate, the test duration, the user request interval, and the interface URL of the large model interface to be tested, thereby determining the streaming interface of the large model to be tested and triggering the concurrent access scenario based on the configured parameters;

[0092] (2) Execute the script;

[0093] (3) Obtain test data;

[0094] (4) Access the specified large model streaming interface to perform performance stress testing;

[0095] (5) If the large model streaming interface can be accessed, the monitoring data is recorded and the performance indicators are calculated based on the monitoring data. The obtained performance indicators are stored in the data file. If the access fails, the log file is recorded and the data file and the log file are output as the test result file.

[0096] Embodiment 2:

[0097] The present invention provides a performance stress testing method applied to a large model streaming interface, characterized in that the performance stress testing of the large model streaming interface is implemented by the system disclosed in Example 1. When executing the performance stress testing, concurrent access parameters are configured to simulate concurrent access scenarios of multiple users, and performance stress testing is performed on concurrent user access requests of the specified large model streaming interface under the concurrent access scenario.

[0098] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A performance stress testing system for a large model streaming interface, characterized in that: It includes test data management module, concurrent access simulation module, performance monitoring module, fault-tolerant processing module, post-processor and result output module; The test data management module is used to construct a test data set based on the data type and business scenario that the streaming interface needs to process, and to manage the test data set; The concurrent access simulation module is used to dynamically configure concurrent access parameters and simulate and generate concurrent access scenarios for multiple users; The performance monitoring module is used to monitor and record the large model streaming interface in the concurrent access scenario based on the customized monitoring indicators and monitoring items to obtain monitoring data; The fault-tolerant processing module analyzes and processes the abnormal conditions of the large model interface in the concurrent access scenario based on the predefined exception handling strategy, and forms an error log file, which records the abnormal conditions and the handling process and results of the abnormal conditions through the error log file; The post-processing module is used to calculate performance indicators based on monitoring data; The result output module is used to construct a test result file based on the error log file and the performance index, and output the test result file.

2. The performance stress testing system for large model streaming interface according to claim 1 is characterized in that: The test data management module interacts with the user through the test data management interface to provide the following services: Supports users to import predefined text files through the data management interface or import text files from authorized third-party interfaces, and supports the import of test data in multiple formats, including JSON and CSV; Supports users to filter the test data in the imported text file to obtain test data that matches the quantity type and business scenario to be processed, and randomly select test data from the matched test data to build a test data set; Provides a data caching mechanism for imported text files and constructed test data sets.

3. The performance stress testing system for large model streaming interface according to claim 1 is characterized in that: The concurrent access control module interacts with the user through the concurrent access control interface to provide the following services: Support users to dynamically configure concurrent access parameters through the concurrent access control interface, including the number of user requests, the IP address and port of each user request, the user generation rate, the test duration, the user request interval time, and the interface URL of the large model interface to be tested; Simulate concurrent access scenarios based on configured concurrent access parameters, and perform concurrency control on user requests in concurrent access scenarios through multi-threading or multi-process technology; Based on the user request retry mechanism, ensure that each user request can be responded to; Record the sending and receiving time of each user request.

4. The performance stress testing system for large model streaming interface according to claim 1 is characterized in that: The performance monitoring module interacts with the user through the performance monitoring interface to provide the following services: Custom monitoring indicators, which include at least first token time, average number of tokens per second, average number of characters per second, response time, and throughput; Based on custom monitoring indicators and custom monitoring items, add new monitoring indicators as needed; When performing performance stress testing on large model streaming interfaces in concurrent scenarios, monitor data is collected and stored in real time; The monitoring data is displayed in real time through the performance monitoring interface.

5. The performance stress testing system for large model streaming interface according to claim 1 is characterized in that: The fault-tolerant processing module interacts with the user through the fault-tolerant processing interface to provide the following services: Users can customize the abnormal situations that may occur in the large model streaming interface during the user's access request through the fault-tolerant processing interface, and configure the exception handling strategy; During the performance stress test of the large model streaming interface in a concurrent scenario, the exceptions that occur during the user access request process of the large model streaming interface are captured and recorded, and the exception type and occurrence time corresponding to the exception are recorded; Handle the abnormal situations based on the exception handling strategy, retry the user request for recoverable abnormal situations based on the automatic retry mechanism, and record the handling process and results of the abnormal situations; If the error caused by the abnormal situation can be recovered based on the retry mechanism, the processing result is a test success. If the error caused by the abnormal situation exceeds expectations, the processing result is a test failure, and the performance stress test of the large model streaming interface in the current concurrent scenario is interrupted; Generate an error log file based on the exception type, occurrence time, resulting error, processing process, and processing results corresponding to the exception situation.

6. The performance stress testing system for large model streaming interface according to claim 1 is characterized in that: The post-processing module stores the calculated key performance indicators in the memory, the key performance indicators including the first token time, the average number of tokens per second, and the average number of characters per second; When calculating the first token time, calculate the time from when the user request is sent to when the first token is received; When calculating the average number of tokens per second, calculate the total number of tokens divided by the test time; To calculate the average characters per second, take the total number of characters and divide it by the test time.

7. The performance stress testing system for large model streaming interface according to claim 1 is characterized in that: The result output module interacts with the user through the result output interface to provide the following services: The result output interface supports users to customize the output format, path and name of the test result file; For a successful performance stress test, output all performance indicators and save them to the sqlite3 database; For failed performance stress tests, output error log files; Read the data in sqlite3 and store it in a predefined data file. The format of the data file includes EXCEL. Output the data file and error log file as test result files according to the predefined name and path.

8. A performance stress testing method applied to a large model streaming interface, characterized in that: The performance stress test of the large model streaming interface is implemented by a performance stress testing system applied to the large model streaming interface as described in any one of claims 1 to 7. When executing the performance stress test, the concurrent access parameters are configured to simulate the concurrent access scenario of multiple users, and the performance stress test is performed on the concurrent user access requests of the specified large model streaming interface under the concurrent access scenario.

Images