Inference type large model first token performance test method, equipment and medium

Abstract

The invention discloses an inference type large model first token performance test method and device and a medium, belongs to the technical field of software development, and aims to solve the technical problem of how to truly simulate a system load in a high-concurrency environment and accurately evaluate the processing performance of a first token of a large model. Based on a first token performance test plug-in, data interaction is realized through an OkHttpClient client by adopting a streaming transmission protocol, and the method specifically comprises the following steps: capturing a receiving timestamp of a first token in an interface response stream, taking the receiving timestamp of the first token as a measurement index of first token response time, immediately terminating current stream transmission, and concurrently triggering multiple test requests through a multi-thread mechanism, the system load in a high-concurrency environment is truly simulated, and the first token processing performance of a large model is accurately evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of software development, and in particular to a method, device, and medium for testing the performance of a first token of a large inference model. Background Art

[0002] In the context of the rapid development of current computer technology, performance testing faces many challenges, especially in the performance testing of large inference models. The performance efficiency test cannot be met by traditional performance testing tools such as LoadRunner and Jmeter.

[0003] Traditional commercial testing tools such as LoadRunner and open source tools such as Jmeter collect performance data based on request-response patterns. The response of large models is based on tokens, and one response contains multiple tokens. The response time and transaction times calculated by traditional tools only consider the push of all tokens as a single transaction, which cannot reflect the actual performance of large models.

[0004] Therefore, how to truly simulate the system load in a high-concurrency environment and accurately evaluate the token processing performance of large models is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The technical task of the present invention is to provide a method, device and medium for testing the performance of the first token of an inference-based large model to solve the problem of how to truly simulate the system load in a high-concurrency environment and accurately evaluate the performance of the first token processing of a large model.

[0006] The technical task of the present invention is achieved in the following way: a reasoning-type large model first token performance test method, which is based on the first token performance test plug-in in the large model interaction scenario, and uses the OkHttpClient client to adopt the streaming transmission protocol to realize data interaction. Specifically, it captures the receiving timestamp of the first token (marker) in the interface response stream, and uses the receiving timestamp of the first token as the measurement indicator of the first token response time, and then immediately terminates the current stream transmission, and triggers multiple test requests concurrently through the multi-threaded mechanism, thereby truly simulating the system load in a high-concurrency environment and accurately evaluating the first token processing performance of the large model.

[0007] As a preferred option, the first token performance test plug-in includes a sampler (CustomSampler) and a sampler GUI configuration (CustomSamplerGui) module. The sampler GUI configuration module manages core parameters. Users configure core parameters through the GUI interface and pass the core parameters to the sampler. Jmeter further passes the current configuration items (such as the number of concurrency, the list of questions in the CSV, etc.) to the sampler. The sampler uses streaming (Streaming) based on OkHttpClient to realize interaction with the large model.

[0008] Preferably, the core parameters managed by the sampler GUI configuration module include the API Endpoint, API Key, Model Name, QA List, Token Position, Timeout, and Debug Mode.

[0009] More optimally, the sampler uses streaming transmission based on OkHttpClient to interact with the large model as follows:

[0010] Each streaming request generates an independent request body, which is implemented through the message encapsulation class;

[0011] When each thread starts, a start timestamp is recorded;

[0012] When the response content returned by the large model reaches the Token position specified in the sampler GUI configuration module, the current thread records the end timestamp and stops the current thread;

[0013] Update the start and end timestamps of SampleResult in JMeter to ensure the accuracy of test results.

[0014] Preferably, the debug mode of the method includes the following two types:

[0015] ① When debug mode = 0, the first token performance test plug-in only returns the time and content of the specified bytes, which is used for fast performance testing;

[0016] ② When debug mode = 1, the first token performance test plug-in returns the entire token location and content, which is convenient for script debugging and problem troubleshooting.

[0017] Preferably, the message encapsulation class includes ChartRequest and ChartResponse; wherein ChartRequest is used to encapsulate the request body; ChartResponse is used to receive streaming message data;

[0018] Among them, the ChartRequest data structure and the ChartResponse data structure both include field names, types, and descriptions;

[0019] The field names of the ChartRequest data structure include model, messages, temperature, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, and user;

[0020] The field names of the ChartResponse data structure include id, model, object, choices, usage, and created.

[0021] More preferably, the sampler GUI configuration module is used to provide an input interface for basic parameters in large model testing;

[0022] Among them, the basic parameters in the large model test include urlField, timeoutSpinner, moduleField, apikeyField, questionField, tkPositionField and debugField;

[0023] urlField is used to receive the large model API interface address;

[0024] timeoutSpinner is used to set the timeout unit of milliseconds for large API responses;

[0025] moduleField is used to receive the name of the large model;

[0026] apikeyField is used to receive the security key of the large model;

[0027] questionField is used to receive the input question of the large model;

[0028] tkPositionField is used to set the token position to the default first token;

[0029] debugField is used to set whether to debug mode.

[0030] Preferably, the sampler receives thread scheduling from Jmeter, implements streaming requests based on okHttpclient, captures the first byte in the byte stream, records the timestamp before each concurrent execution, and records the end timestamp after obtaining the first byte. At the same time, it updates the statistical information data to the SampleResult in Jmeter and refreshes the data in the Jmeter output report.

[0031] An electronic device comprising: a memory and at least one processor;

[0032] Wherein, the memory stores a computer program;

[0033] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the above-mentioned inference-based large model first token performance testing method.

[0034] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned inference-based large model first token performance testing method.

[0035] The inference-based large-scale model first token performance testing method, device, and medium of the present invention have the following advantages:

[0036] (1) The present invention can capture the first token in the interface response stream and use it as a metric for the first token response time, thereby realistically simulating the system load in a high-concurrency environment and accurately evaluating the first token processing performance of large models;

[0037] (2) This invention uses streaming interaction to calculate the response time of the first token, realistically simulating the performance of large models in a high-concurrency environment. This solves the problem that conventional testing tools such as LoadRunner and Jmeter cannot test the performance of large models. It also solves the problem that different projects often develop their own scripts to test the first token of large models, and different development languages ​​and code logic can lead to significant differences in the testing process.

[0038] (3) This invention solves the problem that current mainstream performance testing tools do not support large model first token performance testing;

[0039] (4) The present invention solves the problem of large errors in first token testing tools written in different languages ​​and with different logics for testing first tokens under different models;

[0040] (5) The present invention is based on the Jmeter open source framework, adopts a plug-in approach, has good scalability, and provides accurate test results;

[0041] (6) The present invention expands the original functions through plug-in development, solving the pain point that traditional performance testing tools cannot test the performance of large models. The plug-in has strong scalability and accurate test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below with reference to the accompanying drawings.

[0043] Attachment Figure 1 Schematic diagram of the first token performance testing method for large inference models. DETAILED DESCRIPTION

[0044] The inference-based large-scale model first token performance testing method, device, and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1:

[0046] As attached Figure 1 As shown, this embodiment provides an inference-type large model first token performance test method. The method is based on the first token performance test plug-in in the large model interaction scenario, and uses the streaming transmission protocol through the OkHttpClient client to realize data interaction. Specifically, the receiving timestamp of the first token (marker) in the interface response stream is captured, and the receiving timestamp of the first token is used as a measurement indicator of the first token response time, and then the current stream transmission is immediately terminated, and multiple test requests are triggered concurrently through a multi-threaded mechanism, thereby truly simulating the system load in a high-concurrency environment and accurately evaluating the first token processing performance of the large model.

[0047] The first token performance test plug-in in this embodiment includes a sampler (CustomSampler) and a sampler GUI configuration (CustomSamplerGui) module. The sampler GUI configuration module manages core parameters. Users configure core parameters through the GUI interface and pass the core parameters to the sampler. Jmeter further passes the current configuration items (such as the number of concurrency, the question list in the CSV, etc.) to the sampler. The sampler uses streaming transmission (Streaming) based on OkHttpClient to realize interaction with the large model.

[0048] The core parameters managed by the sampler GUI configuration module in this embodiment include interface address (APIEndpoint), secret key (APIKey), model name (ModelName), question and answer list (QA List), Token position (TokenPosition), timeout (Timeout) and debug mode (Debug Mode).

[0049] The sampler in this embodiment uses OkHttpClient to implement the interaction with the large model in a streaming manner as follows:

[0050] ① Each streaming request generates an independent request body, which is implemented through the message encapsulation class;

[0051] ②When each thread starts, record a start timestamp (Start Timestamp);

[0052] ③ When the response content returned by the large model reaches the Token position specified in the sampler GUI configuration module, the current thread records the end timestamp and stops the current thread;

[0053] ④Update the start and end timestamps of SampleResult in Jmeter to ensure the accuracy of test results.

[0054] The debug mode in this embodiment includes the following two types:

[0055] ① When debug mode = 0, the first token performance test plug-in only returns the time and content of the specified bytes, which is used for fast performance testing;

[0056] ② When debug mode = 1, the first token performance test plug-in returns the entire token location and content, which is convenient for script debugging and problem troubleshooting.

[0057] The message encapsulation class in this embodiment includes ChartRequest and ChartResponse; wherein, ChartRequest is used to encapsulate the request body; ChartResponse is used to receive streaming message data;

[0058] The ChartRequest data structure and the ChartResponse data structure both include field names, types, and descriptions.

[0059] The ChartRequest data structure is shown in Table 1.

[0060] Table 1 ChartRequest data structure

[0061] Field Name type illustrate model string messages List <messages> < / messages> temperature dobule n int stream boolean stop string max_tokens int presence_penalty dobule frequency_penalty dobule user string

[0062] The ChartResponse data structure is shown in Table 2.

[0063] Table 2 ChartResponse data structure

[0064] Field Name type illustrate id string model string object string choices List <chatchoice> < / chatchoice> usage ChatUsage created long

[0065] The sampler GUI configuration module is used to provide an input interface for basic parameters in large model testing;

[0066] Among them, the basic parameters in the large model test are shown in Table 3.

[0067] Table 3 Basic parameters in large model test

[0068]

[0069]

[0070] The sampler in this embodiment receives thread scheduling from Jmeter, implements streaming requests based on okHttpclient, captures the first byte in the byte stream, records the timestamp before each concurrent execution, and records the end timestamp after obtaining the first byte. At the same time, it updates the statistical information data to the SampleResult in Jmeter and realizes the data refresh of the Jmeter output report.

[0071] Part of the code logic is as follows:

[0072]

[0073]

[0074] Example 2:

[0075] This embodiment also provides an electronic device, including: a memory and a processor;

[0076] wherein the memory stores computer-executable instructions;

[0077] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the inference-type large model first token performance testing method in any embodiment of the present invention.

[0078] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0079] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0080] Example 3:

[0081] This embodiment also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the inference-based large-scale model first-token performance testing method according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0082] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0083] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0084] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0085] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing the performance of the first token of a large inference model, characterized in that: This method is based on the first token performance test plug-in in the large model interaction scenario, and uses the OkHttpClient client to adopt the streaming transmission protocol to realize data interaction. Specifically, it captures the reception timestamp of the first token in the interface response stream, and uses the reception timestamp of the first token as the measurement indicator of the first token response time. Then, it immediately terminates the current stream transmission and triggers multiple test requests concurrently through the multi-threaded mechanism, thereby truly simulating the system load in a high-concurrency environment and accurately evaluating the first token processing performance of the large model.

2. The first token performance testing method of a large inference model according to claim 1 is characterized in that: The first token performance test plug-in includes a sampler and a sampler GUI configuration module. The sampler GUI configuration module manages core parameters. Users configure core parameters through the GUI interface and pass the core parameters to the sampler. Jmeter further passes the current configuration items to the sampler. The sampler uses streaming transmission based on OkHttpClient to interact with the large model.

3. The first token performance testing method of the inference large model according to claim 2 is characterized in that: The core parameters managed by the sampler GUI configuration module include interface address, secret key, model name, question and answer list, token location, timeout period, and debug mode.

4. The method for testing the first token performance of a large inference model according to claim 2, characterized in that: The sampler uses OkHttpClient to implement streaming transmission to interact with the large model. The details are as follows: Each streaming request generates an independent request body, which is implemented through the message encapsulation class; When each thread starts, a start timestamp is recorded; When the response content returned by the large model reaches the Token position specified in the sampler GUI configuration module, the current thread records the end timestamp and stops the current thread; Update the start and end timestamps of SampleResult in JMeter to ensure the accuracy of test results.

5. The method for testing the performance of the first token of a large inference model according to claim 1 is characterized in that: The debugging mode of this method includes the following two modes: ① When debug mode = 0, the first token performance test plug-in only returns the time and content of the specified bytes, which is used for fast performance testing; ② When debug mode = 1, the first token performance test plug-in returns the entire token location and content, which is convenient for script debugging and problem troubleshooting.

6. The method for testing the performance of the first token of a large inference model according to claim 4 is characterized in that: The message encapsulation class includes ChartRequest and ChartResponse; ChartRequest is used to encapsulate the request body; ChartResponse is used to receive streaming message data; Among them, the ChartRequest data structure and the ChartResponse data structure both include field names, types, and descriptions; The field names of the ChartRequest data structure include model, messages, temperature, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, and user; The field names of the ChartResponse data structure include id, model, object, choices, usage, and created.

7. The method for testing the performance of the first token of a large inference model according to claim 2, characterized in that: The sampler GUI configuration module is used to provide an input interface for basic parameters in large model testing; Among them, the basic parameters in the large model test include urlField, timeoutSpinner, moduleField, apikeyField, questionField, tkPositionField and debugField; urlField is used to receive the large model API interface address; timeoutSpinner is used to set the timeout unit of milliseconds for large API responses; moduleField is used to receive the name of the large model; apikeyField is used to receive the security key of the large model; questionField is used to receive the input question of the large model; tkPositionField is used to set the token position to the default first token; debugField is used to set whether to debug mode.

8. The method for testing the performance of the first token of a large inference model according to claim 2, characterized in that: The sampler receives thread scheduling from Jmeter, implements streaming requests based on okHttpclient, captures the first byte in the byte stream, records the timestamp before each concurrent execution, and records the end timestamp after obtaining the first byte. At the same time, it updates the statistical information data to the SampleResult in Jmeter and refreshes the data in the Jmeter output report.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the inference-type large model first token performance testing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the inference-type large model first token performance testing method according to any one of claims 1 to 8.

Images