Large language model system pressure test method and computing device

Abstract

The embodiment of the invention provides a large language model system pressure testing method and computing equipment. The large language model system pressure test method comprises the steps of performing length statistical analysis on historical input texts of a target user, and determining the proportion of the input texts with various lengths; selecting pressure test texts from the text database, and constructing a pressure test text set by using the pressure test texts; wherein the proportion of the pressure test texts with various lengths in the pressure test text set is the proportion; and generating a model calling request based on the pressure test text set, performing pressure test on the large language model system by using the model calling request, and obtaining pressure test index data of the large language model system according to test process statistical data. The pressure test text set constructed according to the method is more in line with the actual application scene condition, and the pressure test index data obtained by correspondingly adopting the pressure test text set to carry out the pressure test can also reflect the performance characteristics of a large language model system which has a large number of concurrent requests in the actual scene.

Description

Technical Field

[0001] The present disclosure relates to the technical field of software testing, and in particular to a large language model system stress testing method and computing device. Background Art

[0002] Before the large language model system is officially commercially deployed, reasonable stress testing is required to verify the response performance and reliability of the model in high-concurrency and variable text input scenarios. The stress process needs to consider the unity of the response performance and reliability of the large language model system. However, most of the existing stress tests are only aimed at maintaining the reliability of the large language model system under high traffic pressure. Therefore, the test texts used in the test process are mostly long texts or very long texts, and the test results obtained are only the indicator data reflected in the aforementioned long text scenario, and cannot truly reflect whether the performance indicators in the actual application scenario are met. Summary of the invention

[0003] In order to solve the problem that the indicator data obtained by the existing stress testing method cannot reflect the performance indicators in the actual application scenario, the embodiments of the present disclosure provide a large language model system stress testing method and computing device.

[0004] In a first aspect, an embodiment of the present disclosure provides a large language model system stress testing method, including:

[0005] Perform statistical analysis on the length of historical input texts of target users to determine the proportion of input texts of various lengths;

[0006] Selecting stress testing texts from a text database, and using the stress testing texts to construct a stress testing text set; wherein the proportion of stress testing texts of various lengths in the stress testing text set is the proportion ratio;

[0007] A model call request is generated based on the stress test text set, the large language model system is stress tested using the model call request, and stress test indicator data of the large language model system is obtained according to statistical data of the test process.

[0008] Optionally, before performing stress testing on the large language model system based on the stress testing text set, the method further includes:

[0009] Calculate the number of converted users based on the conversion ratio of target users and the pre-estimated target user volume, and estimate the number of concurrent users based on the model usage time habits of target users and the number of converted users;

[0010] Determine the number of stress testing concurrent terminals according to the number of concurrent users and the number of stress testing texts in the stress testing text set;

[0011] The generating a model call request based on the stress test text set, and using the model call request to perform stress testing on the large language model system, includes:

[0012] Building multiple concurrent clients based on the number of concurrent terminals in the stress test;

[0013] Each concurrent client is used to simultaneously load the stress testing text in the stress testing text set and generate a model call request, and the model call request is sent to the large language model system.

[0014] Optionally, generating a model call request based on the stress testing text set includes:

[0015] Standardizing the stress testing texts in the stress testing text set according to the template format to obtain standardized stress testing texts;

[0016] Use the standardized stress testing text to generate model call requests.

[0017] Optionally, before generating a model call request based on the stress testing text set, the method further includes:

[0018] According to the stop word file, stop words are removed from the stress testing text.

[0019] Optionally, obtaining stress testing indicator data of the large language model system according to test process statistical data includes:

[0020] Record the time when the model call request is sent;

[0021] Record the time when the response data for each model call request is received;

[0022] Calculate the response processing time based on the corresponding request sending time and response receiving time;

[0023] The response processing time of all stress test texts is counted to obtain stress test indicator data of the large language model.

[0024] Optionally, the recording of the time of receiving a response to the processing response data for each model call request includes: recording the time of receiving a response to the first byte feedback for each model call request; and / or recording the time of receiving a response to the last byte feedback for each model call request;

[0025] Determining the response processing duration of the corresponding type according to the corresponding request sending time and the response receiving time includes: determining the first byte response duration according to the corresponding request sending time and the response receiving time of the first byte feedback; and / or determining the complete request processing duration according to the corresponding request sending time and the response receiving time of the last byte feedback;

[0026] The response processing time of all stress testing texts is counted to obtain the stress testing index data of the large language model, including: counting the first byte response time of all stress testing texts to obtain corresponding first byte index data; and / or counting the request processing time of all stress testing texts to obtain index data corresponding to the complete request processing time.

[0027] Optionally, it also includes: obtaining standard response data corresponding to each stress test text, and counting the number of standard response characters contained in the standard response data; the standard response data is response data obtained by the large language model system processing the stress test text in a non-stress test situation;

[0028] Obtain stress test response data obtained by the large language model system processing the stress test text during the stress test, and count the actual number of response characters contained in the stress test response data;

[0029] Calculate the response distortion ratio based on the standard response character count and the actual response character count for each stress test text.

[0030] Perform statistical analysis on the response distortion ratios of all stress test texts to obtain the corresponding distortion index data.

[0031] Optionally, it also includes: obtaining reasonable time response data corresponding to each stress test text, and counting the number of ideal response characters contained in the reasonable time response data, wherein the reasonable time response data is response data obtained within a reasonable waiting time after sending a model call request to the large language model system in a non-stress test situation;

[0032] Obtain stress test response data obtained by the large language model system processing the stress test text within a reasonable waiting time during the stress test, and count the actual number of response characters contained in the stress test response data;

[0033] Calculate the response delay index based on the ideal number of corresponding characters and the actual number of response characters corresponding to each stress test text;

[0034] The response delay index of all stress test texts is counted to obtain the response delay indicator data.

[0035] In a second aspect, the present disclosure provides a stress testing device for a large language model system, including:

[0036] A statistical processing unit, used to perform statistical analysis on the length of the input text of the target user and determine the proportion of input texts of various lengths;

[0037] The test data set construction unit is used to select stress testing texts from the text database and construct a stress testing text set using the stress testing texts; wherein the proportion of stress testing texts of various lengths in the stress testing text set is the proportion of input texts of corresponding lengths;

[0038] A test processing unit is used to generate a model call request based on the stress test text set, use the model call request to perform stress testing on the large language model system, and obtain stress testing indicator data of the large language model system according to test process statistical data.

[0039] In a third aspect, an embodiment of the present disclosure provides a computing device, including a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the large language model system stress testing method as described above.

[0040] The embodiment scheme of the present disclosure determines the proportion of input texts of various lengths based on the historical input texts of the target users for the large language model system, constructs a stress test text set based on the aforementioned proportions, and uses the constructed stress test text set for testing. Because the proportions of input texts of various lengths in the aforementioned stress test text set are determined based on the historical input texts of the target users, it can characterize the overall input feature attributes of the target users. The stress test text set constructed according to the aforementioned method is more in line with actual application scenarios, and the corresponding stress test indicator data obtained by using it for stress testing can also better reflect the performance characteristics of the large language model system in actual scenarios with a large number of concurrent requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work, including:

[0043] Figure 1 is a flow chart of a large language model system stress testing method provided by an embodiment of the present disclosure;

[0044] Figure 2 is a flow chart of a method for determining a model response distortion index in some embodiments of the present disclosure;

[0045] Figure 3 is a flow chart of a method for determining a model procrastination indicator index according to some embodiments of the present disclosure;

[0046] Figure 4 is a schematic diagram of a large language model system stress testing device provided by some embodiments of the present disclosure;

[0047] Figure 5 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0049] The term "including" and its variations used in this document are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. In this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0050] In order to solve the problem that in the existing stress testing of large language model systems, long text or very long text is used as stress testing text, and it is unable to truly reflect whether the large language model meets the performance indicators in actual application scenarios, the present disclosure implements a new stress testing method for large language model systems.

[0051] Figure 1 : is a flow chart of a large language model system stress testing method provided by an embodiment of the present disclosure. Figure 1 As shown, the large language model system stress testing method provided by the embodiment of the present disclosure includes S110-S130.

[0052] Before analyzing the method of the embodiment of the present disclosure, the large language model system is first explained. The large language model system is a system consisting of a large language model algorithm program, a model engine program, hardware devices for executing the aforementioned programs, and auxiliary programs (such as load balancing algorithm programs, cache algorithm programs, and underlying data call programs) for realizing the normal operation of system functions.

[0053] S110: Perform length statistics analysis on the target user's historical input texts to determine the proportions of input texts of various lengths.

[0054] The target user is the target user when designing the large language model, that is, the user who uses the text generation function provided by the large language model. For example, in the case of a large language model created based on scientific data, the target user is the user who searches and retrieves scientific information, or the user who acquires scientific knowledge; in the case of a large language model for foreign language learning, the target user is the learner of the corresponding foreign language.

[0055] In actual applications, before the large language model system is truly commercially deployed, a small-scale test version (Beta version) may be deployed to test the basic functions and user experience of the system. In the process of using the aforementioned test version to test basic functions and user experience, system developers can also obtain a large amount of historical input text entered by target users (equivalent to questions asked to the large language model).

[0056] The amount of historical input text data of the aforementioned target user is huge and typical. Therefore, a statistical analysis can be performed based on the aforementioned input text to determine the length characteristics of the target user's input text. Based on this, the embodiment of the present disclosure performs a length statistical analysis on the input text of the target user to determine the proportion of input texts of various lengths.

[0057] In addition, even if there is no deployed test version of the aforementioned large language model, it is also possible to obtain search data entered by a specific type of user (target user) in the search engine from a search engine provider through commercial means, and then determine the proportion of texts of various lengths entered by the target users based on the aforementioned search data.

[0058] S120: Select stress testing texts from the text database, and use the stress testing texts to construct a stress testing text set; wherein the proportion of stress testing texts of various lengths in the stress testing text set is a proportion ratio.

[0059] The text database is a pre-built database for storing a large number of stress test texts. The text database stores a large number of stress test texts of different lengths. After determining the proportion of input texts of various lengths, the actual stress test text (stress test text) can be selected from the text database according to the aforementioned proportion, so that the proportion of stress test texts of various lengths is the aforementioned proportion. Using the aforementioned selected stress test texts, a stress test text set can be formed.

[0060] S130: Generate a model call request based on the stress test text set, use the model call request to perform stress testing on the large language model system, and obtain stress testing indicator data of the large language model system based on statistical data of the test process.

[0061] After the stress test text set is determined, the basic test data for stress testing is determined. Subsequently, a model call request can be generated based on the stress test text set to stress test the large language model system and obtain stress test indicator data that characterizes the performance indicators of the large language model under stress testing.

[0062] The aforementioned model call request is generated by the test end and is used to trigger the large language model to execute the corresponding model task request. In the specific implementation, the model call request includes the interface address (API) of the large language model system, so that the corresponding model call request can be sent to the correct large language model system. In addition, the model call request also includes a request body formulated in accordance with the standard protocol rules (interface protocol rules formulated by the large language model), and the request body contains the stress test text in the aforementioned stress test text set, so that the large language model system can correctly obtain the stress test text.

[0063] By sending the model call request generated by the stress test text to the large language model system in a short period of time, the large language model system can be triggered to receive a large number of user requests that need to be processed simultaneously in a short period of time. Correspondingly, the large language model system needs to process the aforementioned model call request according to its own pre-set rule strategy, and then generate the corresponding response and return it to the test end.

[0064] In the process of executing the aforementioned request sending task and receiving the response to the request, the test end can obtain the test process statistics. According to the aforementioned test process statistics, the corresponding indicator statistics can be performed to obtain the stress test indicator data that characterizes the performance of the large language model system under stress testing.

[0065] As analyzed above, the embodiment of the present disclosure determines the proportion of input texts of various lengths based on the historical input texts of the target users for the large language model system, constructs a stress test text set based on the aforementioned proportions, and uses the constructed stress test text set for testing. Because the proportions of input texts of various lengths in the aforementioned stress test text set are determined based on the historical input texts of the target users, it can characterize the overall input feature attributes of the target users. Accordingly, the stress test text set constructed according to the aforementioned method is also more in line with the actual application scenario, and the corresponding stress test indicator data obtained by using it for stress testing can also better reflect the performance characteristics of the large language model system in actual scenarios with a large number of concurrent requests.

[0066] The essence of stress testing is to send a large number of concurrent test requests to the system under test in a short period of time to test whether the system under test can bear and digest the concurrent test requests in a short period of time. What stress testing needs to explore is whether the system under test can operate normally without downtime and crashing in the case of a large number of concurrent requests in actual business scenarios, and even whether it can provide high-quality services to users. Accordingly, it is necessary to set a reasonable number of concurrent requests during the stress test to simulate the random concurrency that occurs in actual scenarios. Based on this consideration, in the embodiment of the present disclosure, before executing the aforementioned S130, the following S140-S150 can also be executed.

[0067] S140: Calculate the number of converted users according to the conversion ratio of target users and the pre-estimated target user volume, and estimate the number of concurrent users according to the model usage time habits of target users and the number of converted users.

[0068] As analyzed above, during the early stage of the system test version delivery, the conversion rate of target users can be calculated based on the delivery volume of target users and the number of target users who actually use the system test version. Similarly, an estimated target user volume can be obtained through early time surveys, industry data analysis, and advertising delivery. By multiplying the aforementioned target user conversion rate and target user volume, the number of converted users in the formal commercial delivery process can be obtained.

[0069] At the same time, during the release of the aforementioned system test version, the model usage time habits of the target users can be obtained based on the log data of a large number of target users using the test version. The number of concurrent users can be estimated based on the aforementioned model usage time habits and the number of converted users. For example, through the target users' model usage time habits, it was found that 20% (this data proportion is the largest usage time habit proportion) of users will use the large language model system at 8 pm. Correspondingly, according to 20% and the number of converted users, the number of concurrent users can also be obtained. The aforementioned number of concurrent users corresponds to the number of concurrent requests of the large language model system basically under maximum load.

[0070] In actual applications, stress testing requires a large number of concurrent test requests, which are mostly formed by concurrent virtual clients on the test host, specifically by creating multiple concurrent processes or concurrent threads. Based on this conventional application, after executing S140 to obtain the estimated number of concurrent users, the embodiment of the present disclosure will also execute the following S150.

[0071] S150: Determine the number of concurrent terminals for stress testing according to the number of concurrent users and the number of stress testing texts in the stress testing text set.

[0072] Specifically, the number of stress test concurrent terminals is determined based on the number of concurrent users and the number of stress test texts in the stress test text set. First, the number of times a concurrent terminal will load the stress test text set is determined, and then the number of concurrent terminals is determined based on the aforementioned number of concurrent users, number of loads, and number of stress test texts in the test text set. It is assumed here that in a very short period of time, the target user will not input a new request before receiving feedback from the large language model system. Of course, in actual applications, the user's input frequency characteristics can also be determined based on the historical input text features of the target user, and then the number of concurrent terminals can be determined.

[0073] When executing the aforementioned S140-S150, the aforementioned S130 may specifically include the following S131-S132.

[0074] S131: Build multiple concurrent clients based on the number of concurrent terminals in the stress test.

[0075] S132: Use each concurrent client to simultaneously load the stress testing text in the stress testing text set and generate a model call request, and send the model call request to the large language model system.

[0076] In the specific implementation, the concurrent client is built according to the number of stress test concurrent terminals, that is, a virtualized stress test client is built on the test host according to the number of stress test concurrent terminals, and each stress test client loads the stress test text set according to the indicated low value, so as to generate a model call request according to the set protocol, and send the model call request to the large language model at the same time.

[0077] In some embodiments, the aforementioned S130 in generating a model call request based on the stress testing text set may specifically include the following S1321-S1322.

[0078] S1321: Standardize the stress testing text in the stress testing text set according to the template format to obtain a standardized stress testing text.

[0079] S1322: Generate a model call request using the standardized stress testing text.

[0080] The template format is a standardized format set according to the model interface of a specific large language model. Controlling the concurrent client to perform standardized processing on the stress test text in the stress test text set according to the template format to obtain the standardized stress test text is the process of simulating the user end to process the user input text according to the template format before sending the model call request. The standardized stress test text after standardization is more in line with the processing requirements of the large language model, and the speed of processing it by the corresponding large language model system can also be accelerated. For example, in some large language model systems that use prompt engineering, the stress test text can be processed in response to the prompt engineering template format to obtain a stress test text with questions and prompt content.

[0081] After obtaining the standardized stress test text, it is possible to generate model call requests based on the standardized stress test text. In specific implementation, each concurrent client can load the standardized stress test text to generate model call requests.

[0082] In some embodiments, before performing the foregoing S130, the following S160 may also be performed.

[0083] S160: According to the stop word file, perform stop word removal processing on the stress test text.

[0084] In practical applications, some stress test texts contain stop words that have no actual semantics but affect the processing speed of the large language model or the semantic understanding, such as "on", "of", etc. By removing the stop words, the stress test texts in the stress test text set can be simplified, thereby improving the processing efficiency of the large language model system.

[0085] As analyzed above, during the process of performing a stress test on the large language model, it is necessary to obtain the stress test index data of the large language model based on the statistical data during the test process. In specific implementation, the process of obtaining the stress test index data of the large language model includes S210 - S240.

[0086] S210: Record the request sending time when sending the model call request.

[0087] S220: Record the response receiving time when receiving the processing response data for each model call request.

[0088] In specific implementation, the statistical data packet form sending time of the model call request can be used as the foregoing request sending time, and the time when receiving the response packet for each model call request can be used as the response receiving time.

[0089] S230: Calculate the response processing duration according to the corresponding request sending time and response receiving time.

[0090] After obtaining the foregoing request sending time and response receiving time, subtracting the request sending time from the response receiving time can obtain the response processing duration. If precise analysis is performed, the duration between the foregoing response receiving time and request sending time includes the transmission duration of the request packet and the response packet. However, considering that the transmission durations are basically the same and there will also be transmission duration consumption in actual commercial applications, the difference between the response receiving time and the request sending time is directly used as the response processing duration here without further subdivision.

[0091] S240: Statistically analyze the response processing durations of all stress test texts to obtain the stress test index data of the large language model.

[0092] After obtaining the response processing time for each stress test text, statistical analysis is then performed on all response processing times to obtain the corresponding large language model stress test indicator data.

[0093] In a specific implementation, for a certain type of stress testing indicator data, the corresponding statistical data may include average value, P75 (75% of request time consumption), P90 (90% of request time consumption), P95 (95% of request time consumption), P99 (99% of request time consumption) and maximum time consumption.

[0094] In a specific implementation, the aforementioned S220 records the response reception time of the processing response data received for each model call request, which may include: (1) the response reception time of the first byte feedback received for each model call request; (2) the response reception time of the last byte feedback received for each model call request.

[0095] For the aforementioned (1) and (2), S230 calculates the corresponding response processing time as follows: (1-1) Determine the first byte response time based on the corresponding request sending time and the response receiving time of the first byte feedback; (1-2) Determine the request processing time based on the corresponding request sending time and the response receiving time of the last byte feedback.

[0096] With respect to the aforementioned (1-1) and (1-2), the aforementioned S240 is respectively: (a) counting the first byte response time of all stress testing texts to obtain corresponding first byte indicator data; (b) counting the request processing time of all stress testing texts to obtain indicator data corresponding to the complete request processing time.

[0097] Of course, in addition to counting the aforementioned indicator data, data such as total time consumption, maximum decoding time, minimum decoding time, etc. can also be counted based on the aforementioned indicator data or the corresponding time.

[0098] In some embodiments, for security reasons (to ensure that the large language model system does not crash), when the system's resource consumption rate is too high, some large language model systems will perform specific deletion processing on the stress test text in the model call request (for strategic considerations, for example, if some stress test texts are too long, they will adaptively only use the text segments in front of the stress test text, and discard the subsequent text segments to answer the questions corresponding to the previous text segments first). In this case, the number of response characters actually output by the large language model system may not be the same as the number of response characters that should be output ideally. From the user's perspective, the model has a distorted response (or a lazy response). During the stress test process, the aforementioned issues need to be considered and tested. Figure 2 4 is a flow chart of a method for determining a model response distortion index in some embodiments of the present disclosure. Figure 2As shown, in some embodiments, the following S310 - S330 may be performed.

[0099] S310: Obtain standard response data corresponding to each stress testing text, and count the number of standard response characters contained in the standard response data.

[0100] Standard response data is the response data obtained by the large language model system processing the stress test text in a non-stress test situation. In a specific implementation, before the stress test, the text in each text database can be model processed in a non-stress test manner to obtain and store the corresponding standard response data. After the corresponding text is used as the stress test text, the corresponding standard response data can be processed by character counting to obtain the corresponding standard response character count.

[0101] S320: Obtain stress test response data obtained by the large language model system processing the stress test text during the stress test, and count the actual number of response characters contained in the stress test response data.

[0102] After the large language model system processes the stress test text to obtain stress test response data, the stress test response data may be counted to obtain the actual number of response characters contained in the stress test response data.

[0103] S330: Calculate the response distortion ratio according to the standard response character number and the actual response character number corresponding to each stress test text.

[0104] In a specific implementation, the result of subtracting and dividing the standard response character number and the actual response character number can be used as the response distortion ratio.

[0105] S340: Perform statistical analysis on the response distortion ratios of all stress test texts to obtain corresponding distortion index data.

[0106] Performing statistical analysis on the response-distortion ratio to obtain corresponding distortion index data may be to determine the maximum response-distortion ratio, the minimum response-distortion ratio and the average response-distortion ratio.

[0107] As analyzed above, in high concurrency situations (i.e., stress testing situations), the time it takes for a large language processing model to process a single model call request becomes longer, and in some cases it becomes too long for users to tolerate. To ensure the user-friendliness of the large language model, it is necessary to evaluate whether it can generate reasonable response content within a tolerable time under high concurrency situations and obtain the corresponding response indicator index. Figure 3 is a flow chart of a method for determining a model procrastination index in some embodiments of the present disclosure. Figure 3 As shown, in some embodiments, the following S410 - S440 may be performed.

[0108] S410: Acquire reasonable time length response data corresponding to each stress test text, and count the number of ideal response characters included in the reasonable time length response data.

[0109] In the disclosed embodiment, the reasonable time length can be the time length that the user can tolerate to wait for the model to output a response. The reasonable time length response data is the response data obtained within the reasonable waiting time length after sending a model call request to the large language model system in a non-stress test situation.

[0110] In a specific implementation, before performing stress testing, the texts in each text database may be model processed in a non-stress testing manner to obtain response data of reasonable duration and count the number of ideal response characters contained therein.

[0111] S420: Obtain stress test response data obtained by the large language model system processing the stress test text within a reasonable waiting time during the stress test, and count the actual number of response characters included in the stress test response data.

[0112] In a specific implementation, the method for determining the actual number of response characters is the same as the method for determining the ideal number of corresponding characters in the previous text, and will not be repeated here.

[0113] S430: Calculate a response delay index according to the ideal number of response characters and the actual number of response characters corresponding to each stress test text.

[0114] In a specific implementation, the ideal number of response characters may be divided by the actual number of response characters to obtain the response delay index.

[0115] S440: Collect statistics on the response delay index of all stress test texts to obtain response delay index data.

[0116] Performing statistical analysis on the response delay index to obtain response delay index data may include determining the maximum response delay index, the minimum response delay index, and the average response delay index.

[0117] In addition to providing the aforementioned large language model system stress testing method, the embodiment of the present disclosure also provides a large prediction model system stress testing device. Figure 4 Schematic diagram of a large language model system stress testing device provided by some embodiments of the present disclosure. Figure 4 As shown, the large language model system stress testing device 400 includes a statistical processing unit 401 , a test data set construction unit 402 and a test processing unit 403 .

[0118] The statistical processing unit 401 is used to perform length statistical analysis on the input text of the target user and determine the proportion of input texts of various lengths.

[0119] The test data set construction unit 402 is used to select stress testing texts from the text database and construct a stress testing text set using the stress testing texts; wherein the proportion of stress testing texts of various lengths in the stress testing text set is the proportion of input texts of corresponding lengths.

[0120] The test processing unit 403 is used to generate a model call request based on the stress test text set, perform stress testing on the large language model system using the model call request, and obtain stress test indicator data of the large language model system according to the test process statistical data.

[0121] In some embodiments, the stress testing device also includes a concurrent number estimation unit. The concurrent number estimation unit is used to calculate the number of converted users based on the conversion ratio of the target users and the pre-estimated target user volume, and estimate the number of concurrent users based on the model usage time habits of the target users and the number of converted users; and determine the number of stress testing concurrent terminals based on the number of concurrent users and the number of stress testing texts in the stress testing text set. Accordingly, the test processing unit 403 constructs multiple concurrent clients based on the number of stress testing concurrent terminals, and uses each concurrent client to simultaneously load the stress testing text in the stress testing text set and generate a model call request, and sends the model call request to the large language model system.

[0122] In some embodiments, the test processing unit 403 is further used to perform standardization processing on the stress testing text in the stress testing text set according to the template format to obtain the standardized stress testing text; and then use the standardized stress testing text to generate a model call request.

[0123] In some embodiments, before generating a model call request based on the stress testing text set, the test data set construction unit 402 performs stop word removal processing on the stress testing text according to a stop word file.

[0124] In some embodiments, the test processing unit 403 records the request sending time when the model call request is sent; records the response receiving time when the processing response data for each model call request is received; and calculates the response processing time according to the corresponding request sending time and response receiving time; then, the response processing time of all stress test texts is counted to obtain the stress test indicator data of the large language model.

[0125] In some embodiments, the response reception time of the processing response data received for each model call request is recorded, including: recording the response reception time of the first byte feedback received for each model call request; and / or recording the response reception time of the last byte feedback received for each model call request. Determine the response processing time of the corresponding type according to the corresponding request sending time and the response receiving time, including: determine the first byte response time according to the corresponding request sending time and the response receiving time of the first byte feedback; and / or determine the complete request processing time according to the corresponding request sending time and the response receiving time of the last byte feedback. Statistics are taken for the response processing time of all stress test texts to obtain stress test indicator data of the large language model, including: statistics are taken for the first byte response time of all stress test texts to obtain the corresponding first byte indicator data; and / or statistics are taken for the request processing time of all stress test texts to obtain the indicator data corresponding to the complete request processing time.

[0126] In some embodiments, the test processing unit 403 obtains standard response data corresponding to each stress testing text, and counts the number of standard response characters contained in the standard response data; the standard response data is the response data obtained by the large language model system processing the stress testing text under non-stress testing conditions; then, the stress testing response data obtained by the large language model system processing the stress testing text during the stress testing process is obtained, and the actual number of response characters contained in the stress testing response data is counted; finally, the response distortion ratio is calculated according to the number of standard response characters and the actual number of response characters corresponding to each stress testing text, and the response distortion ratios of all stress testing texts are statistically analyzed to obtain corresponding distortion indicator data.

[0127] In some embodiments, the test processing unit 403 obtains reasonable time response data corresponding to each stress testing text, and counts the ideal number of response characters contained in the reasonable time response data, where the reasonable time response data is the response data obtained within a reasonable waiting time after sending a model call request to the large language model system under non-stress testing conditions; obtains stress testing response data obtained by the large language model system processing the stress testing text within a reasonable waiting time during the stress testing process, and counts the actual number of response characters contained in the stress testing response data; calculates a response delay index based on the ideal number of corresponding characters and the actual number of response characters corresponding to each stress testing text; and counts the response delay index of all stress testing texts to obtain response delay indicator data.

[0128] The embodiment of the present disclosure also provides a computing device for implementing the aforementioned method. Figure 5 Schematic diagram of the structure of the computing device provided by the embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of a computing device 500 suitable for implementing the embodiments of the present disclosure. Figure 5The computing device shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0129] like Figure 5 As shown, the computing device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 502 or a program loaded from a storage device 508 to a random access memory RAM 503. Various programs and data required for the operation of the computing device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output I / O interface 505 is also connected to the bus 504.

[0130] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a camera, a microphone, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the computing device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The computing device 500 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0131] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0132] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination of the above two.

[0133] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0135] In some embodiments, the client and system can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0136] The computer-readable medium may be included in the computing device, or may exist independently without being incorporated into the computing device.

[0137] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tester's computer, partially on the tester's computer, as a separate software package, partially on the tester's computer and partially on a remote computer, or entirely on a remote computer or system. In cases involving a remote computer, the remote computer may be connected to the tester's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0138] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system based on hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The name of the unit does not, in some cases, constitute a limitation on the unit itself. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0140] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model system stress testing method, characterized in that: include: Perform statistical analysis on the length of historical input texts of target users to determine the proportion of input texts of various lengths; Selecting stress testing texts from a text database, and using the stress testing texts to construct a stress testing text set; wherein the proportion of stress testing texts of various lengths in the stress testing text set is the proportion ratio; A model call request is generated based on the stress test text set, the large language model system is stress tested using the model call request, and stress test indicator data of the large language model system is obtained according to statistical data of the test process.

2. The pressure testing method according to claim 1, characterized in that: Before performing stress testing on the large language model system based on the stress testing text set, the method further includes: Calculate the number of converted users based on the conversion ratio of target users and the pre-estimated target user volume, and estimate the number of concurrent users based on the model usage time habits of target users and the number of converted users; Determine the number of stress testing concurrent terminals according to the number of concurrent users and the number of stress testing texts in the stress testing text set; The generating a model call request based on the stress test text set, and using the model call request to perform stress testing on the large language model system, includes: Building multiple concurrent clients based on the number of concurrent terminals in the stress test; Each concurrent client is used to simultaneously load the stress testing text in the stress testing text set and generate a model call request, and the model call request is sent to the large language model system.

3. The pressure testing method according to claim 1, characterized in that: Generate a model call request based on the stress testing text set, including: Standardizing the stress testing texts in the stress testing text set according to the template format to obtain standardized stress testing texts; Use the standardized stress testing text to generate model call requests.

4. The pressure testing method according to claim 1, characterized in that: Before generating a model call request based on the stress testing text set, the method further includes: According to the stop word file, stop words are removed from the stress testing text.

5. The pressure testing method according to any one of claims 1 to 4, characterized in that: The stress testing indicator data of the large language model system is obtained according to the statistical data of the test process, including: Record the time when the model call request is sent; Record the time when the response data for each model call request is received; Calculate the response processing time based on the corresponding request sending time and response receiving time; The response processing time of all stress test texts is counted to obtain stress test indicator data of the large language model.

6. The pressure testing method according to claim 5, characterized in that: The recording of the time of receiving the response of the processing response data for each model call request includes: recording the time of receiving the response of the first byte feedback for each model call request; and / or recording the time of receiving the response of the last byte feedback for each model call request; Determining the response processing duration of the corresponding type according to the corresponding request sending time and the response receiving time includes: determining the first byte response duration according to the corresponding request sending time and the response receiving time of the first byte feedback; and / or determining the complete request processing duration according to the corresponding request sending time and the response receiving time of the last byte feedback; The response processing time of all stress testing texts is counted to obtain the stress testing index data of the large language model, including: counting the first byte response time of all stress testing texts to obtain corresponding first byte index data; and / or counting the request processing time of all stress testing texts to obtain index data corresponding to the complete request processing time.

7. The pressure testing method according to any one of claims 1 to 4, characterized in that: Also includes: Obtain the standard response data corresponding to each stress test text, and count the number of standard response characters contained in the standard response data; The standard response data is the response data obtained by the large language model system processing the stress test text in a non-stress test situation; Obtain stress test response data obtained by the large language model system processing the stress test text during the stress test, and count the actual number of response characters contained in the stress test response data; Calculate the response distortion ratio based on the standard response character count and the actual response character count for each stress test text. Perform statistical analysis on the response distortion ratios of all stress test texts to obtain the corresponding distortion index data.

8. The pressure testing method according to claim 7, characterized in that: Also includes: Obtain the reasonable time response data corresponding to each stress test text, and count the number of ideal response characters contained in the reasonable time response data, wherein the reasonable time response data is the response data obtained within the reasonable waiting time after sending a model call request to the large language model system in a non-stress test situation; Obtain stress test response data obtained by the large language model system processing the stress test text within a reasonable waiting time during the stress test, and count the actual number of response characters contained in the stress test response data; Calculate the response delay index based on the ideal number of corresponding characters and the actual number of response characters corresponding to each stress test text; The response delay index of all stress test texts is counted to obtain the response delay indicator data.

9. A stress testing device for a large language model system, characterized in that: include: A statistical processing unit, used to perform statistical analysis on the length of the input text of the target user and determine the proportion of input texts of various lengths; The test data set construction unit is used to select stress testing texts from the text database and construct a stress testing text set using the stress testing texts; wherein the proportion of stress testing texts of various lengths in the stress testing text set is the proportion of input texts of corresponding lengths; The test processing unit is used to generate a model call request based on the stress test text set, use the model call request to perform stress testing on the large language model system, and obtain stress testing indicator data of the large language model system according to test process statistical data.

10. A computing device, characterized in that: It includes a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the large language model system stress testing method as described in any one of claims 1 to 8.

Images