Method, device, computer equipment, storage medium and program product for testing large language model

Abstract

The present disclosure provides a method, device, computer equipment, storage medium and program product for testing a large language model, and relates to the field of computer and artificial intelligence. The implementation scheme is as follows: setting a target hyperparameter of a target large language model via an interactive interface server, the target hyperparameter being used to adjust a probability distribution of a word token predicted by the target large language model; performing multiple rounds of testing, for each round of testing: providing a test case to the target large language model via a first distribution unit and a second distribution unit through the interactive interface server; obtaining an unnormalized original prediction score of a word token predicted by the target large language model in response to the test case; converting the unnormalized original prediction score into a byte string; comparing the byte strings obtained in each round of the multiple rounds of testing to determine whether there is an inconsistent byte string; and in response to determining that there is an inconsistent byte string, determining that the target large language model has an output determinacy anomaly on a test link.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and artificial intelligence technology, particularly to the fields of large language models (LLM) and data testing, and especially to a method, apparatus, computer device, computer-readable storage medium, and computer program product for testing large language models. Background Technology

[0002] In recent years, the emergence of large language models (also known as large models) has greatly promoted the development of the field of artificial intelligence. Some of these large language models, which possess thinking processes, can also provide more powerful problem-solving capabilities. However, the industry's attention to and development of testing methods for large language models remains relatively limited.

[0003] Current traditional testing methods are generally based on black-box testing schemes, where questions are posed on one side of a large language model and the model's response is obtained on the other side. The test results are then determined based on the analysis of this response. However, the testing depth of this traditional method is increasingly unable to meet the demands of technological iteration. Furthermore, current traditional testing methods lack end-to-end testing mechanisms, resulting in a gap between them and real-world user scenarios. Therefore, there is an urgent need for more effective large language model testing methods to adapt to the continuous evolution of large language models themselves. Summary of the Invention

[0004] This disclosure provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for testing large language models.

[0005] According to one aspect of this disclosure, a method for testing a large language model is provided, which is executed on at least two test links, each test link deploying at least one target large language model in a large language model via an inference engine server, and each test link sequentially setting up a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface servers communicatively connected to the inference engine server, the second number being N times the first number, where N is a natural number greater than or equal to 2, wherein the method includes: setting target hyperparameters of the target large language model via the interactive interface servers, wherein the target hyperparameters are used to adjust the probability distribution of predicted words in the target large language model; executing multiple rounds of testing, for each round of testing: providing test cases to the target large language model via the first and second distribution units through the interactive interface servers; obtaining unnormalized raw prediction scores obtained by the target large language model in response to predicted words in the test cases; converting the unnormalized raw prediction scores into byte strings; comparing the byte strings obtained from each of the multiple rounds of testing to determine whether there are inconsistent byte strings; and determining, in response to determining that there are inconsistent byte strings, that the target large language model has an output deterministic anomaly on the test link.

[0006] In some embodiments, the method further includes: for at least one round of testing: obtaining the actual hyperparameters used by the target large language model in predicting lexical units in response to test cases; determining whether the actual hyperparameters are the same as the target hyperparameters; and determining, in response to the difference between the actual hyperparameters and the target hyperparameters, that the target large language model has a parameter pass-through anomaly in the test chain.

[0007] In some embodiments, the method further includes: in response to determining that at least one of an output deterministic anomaly or a parameter pass-through anomaly occurs, locating the potential location of the anomaly on the test link to the interactive interface server or the inference engine server via application programming interface debugging.

[0008] In some embodiments, providing test cases to the target large language model via the first distribution unit and the second distribution unit through the interactive interface server includes: placing test cases into a test queue via the first distribution unit; determining whether the number of test cases being processed by the target large language model is less than a preset maximum pending threshold; in response to determining that the number of test cases being processed by the target large language model is less than the maximum pending threshold, inputting test cases from the test queue into the interactive interface server via the second distribution unit; and in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold, pausing the acquisition of test cases from the test queue.

[0009] In some embodiments, test cases include prompts to guide the target large language model to output the processing results in a predetermined format.

[0010] In some embodiments, the test case includes at least one test question having a predetermined answer, wherein the method further includes: for at least one round of testing: obtaining the processing result of the target large language model in response to the test case generation from the interactive interface server; determining whether the processing result matches the predetermined answer; and determining that the target large language model has an accuracy anomaly in response to determining that the processing result does not match the predetermined answer.

[0011] According to one aspect of this disclosure, an apparatus for testing large language models is provided, for execution on at least two test links, each test link deploying at least one target large language model in a large language model via an inference engine server, and each test link sequentially providing a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface servers communicatively connected to the inference engine server, the second number being N times the first number, where N is a natural number greater than or equal to 2, wherein the apparatus includes: a hyperparameter setting module configured to set target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of predicted lexical units of the target large language model; a first test execution module configured to execute multiple rounds of testing, for each round of testing; and a test case providing module configured to provide test cases via... The first and second distribution units provide test cases to the target large language model through the interactive interface server; the first result acquisition module is configured to acquire the unnormalized raw prediction scores obtained by the target large language model in response to the predicted tokens of the test cases; the result conversion module is configured to convert the unnormalized raw prediction scores into byte strings; the first result comparison module is configured to compare the byte strings obtained from each of the multiple rounds of testing to determine whether there are inconsistent byte strings; and the first anomaly determination module is configured to determine that the target large language model has an output deterministic anomaly in the test chain in response to the determination that there are inconsistent byte strings.

[0012] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0013] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0015] According to one or more embodiments of this disclosure, a more effective testing architecture for large language models can be provided, which not only breaks through the traditional limitations of testing depth, but also is closer to user scenarios to achieve true end-to-end testing.

[0016] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0017] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 This is a schematic diagram illustrating a test link according to an exemplary embodiment.

[0019] Figure 2 This is a flowchart illustrating a method for testing a large language model according to an exemplary embodiment.

[0020] Figure 3 This is a schematic diagram illustrating the location of potential anomalies on a test link according to an exemplary embodiment.

[0021] Figure 4 This is a schematic diagram illustrating the distribution of test cases via a first distribution unit and a second distribution unit according to an exemplary embodiment.

[0022] Figure 5 This is a schematic block diagram illustrating an apparatus for testing a large language model according to an exemplary embodiment.

[0023] Figure 6 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0026] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0027] In related technologies, testing large language models typically involves accuracy testing. Traditional accuracy testing usually employs black-box testing methods, but in this approach, the black box of the model deployed on the server is invisible, resulting in insufficient testing depth and efficiency. Furthermore, traditional accuracy testing often relies on script tools to call the large language model, failing to consider everyday user scenarios and thus not constituting true end-to-end testing.

[0028] Therefore, embodiments of this disclosure provide a more effective testing architecture for large language models, which not only breaks through the limitations of traditional testing depth, but also is closer to user scenarios to achieve true end-to-end testing.

[0029] Figure 1 This is a schematic diagram illustrating a test link according to an exemplary embodiment.

[0030] The method for testing large language models according to embodiments of this disclosure is executed on at least two test links. On each test link, at least one large language model to be tested, i.e., the target large language model, is deployed via an inference engine server. Furthermore, each test link sequentially includes a first number of first distribution units and a second number of second distribution units for distributing test cases, as well as a second number of interactive interface servers communicatively connected to the inference engine server. The second number is N times the first number, where N is a natural number greater than or equal to 2.

[0031] like Figure 1As shown, an exemplary description is provided using two test links, namely the first test link 110 and the second test link 120, and N = 4. It is understood that in practical applications, the number of test links and the value of N can be determined according to actual testing requirements, and the scope of this disclosure is not limited thereto.

[0032] On the first test link 110, the target large language model is deployed via the inference engine server 112. Furthermore, on the first test link 110, one first distribution unit 114, four second distribution units 116, and four interactive interface servers 118 are sequentially configured. These four interactive interface servers 118 are communicatively connected to the inference engine server 112. Figure 1 As shown, the arrangement of the second test link 120 is similar to that of the first test link 110, so it will not be described in detail here.

[0033] In the example, the inference engine servers on the first test link 110 and the second test link 120 can use the same inference engine service or different inference engine services, such as vLLM service, SGLang service, etc. Since the large language model itself exists only as serialized weights and configuration files, it needs to be loaded and instantiated by the inference engine server before it can provide inference capabilities. Therefore, the target large language model to be tested needs to be deployed on the inference engine server.

[0034] In the example, the deployment of the inference engine service on the server can be precise down to the GPU board level. This means that the number of inference engine services that can be deployed on the same server can be determined based on the actual number of GPUs on the server. For example, assuming a server has 8 GPUs, 4 GPUs can be used to deploy one inference engine service, and the other 4 GPUs can be used to deploy another inference engine service. If the large language model itself is large, it is also possible to deploy one inference engine service across 8 GPUs.

[0035] In the example, the target large language model to be tested on the first test link 110 and the second test link 120 can be set to the same large language model, or they can be set to different large language models, such as DeepSeek-V3, DeepSeek-R1, etc. When set to the same large language model, it means that the same large language model can be tested simultaneously using the first test link 110 and the second test link 120 to improve testing efficiency.

[0036] In the example, the first distribution unit 114 and the second distribution unit 116 can also be referred to as the producer and consumer, respectively. They work together to collaboratively distribute test cases. The number of second distribution units is greater than the number of first distribution units.

[0037] In the example, the number of second distribution units 116 and interactive interface servers 118 are the same, i.e., they correspond to each other. Thus, one second distribution unit 116 can interact with the target large language model on a corresponding interactive interface server 118 by simulating user operation behavior.

[0038] In the example, the interactive interface server 118 can be a web-based interactive interface (WebUI), through which users can perform operations such as input, output, and parameter settings with the target large language model. For example, test cases can be input into the interactive interface server 118 by simulating user clicks and inputs on the page of the interactive interface server 118.

[0039] In the example, the first test link 110 and the second test link 120 can be monitored by a monitor 130. The monitor 130 can monitor the running status of the inference engine service on each test link, and can also expose the running status to the second distribution unit 116 in the form of a RESTful API (Representation State Transition Application Programming Interface) to determine whether to continue distributing test cases.

[0040] Figure 2 This is a flowchart illustrating a method for testing a large language model according to an exemplary embodiment.

[0041] In embodiments of this disclosure, the method is used in conjunction with... Figure 1 The test is executed on at least two test links as shown. On each test link, a first distribution unit, a second distribution unit, an interactive interface server, and an inference engine server are sequentially configured, wherein the interactive interface server is communicatively connected to the inference engine server, through which the target large language model to be tested is deployed. The number of second distribution units and interactive interface servers is the same, and is N times the number of first distribution units (N is a natural number greater than or equal to 2).

[0042] like Figure 2 As shown, the method includes steps S201, S202, S203 and S204, wherein step S202 further includes sub-steps S2021, S2022 and S2023.

[0043] In step S201, the target hyperparameters of the target large language model are set via the interactive interface server. These target hyperparameters are used to adjust the probability distribution of predicted words by the target large language model.

[0044] In the example, the target hyperparameter can refer to the temperature hyperparameter of the large language model, usually denoted by Temperature or T. It's a hyperparameter used by the large language model to control randomness when generating text. A word, also known as a token, is the text before decoding. During text generation, the large language model predicts the probability of each candidate word in its vocabulary to select the word to output. Correspondingly, each candidate word has a certain probability distribution, and the target hyperparameter can be used to adjust this probability distribution. For example, when the target hyperparameter is 0, the randomness of the large language model's output is turned off, always selecting the word with the highest probability, thus the output is deterministic. When the target hyperparameter is between 0 and 1, the large language model's output is more conservative, with lower randomness. When the target hyperparameter is greater than 1, the large language model's output is more random, considering lower-probability words more, increasing the diversity of the generated results. Therefore, to test the large language model, the target hyperparameter can be set to a specific value, such as 0, to turn off the randomness of the large language model's output. The reason for this is that, in some cases, although the text generated by the large language model is the same, the underlying probability distribution may be different, and the testing method in this embodiment is precisely to find out whether such anomalies exist.

[0045] In the example, the target hyperparameters of the target large language model can be set by sending a parameter setting request to the inference engine server via the interactive interface server. At the same time, other hyperparameters in the inference stage, such as top_k and top_p, can be fixed. These hyperparameters are also used to control the rules for selecting lexical units from the probability distribution, but they do not affect the weights of the large language model itself.

[0046] In step S202, multiple rounds of testing are performed, and sub-steps S2021, S2022 and S2023 are executed for each round of testing.

[0047] In sub-step S2021, test cases are provided to the target large language model via the interactive interface server through the first distribution unit and the second distribution unit.

[0048] In this example, a test case may include one or more test questions. In each round of testing, the target large language model answers these test questions to provide one or more corresponding solutions. In this embodiment, the multiple corresponding test cases used in multiple rounds of testing may be the same or substantially the same, that is, they all involve the same one or more test questions. Accordingly, the current round of testing is completed when the target large language model provides an answer.

[0049] In sub-step S2022, obtain the unnormalized raw prediction scores of the target large language model in response to the predicted lexical units of the test cases.

[0050] In the example, the unnormalized raw prediction score is often referred to as "logits," which is transformed into a probability distribution after being processed by softmax (normalization exponential function). In the testing method of this disclosure embodiment, in order to find out whether there are anomalies where the generated text is the same but the underlying probabilities are different, the unnormalized raw prediction score is obtained for comparison.

[0051] In the example, the corresponding data can be obtained by sending a request to the inference engine server via the interactive interface server to return the unnormalized raw prediction scores.

[0052] In substep S2023, the unnormalized original prediction score is converted into a byte string.

[0053] In the example, depending on the number of candidate lexical units contained in the vocabulary of the target large language model, a corresponding number of unnormalized raw prediction scores can be obtained, forming an array containing multiple elements. The value of each element can be converted into several bytes (e.g., 4 bytes) and then concatenated sequentially to obtain the byte string corresponding to the entire array.

[0054] In step S203, the byte strings obtained from each of the multiple rounds of testing are compared to determine whether there are any inconsistent byte strings.

[0055] In the example, MD5 (Message Digest) can be used to verify whether there are inconsistent byte strings among multiple obtained byte strings. MD5 is a hash algorithm used to map data of arbitrary length to a fixed 128-bit digital fingerprint. Due to the high sensitivity of the MD5 algorithm, even changing just one bit will completely change the fingerprint.

[0056] In step S204, in response to determining that there is an inconsistent byte string, it is determined that the target large language model has an output deterministic anomaly on the test link.

[0057] In the example, with the target hyperparameter set to 0, the randomness of the target large language model's output should be off, and the output should be deterministic, without any output deterministic anomalies. However, if inconsistent byte strings are found in the verification results, it means that the target large language model has experienced output deterministic anomalies in the test chain. This anomaly can reflect non-deterministic factors in the inference chain, such as those originating from the inference engine server or the interaction interface server.

[0058] Therefore, the method for testing large language models according to embodiments of this disclosure provides a testing mechanism that supports bit-by-bit matching at unnormalized raw prediction score levels. Its testing depth breaks through traditional black-box testing schemes, penetrating into the interior of the large language model, enabling more precise and in-depth testing of the model's accuracy. Furthermore, the method for testing large language models according to embodiments of this disclosure also provides a fully automated testing mechanism. By leveraging the interaction between the first and second distribution units and the interactive interface server, user operation behavior is simulated, making the testing method closer to user scenarios and thus achieving true end-to-end testing. Additionally, since the test supports board-level deployment of large language models, the testing mechanism becomes more flexible, improving testing efficiency.

[0059] In some embodiments, the method for testing a large language model may further include the following steps, namely, for at least one round of multi-round testing: obtaining the actual hyperparameters used by the target large language model in the process of predicting lexical units in response to test cases; determining whether the actual hyperparameters are the same as the target hyperparameters; and determining that the target large language model has a parameter pass-through anomaly in the test link in response to the difference between the actual hyperparameters and the target hyperparameters.

[0060] In the example, the corresponding data can be obtained by sending a request to the inference engine server to retrieve the actual hyperparameters via the interactive interface server. If the actual hyperparameters differ from the target hyperparameters, it means that the target large language model did not actually use the set target hyperparameters when predicting lexical units in response to test cases, resulting in a parameter pass-through anomaly. This anomaly may originate from the inference engine server or the interactive interface server, etc.

[0061] Therefore, parameter pass-through testing can further break through the traditional black-box testing scheme, thereby ensuring the reliability of test results, the localization of problems, and the reproducibility of iterations.

[0062] In some embodiments, the method for testing a large language model may further include the following steps: in response to determining that at least one of an output deterministic anomaly or a parameter pass-through anomaly occurs, the potential location of the anomaly on the test chain is located to the interactive interface server or the inference engine server via application programming interface (API) debugging.

[0063] In the example, assuming a parameter pass-through exception occurs, you can bypass the interactive interface server via API debugging, directly set the target hyperparameter, and then retrieve the actual hyperparameter again. If the actual hyperparameter becomes the same as the target hyperparameter at this point, it means there was a problem when the parameter information was sent on the interactive interface server side. However, if the actual hyperparameter is still different from the target hyperparameter, it means there is a problem on the inference engine server side, such as due to default value overwriting or precision loss.

[0064] In the example, assuming a deterministic output anomaly occurs, you can bypass the interactive server-side debugging via API, directly set the target hyperparameters, and then compare the byte strings obtained from multiple rounds of testing. If inconsistent byte strings still appear at this point, it means there is a problem on the inference engine server side, such as due to parallel accumulation. If the inconsistent byte strings disappear at this point, it means there is a problem on the interactive server side, such as due to input rewriting or parameter tampering.

[0065] Because the entire process is quite long, debugging via API can help locate potential problems, which in turn facilitates rapid improvement and optimization after testing.

[0066] Figure 3 This is a schematic diagram illustrating the location of potential anomalies on a test link according to an exemplary embodiment.

[0067] like Figure 3 As shown, taking a parameter pass-through anomaly as an example, during the test, the target hyperparameter T was set to 0 on the interactive interface server, but the actual hyperparameter obtained on the inference engine server was not 0. Therefore, API debugging can be used to bypass the interactive interface server and directly send a request to the inference engine server to set the target hyperparameter T to 0, and then obtain the actual hyperparameter again. If the returned target hyperparameter T is 0, it means the problem originates from the interactive interface server; if the returned target hyperparameter T is not 0, it means the problem originates from the inference engine server. This allows for locating the potential location of the anomaly in the test chain.

[0068] In some embodiments, the step described above, of providing test cases to the target large language model via the interactive interface server through the first and second distribution units (such as combining...) Figure 2 The sub-step S2021 shown may include the following steps: placing test cases into a test queue via a first distribution unit; determining whether the number of test cases being processed by the target large language model is less than a preset maximum pending threshold; in response to determining that the number of test cases being processed by the target large language model is less than the maximum pending threshold, inputting test cases from the test queue into the interactive interface server via a second distribution unit; and in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold, pausing the acquisition of test cases from the test queue.

[0069] In the example, the test pipeline monitor can obtain the running status of the target large language model and expose this status to the second distribution unit. The second distribution unit can then determine whether the number of test cases being processed by the target large language model is less than the maximum pending threshold. The second distribution unit will only continue to distribute test cases if the target large language model still has remaining processing capacity. If there is no remaining processing capacity, test cases will not be retrieved from the test queue until the target large language model generates new processing capacity. Alternatively, the first distribution unit can be notified to pause adding test cases to the test queue.

[0070] Figure 4 This is a schematic diagram illustrating the distribution of test cases via a first distribution unit and a second distribution unit according to an exemplary embodiment.

[0071] like Figure 4 As shown, the first distribution unit can add test cases to the test queue, and the second distribution unit can retrieve test cases from the queue and distribute them to the target large language model under test for processing. However, since a maximum pending threshold is set to determine whether to continue distribution, if the second distribution unit learns from the information exposed by the monitor that the number of test cases being processed by the target large language model is less than the maximum pending threshold, it can continue distributing test cases. Conversely, if the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold, the distribution of test cases is paused until the target large language model recovers its remaining processing capacity.

[0072] In this way, the collaboration between the first and second distribution units can be used to achieve a balance in test case distribution, avoiding overdrawing the processing capacity of the target large language model during the testing process, thereby ensuring the smoothness and effectiveness of the testing process.

[0073] In some embodiments, test cases may include prompts to guide the target large language model under test to output the processing results in a predetermined format.

[0074] In the example, the dataset used for testing can be processed to have the same format, which can refer to the way real users ask questions, such as "Please help me answer this question, the answer should be output in the format of...", etc.

[0075] In this way, the target large language model under test can output processing results in a uniform format according to the indicated requirements, which helps to simplify the comparison process of processing results and improve testing efficiency.

[0076] In some embodiments, a test case may include at least one test question, which may have a predetermined answer. Accordingly, the method for testing a large language model may further include the steps of, for at least one round of testing: obtaining the processing result of the target large language model in response to the test case generation from an interactive interface server; determining whether the processing result matches the predetermined answer; and determining that the target large language model has an accuracy anomaly in response to determining that the processing result does not match the predetermined answer.

[0077] In the example, dialogue logs can be retrieved from the completed test interface server. After all test interfaces for the same target large language model have been completed, these logs are merged to obtain a merged dialogue log. Then, the fields in this merged dialogue log can be analyzed to obtain the test questions and the answers provided by the target large language model—that is, the processing results of the target large language model in response to the test cases. For large language models that provide thought processes, procedural answers during the thought process can be filtered out from the merged dialogue log to avoid affecting the test results.

[0078] In the example, the answer given by the target large language model can be compared with the predetermined answer using either exact match or maximum similarity match (the threshold can be set to 0.9). When testing multiple large language models, the output accuracy of each model can also be compared to obtain an accuracy comparison report.

[0079] Embodiments of this disclosure also provide an apparatus for testing large language models.

[0080] Figure 5 This is a schematic block diagram illustrating an apparatus 500 for testing a large language model according to an exemplary embodiment.

[0081] In some embodiments, the apparatus 500 is configured to execute on at least two test links. On each test link, at least one target large language model from a large language model is deployed via an inference engine server. Furthermore, each test link sequentially includes a first number of first distribution units and a second number of second distribution units for distributing test cases, as well as a second number of interactive interface servers communicatively connected to the inference engine server. The second number is N times the first number, where N is a natural number greater than or equal to 2.

[0082] The device 500 includes a hyperparameter setting module 501, a first test execution module 502, a first result comparison module 503, and a first anomaly determination module 504. The first test execution module 502 further includes a test case providing module 5021, a first result acquisition module 5022, and a result conversion module 5023.

[0083] The hyperparameter setting module 501 is configured to set the target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of the predicted words of the target large language model.

[0084] The first test execution module 502 is configured to execute multiple rounds of testing. For each round of testing: the test case providing module 5021 is configured to provide test cases to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit; the first result acquisition module 5022 is configured to acquire the unnormalized raw prediction scores obtained by the target large language model in response to the predicted words of the test cases; and the result conversion module 5023 is configured to convert the unnormalized raw prediction scores into byte strings.

[0085] The first result comparison module 503 is configured to compare the byte strings obtained from each of the multiple rounds of testing to determine whether there are any inconsistent byte strings.

[0086] The first anomaly determination module 504 is configured to determine that the target large language model has an output deterministic anomaly on the test link in response to the determination that an inconsistent byte string exists.

[0087] The operations of the aforementioned hyperparameter setting module 501, first test execution module 502, first result comparison module 503, and first anomaly determination module 504 can be combined. Figure 2 The operations in steps S201, S202, S203, and S204 are identical. Furthermore, the test case providing module 5021, the first result acquisition module 5022, and the result conversion module 5023 can be combined with... Figure 2 The sub-steps S2021, S2022, and S2023 are identical. Therefore, details of each aspect will not be elaborated here.

[0088] In some embodiments, the apparatus 500 may further include a second test execution module 505, a second result comparison module 506, and a second anomaly determination module 507. For any round of multi-round testing: the second test execution module 505 is configured to obtain the actual hyperparameters used by the target large language model in the process of predicting lexical units in response to test cases; the second result comparison module 506 is configured to determine whether the actual hyperparameters are the same as the target hyperparameters; and the second anomaly determination module 507 is configured to determine that a parameter pass-through anomaly has occurred in the test link of the target large language model in response to the difference between the actual hyperparameters and the target hyperparameters.

[0089] In some embodiments, the apparatus 500 may further include: an anomaly localization module 508, configured to, in response to determining that at least one of an output deterministic anomaly or a parameter pass-through anomaly has occurred, locate the potential location of the anomaly on the test link to the interactive interface server or the inference engine server via API debugging.

[0090] In some embodiments, the test case providing module 5021 may include: a first providing module 5021a, configured to place test cases into a test queue via a first distribution unit; a threshold determining module 5021b, configured to determine whether the number of test cases being processed by the target large language model is less than a preset maximum pending threshold; a second providing module 5021c, configured to input test cases from the test queue into the interactive interface server via a second distribution unit in response to determining that the number of test cases being processed by the target large language model is less than the maximum pending threshold; and a pause providing module 5021d, configured to pause obtaining test cases from the test queue in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold.

[0091] In some embodiments, a test case may include at least one test question, which may have a predetermined answer. Accordingly, the apparatus 500 may also include a third test execution module 509, a third result comparison module 510, and a third anomaly determination module 511. For at least one round of multi-round testing: the third test execution module 509 is configured to obtain the processing result of the target large language model in response to the generation of the test case from the interactive interface server; the third result comparison module 510 is configured to determine whether the processing result matches the predetermined answer; and the third anomaly determination module 511 is configured to determine that the target large language model has an accuracy anomaly in response to determining that the processing result does not match the predetermined answer.

[0092] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0093] It should also be understood that the above regarding Figure 5The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0094] According to one aspect of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0095] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0096] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0097] In the following text, combined with Figure 6 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0098] Figure 6 An example configuration of a computer device 600 that can be used to implement the methods described herein is shown.

[0099] Computer device 600 can be a variety of different types of devices. Examples of computer device 600 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0100] Computer device 600 may include at least one processor 602, memory 604, multiple communication interfaces 606, display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612 capable of communicating with each other, such as via system bus 614 or other suitable connections.

[0101] Processor 602 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 602 may be configured to acquire and execute computer-readable instructions stored in memory 604, mass storage device 612, or other computer-readable media, such as program code of operating system 616, program code of application program 618, program code of other program 620, etc.

[0102] Memory 604 and mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by processor 602 to perform the various functions described above. For example, memory 604 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 612 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 604 and mass storage device 612 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 602 as a specific machine configured to perform the operations and functions described in the examples herein.

[0103] Multiple programs can be stored on mass storage device 612. These programs include operating system 616, one or more application programs 618, other programs 620, and program data 622, and they can be loaded into memory 604 for execution. Examples of such application programs or program modules may include... Figure 2 The computer program logic (e.g., computer program code or instructions) of the methods shown and / or other embodiments described herein.

[0104] Although Figure 6The modules 616, 618, 620, and 622, or portions thereof, are illustrated as being stored in memory 604 of computer device 600; however, modules 616, 618, 620, and 622 may be implemented using any form of computer-readable medium accessible by computer device 600. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0105] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0106] One or more communication interfaces 606 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 606 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 606 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0107] In some examples, a display device 608, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0108] The technologies described herein can be supported by these various configurations of computer device 600, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 600. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 600 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computer device 600 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. A method for testing a large language model, characterized by, The method is used for execution on at least two test links, each test link deploying a target large language model in at least one large language model via an inference engine service end, and each test link being sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface service ends in communication connection with the inference engine service end, the second number being N times of the first number, N being a natural number greater than or equal to 2, wherein the interaction operation of the first distribution unit and the second distribution unit with the interactive interface service end is used to simulate user operation behavior, wherein the method comprises: setting a target hyperparameter of the target large language model via the interactive interface service end, wherein the target hyperparameter comprises a temperature hyperparameter used to adjust the probability distribution of the target large language model to predict a token, wherein the temperature hyperparameter is set to zero, so that other hyperparameters of the inference stage except the temperature hyperparameter are fixed; performing multiple rounds of testing, for each round of testing: providing the test cases to the target large language model via the first distribution unit and the second distribution unit through the interactive interface service end; obtaining an unnormalized original prediction score obtained by the target large language model in response to the test cases to predict a token; converting the unnormalized original prediction score into a byte string; comparing the byte strings obtained in each of the multiple rounds of testing to determine whether there is an inconsistent byte string; in response to determining that there is an inconsistent byte string, determining that the target large language model has an output determinacy abnormality on the test link; and locating a potential position of the abnormality on the test link to the interactive interface service end or the inference engine service end via an application programming interface debugging, wherein: skipping the interactive interface service end, directly setting the target hyperparameter, and again obtaining the byte strings obtained in each of the multiple rounds of testing for comparison via the application programming interface debugging; in response to still having an inconsistent byte string, locating an abnormality on the inference engine service end side; and in response to the inconsistent byte string disappearing, locating an abnormality on the interactive interface service end side.

2. The method of claim 1, wherein, The method further comprises: for at least one of the multiple rounds of testing: obtaining an actual hyperparameter used by the target large language model in the process of responding to the test cases to predict a token; determining whether the actual hyperparameter is the same as the target hyperparameter; and in response to the actual hyperparameter being different from the target hyperparameter, determining that the target large language model has a parameter transmission abnormality on the test link; locating a potential position of the abnormality on the test link to the interactive interface service end or the inference engine service end via an application programming interface debugging, wherein: skipping the interactive interface service end, directly setting the target hyperparameter, and again obtaining the actual hyperparameter via the application programming interface debugging; in response to the actual hyperparameter becoming the same as the target hyperparameter, determining that an exception occurs when the parameter information is issued on the interactive interface server side; and in response to the actual hyperparameter still being different from the target hyperparameter, determining that an exception occurs on the inference engine server side.

3. The method according to claim 1 or 2, characterized in that, The providing of the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit includes: putting the test case into a test queue via the first distribution unit; determining whether the number of test cases being processed by the target large language model is less than a preset maximum pending threshold; in response to determining that the number of test cases being processed by the target large language model is less than the maximum pending threshold, inputting the test case from the test queue to the interactive interface server via the second distribution unit; and in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold, pausing the acquisition of the test case from the test queue.

4. The method according to claim 1 or 2, characterized in that, The test case includes a prompt word for guiding the target large language model to output a processing result in a predetermined format.

5. The method of claim 4, wherein, The test case includes at least one test question having a predetermined answer, wherein the method further includes: for at least one round of the multiple rounds of testing: acquiring the processing result generated by the target large language model in response to the test case from the interactive interface server; determining whether the processing result matches the predetermined answer; and in response to determining that the processing result does not match the predetermined answer, determining that the target large language model has a precision exception.

6. An apparatus for testing a large language model, the apparatus comprising: The device is used to be executed on at least two test links, a target large language model in at least one large language model is deployed on each test link via an inference engine server, and a first number of first distribution units and a second number of second distribution units for distributing test cases are sequentially arranged on each test link, and the second number is N times of the first number, N is a natural number greater than or equal to 2, and the second number of interactive interface servers in communication connection with the inference engine server, wherein the interaction operation of the first distribution unit and the second distribution unit with the interactive interface server is used to simulate user operation behavior, wherein the device includes: a hyperparameter setting module configured to set a target hyperparameter of the target large language model via the interactive interface server, wherein the target hyperparameter includes a temperature hyperparameter for adjusting the probability distribution of the target large language model to predict a word element, wherein the temperature hyperparameter is set to zero, so that other hyperparameters of an inference stage except the temperature hyperparameter are fixed; a first test execution module configured to perform multiple rounds of testing, for each round of testing: a test case providing module configured to provide the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit; a first result obtaining module configured to obtain unnormalized original prediction scores obtained by the target large language model in response to the test case prediction tokens; a result conversion module configured to convert the unnormalized original prediction scores into byte strings; a first result comparison module configured to compare the byte strings obtained by performing the multiple rounds of tests respectively to determine whether there is an inconsistent byte string; a first exception determination module configured to determine that the target large language model has output determinacy exception on the test link in response to a determination that there is the inconsistent byte string; and an exception positioning module configured to locate a potential position of the exception on the test link to the interactive interface server or the inference engine server via an application programming interface debugging, wherein: the application programming interface debugging skips the interactive interface server, directly sets the target hyperparameters, and again obtains the byte strings obtained by performing the multiple rounds of tests respectively for comparison; in response to the inconsistent byte string still existing, locating the exception on the inference engine server side; and in response to the inconsistent byte string disappearing, locating the exception on the interactive interface server side.

7. A computer device, characterized by The computer device comprises: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, causes the processor to perform the method of any one of claims 1-5.

9. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program, when executed by a processor, causes the processor to perform the method of any one of claims 1-5.

Images