Method and device for testing large language model, computer equipment, storage medium and program product

Abstract

The invention provides a method and device for testing a large language model, computer equipment, a storage medium and a program product, and relates to the field of computers and artificial intelligence. According to the implementation scheme, target hyper-parameters of a target large language model are set through an interaction interface server side, wherein the target hyper-parameters are used for adjusting probability distribution of prediction lexical elements of the target large language model; multiple rounds of testing are executed, and for each round of testing, the first distribution unit and the second distribution unit provide the test cases for the target large language model through the interactive interface server side; obtaining a non-normalized original prediction score obtained by the target large language model in response to the test case prediction lexical element; converting the original prediction score which is not normalized into a byte string; comparing the byte strings obtained by executing multiple rounds of tests to determine whether inconsistent byte strings exist or not; and in response to determining that the inconsistent byte strings exist, determining that the target large language model has an output certainty anomaly on the test link.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer and artificial intelligence technology, in particular to the fields of large language models (LLMs), data testing, and more particularly to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for testing large language models. Background Art

[0002] In recent years, the emergence of large language models (also known as megamodels) has significantly boosted the field of artificial intelligence. Some of these megamodels, which incorporate thought processes, can also provide more powerful problem-solving capabilities. However, the industry's attention and development of testing methods for these models remains limited.

[0003] At present, traditional testing methods are generally based on black-box testing solutions, in which questions are raised on one side of the large language model and the answers to the questions are obtained on the other side. The test results are determined based on the analysis of the answers. However, the testing depth of this traditional testing method is increasingly unable to meet the requirements of technological iteration. In addition, the current traditional testing methods have not yet implemented an end-to-end testing mechanism, and there is still a certain gap with the actual user usage scenarios. Therefore, there is an urgent need for more effective large language model testing methods to adapt to the continuous evolution of the large language model itself. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for testing a large language model.

[0005] According to one aspect of the present disclosure, a method for testing a large language model is provided, the method being executed on at least two test links, wherein a target large language model of at least one large language model is deployed on each test link via an inference engine server, and each test link is sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface servers communicatively connected to the inference engine server, wherein the second number is N times the first number, where N is a natural number greater than or equal to 2. The method comprises: setting target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of word units predicted by the target large language model; performing multiple rounds of testing, and for each round of testing: providing the test case to the target large language model via the interactive interface server via the first distribution unit and the second distribution unit; obtaining an unnormalized raw prediction score obtained by the target large language model in response to the test case word unit prediction; converting the unnormalized raw prediction score into a byte string; comparing the byte strings obtained from each of the multiple rounds of testing to determine whether there is an inconsistent byte string; and in response to determining that there is an inconsistent byte string, determining that an output determinism anomaly occurs in the target large language model on the test link.

[0006] In some embodiments, the method further includes: for at least one round of multiple rounds of testing: obtaining actual hyperparameters used by the target large language model in the process of predicting words in response to test cases; determining whether the actual hyperparameters are the same as the target hyperparameters; and in response to the actual hyperparameters being different from the target hyperparameters, determining that a parameter transmission anomaly occurs in the target large language model on the test link.

[0007] In some embodiments, the method further includes: in response to determining that at least one of an output deterministic anomaly or a parameter transmission anomaly occurs, locating the potential location of the anomaly on the test link to the interactive interface server or the inference engine server via application programming interface debugging.

[0008] In some embodiments, providing the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit includes: placing the test case into the test queue via the first distribution unit; determining whether the number of test cases being processed by the target large language model is less than a preset maximum pending threshold; in response to determining that the number of test cases being processed by the target large language model is less than the maximum pending threshold, inputting the test case from the test queue into the interactive interface server via the second distribution unit; and in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold, pausing the acquisition of test cases from the test queue.

[0009] In some embodiments, the test case includes prompt words for guiding the target large language model to output processing results in a predetermined format.

[0010] In some embodiments, the test case includes at least one test question, and the test question has a predetermined answer, wherein the method further includes: for at least one round of multiple rounds of testing: obtaining a processing result generated by the target large language model in response to the test case from the interactive interface server; determining whether the processing result matches the predetermined answer; and in response to determining that the processing result does not match the predetermined answer, determining that an accuracy abnormality occurs in the target large language model.

[0011] According to one aspect of the present disclosure, a device for testing a large language model is provided, which is used to execute on at least two test links, and each test link deploys a target large language model in at least one large language model via an inference engine server, and each test link is sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface servers communicatively connected to the inference engine server, the second number being N times the first number, and N being a natural number greater than or equal to 2, wherein the device includes: a hyperparameter setting module, configured to set target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of word units predicted by the target large language model; a first test execution module, configured to perform multiple rounds of testing, and for each round of testing: a test case providing module, configured to provide test cases via The first distribution unit and the second distribution unit provide the test case to the target large language model through the interactive interface server; the first result acquisition module is configured to obtain the unnormalized original prediction score obtained by the target large language model in response to the test case word prediction; the result conversion module is configured to convert the unnormalized original prediction score into a byte string; the first result comparison module is configured to compare the byte strings obtained from each of the multiple rounds of testing to determine whether there are inconsistent byte strings; and the first abnormality determination module is configured to determine that the target large language model has an output deterministic abnormality on the test link in response to determining that there are inconsistent byte strings.

[0012] According to another aspect of the present disclosure, a computer device is provided, comprising: at least one processor; and a memory on which a computer program is stored. When the computer program is executed by the at least one processor, the at least one processor executes the method provided above in the present disclosure.

[0013] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the method provided above in the present disclosure.

[0014] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the processor is caused to perform the method provided above in the present disclosure.

[0015] According to one or more embodiments of the present disclosure, a more effective testing architecture for large language models can be provided, which not only breaks through the depth limitations of traditional testing, but also is closer to user usage scenarios to achieve true end-to-end testing.

[0016] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the text of the specification, they are used to explain exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the present disclosure. In all figures, the same reference numerals refer to similar, but not necessarily identical, elements.

[0018] Figure 1 is a schematic diagram illustrating a test link according to an exemplary embodiment.

[0019] Figure 2 is a flowchart illustrating a method for testing a large language model according to an exemplary embodiment.

[0020] Figure 3 is a schematic diagram illustrating locating a potential location where an anomaly occurs on a test link according to an exemplary embodiment.

[0021] Figure 4 is a schematic diagram illustrating distribution of test cases via a first distribution unit and a second distribution unit according to an exemplary embodiment.

[0022] Figure 5 is a schematic block diagram illustrating an apparatus for testing a large language model according to an exemplary embodiment.

[0023] Figure 6 is a block diagram illustrating an exemplary computer device that can be used with the exemplary embodiments. DETAILED DESCRIPTION

[0024] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0026] The terms used in the description of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. As used herein, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on". In addition, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations.

[0027] In related technologies, testing large language models generally involves accuracy testing. Traditional accuracy testing typically uses a black-box testing approach. However, this approach hides the model black box deployed on the server, resulting in insufficient testing depth and efficiency. Furthermore, traditional accuracy testing typically relies on scripting tools to call large language models, failing to consider daily user scenarios and failing to implement true end-to-end testing.

[0028] To this end, the embodiments of the present disclosure provide a more effective testing architecture for large language models, which not only breaks through the traditional testing depth limitations, but also is closer to user usage scenarios to achieve true end-to-end testing.

[0029] Figure 1 is a schematic diagram illustrating a test link according to an exemplary embodiment.

[0030] The method for testing a large language model in an embodiment of the present disclosure is used to be executed on at least two test links. On each test link, at least one large language model to be tested in a large language model, i.e., a target large language model, is deployed via an inference engine server. In addition, each test link is sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface servers communicatively connected to the inference engine server. The second number is N times the first number, where N is a natural number greater than or equal to 2.

[0031] like Figure 1As shown, an exemplary description is given with two test links, namely a first test link 110 and a second test link 120, and N being 4. It is understandable that in actual applications, the number of test links and the value of N can be determined according to actual test requirements, and the scope of this disclosure is not limited thereto.

[0032] On the first test link 110, the target large language model is deployed via the inference engine server 112. In addition, on the first test link 110, a first distribution unit 114, four second distribution units 116 and four interactive interface servers 118 are sequentially arranged. The four interactive interface servers 118 are in communication with the inference engine server 112. Figure 1 As shown, the arrangement of the second test link 120 is similar to that of the first test link 110, so they are not described in detail here.

[0033] In this example, the inference engine servers on the first test link 110 and the second test link 120 can use the same inference engine service, or different inference engine services, such as vLLM services, SGLang services, etc. Because the large language model itself only exists in the form of serialized weights and configuration files, it must be loaded and instantiated by the inference engine server before it can provide inference capabilities to the outside world. Therefore, the target large language model to be tested needs to be deployed on the inference engine server.

[0034] In this example, the deployment of inference engine services on servers can be precisely tailored to the GPU board level. This means that the number of inference engine services that can be deployed on the same server can be determined based on the actual number of GPUs on the server. For example, if a server has eight GPUs, one inference engine service can be deployed on four of them, and one on the remaining four. If the large language model itself is relatively large, one inference engine service can also be deployed on eight GPUs.

[0035] In this example, the target large language model to be tested on the first test link 110 and the second test link 120 can be set to the same large language model, or different large language models, such as DeepSeek-V3, DeepSeek-R1, etc. When set to the same large language model, it means that the first test link 110 and the second test link 120 can simultaneously test the same large language model, thereby improving test efficiency.

[0036] In an example, the first distribution unit 114 and the second distribution unit 116 may also be referred to as a producer and a consumer, respectively, which are combined to collaboratively distribute test cases. The number of the second distribution units is greater than the number of the first distribution units.

[0037] In this example, the number of second distribution units 116 and interactive interface servers 118 is the same, that is, they are in a corresponding relationship. Therefore, one second distribution unit 116 can interact with the target large language model on a corresponding interactive interface server 118 by simulating user operation behavior.

[0038] In this example, the interactive interface server 118 may be a web UI (WebUI), through which operations such as input, output, and parameter setting can be performed with the target large language model. For example, test cases can be input into the interactive interface server 118 by simulating user clicks and input on the page of the interactive interface server 118.

[0039] In an example, the first test link 110 and the second test link 120 can be monitored by a monitor 130. The monitor 130 can monitor the running status of the inference engine service on each test link and can also expose the running status to the second distribution unit 116 in a form such as a RESTful API (Representational State Transfer Application Programming Interface) to determine whether to continue distributing test cases.

[0040] Figure 2 is a flowchart illustrating a method for testing a large language model according to an exemplary embodiment.

[0041] In the embodiment of the present disclosure, the method is combined with Figure 1 The test is performed on at least two test links shown. On each test link, a first distribution unit, a second distribution unit, an interactive interface server, and an inference engine server are sequentially provided. The interactive interface server is communicatively connected to the inference engine server, and the target large language model to be tested is deployed via the inference engine server. The number of second distribution units is the same as the number of interactive interface servers, and is N times the number of first distribution units (N is a natural number greater than or equal to 2).

[0042] like Figure 2 As shown, the method includes steps S201, S202, S203 and S204, wherein step S202 further includes sub-steps S2021, S2022 and S2023.

[0043] In step S201, target hyperparameters of a target large language model are set via the interactive interface server. The target hyperparameters are used to adjust the probability distribution of word units predicted by the target large language model.

[0044] In this example, the target hyperparameter can refer to the temperature hyperparameter of the large language model, commonly denoted by Temperature or T. This hyperparameter controls the randomness of the large language model when generating text. A word, also commonly referred to as a token, is the form of text before it is decoded. During text generation, the large language model predicts the probabilities of each candidate word in its vocabulary to select the word to output. Accordingly, each candidate word has a certain probability distribution, and the target hyperparameter can be used to adjust this probability distribution. For example, when the target hyperparameter is 0, the large language model's output randomness is disabled, and the word with the highest probability is always selected, resulting in a deterministic output. When the target hyperparameter is between 0 and 1, the large language model's output is more conservative and less random. When the target hyperparameter is greater than 1, the large language model's output is more random, considering low-probability words more, increasing the diversity of the generated results. Therefore, to test the large language model, the target hyperparameter can be set to a specific value, such as 0, to disable the randomness of the large language model's output. The reason for doing this is that, in some cases, although the text generated by the large language model is the same, the underlying probability distribution may be different, and the testing method in this embodiment is to find out whether such anomalies exist.

[0045] In this example, the target hyperparameters of the target large language model can be set by sending a parameter setting request to the inference engine server via the interactive interface server. Other hyperparameters of the inference phase, such as top_k and top_p, can also be fixed. These hyperparameters also control the rules for selecting tokens from the probability distribution but do not affect the weights of the large language model itself.

[0046] In step S202 , multiple rounds of testing are performed, and sub-steps S2021 , S2022 , and S2023 are performed for each round of testing.

[0047] In sub-step S2021, the test case is provided to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit.

[0048] In this example, a test case may include one or more test questions. In each round of testing, the target large language model answers the one or more test questions to provide one or more corresponding answers. In this embodiment, the multiple corresponding test cases used in multiple rounds of testing can be the same or substantially the same, that is, they all involve the same one or more test questions. Accordingly, when the target large language model provides an answer, the current round of testing is completed.

[0049] In sub-step S2022 , the unnormalized original prediction score obtained by the target large language model in response to the word-gram prediction of the test case is obtained.

[0050] In this example, the unnormalized raw prediction scores, also known as "logits," are converted into probability distributions after softmax (a normalized exponential function). In the testing method of the disclosed embodiment, to identify unusual situations where the generated text is identical but the underlying probabilities differ, the unnormalized raw prediction scores are obtained for comparison.

[0051] In the example, the corresponding data can be obtained by sending a request for returning unnormalized raw prediction scores to the inference engine server via the interactive interface server.

[0052] In sub-step S2023, the unnormalized raw prediction score is converted into a byte string.

[0053] In this example, depending on the number of candidate tokens in the vocabulary of the target large language model, a corresponding number of unnormalized raw prediction scores can be obtained, thereby forming an array containing multiple elements. The value of each element can be converted into a number of bytes (for example, 4 bytes) and then concatenated sequentially to obtain the byte string corresponding to the entire array.

[0054] In step S203, the byte strings obtained from the multiple rounds of testing are compared to determine whether there are inconsistent byte strings.

[0055] In this example, we can use the MD5 (Message Digest) algorithm to verify whether there are any inconsistencies in the resulting byte strings. The MD5 algorithm is a hash algorithm that maps data of any length into a fixed 128-bit digital fingerprint. Due to the high sensitivity of the MD5 algorithm, even a single bit change will result in a completely different fingerprint.

[0056] In step S204 , in response to determining that there are inconsistent byte strings, it is determined that an output determinism anomaly occurs in the target large language model on the test link.

[0057] In this example, when the target hyperparameter is set to 0, the output randomness of the target large language model should be disabled, the output should be deterministic, and no output determinism anomalies should occur. However, if the verification results reveal inconsistent byte strings, this means that the target large language model has experienced an output determinism anomaly on the test link. This anomaly may reflect the presence of non-deterministic factors in the inference link, such as those from the inference engine server or the interactive interface server.

[0058] Therefore, the method for testing a large language model according to an embodiment of the present disclosure provides a test mechanism for supporting bit-by-bit matching at the level of unnormalized raw prediction scores. The depth of the test breaks through the traditional black box test solution and penetrates into the large language model, which can test the accuracy of the large language model more accurately and in-depth. In addition, the method for testing a large language model according to an embodiment of the present disclosure also provides a full-link automated testing mechanism, which simulates user operation behavior by means of the interactive operation of the first distribution unit and the second distribution unit with the interactive interface server, so that the test method is closer to the user usage scenario, thereby realizing true end-to-end testing. In addition, since the test supports the deployment of large language models at the board level, the test mechanism can be made more flexible and the test efficiency can be improved.

[0059] In some embodiments, the method for testing a large language model may further include the following steps, namely, for at least one round of multiple rounds of testing: obtaining actual hyperparameters used by the target large language model in the process of predicting words in response to test cases; determining whether the actual hyperparameters are the same as the target hyperparameters; and in response to the actual hyperparameters being different from the target hyperparameters, determining that a parameter transmission anomaly occurs in the target large language model on the test link.

[0060] In this example, the interactive interface server can retrieve the corresponding data by sending a request to the inference engine server for the actual hyperparameters. If the actual hyperparameters differ from the target hyperparameters, this means that the target large language model did not actually use the set target hyperparameters when predicting tokens in response to the test case, resulting in a parameter passthrough anomaly. This anomaly may originate from the inference engine server or the interactive interface server.

[0061] Therefore, parameter transparent testing can further break through the traditional black box testing solution, thereby ensuring that the test results are reliable, the problems can be located, and the iterations can be reproduced.

[0062] In some embodiments, the method for testing a large language model may further include the following steps: in response to determining that at least one of an output deterministic anomaly or a parameter transmission anomaly occurs, locating the potential location of the anomaly on the test link to the interactive interface server or the inference engine server via application programming interface (API) debugging.

[0063] In this example, if a parameter passthrough exception occurs, you can bypass the interactive interface server through API debugging, directly set the target hyperparameters, and then retrieve the actual hyperparameters again. If the actual hyperparameters now match the target hyperparameters, this indicates a problem with the interactive interface server sending the parameter information. If the actual hyperparameters still differ from the target hyperparameters, this indicates a problem with the inference engine server, such as overwriting default values ​​or losing precision.

[0064] In this example, if an output determinism anomaly occurs, you can bypass the interactive interface server through API debugging, directly set the target hyperparameters, and re-obtain the byte strings obtained from multiple rounds of testing for comparison. If inconsistent byte strings still appear, this indicates a problem on the inference engine server side, such as due to parallel accumulation. If the inconsistent byte strings disappear, this indicates a problem on the interactive interface server side, such as due to input rewriting or parameter tampering.

[0065] Since the entire link is long, API debugging can help locate the potential location of the problem, which in turn facilitates rapid improvement and optimization after testing.

[0066] Figure 3 is a schematic diagram illustrating locating a potential location where an anomaly occurs on a test link according to an exemplary embodiment.

[0067] like Figure 3 As shown, the parameter pass-through exception is described as an example, in which the target hyperparameter T is set to 0 on the interactive interface server during the test, but the actual hyperparameter obtained on the inference engine server is not 0. Therefore, through API debugging, you can skip the interactive interface server and send a request directly to the inference engine server to set the target hyperparameter T to 0, and obtain the actual hyperparameter again. If the returned target hyperparameter T is 0, it means that the problem comes from the interactive interface server, and if the returned target hyperparameter T is not 0, it means that the problem comes from the inference engine server. In this way, the potential location of the exception can be located on the test link.

[0068] In some embodiments, the step of providing the test case to the target large language model via the interactive interface server via the first distribution unit and the second distribution unit (such as combining Figure 2 Sub-step S2021 shown) may include the following steps: placing the test case into a test queue via a first distribution unit; determining whether the number of test cases being processed by the target large language model is less than a preset maximum threshold to be processed; in response to determining that the number of test cases being processed by the target large language model is less than the maximum threshold to be processed, inputting the test case from the test queue into the interactive interface server via a second distribution unit; and in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum threshold to be processed, pausing the acquisition of test cases from the test queue.

[0069] In the example, the monitor of the test link can obtain the operating status of the target large language model and expose the operating status to the second distribution unit, so that the second distribution unit can determine whether the number of test cases being processed by the target large language model is less than the maximum pending threshold. The second distribution unit will continue to distribute test cases only when the target large language model still has residual processing capacity. If there is no residual processing capacity, test cases will not be temporarily obtained from the test queue until the target large language model generates new processing capacity. In addition, the first distribution unit can also be notified to suspend placing test cases into the test queue.

[0070] Figure 4 is a schematic diagram illustrating distribution of test cases via a first distribution unit and a second distribution unit according to an exemplary embodiment.

[0071] like Figure 4 As shown, the first distribution unit can put the test case into the test queue, and the second distribution unit can take the test case out of the test case and distribute it to the target large language model to be tested for processing. However, since a maximum pending threshold is set for determining whether to continue distribution, when the second distribution unit knows through the information exposed by the monitor that the number of test cases being processed by the target large language model is less than the maximum pending threshold, the distribution of the test case can continue. When the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold, the distribution of the test case is suspended until the target large language model recovers its remaining processing capacity.

[0072] In this way, the collaboration between the first distribution unit and the second distribution unit can be used to achieve a balanced distribution of test cases, avoiding overdrawing the processing capacity of the target large language model to be tested during the test process, thereby ensuring the smoothness and effectiveness of the test process.

[0073] In some embodiments, the test case may include prompt words for guiding the target large language model to be tested to output processing results in a predetermined format.

[0074] In an example, a data set used for testing may be processed to have the same format, wherein reference may be made to the way in which real users ask questions, such as “Please help me answer this question, the answer should be output in the format of…”.

[0075] In this way, the target large language model to be tested can output processing results in a unified format according to the indicated requirements, which helps to simplify the comparison process of processing results and improve test efficiency.

[0076] In some embodiments, a test case may include at least one test question, which may have a predetermined answer. Accordingly, the method for testing a large language model may further include the following steps: obtaining, for at least one of the multiple rounds of testing, a processing result generated by the target large language model in response to the test case from the interactive interface server; determining whether the processing result matches the predetermined answer; and, in response to determining that the processing result does not match the predetermined answer, determining that an accuracy anomaly exists in the target large language model.

[0077] In this example, conversation logs can be obtained from an interactive interface server that has completed testing. After all interactive interface servers for the same target large language model have completed testing, the conversation logs are merged to produce a merged conversation log. The fields in this merged conversation log can then be analyzed to obtain the test questions and the answers given by the target large language model, i.e., the processing results generated by the target large language model in response to the test case. For large language models that provide thought processes, procedural answers from the thought process can also be filtered out from the merged conversation log to avoid affecting the test results.

[0078] In this example, you can compare the answer given by the target large language model with the predetermined answer using either an exact match or a maximum similarity match (with a threshold set to 0.9). When testing multiple large language models, you can also compare the output accuracy of each language model to generate an accuracy comparison report.

[0079] An embodiment of the present disclosure also provides a device for testing a large language model.

[0080] Figure 5 is a schematic block diagram illustrating an apparatus 500 for testing a large language model according to an exemplary embodiment.

[0081] In some embodiments, the apparatus 500 is configured to execute on at least two test links. A target large language model from at least one large language model is deployed on each test link via an inference engine server. Furthermore, each test link is sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and a second number of interactive interface servers communicatively connected to the inference engine server. The second number is N times the first number, where N is a natural number greater than or equal to 2.

[0082] The apparatus 500 includes a hyperparameter setting module 501, a first test execution module 502, a first result comparison module 503, and a first anomaly determination module 504. The first test execution module 502 further includes a test case providing module 5021, a first result obtaining module 5022, and a result conversion module 5023.

[0083] The hyperparameter setting module 501 is configured to set target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of word units predicted by the target large language model.

[0084] The first test execution module 502 is configured to perform multiple rounds of testing. For each round of testing: the test case providing module 5021 is configured to provide the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit; the first result acquisition module 5022 is configured to obtain the unnormalized original prediction score obtained by the target large language model in response to the test case word prediction; the result conversion module 5023 is configured to convert the unnormalized original prediction score into a byte string.

[0085] The first result comparison module 503 is configured to compare the byte strings obtained from executing multiple rounds of tests to determine whether there are inconsistent byte strings.

[0086] The first anomaly determination module 504 is configured to determine that an output deterministic anomaly occurs in the target large language model on the test link in response to determining that an inconsistent byte string exists.

[0087] The operations of the above-mentioned hyperparameter setting module 501, the first test execution module 502, the first result comparison module 503 and the first abnormality determination module 504 can be combined with Figure 2 The operations of steps S201, S202, S203 and S204 are the same. In addition, the test case providing module 5021, the first result obtaining module 5022 and the result conversion module 5023 can be combined with Figure 2 The operations of sub-steps S2021, S2022 and S2023 are the same, so the details of each aspect are not repeated here.

[0088] In some embodiments, the apparatus 500 may further include a second test execution module 505, a second result comparison module 506, and a second anomaly determination module 507. For any round of multiple test rounds: the second test execution module 505 is configured to obtain actual hyperparameters used by the target large language model in the process of predicting word units in response to test cases; the second result comparison module 506 is configured to determine whether the actual hyperparameters are the same as the target hyperparameters; and the second anomaly determination module 507 is configured to determine that a parameter transparent transmission anomaly occurs in the target large language model on the test link in response to the actual hyperparameters being different from the target hyperparameters.

[0089] In some embodiments, the device 500 may also include: an exception locating module 508, which is configured to locate the potential location of the exception on the test link to the interactive interface server or the inference engine server via API debugging in response to determining that at least one of an output deterministic exception or a parameter transmission exception occurs.

[0090] In some embodiments, the test case providing module 5021 may include: a first providing module 5021a, configured to place the test case into the test queue via a first distribution unit; a threshold judgment module 5021b, configured to determine whether the number of test cases being processed by the target large language model is less than a preset maximum pending threshold; a second providing module 5021c, configured to input the test case from the test queue into the interactive interface server via the second distribution unit in response to determining that the number of test cases being processed by the target large language model is less than the maximum pending threshold; and a pause providing module 5021d, configured to pause obtaining the test case from the test queue in response to determining that the number of test cases being processed by the target large language model is greater than or equal to the maximum pending threshold.

[0091] In some embodiments, the test case may include at least one test question, which may have a predetermined answer. Accordingly, the apparatus 500 may further include a third test execution module 509, a third result comparison module 510, and a third anomaly determination module 511. For at least one round of multiple rounds of testing: the third test execution module 509 is configured to obtain the processing result generated by the target large language model in response to the test case from the interactive interface server; the third result comparison module 510 is configured to determine whether the processing result matches the predetermined answer; and the third anomaly determination module 511 is configured to determine that the target large language model has an accuracy anomaly in response to determining that the processing result does not match the predetermined answer.

[0092] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be separated into multiple modules, and / or at least some functions of multiple modules may be combined into a single module. A specific module as discussed herein performing an action may include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Thus, a specific module that performs an action may include the specific module itself that performs the action and / or another module that the specific module calls or otherwise accesses to perform the action.

[0093] It should also be understood that the above Figure 5The various modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions that are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuits. For example, these modules can be implemented together in a system on chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform the functions.

[0094] According to one aspect of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any one of the method embodiments described above.

[0095] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method embodiment described above are implemented.

[0096] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of any one of the method embodiments described above are implemented.

[0097] In the following, combined Figure 6 Illustrative examples of such a computer device, non-transitory computer-readable storage medium, and computer program product are described.

[0098] Figure 6 An example configuration of a computer device 600 is shown that may be used to implement the methods described herein.

[0099] Computer device 600 can be a variety of different types of devices. Examples of computer device 600 include, but are not limited to, a desktop computer, a server computer, a laptop or netbook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless phone (e.g., a smartphone), a notepad computer, a mobile station), a wearable device (e.g., eyeglasses, a watch), an entertainment device (e.g., an entertainment appliance, a set-top box communicatively coupled to a display device, a game console), a television or other display device, an automobile computer, and the like.

[0100] Computer device 600 may include at least one processor 602, memory 604, communication interface(s) 606, a display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612, all capable of communicating with one another, such as through a system bus 614 or other appropriate connections.

[0101] The processor 602 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. Among other capabilities, the processor 602 may be configured to retrieve and execute computer-readable instructions stored in the memory 604, mass storage device 612, or other computer-readable media, such as program code for an operating system 616, program code for application programs 618, program code for other programs 620, and the like.

[0102] Memory 604 and mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by processor 602 to implement the various functions described above. For example, memory 604 may generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). Additionally, mass storage device 612 may generally include a hard drive, a solid-state drive, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, and the like. Memory 604 and mass storage device 612 may be collectively referred to herein as memory or computer-readable storage media and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 602 as a specific machine configured to implement the operations and functions described in the examples herein.

[0103] A number of programs may be stored on the mass storage device 612. These programs include an operating system 616, one or more application programs 618, other programs 620, and program data 622, and may be loaded into the memory 604 for execution. Examples of such applications or program modules may include, for example, Figure 2 Computer program logic (eg, computer program code or instructions) of the illustrated methods and / or other embodiments described herein.

[0104] Although Figure 66 as being stored in the memory 604 of the computer device 600, but the modules 616, 618, 620, and 622, or portions thereof, may be implemented using any form of computer-readable media accessible by the computer device 600. As used herein, "computer-readable media" includes at least two types of computer-readable media, namely, computer-readable storage media and communication media.

[0105] Computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVDs), or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices, or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computer device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. Computer-readable storage media, as defined herein, does not include communication media.

[0106] One or more communication interfaces 606 are used to exchange data with other devices, such as through a network, a direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), a wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth TM The communication interface 606 may include a wireless network interface, a near field communication (NFC) interface, and the like. The communication interface 606 may facilitate communication within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, and the like. The communication interface 606 may also provide for communication with external storage devices (not shown) such as storage arrays, network attached storage, storage area networks, and the like.

[0107] In some examples, a display device 608 such as a monitor may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include a touch input device, a gesture input device, a camera, a keyboard, a remote control, a mouse, a printer, an audio input / output device, and the like.

[0108] The techniques described herein can be supported by these various configurations of computer device 600 and are not limited to the specific examples of the techniques described herein. For example, the functionality can also be implemented in whole or in part in a "cloud" using a distributed system. The cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud's hardware (e.g., servers) and software resources. Resources can include applications and / or data that can be used when performing computing processing on servers remote from computer device 600. Resources can also include services provided over the Internet and / or through subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 600 with other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, functionality can be implemented partially on computer device 600 and partially through a platform that abstracts the functionality of the cloud.

Claims

1. A method for testing a large language model, characterized in that: The method is configured to be executed on at least two test links, wherein a target large language model of at least one large language model is deployed on each test link via an inference engine server, and each test link is sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and the second number of interactive interface servers communicatively connected to the inference engine server, wherein the second number is N times the first number, where N is a natural number greater than or equal to 2. The method includes: Setting target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of word units predicted by the target large language model; Perform multiple rounds of testing, and for each round: Providing the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit; Obtaining an unnormalized raw prediction score obtained by the target large language model in response to predicting a word unit of the test case; Converting the unnormalized raw prediction scores into a byte string; Comparing the byte strings obtained from executing the multiple rounds of testing to determine whether there are inconsistent byte strings; and In response to determining that the inconsistent byte string exists, it is determined that an output deterministic anomaly occurs in the target large language model on the test link.

2. The method according to claim 1, characterized in that The method further comprises: For at least one round of the multiple testing rounds: Obtaining actual hyperparameters used by the target large language model in a process of predicting word units in response to the test case; determining whether the actual hyperparameter is the same as the target hyperparameter; and In response to the actual hyperparameter being different from the target hyperparameter, it is determined that a parameter transparent transmission anomaly occurs in the target large language model on the test link.

3. The method according to claim 2, characterized in that The method further comprises: In response to determining that at least one of the output deterministic anomaly or the parameter transparent transmission anomaly occurs, the potential location of the anomaly on the test link is located to the interactive interface server or the inference engine server via application programming interface debugging.

4. The method according to any one of claims 1 to 3, characterized in that Providing the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit includes: placing the test case into a test queue via the first distribution unit; Determining whether the number of the test cases being processed by the target large language model is less than a preset maximum threshold to be processed; In response to determining that the number of the test cases being processed by the target large language model is less than the maximum to-be-processed threshold, inputting the test cases from the test queue to the interactive interface server via the second distribution unit; and In response to determining that the number of the test cases being processed by the target large language model is greater than or equal to the maximum to-be-processed threshold, pausing obtaining the test cases from the test queue.

5. The method according to any one of claims 1 to 3, characterized in that The test case includes prompt words for guiding the target large language model to output processing results in a predetermined format.

6. The method according to claim 5, characterized in that The test case includes at least one test question, the test question having a predetermined answer, wherein the method further comprises: For at least one round of the multiple testing rounds: Obtaining, from the interactive interface server, the processing result generated by the target large language model in response to the test case; determining whether the processed result matches the predetermined answer; and In response to determining that the processing result does not match the predetermined answer, it is determined that an accuracy abnormality occurs in the target large language model.

7. A device for testing a large language model, characterized in that: The apparatus is configured to execute on at least two test links, wherein a target large language model of at least one large language model is deployed on each test link via an inference engine server, and each test link is sequentially provided with a first number of first distribution units and a second number of second distribution units for distributing test cases, and the second number of interactive interface servers communicatively connected to the inference engine server, wherein the second number is N times the first number, where N is a natural number greater than or equal to 2, wherein the apparatus includes: a hyperparameter setting module, configured to set target hyperparameters of the target large language model via the interactive interface server, wherein the target hyperparameters are used to adjust the probability distribution of word units predicted by the target large language model; The first test execution module is configured to execute multiple rounds of testing, and for each round of testing: a test case providing module, configured to provide the test case to the target large language model through the interactive interface server via the first distribution unit and the second distribution unit; A first result acquisition module is configured to obtain an unnormalized original prediction score obtained by the target large language model in response to predicting a word unit of the test case; A result conversion module configured to convert the unnormalized raw prediction score into a byte string; A first result comparison module is configured to compare the byte strings obtained from executing the multiple rounds of testing to determine whether there are inconsistent byte strings; and The first abnormality determination module is configured to determine that an output deterministic abnormality occurs in the target large language model on the test link in response to determining that the inconsistent byte string exists.

8. A computer device, characterized in that: The computer device comprises: at least one processor; and A memory having a computer program stored thereon, wherein when the computer program is executed by the at least one processor, the at least one processor is caused to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

Images