A large language model evaluation method, server and computer readable storage medium

Abstract

The application provides a large language model evaluation method, a server and a computer readable storage medium, and relates to the technical field of computer processing. The method comprises the following steps: obtaining a first test set; the first test set comprises a plurality of first test data; the first test data comprises first format information; the first format information is used for specifying that the output format of a language model is a first format; obtaining a first result set output by a first large language model based on the first test set; the first result set comprises a plurality of first result data; the plurality of first result data correspond to the plurality of first test data one by one; the format of the first result data is the first format; obtaining a first evaluation result based on the first result set and a label set; the first evaluation result is used for evaluating the capability of the first large language model; the label set comprises a plurality of label data; the plurality of label data correspond to the plurality of first test data; and the format of the label data is the first format. Through the above method, the efficiency of large language model evaluation can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer processing technology, and in particular to a method for evaluating large language models, a server, and a computer-readable storage medium. Background Technology

[0002] Large language models (LLMs) are deep learning models trained on large amounts of text data. They can generate corresponding natural language text based on input natural language text, or understand the meaning of input natural language text. Currently, LLMs can be used to handle tasks such as text classification, question answering, and dialogue.

[0003] The development and completion of a large language model includes pre-training, fine-tuning, and deployment. At each stage, the capabilities of the large language model need to be evaluated. Because large language models differ at different stages, developers need to manually evaluate the model at each stage. This results in low efficiency for evaluating large language models. Summary of the Invention

[0004] In view of this, this application provides a method for evaluating large language models, a server, and a computer-readable storage medium, which improves the efficiency of large language model evaluation.

[0005] In a first aspect, this application provides a method for evaluating a large language model. The method includes: obtaining a first test set; the first test set includes multiple first test data, each first test data including first format information, the first format information being used to specify that the output format of the language model is a first format; obtaining a first result set output by a first large language model based on the first test set; the first result set includes multiple first result data, each first result data corresponding one-to-one with each of the multiple first test data, the first result data being in the first format; obtaining a first evaluation result based on the first result set and a label set; the first evaluation result being used to evaluate the capabilities of the first large language model; the label set includes multiple labeled data, each labeled data corresponding one-to-one with each of the multiple first test data, the labeled data being in the first format.

[0006] The first test data can be of any type that the large text language model can process; there are no restrictions on the specific type of the first test data here.

[0007] The capabilities of a large language model include one or more of the following: factual question answering, reading comprehension, frame generation, paragraph rewriting, summary extraction, mathematical problem-solving, reasoning, poetry generation, or programming domains. Understandably, the capabilities of a large language model are not limited to the examples listed above, and there may be similarities between two of its capabilities. For instance, both reading comprehension and summary extraction capabilities of a large language model may require information extraction. Furthermore, the capabilities of a large language model can be described in other ways, such as information extraction capabilities and sentiment analysis capabilities.

[0008] In the above method, the first result data is the output of the first large language model processing the first test data, so the first result data and the first test data have a one-to-one correspondence. Furthermore, since the first test data and the labeled data also have a one-to-one correspondence, the first result data and the labeled data also have a one-to-one correspondence. In addition, both the first result data and the labeled data are in the same format, so the server can efficiently process result data and labeled data with the same format, thereby efficiently obtaining the first evaluation result. Thus, the above method can effectively improve the efficiency of large language model evaluation.

[0009] In one possible implementation of the first aspect, obtaining the first test set includes: obtaining a first dataset, a first prompt word template set, and a first format; the first dataset includes multiple data sets, and the first prompt word template set includes at least one first prompt word template; and generating the first test set based on the first dataset, the first prompt word template set, and the first format.

[0010] In the above implementation, the server can first merge the data in the first dataset, the first prompt word templates in the first prompt word template set, and the first format to generate multiple first test data sets. This allows for the generation of a first test set containing multiple first test data sets. By combining a small amount of data with a small number of first prompt word templates, a large amount of first test data can be obtained. Using these multiple first test data sets allows for more effective evaluation of large language models.

[0011] In one possible implementation of the first aspect, the first format is a pre-configured format; or, the first format is a custom format.

[0012] In one possible implementation of the first aspect, the method further includes: receiving a first instruction from an electronic device; the first instruction is used to specify the output format of the language model as a first format.

[0013] In one possible implementation of the first aspect, obtaining the first evaluation result based on the first result set and the annotation set includes: using test code to compare and analyze each first result data in the first result set with the corresponding annotation data in the annotation set to obtain multiple comparison results; and obtaining the first evaluation result based on the multiple comparison results and evaluation indicators.

[0014] Since the format of the first result data in the first result set is the first format and the format of the annotation data in the annotation set is the first format, test code can be used to compare and analyze the first result data and annotation data with the same format, thereby effectively improving the efficiency of evaluating large language models.

[0015] In one possible implementation of the first aspect, the first major language model is able to understand the first format information;

[0016] Before obtaining the first test set, the method also includes: obtaining a second test set, which is used to test the ability of the first language model to understand the first format information; and testing the ability of the first language model to understand the first format information based on the second test set.

[0017] In the above implementation process, when the server tests the ability of the first language model to understand the first format information based on the second test set, it can first obtain the test results output by the first language model based on the second test set, and determine the ability of the first language model to understand the format information based on the test results; wherein, the test data included in the second test set contains format information, and the test results include the test result data output by the first language model based on the test data in the input second test set; if the format of the test result data is the first format, it indicates that the first language model can understand the format information, and if the format of the test result data is not the first format, it indicates that the first language model cannot understand the format information.

[0018] When the first language model can understand the format information, it can be guaranteed that the format of the first result data output by the first language model is the first format. Thus, the first evaluation result can be obtained efficiently from the first result set and the annotation set, which are both in the first format.

[0019] In one possible implementation of the first aspect, the method further includes: in response to the first language model's inability to understand the first format information, obtaining a third test set, the third test set comprising multiple second test data, generated based on the first dataset and the first prompt word template set; obtaining a second result set output by the first language model based on the third test set; the second result set comprising multiple second result data, each corresponding one-to-one with multiple second test data; obtaining a fourth test set based on the third test set, the second result set, and the second format information; the fourth test set being used to indicate the ability of the second language model to analyze the first language model, the fourth test set comprising multiple third test data, each third test data comprising one second test data, second result data corresponding to the second test data, and second format information; the second format information being used to specify that the language model's output format is a second format; obtaining a third result set output by the second language model based on the fourth test set; the third result set comprising multiple third result data, each corresponding one-to-one with multiple third test data, the third result data being in the second format; and obtaining a second evaluation result based on the third result set. The second format information and the first format information may be the same or different. Specifically, the first format information and the second format information can be determined based on the user's actual needs.

[0020] In one possible implementation of the first aspect, obtaining the second evaluation result based on the third result set includes: obtaining the second evaluation result based on the third result set and the evaluation metrics.

[0021] In one possible implementation of the first aspect, the method further includes receiving evaluation metrics from an electronic device. The evaluation metrics may include at least one of accuracy, precision, and recall.

[0022] In the above implementation, based on the third result set and the evaluation metric, the server can determine the number of correct results and the number of incorrect results in the third result set representing the output of the first language model based on the third test set from the second result set. Then, based on the evaluation metric, the server determines the evaluation result. For example, if the evaluation metric is accuracy, the server can divide the number of correct results in the third result set representing the output of the first language model based on the third test set from the total number of results in the third result set, thus obtaining the evaluation result represented by accuracy.

[0023] In one possible implementation of the first aspect, the method further includes: obtaining a second prompt word template set; the second prompt word template set includes at least one second prompt word template; obtaining a fifth test set based on a first dataset, the second prompt word template set, and a first format; obtaining a fourth result set output by the first large language model based on the fifth test set; the format of the result data included in the fourth result set is the first format; obtaining a third evaluation result based on the fourth result set and the annotation set; the third evaluation result is used to evaluate the capability of the first large language model; obtaining prompt word evaluation results based on the first evaluation result and the third evaluation result, the prompt word evaluation results are used to evaluate the capability of the first prompt word template and the second prompt word template.

[0024] In a second aspect, this application provides a server, which includes a communication module, a memory, and one or more processors; the communication module, the memory, and the processors are coupled; the communication module is used to establish a communication connection and send and receive data through the communication connection, and the memory is used to store computer program code, which includes computer instructions; when the processor executes the computer instructions, it causes the server to perform the method described in the first aspect and any of its possible design embodiments.

[0025] Thirdly, this application provides a computer-readable storage medium including computer instructions that, when executed on a server, cause the server to perform the method described in the first aspect and any of its possible design embodiments.

[0026] Fourthly, this application provides a computer program product that, when run on a server, causes the server to perform the method described in the first aspect above and any possible design of the method.

[0027] Fifthly, this application provides an apparatus included in a server, which has the function of implementing the server behavior in any of the above aspects and possible implementations. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function. For example, an allocation module or unit, a scanning module or unit, a recycling module or unit, a moving module or unit, and a storage module or unit, etc.

[0028] Sixthly, embodiments of this application provide a chip system including a processor and potentially a memory, for implementing any method provided by the first aspect and any of its possible design embodiments. The chip system may be composed of chips or may include chips and other discrete devices.

[0029] Understandably, the server described in the second aspect and any possible design of the above, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0030] Figure 1 A schematic diagram illustrating the development process of a large language model provided in this application embodiment;

[0031] Figure 2 A schematic diagram illustrating a large language model evaluation method provided in an embodiment of this application;

[0032] Figure 3 A schematic diagram illustrating a large language model evaluation method provided in an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of a model evaluation system provided in an embodiment of this application;

[0034] Figure 5 A schematic diagram of the hardware structure of a server provided in an embodiment of this application;

[0035] Figure 6 A flowchart illustrating a large language model evaluation method provided in this application embodiment. Figure 1 ;

[0036] Figure 7 This is a schematic diagram of an operation display interface for large language model evaluation provided in an embodiment of this application;

[0037] Figure 8 A schematic diagram of a large language model evaluation provided in this application embodiment. Figure 1 ;

[0038] Figure 9 A flowchart illustrating a large language model evaluation method provided in this application embodiment. Figure 2 ;

[0039] Figure 10 A schematic diagram of a large language model evaluation provided in this application embodiment. Figure 2 ;

[0040] Figure 11 A flowchart illustrating a large language model evaluation method provided in this application embodiment. Figure 3 ;

[0041] Figure 12 A schematic diagram of a large language model evaluation provided in this application embodiment. Figure 3 . Detailed Implementation

[0042] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships may exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0043] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0044] Before introducing the embodiments of this application, the technologies involved in the embodiments of this application will be described in detail.

[0045] 1. Large Language Model

[0046] Large language models are deep learning models capable of processing and generating complex natural language. The training process of large language models utilizes massive amounts of text data; therefore, they are able to generate accurate response text for input natural language.

[0047] 2. Prompt

[0048] The input text for a large language model is also known as a prompt. A prompt can be understood as a way for users to interact with the large language model based on natural language. In related technologies, a prompt can be summarized as input text that includes any one of the following: instruction, context, input data, or output indicator.

[0049] To achieve better evaluation of a large language model using limited data during the evaluation process, the prompt in this embodiment can be understood as input text comprising two parts: data (as described above) and a prompt template (as described above). The data can be understood as the question posed to the large language model. The prompt template can be understood as supplementing the question with examples or background knowledge before and after it. The main purpose of the prompt template is to stimulate the large language model's abilities in contextual few-shot learning, zero-shot learning, and thought chaining, guiding it to generate content that satisfies the user in response to the question. Thus, by combining a small amount of data and a small amount of prompt templates, a large amount of evaluation data can be obtained, which can then be used to more effectively evaluate the large language model.

[0050] Example 1:

[0051] Data: What is a cat?

[0052] prompt template: Can you explain () to me?

[0053] The above data and the prompt template can be combined to form prompt1: Can you explain to me what a cat is?

[0054] Using prompt1 as input text for a large speech model, the large language model can output the resulting text. The output text of the large speech model can be called Result Text 1, specifically: Cats belong to the Felidae family. Cats are small, with coat colors ranging from blue-gray to brownish-yellow. They are slender, 0.3-0.5 meters in length, with dense, soft fur, small collarbones, short muzzles, round eyes, thick necks, relatively short limbs, and several spherical paw pads; their tongues are covered with keratinized, filamentous, hook-shaped papillae. Male and female cats are similar, except that males have thicker, rounder heads. Cats have a long reproductive and fertile period, with a lifespan of 12-17 years.

[0055] Example 2:

[0056] Data: What is a cat?

[0057] Prompt template: If I were a primary school student and you were a teacher, could you explain () to me?

[0058] The above data and the prompt template can be combined to form prompt2: If I were a primary school student and you were a teacher, could you explain to me what a cat is?

[0059] Using prompt2 as input text for the large speech model, the large language model can output the result text. The result text output by the large language model can be called result text 2, specifically: Cats are cute animals. They have pointed ears, wet noses, small mouths, and a pair of bright, piercing eyes that sparkle in the dark.

[0060] In prompt1 of Example 1, the questioner only provided the question to the large language model without informing it of their identity. Therefore, after processing prompt1, the large language model returned an answer that did not consider the questioner's identity, outputting information about cats as the result text. As can be seen in the result text 1 output by the large language model in Example 1 above, it describes the cat from biological characteristics.

[0061] In prompt2 of Example 2, the questioner provides the same question to the large language model and informs the large language model that the questioner is a primary school student. That is to say, prompt1 and prompt2 are composed of different prompt templates. Therefore, after processing prompt2, the large language model outputs text2, which uses language that is easier for primary school students to understand to introduce the cat.

[0062] Based on the two examples above, it can be seen that when the questioner wants to learn about "cats", the response of the large language model changes because the text input by the questioner includes different knowledge backgrounds, i.e., the prompt template changes.

[0063] The following describes the large language model evaluation scheme provided in the embodiments of this application with reference to the accompanying drawings.

[0064] The development process of a large language model includes a pre-training phase and a fine-tuning phase. In the pre-training phase, developers train the model using large-scale unlabeled natural language text, enabling it to learn common language patterns and thus improve its expressive and generalization abilities. In the fine-tuning phase, developers train the model using carefully designed question-and-answer dialogues (users mimicking human-language interaction) to allow it to process the input text more accurately and output higher-quality text. For example... Figure 1 As shown, before the pre-training phase, developers need to evaluate the large language model A in order to formulate corresponding pre-training strategies based on the evaluation results to obtain the large language model B. Before the fine-tuning phase, developers need to evaluate the large language model B in order to formulate corresponding fine-tuning strategies based on the evaluation results to continue training the model to obtain the large language model C.

[0065] During the pre-training or fine-tuning phases, the capabilities of the large language model change as it is trained. Consequently, the output format of the large language model for the same input will vary. Therefore, during the pre-training or fine-tuning phases, developers need to customize corresponding test code for the different output formats of the large language model to analyze the different output formats at different stages, and then evaluate the model based on the analysis results.

[0066] In addition, before the large language model goes live, developers may need to evaluate it to make a final confirmation of its capabilities before deployment. For example... Figure 1 As shown, developers need to evaluate the pre-deployment large language model C to formulate corresponding pre-deployment adjustment strategies based on the evaluation results, thereby adjusting the large language model C to obtain the large language model D. After obtaining the large language model D, developers may also need to evaluate the deployed large language model D to facilitate timely maintenance or modification of it. During the above process, differences may exist between the large speech model A, large speech model B, large language model C, and large language model D. Therefore, developers also need to customize test code to analyze the different formats of the output text from the large language models at different stages, facilitating the determination of evaluation results for the large language models at different stages based on the analysis results.

[0067] For example, before evaluating a large language model, developers need to determine the test input set and annotation set used for testing the model. The test input set includes multiple input texts that can be used as input to the large language model. The annotation set includes annotation text corresponding to each input text in the test input set. After inputting the input texts from the test input set into the large language model, the model can output corresponding response texts, such as result texts. By comparing the annotation texts corresponding to each input text in the test input set with the result texts, developers can determine the differences between the annotation texts and the result texts, and thus evaluate the capabilities of the large language model based on these differences.

[0068] Example 3:

[0069] Input text: Please determine the time and location from the following given text. Text: "Please go to the park tonight."

[0070] Result text: Tonight; the park.

[0071] Label text: Time: Tonight; Location: Park.

[0072] Example 4:

[0073] Input text: Please determine the time and location from the following given text. Text: "Please go to the park tonight."

[0074] Result text: Evening; Park.

[0075] Label text: Time: Tonight; Location: Park.

[0076] In Example 3 above, comparing the result text and the annotated text reveals that the large language model can accurately obtain the desired result text based on the input text. Therefore, it can be concluded that the large language model's capabilities meet the user's needs. In Example 4 above, comparing the result text and the annotated text reveals that the large language model's result text, based on the input text, only specifies "evening" for the time field, without accurately specifying "tonight." Therefore, it can be concluded that the large language model's capabilities are insufficient, and developers need to continue training it to better meet user needs.

[0077] The example above illustrates how to evaluate the capabilities of a large language model using a single input text. Understandably, to more accurately evaluate the capabilities of a large language model, the test input set includes a large number of input texts, and correspondingly, the labeled text set also contains a large amount of data. For example... Figure 2 As shown, developers input multiple input texts from the test input set into a large language model, causing the model to output corresponding result texts. These multiple output result texts can form a result set. Due to the large number of texts, manually comparing each result text in the result set with its corresponding labeled text in the annotation set would be extremely time-consuming. Therefore, developers need to write appropriate test code for the result set containing multiple result texts and their corresponding labeled texts. This test code can then compare the labeled text in the annotation set with the result text in the result set, analyze the differences, and thus evaluate the capabilities of the large language model based on the comparison results.

[0078] However, when the large language model changes, Figure 2 The input text from the test input set shown is fed into the modified large language model. Due to the change in the language processing capabilities of the modified large language model, the modified large language model outputs a new result set based on this test input set. Figure 2 The result set shown is different; that is, the format of the result text in the new result set is different from the format of the annotation text. This causes developers to... Figure 2 The test code written in the example cannot compare the result text in the new result set with the labeled text in the labeled set, and therefore cannot be used to evaluate the capabilities of the changed large language model.

[0079] To evaluate the changed large language model, developers need to rewrite new test code based on the new result set and annotation set. Furthermore, developers need to rewrite the test code every time the large language model changes, which reduces the efficiency of evaluating the large language model.

[0080] Therefore, embodiments of this application provide a method for evaluating large language models, such as... Figure 3 As shown, developers can specify the output format of a large language model. When input text from the test input set is fed into the large language model, it outputs the resulting text according to the specified format. Because the format of the resulting text is predefined, the server can execute pre-written test code to perform a differential comparison between the output text and the labeled text, thereby evaluating the capabilities of the large language model. Furthermore, even if the large language model changes, and the actual content of the changed resulting text also changes, the server can still use pre-written test code to perform a differential comparison between the changed output text and the labeled text, thus evaluating the capabilities of the changed large language model. In this way, developers do not need to write different test code for different large language models, improving the efficiency of large language model evaluation.

[0081] The above description uses the example where the input format of the large language model is specified by the developers. In other examples, the output format of the large language model can also be a pre-defined default format.

[0082] The large language model evaluation method provided in this application embodiment can be applied to model evaluation systems.

[0083] In some embodiments, such as Figure 4 As shown, the model evaluation system may include at least one electronic device 100 and a server 200. A large language model can be deployed on the server 200. Users can send specified output format instructions to the large language model on the server 200 via the electronic device 100, so that the large language model can output the result text according to the specified output format. Additionally, users can also input input text into the large language model on the server 200 via the electronic device 100, so that the large language model outputs corresponding result text based on the input text. Of course, the format of this result text is the specified output format. Evaluation results can then be obtained by analyzing the differences between the result text in the specified output format and the corresponding annotated text. Afterwards, the server 200 sends the evaluation results back to the electronic device 100, so that the electronic device 100 can display the evaluation results to the user.

[0084] In some examples, a user can send evaluation instructions to a large language model in server 200 via electronic device 100. These instructions include the aforementioned instructions specifying the output format of the large language model, multiple input texts, and multiple corresponding labeled texts. Server 200 can obtain test input text based on each input text and the instructions specifying the output format, enabling the large language model in server 200 to process the test input text and obtain result text in the specified output format. Then, server 200 can use generic test code to analyze the differences between the result text in the specified output format and the labeled text, and obtain evaluation results based on the analysis. Server 200 can then send the evaluation results to electronic device 100 for display to the user.

[0085] In some examples, the electronic devices in this model evaluation system can specifically be mobile phones, tablets, smart screens, laptops, in-vehicle devices, wearable devices (such as smartwatches), ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), artificial intelligence devices, and other electronic devices. This application does not limit the specific type of electronic device or the operating system installed.

[0086] In some examples, the server in the model evaluation system can be a cloud server or a web server, or a device with natural language processing capabilities. This server can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0087] The following describes the server's hardware structure.

[0088] like Figure 5 As shown, server 200 may include processor 210, memory 220 and communication module 230.

[0089] Processor 210 can be used to read and execute computer-readable instructions. Specifically, processor 210 may include a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In specific implementations, the hardware architecture of processor 210 can be an application-specific integrated circuit (ASIC) architecture, a MIPS (microprocessor without interlocked piped stages) architecture, an ARM (advanced RISC machines) architecture, or a net processor (NP) architecture, etc.

[0090] Memory 220 is coupled to processor 210 and is used to store various software programs and / or sets of instructions. In specific implementations, memory 220 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 may store an operating system, such as uCOS, VxWorks, RTLinux, or other embedded operating systems.

[0091] Communication module 230 can be used to establish communication between cloud server 100 and other communication terminals (such as...) via a network. Figure 4 Communication connections between multiple electronic devices 100 in the network, and for sending and receiving data over the network.

[0092] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on server 200. In other embodiments, server 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0093] Figure 6 A flowchart illustrating a large language model evaluation method provided in this application embodiment. Figure 1 Before the large language model is fully trained but not yet deployed, or after the large language model is deployed, the server can access it via methods such as... Figure 6 The steps shown demonstrate the evaluation of a large language model. This large language model evaluation method includes the following steps:

[0094] S601: The server retrieves instruction 1 for specifying the output format of the large language model.

[0095] Instruction 1 (i.e., the aforementioned "first instruction") is used to specify the output format of the large language model (i.e., the aforementioned "first format"). After the server obtains Instruction 1, it can specify the output format of the large language model through Instruction 1. This Instruction 1 can also be called a general instruction, etc.

[0096] In some embodiments, the specified output format can be any of the following: JavaScript Object Notation (JSON), Extensible Markup Language (XML), or YAML (Yain't Markup Language). JSON format is a sequence of tags that may include six constructor characters and a value. The six constructor characters include a left square bracket ([), a left curly brace ({), a right square bracket (]), a right curly brace (}), a colon (:), and a comma (,). The value can be an object, an array, a number, a string, or one of three literal values ​​(false, null, true).

[0097] Specifically, an object is enclosed in curly braces and consists of comma-separated members, which are strings and comma-separated key-value pairs, as described above. An array consists of a group of values ​​enclosed in square brackets. A string is a collection of any number of Unicode characters enclosed in double quotes ("").

[0098] For explanations of the different formats mentioned above, please refer to relevant technologies; this embodiment will not repeat them here.

[0099] In some embodiments, the output format of the large speech model specified in instruction 1 may be a default format built into the server ("built-in default format" may also be described as "pre-configured format") or a user-defined output format ("user-defined output format" may also be described as "custom format").

[0100] For example, Figure 7 This diagram illustrates an operation display interface for evaluating a large language model. This interface, displayed on an electronic device, is used by the user to configure model evaluation parameters. For example, the interface includes multiple configuration items, including a model output format configuration item, used by the user to configure the output format of the large language model. Figure 7In the operation display interface shown, the model output format configuration item has two options: a default format and a custom format. When the user selects the default format, the electronic device sends the default format as instruction 1 to the server. The server receives instruction 1, which specifies that the large language model uses the default format as its output format. When the user selects the custom format, the user can also input a custom output format into the electronic device. In this case, the electronic device sends the user-input output format as instruction 1 to the server, which receives instruction 1. This instruction 1 specifies that the large language model uses the user-defined output format as its output format. For example, in response to the user selecting... Figure 7 The control 701 corresponding to the output format configuration item of the model, and the format entered in the input box 702, such as JSON format, allow the electronic device to send the JSON format as instruction 1 to the server. The server can then obtain the output format of the specified large language model as the user-defined JSON format.

[0101] S602: The server obtains multiple test input texts based on instruction 1 and multiple input texts.

[0102] It should be noted that the test input text is the aforementioned "first test data". The first test data can be in the form of a text-based large language model. For ease of understanding, the following embodiments of this application will use the first test data as a text type to introduce the large language model evaluation scheme.

[0103] In some embodiments, the server can obtain the output format of the large language model based on instruction 1. Then, the server can merge the input text and the output format into test input text. Understandably, the server can merge instruction 1 and the input text in any manner, and this application embodiment does not limit this. As described in the foregoing embodiments, in some examples, the input text may include both a prompt template and data.

[0104] In some examples, the server can obtain format specification text for the output format of the specified large language model based on the output format specified in instruction 1. The format specification text can be in the form of natural language text or a format sample. The server can then merge this format specification text with the input text to obtain the test input text.

[0105] For example, if instruction 1 specifies that the output format of the large language model is JSON, the server can obtain the specified output format of the large language model as JSON based on instruction 1. Then, the server can obtain the corresponding natural language text, such as: "Please output in JSON format," which is the specified format text (i.e., the aforementioned "first format information") used to specify the output format of the large language model.

[0106] Let's take the input text as: "Please determine the time and location in the following given text." For example, the text: "Please go to the park tonight." Understandably, the input text: "Please determine the time and location in the following given text." The text: "" represents the prompt template; "Please go to the park tonight" represents the data.

[0107] The test input text received by the server could be: Please determine the time and location in the following given text. Text: "Please go to the park tonight." Please output in JSON format.

[0108] Alternatively, the test input text received by the server could be: Please output in JSON format. Please determine the time and location in the following given text. Text: "Please go to the park tonight."

[0109] For example, if instruction 1 specifies that the output format of the large language model is JSON, the server can obtain the specified output format of the large language model as JSON based on instruction 1. Then, the server can obtain the corresponding format example, such as: Please output in JSON format, the output format example is {"time": "tomorrow morning", "location": "school"}, which is the specified text for the above format, used to specify the output format of the large language model.

[0110] Given the input text: Please determine the time and location in the following given text. For example, the text: "Please go to the park tonight."

[0111] The test input text received by the server could be: Please determine the time and location in the following text. Text: "Please go to the park tonight." Please output in JSON format, with an example output format of {"time": "tomorrow morning", "location": "school"}.

[0112] Alternatively, the test input text received by the server could be: Please output in JSON format, with an example output format of {"time": "tomorrow morning", "location": "school"}. Please determine the time and location in the following given text. Text: "Please go to the park tonight."

[0113] In some embodiments, to evaluate the capabilities of a large language model, the server needs to input multiple input texts into the large language model and obtain corresponding result texts. Then, the server analyzes the differences between each result text and its corresponding labeled text, thus obtaining analysis results for multiple result texts. Based on these analysis results, metrics that can be used to evaluate the capabilities of the large language model can be calculated. As mentioned earlier, the server can obtain a large amount of input text based on a small amount of data and a small combination of prompt templates.

[0114] For example, the dataset (i.e., the aforementioned "first dataset") includes the following four types of data (i.e., the aforementioned "first data"):

[0115] Please go to the park tonight, please go to the park tomorrow morning, please go to school tonight, please go to school tomorrow morning.

[0116] The prompt template set (i.e., the aforementioned "first prompt word template set") includes the following two prompt templates (i.e., the aforementioned "first prompt word templates"):

[0117] Please determine the time and location in the following text. Please indicate the time and location in the following text.

[0118] Therefore, the server can obtain the following eight input texts:

[0119] Input Text 1: Please determine the time and location in the following given text. Text: "Please go to the park tonight."

[0120] Input Text 2: Please determine the time and location from the following given text. Text: "Please go to the park tomorrow morning."

[0121] Input text 3: Please determine the time and location in the following given text. Text: "Please go to school tonight."

[0122] Input text 4: Please determine the time and location in the following given text. Text: "Please go to school tomorrow morning."

[0123] Input text 5: Please indicate the time and location in the following text. Text: "Please go to the park tonight."

[0124] Input text 6: Please indicate the time and location in the following text. Text: "Please go to the park tomorrow morning."

[0125] Input text 7: Please indicate the time and location in the following text. Text: "Please go to school tonight."

[0126] Input text 8: Please indicate the time and location in the following text. Text: "Please go to school tomorrow morning."

[0127] In some embodiments, such as Figure 7 The operation display interface shown may also include: dataset configuration items and prompt word template configuration items. The dataset configuration items are used by the user to configure the data in the input text of the large language model, and the prompt word template configuration items are used by the user to configure the prompt templates in the input text of the large language model. For example... Figure 7 In the operation display interface shown, when the user selects dataset A as the dataset configuration item and prompt template A as the prompt template set, the electronic device can send dataset A and prompt template set A to the server. Therefore, the server can obtain dataset A and prompt template set A. Subsequently, the server can obtain multiple input texts based on the data in dataset A and the prompt templates in prompt template set A.

[0128] For example, combining Figure 8 As shown, after the server receives instruction 1, it can obtain a test input text based on an input text and instruction 1. If there are multiple input texts, the server can obtain a test input set A, which includes multiple test input texts, i.e., the aforementioned "first test set".

[0129] S603: The server inputs multiple test input texts into the large language model to be evaluated, and obtains multiple result texts output by the large language model to be evaluated in a specified format.

[0130] As in the example in S602, for scenarios with multiple input texts, the server can obtain a test input set A that includes multiple test input texts. After obtaining the test input set A, the server can input the multiple test input texts included in the test input set A into the large language model to be tested. Then, the large language model to be tested can output the result text corresponding to each test input text in the test input set A according to a specified format. The server can then obtain the result text corresponding to each test input text in the test input set A output according to the specified format (i.e., the aforementioned "first result data"), thus obtaining the result set (i.e., the aforementioned "first result set").

[0131] Continuing with the example in S602 above, the test input text is: Please determine the time and location in the following given text. Text: "Please go to the park tonight." Please output in JSON format, with an example output format of {"time": "tomorrow morning", "location": "school"}.

[0132] Therefore, after the server inputs the above test input text into the evaluation large language model, the output result text in the specified format can be obtained as: {"time": "Tonight", "location": "Park"}.

[0133] For example, the test input text is: Please determine the time and location in the following text. Text: "Please go to school tomorrow morning." Please output it in JSON format.

[0134] Therefore, after the server inputs the above test input text into the evaluation large language model, the output result text in the specified format can be obtained as: {"time": "tomorrow morning", "location": "school"}.

[0135] Understandably, after inputting test input text into a large language model, the large language model can obtain result text in a specified format based on the input test input text. Therefore, as... Figure 8 As shown, by inputting multiple test input texts from the test input set A into the large language model, a result text in a specified format corresponding to each test input text can be obtained. In other words, the server can obtain a result set that includes multiple result texts in a specified format.

[0136] S604: The server analyzes each result text in a specified format and the corresponding annotation text to obtain multiple analysis results.

[0137] like Figure 8 As shown, the server analyzes each result text in a specified format and the corresponding annotation text in the result set, and can obtain multiple analysis results.

[0138] The annotation text refers to the test input text used as input to the large language model, representing the user's expected correct output. Typically, the annotation text is manually annotated by annotators after analyzing the features of the input text, and its format matches the output format specified in instruction 1 above. Thus, after the server inputs the test input text into the large language model, and the model outputs the result text in the specified format, the server can compare the result text in the specified format with the corresponding annotation text to obtain the analysis results. Subsequently, the server determines the language analysis capabilities of the large language model based on the analysis results, thus completing the evaluation of the large language model.

[0139] In some embodiments, since the format of the specified result text is consistent with the format of the annotation text, the server can execute pre-written test code to analyze the differences between multiple specified result texts and multiple annotation texts. It should be noted that because the format of the specified result text is consistent with the format of the annotation text, and the pre-written test code is used to compare the result text and annotation text, the pre-written test code is general-purpose code and can be used to analyze the differences between result texts in other formats and their corresponding annotation texts. For example, the server can use the test code to analyze JSON format result text and JSON format annotation text, and it can also analyze XML format result text and XML format annotation text.

[0140] For example, the resulting text is: {"time": "Tonight", "location": "Park"}.

[0141] The annotation text is: {"time": "Tonight", "location": "Park"}.

[0142] The server can then execute pre-written test code to compare the differences between the two and obtain a comparison result. For example, during the execution of the test code, the server can extract the value of "time" ("tonight") and the value of "location" ("park") from the result text. Furthermore, the server can also extract the values ​​of "time" ("tonight") and "location" ("park") from the annotation text. After comparison, the server can obtain a result indicating that the result text and the annotation text are identical, thus indicating that the result text is correct.

[0143] For example, the resulting text is: {"time": "evening", "location": "park"}.

[0144] The annotation text is: {"time": "Tonight", "location": "Park"}.

[0145] So, during the execution of the test code, the server can extract the time value "evening" and the location value "park" from the result text. Furthermore, the server can also extract the time value "tonight" and the location value "park" from the annotation text. After comparison, the server can obtain a comparison result indicating that the result text is different from the annotation text, meaning the result text is incorrect. Understandably, the comparison result in the above example is the analysis result.

[0146] S605: The server obtains the evaluation result based on multiple analysis results (i.e., the aforementioned "first evaluation result").

[0147] In some embodiments, such as Figure 8 As shown, the test input set A includes multiple test input texts, and the large language model will correspondingly output multiple result texts in specified formats. The server can analyze each specified format result text in the result set separately, and the analysis results include comparison results between multiple specified format result texts and their corresponding labeled texts. Subsequently, based on the multiple comparison results in the analysis results, the server can calculate the metrics used to evaluate the large language model, i.e., the evaluation results.

[0148] In some examples, the server can send the evaluation results to an electronic device, which can then display the evaluation results to the user in any one or more forms such as graphics, data, and text.

[0149] In some examples, metrics used to evaluate large language models may include one or more of the following: accuracy, precision, recall, and F1 parameter (F1 parameter = 2 * (precision * recall) / (precision + recall), where F1 parameter is the weighted harmonic average of precision and recall). For details on the above evaluation metrics, please refer to the relevant technical documentation, which will not be elaborated here.

[0150] Optionally, specific evaluation metrics can be set according to the user's actual needs. For example... Figure 7 The operation display interface shown may also include evaluation indicator configuration items. When the user selects indicator A as the evaluation indicator configuration item, the electronic device can send indicator A to the server, instructing the server to calculate indicator A based on the analysis results.

[0151] For example, when the evaluation metric is accuracy, if the test input set A includes N test input texts, then the large language model will output N result texts. If the server finds that M of the N result texts are incorrect when comparing them with their corresponding labeled texts, where M ≤ N, then the server can calculate the evaluation metric accuracy = M / N.

[0152] In some embodiments, the server analyzes the result text and labeled text in a specified format. After obtaining the analysis results, it can compare the current evaluation results with preset evaluation results or historical evaluation results to determine the changes in the current large language model's capabilities. Based on these changes, it can then determine whether further training of the model is necessary. Alternatively, the server can compare the current evaluation results of the large language model with the evaluation results of other large language models to identify the most capable large language model among multiple models.

[0153] In some examples, after a pre-training phase, a large language model V1 version is obtained. To test the information extraction capability of the large language model V1 version, the server can obtain multiple test input texts for information extraction, and these test input texts include information in a specified large language model output format. Then, by inputting the obtained multiple test input texts into the large language model V1 version, it outputs result text in a specified format, that is, extracting information from the test input texts and outputting result text including the extraction results according to the specified format. In this way, the server can obtain multiple result texts in the specified format, i.e., result set A. The server can use general test code to analyze each result text in result set A and the corresponding annotation text in the annotation set to obtain analysis results. Then, the server can obtain corresponding metrics based on the analysis results. Based on the obtained metrics, the server can analyze whether the information extraction capability of the large language model V1 version meets the standards. For example, the server inputs 10 test input texts into the large language model V1 version, causing the large language model V1 version to output 10 result texts in a specified format. Afterwards, the server used test code to compare the 10 result texts with the 10 labeled texts. It found that 5 out of the 10 result texts in result set A were correct. Therefore, the information extraction accuracy of the large language model V1 version can be determined to be 5 / 10 = 50%. If the information extraction accuracy of a large language model is above 70%, it is considered to have satisfactory information extraction capability. In the above example, the information extraction accuracy of the large language model V1 version did not reach 70%, so the server can consider that the information extraction capability of the large language model V1 version is unsatisfactory.

[0154] Accordingly, if the server deems the information extraction capability of the large language model V1 version insufficient, the server can obtain more training texts that can enhance the information extraction capability of the large language model, and use these training texts to continue training the large language model V1 version in order to improve the information extraction capability of the large language model V1 version.

[0155] Continuing the example above, after fine-tuning the large language model V1, the developers obtained large language model V2. Similarly, the server inputs the multiple test input texts used for information extraction into large language model V2, resulting in result set B. If the server analyzes and finds that 8 out of the 10 result texts in result set B are correct, then the information extraction accuracy of large language model V2 can be determined to be 80%. If an information extraction accuracy of over 70% is considered sufficient for a large language model, then in the example above, large language model V2 achieved an accuracy of over 70%, and the server can conclude that large language model V2's information extraction capability meets the standard.

[0156] In other examples, the server evaluates different versions of the large language model and obtains corresponding evaluation metrics. If the server determines that the evaluation metrics of the earlier version of the large language model are better than those of the later version, it indicates that the earlier version is more capable than the later version. In this case, it suggests that the earlier version of the large language model's ability has actually decreased after training, possibly due to problems in the training process, such as issues with the training text used. In other words, the evaluation results can also determine whether there are problems with the training text used to train the large language model and whether adjustments are needed.

[0157] It should be noted that the same test code, or a universal test code, was used when evaluating different versions of the large language model. This is because although the capabilities of different versions of the large language model have changed, and the content of their output text may also change, they all output the text according to the specified output format, so the same test code can be used for evaluation.

[0158] In some examples, after a pre-training phase, a large language model V1 version is obtained. To test the sentiment analysis capabilities of the large language model V1 version, the server can obtain multiple test input texts for sentiment analysis, each containing information in a specified large language model output format. Then, by inputting these multiple test input texts into the large language model V1 version, it outputs result text in a specified format; that is, it performs sentiment analysis on the information in the test input texts and outputs result text including the sentiment analysis results in a specified format. Thus, the server obtains multiple result texts in a specified format, resulting in a result set C. The server can use the same test code used in the examples above to test the information extraction capabilities of the large language model—that is, the general test code—to analyze each result text in the result set C and the corresponding annotation text in the annotation set to obtain the analysis results. The server can then obtain corresponding metrics based on the analysis results. Based on the obtained metrics, the server can analyze whether the sentiment analysis capabilities of the large language model V1 version meet the standards. For example, the server inputs 10 test input texts into the large language model V1 version, causing it to output 10 result texts in a specified format. Afterwards, the server used test code to compare the 10 result texts with the 10 labeled texts. The result set C contained 8 correct result texts out of the 10. Therefore, the sentiment analysis accuracy of the Large Language Model V1 version can be determined to be 8 / 10 = 80%. If a sentiment analysis accuracy of over 70% is considered satisfactory, then in the above example, the sentiment analysis accuracy of the Large Language Model V1 version is over 70%, and the server can consider the sentiment analysis capability of the Large Language Model V1 version to be satisfactory.

[0159] It is understandable that for different capabilities of the same large language model, or the same capability of different large language models (such as different versions of large language models, or large language models with different functions), since the output format of the large language model is specified, the server can use the same test code to analyze the result text and annotation text with the same format to obtain the analysis results. This allows the server to calculate the corresponding evaluation indicators based on the analysis results, thus effectively improving the evaluation efficiency.

[0160] In some embodiments, the server can utilize different types of test input text to evaluate large language models in different domains. The server obtains the test input text based on the input text and instruction 1. The input text includes data and a prompt template; the type of test input text changes as the type of data changes. For example, Figure 7The operation display interface shown includes dropdown options for the dataset configuration item, which may include information extraction and sentiment analysis capabilities corresponding to the above embodiments. Furthermore, the capabilities of the large language model can be described in other ways. For example, the capabilities of the large language model can be described as factual question answering, reading comprehension, frame generation, paragraph rewriting, summary extraction, mathematical problem-solving, reasoning, poetry generation, or programming domain capabilities. Correspondingly, the dropdown options for the dataset configuration item may include different types of datasets for factual question answering, reading comprehension, frame generation, paragraph rewriting, summarizing, mathematical problem-solving, reasoning, poetry generation, and programming. Users can then select the corresponding dataset in the dataset configuration item of the operation display interface according to their actual evaluation needs, thereby enabling the evaluation of the specified capabilities of the large language model. For example, if a reading comprehension dataset is selected in the dataset configuration item, then the test input text obtained by the server is of the reading comprehension type. Using this type of test input text, the reading comprehension ability of the large language model can be evaluated.

[0161] It's important to note that developers can set the dataset type based on the actual application scenario of the large language model to evaluate the capabilities that the model emphasizes in that specific scenario. For example, a large language model used in psychological counseling scenarios may focus more on sentiment analysis capabilities; therefore, a sentiment analysis dataset could be used to evaluate it. Conversely, a large language model used in data statistics and processing scenarios may focus more on information extraction capabilities; therefore, an information extraction dataset could be used to evaluate it.

[0162] Figure 9 A flowchart illustrating a large language model evaluation method provided in this application embodiment. Figure 2 .

[0163] Before the large language model is trained and deployed, or after the large speech model is deployed, the server can use methods such as... Figure 9 The steps shown demonstrate the evaluation of a large language model. This large language model evaluation method includes the following steps:

[0164] S901: The server inputs the input text into the large language model Mt to obtain the output text of the large language model Mt.

[0165] The input text can be obtained based on the dataset and the prompt template. For details, please refer to the description of the input text in the above embodiments, which will not be repeated here.

[0166] like Figure 10As shown, the server inputs each input text (i.e., the aforementioned "second test data") from the test input set (i.e., the aforementioned "third test set") into the large language model Mt, and can obtain multiple result texts (i.e., the aforementioned "second result data") output by the large language model Mt, thus obtaining the result set (i.e., the aforementioned "second result set").

[0167] After inputting the input text into the large language model Mt, Mt can process the input text and obtain the corresponding result text. Since the output format of the large language model Mt is not specified, Mt actually generates the result text according to its logic of processing natural language, and the generated result text is not in a fixed format.

[0168] For example, enter text 1:

[0169] Please determine the time and location in the following text: Text: "Please go to the park tonight."

[0170] Result text 1 corresponding to input text 1:

[0171] The time is tonight, and the place is the park.

[0172] Input text 2:

[0173] Please identify the time and place in the following given text: Text: "Please go to school tomorrow morning."

[0174] Result text 2 corresponding to input text 2:

[0175] {"time": "tomorrow morning", "location": "school"}

[0176] Understandably, in the above example, given two different input texts, text 1 and text 2, for the same large language model, the model might interpret the difference between the actions "determine" in text 1 and "point" in text 2 as requiring different outputs. For example, the model might assume the result text corresponding to the action "determine" in text 1 should be expressed in natural language, thus outputting natural language as shown in text 1. Similarly, the model might assume the result text corresponding to the action "point" in text 2 should be expressed in JSON format, thus outputting JSON as shown in text 2.

[0177] It should be noted that, for the two input texts with minor differences in the above example, when the large language model has strong information extraction capabilities, the probability of content and / or format differences between the output text 1 and output text 2 is low. However, when the large language model has weak information extraction capabilities, the probability of format differences between the output text 1 and output text 2 is high. Thus, by inputting multiple input texts into the large language model, the server can obtain multiple output texts, and then analyze the information extraction capabilities of the large language model based on these multiple output texts.

[0178] The two result texts are in different formats. Result text 1 contains only text, and the server cannot parse the keywords within it using code; therefore, the server cannot analyze result text 1. Result text 2, on the other hand, is in JSON format, and therefore, the server can analyze it.

[0179] Because the server can only analyze a portion of the output text of the large language model Mt and cannot analyze the other portion, it cannot use the output text to evaluate the capabilities of the large language model Mt. Therefore, the server can perform the following steps (S902-S905):

[0180] S902: The server obtains instruction 2, which instructs the large language model Ma to perform analysis operations and specifies the output format of the large language model Ma.

[0181] Instruction 2 is used to instruct the large language model Ma to perform analysis operations and to instruct the large language model Ma to output the analysis results in the specified output format.

[0182] After the server receives instruction 2, it can perform analysis operations on the large language model Ma by following these steps.

[0183] S903: The server combines instruction 2, the result text, and the input text to obtain the analyzed text.

[0184] It should be noted that the specific implementation of server merge instruction 2, the result text, and the input text can be found in [reference needed]. Figure 6 The specific implementation details of the corresponding content in the illustrated embodiments will not be elaborated here.

[0185] like Figure 10As shown, the server can obtain an analysis input set (i.e., the aforementioned "second result set") that includes multiple result texts (i.e., the aforementioned "second result data"), a test input set (i.e., the aforementioned "third test set") that includes multiple input texts (i.e., the aforementioned "second test data"), and instruction 2 (i.e., the aforementioned "second format information"), which includes multiple analysis texts (i.e., the aforementioned "third test data"), and the aforementioned "fourth test set".

[0186] S904: The server inputs the analyzed text into the large language model Ma to obtain the analysis results output by the large language model Ma.

[0187] In some embodiments, the server can obtain, based on instruction 2, an analysis instruction for instructing the large language model Ma to perform analysis operations and an output format for instructing the output of the large language model Ma. The server can merge the analysis instruction, output format, input text of the large language model Mt, and output text of the large language model Mt into an analysis text. It is understood that the server can merge the analysis instruction, output format, input text of the large language model Mt, and output text of the large language model Mt in any way, and the embodiments of this application do not limit this.

[0188] like Figure 10 As shown, the server inputs the merged analysis text (i.e., the aforementioned "third test data") into the large language model Ma (i.e., the aforementioned "second large language model"). The large language model Ma can output the analysis result (i.e., the aforementioned "third result data"), and the server obtains the analysis result.

[0189] For example, the input text for the large language model Mt could be: Please extract the time and location from the text, such as: Today I want to eat Haidilao hot pot.

[0190] The output text of the large language model Mt can be: the time is today, and the location is Haidilao.

[0191] Instruction 2 can be: The following is a model's response to the input. Please analyze the correctness of the result text, with 1 for correct and 0 for incorrect. Return in JSON format. For example: {"response": "1"}.

[0192] In the example of instruction 2 above, it can be seen that instruction 2, "The following is the response of a certain model to the input. Please analyze the correctness of the result text," instructs the analysis of the large language model Ma to analyze the correctness of the input text and corresponding output text of a certain model (i.e., the large language model Mt to be evaluated). Furthermore, instruction 2, "Correct is 1, incorrect is 0. Return JSON format. For example: {"response": "1"}," instructs the analysis of the large language model Ma to provide analysis results in a specified format after analysis. In instruction 2, the specified format is JSON.

[0193] The analysis text can be: The following is a model's response to the input. Please analyze the correctness of the result, with 1 for correct and 0 for incorrect. {Model Input: Please extract the time and location from the text. Text: Today I want to eat Haidilao hot pot. Model Output: Time is today, location is Haidilao}. Please return in JSON format, for example: {"response": "1"}.

[0194] After the above analysis text is input into the large language analysis model Ma, the analysis result output by the large language analysis model Ma is: {"response": "1"}.

[0195] The above analysis results indicate that the analysis of the large language model Ma and Mt shows that the returned text of the large language model Mt is correct for the input text.

[0196] For example, the input text for the large language model Mt could be: Please give the characteristics of a cat.

[0197] The output text of the large language model Mt can be: Cats are cute animals. They have pointed ears, wet noses, small mouths, and a pair of bright, sparkling eyes that shine in the dark.

[0198] Instruction 2 can be: The following is a model's response to the input. Please evaluate the result text based on the labeled text and give it a score. Return a JSON format, where the evaluation result indicates whether it is easy to understand, and the score represents the rating, ranging from 0 to 100. For example: {"Evaluation Result": "Difficult to understand", "score": "60"}.

[0199] Based on the input text of the large language model Mt, the output text of the large language model Mt, and instruction 2, the server can obtain the analysis text as follows: The following is a model's response to the input. Please evaluate the result text and give it a score. Return in JSON format, where the evaluation result indicates whether it is easy to understand, and the score represents the rating, ranging from 0 to 100. For example: {"Evaluation Result": "Easy to understand", "score": "80"}. {Model Input: Please describe the characteristics of a cat. Model Output: Cats are cute animals; they have pointed ears, wet noses, small mouths, and a pair of bright, sparkling eyes that shine in the dark}.

[0200] After the above analysis text is input into the large language analysis model Ma, the analysis results output by the large language analysis model Ma are: {"Evaluation result": "Easy to understand", "score": "80"}.

[0201] The above analysis results indicate that the analysis of the large language model Ma found that the result text returned by the large language model Mt for the input text is easy to understand, and the comprehensibility score of the result text returned by the large language model Mt for the input text is 80 points.

[0202] For example, combining Figure 10 As shown, after the server receives instruction 2, it can input text into the large language model Mt. The large language model Mt outputs result text based on the input text and instruction 2 to obtain the analysis text. When there are multiple input texts into the large language model Mt, there will also be multiple output texts from the large language model Mt, allowing the server to obtain an analysis input set containing multiple analysis texts. The server then inputs each analysis text from the analysis input set into the large language model Ma, obtaining multiple analysis results output by the large language model Ma.

[0203] S905: The server obtains evaluation results based on the analysis results.

[0204] As in the example in S904, in a scenario where the number of input texts for testing the large language model Mt is multiple, the number of output texts from the large language model Mt is also multiple. Therefore, the number of analysis texts obtained by the server through combining instruction 2 with the output texts and input texts is also multiple. Thus, the large language model Ma can obtain multiple analysis results for each of the multiple input analysis texts. Figure 10 As shown, the server can obtain the evaluation results (i.e., the aforementioned "third result data") based on multiple analysis results (i.e., the aforementioned "third result set").

[0205] In the above embodiments, when the analysis results are used to represent the correctness of the output text of the large language model Mt in response to the input text, the server can determine from multiple analysis results the number of correctly output texts and the number of incorrectly output texts by the large language model Mt. Thus, the server can calculate the accuracy of the output of the large language model Mt. Understandably, the evaluation results can also be other metrics, such as recall, F1 score, etc. Furthermore, the evaluation results can also be other metrics used to represent the capabilities of the large language model Mt, such as the comprehensibility rate, which represents the difficulty of understanding the output of the large language model Mt.

[0206] For example, when testing a large language model Mt with multiple input texts, if the large language model Ma analyzes these multiple input texts and obtains a total of 10 analysis results, and among these 10 results, one result indicates that the resulting text is easy to understand and scores 90 or higher; six results indicate that the resulting text is easy to understand and scores between 80 and 90; one result indicates that the resulting text is easy to understand and scores between 70 and 80; and two results indicate that the resulting text is not easy to understand and scores below 70. Based on these analysis results, the server can obtain the following evaluation results. For example, the comprehensibility rate of the resulting text output by the large language model Mt is (1+6+1) / 10 = 80%. Also, among the easily understood resulting texts output by the large language model Mt, the probability of a high-scoring result (90 or higher) is 1 / (1+6+1) = 12.5%.

[0207] In the above embodiments, when the analysis results are used to represent the comprehensibility of the output text of the large language model Mt for the input text, the server can determine from multiple analysis results the number of comprehensible output texts and the number of incomprehensible output texts of the large language model Mt for the input text. Thus, the server can calculate the probability that the large language model Mt outputs comprehensible output text as an evaluation result.

[0208] It should be noted that the server can obtain different evaluation results based on different analysis results. This embodiment does not limit the specific type of evaluation results.

[0209] Understandably, for ease of calculation, the amount of text given in the above example is relatively small. In real-world applications, the amount of input text is not limited by the above example.

[0210] Figure 11 The flowchart of a large language model evaluation method provided in the embodiments of this application Figure 3 .

[0211] After the user sets the output format of the large language model Mt to the specified format, the user can click as follows: Figure 7 The test controls shown are used to test whether the large language model Mt can understand the specified format set by the user.

[0212] When the server responds to a user clicking the test control, it receives the test command and begins executing the following steps (e.g.) Figure 11 (S1101-S1103 shown).

[0213] S1101: The server obtains the specified format and the format test input text.

[0214] The specified format (i.e., the aforementioned "first format") can be user-defined or a default format built into the server. For example, combined with... Figure 7 As shown, when the default format corresponding to the model output format configuration item is selected, it indicates that the user expects the large language model to use the default format as the output format. Therefore, the electronic device can send the default format to the server as the specified format, and the server can accordingly obtain the default format as the specified format. Figure 7 When the custom format corresponding to the model output format configuration item shown is selected, it indicates that the user expects the large language model to use a custom format as the output format. For example, if the format entered in input box 702 is JSON, the electronic device can send the JSON format as the specified format to the server, and the server can obtain the user-defined JSON format as the specified format.

[0215] The format test input text can be pre-set on the server.

[0216] S1102: The server merges the specified format and the format test input text into a merged input text and inputs it into the large language model Mt, and obtains the format test result text output by the large language model Mt.

[0217] In the above steps, the second test set includes test data, namely, the merged input text obtained by the server combining the specified format and the format test input text. Here, the specified format is the aforementioned "format information", the large language model Mt is the aforementioned "first large language model", and the format test result text output by the large language model Mt is the aforementioned "test result".

[0218] S1103: The server determines whether the large language model Mt can understand the specified format based on the format of the format test result text.

[0219] Given that the large language model Mt understands the specified format, the server can execute... Figure 6 The large language model evaluation steps (S601-S605) are shown.

[0220] When the large language model Mt does not understand the specified format, execution is performed. Figure 9 The large language model evaluation steps (S901-S905) are shown.

[0221] It should be noted that the specific implementation of merging the specified format and format test input text on the server can be found in [reference needed]. Figure 6 The specific implementation details of the corresponding content in the illustrated embodiments will not be elaborated here.

[0222] For example, the server's built-in format test input text is: Please indicate the time and location of the given text. Text: "I want to go home for hot pot tonight."

[0223] The server retrieves data in JSON format.

[0224] The server will merge the specified format and the format test input text into a merged input text: Please indicate the time and location of the given text. Text: "I want to go home for hot pot tonight." The output format is JSON.

[0225] Afterwards, the server inputs the merged input text into the large language model Mt, and then obtains the format test result text output by the large language model Mt. For example, the obtained format test result text is: {"time": "Today", "location": "Home"}.

[0226] Based on the format of the test result text, the server determines whether the large language model Mt can understand the specified format.

[0227] In some examples, combining the format test result text from the above examples, the server obtains the keyword "today" for "time" and "home" for "location" from the format test result text. Then, the server can determine that the format of the above format test result text is JSON format, and the specified format obtained by the server is also JSON format. Alternatively, the format describing the test result data can be replaced with the first format. Then, the server can determine that the large language model Mt understands the specified format, that is, it can determine that the large language model Mt can output the result text in the specified format.

[0228] In other examples, if the server inputs the merged input text from the above examples into the large language model Mt, and the formatted test result text output by the large language model Mt is: "Time is today, Location is home," the server cannot obtain the names and corresponding values ​​in JSON format from this formatted test result text. In other words, the server cannot analyze this test result text in JSON format. Therefore, the server can determine that the formatted test result text is not in JSON format, and thus can conclude that the large language model Mt does not understand the specified format, i.e., the large language model Mt cannot output result text in the specified format.

[0229] In some embodiments, if it is determined that the large language model Mt can output result text in a specified format, the server can send a message confirming that the large language model Mt can output the specified format to the electronic device, so that the electronic device displays a prompt interface to inform the user that the large language model Mt can output the specified format. Conversely, if it is determined that the large language model Mt cannot output result text in a specified format, the server can send a message confirming that the large language model Mt cannot output the specified format to the electronic device, so that the electronic device displays a prompt interface to inform the user that the large language model Mt cannot output the specified format.

[0230] In some embodiments, when a user needs to evaluate a large language model Mt, the user can directly click the evaluation control on the operation display interface of the electronic device. In response to the user clicking the evaluation control, the electronic device can send an evaluation command to the server. Upon receiving the evaluation command, the server can first execute steps S1101-S1103 to determine whether the large language model Mt can output result text in the specified format. If it is determined that the large language model Mt can output result text in the specified format, that is, after step S1103, the server can continue to execute... Figure 6 As shown in S601-S605. If it is determined that the large language model Mt cannot output the result text in the specified format, i.e., after S1103, the server can continue to execute as follows: Figure 9 S901-S905 are shown.

[0231] The evaluation method described above can also be used to evaluate prompt templates. For example, while keeping the dataset and the instructions for specifying the output format unchanged, different prompt templates can be used to merge different test input texts. Different analysis results can be obtained using these different test input texts. By analyzing the obtained analysis results, the prompt templates can be evaluated.

[0232] In some embodiments, such as Figure 12As shown, the server obtains multiple input texts A by combining data from the dataset (i.e., the aforementioned "first dataset") and at least one prompt template A (i.e., the aforementioned "first prompt word template") from the prompt template set A (i.e., the aforementioned "first prompt word template set"). After obtaining instruction 1 for specifying the output format of the large language model, the server can merge each input text A with instruction 1 to obtain a test input set A including multiple test input texts. The server inputs each test input text in the test input set A into the large language model, and can obtain multiple result texts in the specified format output by the large language model. In this way, the server can use test code to analyze the differences between each result text in the result set A and the corresponding labeled text to obtain analysis result A. Based on the analysis result A, the server can obtain an evaluation result (i.e., the aforementioned "first evaluation result"). Similarly, the server uses data from the same dataset as in the example above (i.e., the aforementioned "first dataset") and at least one prompt template B from another prompt template set B (i.e., the aforementioned "second prompt template") to obtain multiple input texts B. Based on each input text B and instruction 1 (instruction 1 specifies the output format of the large language model, i.e., specifies the aforementioned "first format"), the server merges them to obtain test input texts. Then, the server inputs each test input text from the test input set B (i.e., the aforementioned "fifth test set") into the large language model, obtaining a result set B (i.e., the aforementioned "fourth result set") containing multiple result texts. The server uses test code to analyze the differences between each result text in result set B and its corresponding labeled text, obtaining analysis result B. Based on analysis result B, the server can obtain an evaluation result (i.e., the aforementioned "third evaluation result"). Therefore, the server can analyze the differences between the two evaluation results to obtain the difference between the two large language model evaluations. Since this difference is caused by changes in the prompt template, an evaluation result for the prompt template can be obtained based on the two evaluation results.

[0233] Optionally, by analyzing analysis result A and analysis result B, the server can obtain the difference between the two large language model evaluations. Since this difference is caused by the change in the prompt template, the evaluation result obtained by the server by analyzing analysis result A and analysis result B can actually be regarded as the evaluation result of the prompt template.

[0234] For example, if the data in the dataset is combined with prompt template A to form input text A, then analyzing the output text A of the large language model at this time, and analyzing each result text A in result set A based on the labeled text, the server can determine that the accuracy of the large language model in this evaluation is 80%. If the data in the dataset is combined with prompt template B to form input text B, then analyzing the output text B of the large language model at this time, and analyzing each result text B in result set B based on the labeled text, the server can determine that the accuracy of the large language model in this evaluation is 90%. Thus, the server can conclude that the accuracy of the large language model in the first evaluation is higher than that in the second evaluation. Since the large language model itself did not change during the two evaluations, but the prompt template changed, the server's final evaluation result can be: the prompt template B used in the second evaluation of the large language model yielded more accurate results.

[0235] This application also provides a computer-readable storage medium including computer instructions that, when executed on the server, cause the server to perform the various functions or steps described in the method embodiments.

[0236] This application also provides a computer program product, including a computer program that, when run on a server, causes the server to perform the various functions or steps described in the above method embodiments.

[0237] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0241] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for evaluating large language models, characterized in that, The method includes: Obtain the second test set, which is used to test the first language model's ability to understand information in the first format. The first language model's ability to understand the first format information is tested based on the second test set. If the first large language model cannot understand the first format information, a third test set is obtained. The third test set includes multiple second test data sets, and the third test set is generated based on the first dataset and the first prompt word template set. Obtain a second result set output by the first large language model based on the third test set; the second result set includes multiple second result data, and the multiple second result data correspond one-to-one with the multiple second test data; Based on the third test set, the second result set and the second format information are used to obtain a fourth test set; the fourth test set is used to instruct the second language model to analyze the ability of the first language model; the fourth test set includes multiple third test data, each third test data includes a second test data, a second result data and the second format information; the second test data corresponds to the second result data, and the second format information is used to specify the output format of the language model as the second format. Obtain the third result set output by the second language model based on the fourth test set; the third result set includes multiple third result data, each of which corresponds one-to-one with the multiple third test data, and the format of the third result data is the second format; The second evaluation result is obtained based on the third result set; If the first language model can understand the first format information, a first test set is obtained; the first test set includes multiple first test data, and the first test data includes first format information, which is used to specify that the output format of the language model is the first format. Obtain the first result set output by the first language model based on the first test set; the first result set includes multiple first result data, each of which corresponds to one of the multiple first test data, and the format of the first result data is the first format. A first evaluation result is obtained based on the first result set and the annotation set; the first evaluation result is used to evaluate the capability of the first large language model; the annotation set includes multiple annotation data, each of which corresponds one-to-one with the multiple first test data, and the format of the annotation data is the first format.

2. The method according to claim 1, characterized in that, The process of obtaining the first test set includes: Obtain a first dataset, a first prompt word template set, and the first format; the first dataset includes multiple data sets, and the first prompt word template set includes at least one first prompt word template. The first test set is generated based on the first dataset, the first prompt word template set, and the first format.

3. The method according to claim 2, characterized in that, The first format is a pre-configured format; or, the first format is a custom format.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Receive a first instruction from an electronic device; the first instruction is used to specify the output format of the language model as the first format.

5. The method according to claim 4, characterized in that, The step of obtaining the first evaluation result based on the first result set and the annotation set includes: The test code is used to compare and analyze each first result data in the first result set with the corresponding labeled data in the labeled set to obtain multiple comparison results; Based on the multiple comparison results and evaluation indicators, the first evaluation result is obtained.

6. The method according to claim 1, characterized in that, The process of obtaining the second evaluation result based on the third result set includes: Based on the third result set and evaluation metrics, the second evaluation result is obtained.

7. The method according to claim 5 or 6, characterized in that, The method further includes: Receive the evaluation metrics from the electronic device.

8. The method according to claim 2, characterized in that, The method further includes: Obtain a second prompt word template set; the second prompt word template set includes at least one second prompt word template; Based on the first dataset, the second prompt word template set, and the first format, obtain the fifth test set; Obtain the fourth result set output by the first large language model based on the fifth test set; the format of the result data included in the fourth result set is the first format; The third evaluation result is obtained based on the fourth result set and the annotation set; the third evaluation result is used to evaluate the capability of the first large language model; Based on the first evaluation result and the third evaluation result, a prompt word evaluation result is obtained, which is used to evaluate the capabilities of the first prompt word template and the second prompt word template.

9. A server, characterized in that, The server includes a communication module, a memory, and one or more processors; the communication module, the memory, and the processors are coupled; the communication module is used to establish a communication connection and send and receive data through the communication connection; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the server performs the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on a server, cause the server to perform the method as described in any one of claims 1-8.

Images