Assessment method, device and equipment for large language model output performance

Abstract

The invention provides an evaluation method, device and equipment for large language model output performance, and relates to the technical field of natural language process.The evaluation method for the large language model output performance comprises the steps that a structure-related instruction is generated based on a regular expression corresponding to a target output format; generating a task related instruction based on the target application scene; fusing the task-related instruction with the structure-related instruction to generate target test data, acquiring output data generated by the to-be-tested large language model based on the target test data, and comparing the output data with a corresponding regular expression to obtain the target test data. Therefore, the output performance of the to-be-tested large language model is determined based on the consistency or difference, and the reliability of the to-be-tested large language model is ensured.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method, device and equipment for evaluating the output performance of a large language model. Background Art

[0002] With the widespread application of LLMs (Large Language Models) in various fields, such as text generation, sentiment analysis, and automatic question answering systems, the evaluation of their performance and capabilities has become particularly important. Current evaluation benchmarks such as SuperGLUE and MMLU (Mean Multi-Language Understanding) mainly focus on the performance of LLMs in understanding and generation tasks, but often ignore the importance of structured output.

[0003] Although LLMs perform well in generating natural language text, they face challenges in generating accurate structured data. These challenges include but are not limited to format errors, irrelevant information, and lack of consistency, which often have a negative impact on downstream tasks. Therefore, how to accurately evaluate the performance of LLMs' structured output is particularly important. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that it is difficult to accurately evaluate the structured output performance of LLMs. The present invention provides a method, device, equipment and storage medium for evaluating the output performance of a large language model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] A first aspect of an embodiment of the present application provides a method for evaluating the output performance of a large language model, including:

[0007] Generate structure-related instructions based on the regular expression corresponding to the target output format; the formula corresponding to the structure-related instructions is as follows: Among them, t i is the type of target output format, For its corresponding regular expression, the f function is implemented through a large language model;

[0008] Generate task-related instructions based on the target application scenario;

[0009] Merging the task-related instructions with the structure-related instructions to generate target test data;

[0010] The target test data is used to test the large language model to be tested, and the output performance of the large language model to be tested is determined.

[0011] Optionally, the target output format includes at least one of a JSON format and an XML format; the generating structure-related instructions based on the regular expression corresponding to the target output format includes:

[0012] Determine the regular expressions corresponding to each type of target output format;

[0013] A corresponding structure-related instruction is generated based on each of the regular expressions.

[0014] Optionally, the formula corresponding to the target test data is as follows:

[0015] P i =n(S i , G i );

[0016] Where S i For structure-related instructions, G i For task-related instructions.

[0017] Optionally, the using the target test data to test the large language model to be tested to determine the output performance of the large language model to be tested includes:

[0018] Inputting the target test data into the large language model to be tested to obtain target output data;

[0019] Based on the comparison result between the target output data and the corresponding regular expression, the output performance of the large language model is determined.

[0020] Optionally, determining the output performance of the large language model based on a comparison result between the target output data and a corresponding regular expression includes:

[0021] When the target output format is a JSON format, determining a first regular expression corresponding to the JSON format;

[0022] Based on the comparison result between the target output data and the corresponding first regular expression, the output performance of the large language model is determined.

[0023] Optionally, determining the output performance of the large language model based on a comparison result between the target output data and a corresponding regular expression includes:

[0024] When the target output format is XML format, determining a second regular expression corresponding to the XML format;

[0025] Based on the comparison result between the target output data and the corresponding second regular expression, the output performance of the large language model is determined.

[0026] The second aspect of the embodiment of the present application provides an evaluation device for the output performance of a large language model, comprising: a first generation module, a second generation module and a fusion module; wherein:

[0027] The first generation module is configured to generate structure-related instructions based on a regular expression corresponding to the target output format; the formula corresponding to the structure-related instructions is as follows: Among them, t i is the type of target output format, For its corresponding regular expression, the f function is implemented through a large language model;

[0028] The second generating module is configured to generate task-related instructions based on the target application scenario;

[0029] The fusion module is configured to fuse the task-related instructions with the structure-related instructions to generate target test data;

[0030] The determination module is configured to test the large language model to be tested using the target test data to determine the output performance of the large language model to be tested.

[0031] A third aspect of an embodiment of the present application provides an electronic device, comprising a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the method for evaluating the output performance of a large language model as described in the first aspect.

[0032] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0033] Compared with the prior art, the technical solution provided by this application has the following beneficial effects:

[0034] The present invention provides a method, device and equipment for evaluating the output performance of a large language model. By comprehensively considering structure-related instructions related to a target output format and task-related instructions related to an application scenario, target test data is obtained, and output data generated by the large language model to be tested based on the target test data is obtained. The output data is compared with a corresponding regular expression, so as to determine the output performance of the large language model to be tested based on the consistency or difference between the two, thereby ensuring the reliability of the large language model to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a method for evaluating the output performance of a large language model provided in an embodiment of the present application;

[0036] Figure 2A schematic diagram of the structure of a device for evaluating the output performance of a large language model provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.

[0039] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0040] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0041] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.

[0042] The present application embodiment provides a method for evaluating the output performance of a large language model, including:

[0043] S110, generating structure-related instructions based on the regular expression corresponding to the target output format; the formula corresponding to the structure-related instructions is as follows: Among them, t i is the type of target output format, For its corresponding regular expression, the f function is implemented through a large language model.

[0044] S120, generating task-related instructions based on the target application scenario.

[0045] S130, integrating the task-related instructions with the structure-related instructions to generate target test data.

[0046] S140, using the target test data to test the large language model to be tested, and determining the output performance of the large language model to be tested.

[0047] Here, the target output format can be a data exchange format, that is, a format for data transmission and exchange between the server and the client. A regular expression is a set of special texts composed of letters and symbols, which can be used to find sentences in a specific format from a text. There is a corresponding regular expression for each type of target output format. There is a mapping relationship between the target output format and the regular expression. The target application scenario can be a variety of application scenarios, such as text formatting, converting the current text into a text in a specific format, and for sentiment analysis tasks, it is often necessary to design instructions to handle different sentiment categories, handle the influence of negative words, etc. Structure-related instructions can be a programming method that emphasizes the clarity and maintainability of the program structure. Task-related instructions represent instructions that are directly related to specific tasks or functions in a specific system or application. These instructions may involve multiple aspects such as task scheduling, hardware function control, and data processing of multi-core systems.

[0048] In one example, the target application scenario is to format JSON (JavaScript Object Notation) text for easy reading. Correspondingly, the task-related instructions are JSON formatting instructions, which may specifically include indentation instructions and line break instructions.

[0049] In another example, the target application scenario is to format XML (eXtensible Markup Language) text to improve its readability. Correspondingly, the task-related instructions are XML formatting instructions, which may specifically include indentation instructions, line break instructions, and tag closing instructions.

[0050] In this embodiment, the target test data is obtained by fusing the structure-related instructions related to the target output format and the task-related instructions related to the application scenario, and then the target test data is used to test the large language model to be tested to determine the output performance of the large language model to be tested. In an ideal case, the target test data is input into the large language model to be tested to obtain output data. If the output data matches the corresponding regular expression, it can be determined that the output performance of the large language model to be tested is good. Otherwise, it is determined that the output performance of the large language model to be tested does not meet the requirements.

[0051] In some embodiments, the target output format includes at least one of a JSON format and an XML format; S110, generating structure-related instructions based on a regular expression corresponding to the target output format, including:

[0052] Determine the regular expressions corresponding to each type of target output format;

[0053] Generates corresponding structure-related instructions based on each regular expression.

[0054] JSON format is a lightweight data exchange format. Its advantages lie in its lightweight characteristics, easy parsing and generation, easy reading and writing, and support for multiple programming languages. The data structure is simple and nested, so it is widely used in Web development and mobile applications. XML format is an extensible markup language used to describe and transmit structured data. It is often used in Web services and configuration files. It is self-descriptive, highly extensible, and supports multiple programming languages. XML has a clear structure and good readability, and is suitable for the exchange of complex data structures and documents. It should be noted that in addition to JSON format and XML format, there are other data exchange formats, such as HTML, etc., which are not enumerated.

[0055] In this embodiment, different types of target output formats have corresponding regular expressions that are partially the same or completely different. Accordingly, the structure-related instructions generated based on the regular expressions are also different.

[0056] In some embodiments, the formula corresponding to the target test data is as follows:

[0057] P i =n(S i , G i );

[0058] Where S i For structure-related instructions, G i For task-related instructions.

[0059] Correspondingly, there is also a mapping relationship between the structure-related instructions, the task-related instructions and the target test data, and the target test data can change with the changes of the structure-related instructions and the task-related instructions.

[0060] In some embodiments, S140, using the target test data to test the large language model to be tested to determine the output performance of the large language model to be tested includes:

[0061] Input the target test data into the large language model to be tested to obtain the target output data;

[0062] Based on the comparison result between the target output data and the corresponding regular expression, the output performance of the large language model to be tested is determined.

[0063] In this embodiment, the target test data is input into the large language model to be tested, the output data generated by the large language model to be tested based on the target test data is obtained, and the output data is compared with the corresponding regular expression, so as to determine the output performance of the large language model to be tested based on the consistency or difference between the two. If the output data is compatible with the corresponding regular expression, it can be determined that the output performance of the large language model to be tested is good, otherwise, it is determined that the output performance of the large language model to be tested does not meet the requirements.

[0064] In some embodiments, based on the comparison result between the target output data and the corresponding regular expression, determining the output performance of the large language model to be tested includes:

[0065] When the target output format is JSON format, determining a first regular expression corresponding to the JSON format;

[0066] Based on the comparison result between the target output data and the corresponding first regular expression, the output performance of the large language model to be tested is determined.

[0067] In this embodiment, the target output format is JSON format. Therefore, the first regular expression generated based on the target output format, the structure-related instructions and the final target output data are all in JSON format. Therefore, the target output data is compared with the corresponding first regular expression. If the output data matches the first regular expression corresponding to the JSON format, it can be determined that the output performance of the large language model to be tested is good. Otherwise, it is determined that the output performance of the large language model to be tested does not meet the requirements.

[0068] In some embodiments, based on the comparison result between the target output data and the corresponding regular expression, determining the output performance of the large language model to be tested includes:

[0069] When the target output format is XML format, determining a second regular expression corresponding to the XML format;

[0070] Based on the comparison result between the target output data and the corresponding second regular expression, the output performance of the large language model to be tested is determined.

[0071] In this embodiment, the target output format is XML format. Therefore, the second regular expression, structure-related instructions and final target output data generated based on the target output format are all in XML format. Therefore, the target output data is compared with the corresponding second regular expression. If the output data matches the second regular expression corresponding to the XML format, it can be determined that the output performance of the large language model to be tested is good. Otherwise, it is determined that the output performance of the large language model to be tested does not meet the requirements.

[0072] It should be noted that when the target output format is other data exchange formats, the principle is the same as that of the above embodiment, and will not be repeated here.

[0073] In a specific example, the target test data is used to test the large language model to determine the output performance of the large language model to be tested. The corresponding regex pattern can be used for automatic verification. The verification formula is as follows:

[0074]

[0075] Where N is the total number of target test data, outputi is the target output data corresponding to the i-th target test data of the large language model (LLM), regexi represents the regular expression corresponding to the i-th target test data prompt, match_length(regexi,outputi) represents the length of the paragraph matching the regular expression in the target output data, and total_length(outputi) represents the total length of the LLMs output for the i-th target test data. The larger the U value, the better the output performance of the large language model to be tested.

[0076] The embodiment of the present application obtains test data by comprehensively considering structure-related instructions related to the target output format and task-related instructions related to the application scenario, and obtains output data generated by the large language model to be tested based on the target test data, and compares the output data with the corresponding regular expression, thereby determining the output performance of the large language model to be tested based on the consistency or difference between the two, thereby ensuring the reliability of the large language model to be tested.

[0077] In some embodiments, see Figure 2 , Figure 2 A schematic diagram of a structure of an evaluation device for the output performance of a large language model provided in an embodiment of the present application. An evaluation device 200 for the output performance of a large language model provided in an embodiment of the present application includes: a first generation module 210, a second generation module 220 and a fusion module 230; wherein,

[0078] The first generating module 210 is configured to generate structure-related instructions based on the regular expression corresponding to the target output format; the formula corresponding to the structure-related instructions is as follows: Among them, t i is the type of target output format, For its corresponding regular expression, the f function is implemented through a large language model;

[0079] A second generating module 220, configured to generate task-related instructions based on a target application scenario;

[0080] A fusion module 230 configured to fuse the task-related instructions with the structure-related instructions to generate target test data;

[0081] The determination module 240 is configured to test the large language model to be tested using the target test data to determine the output performance of the large language model to be tested.

[0082] In some embodiments, the target output format includes at least one of the JSON format and the XML format; the first generation module 210 is specifically configured as follows:

[0083] Determine the regular expressions corresponding to each type of target output format;

[0084] Generates corresponding structure-related instructions based on each regular expression.

[0085] In some embodiments, the formula corresponding to the target test data is as follows:

[0086] P i =n(S i , G i );

[0087] Where S i For structure-related instructions, G i For task-related instructions.

[0088] In some embodiments, the determination module 240 is configured to:

[0089] Input the target test data into the large language model to be tested to obtain the target output data;

[0090] Based on the comparison result between the target output data and the corresponding regular expression, the output performance of the large language model to be tested is determined.

[0091] In some embodiments, the determination module 230 is specifically configured as follows:

[0092] When the target output format is JSON format, determining a first regular expression corresponding to the JSON format;

[0093] Based on the comparison result between the target output data and the corresponding first regular expression, the output performance of the large language model to be tested is determined.

[0094] In some embodiments, the determination module 240 is further configured to:

[0095] When the target output format is XML format, determining a second regular expression corresponding to the XML format;

[0096] Based on the comparison result between the target output data and the corresponding second regular expression, the output performance of the large language model to be tested is determined.

[0097] The evaluation device for the output performance of a large language model provided in the embodiment of the present application can implement the various processes in the embodiments corresponding to the above-mentioned evaluation method for the output performance of a large language model. To avoid repetition, they will not be described again here.

[0098] It should be noted that the evaluation device for the output performance of a large language model provided in the embodiment of the present application and the evaluation method for the output performance of a large language model provided in the embodiment of the present application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned evaluation method for the output performance of a large language model, and the repeated parts will not be repeated.

[0099] In some embodiments, see Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. An electronic device 300 provided in an embodiment of the present application includes a processor 310 and a memory 320; the memory 320 stores a computer program, wherein the computer program implements the above-mentioned evaluation method for the output performance of a large language model when executed by the processor.

[0100] Specifically, the processor 310 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 310 may also include an onboard memory for cache purposes. The processor 310 may be a single processing unit or multiple processing units for executing different actions of the method flow according to the embodiment of the present application.

[0101] The memory 320 may be any medium capable of containing, storing, conveying, propagating or transmitting instructions. For example, the memory 320 may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device, component or propagation medium. Specific examples of the memory 320 include: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a random access memory (RAM) or flash memory; and / or a wired / wireless communication link.

[0102] The present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for evaluating the output performance of a large language model is implemented. The computer-readable medium may be included in the device / apparatus / system described in the above-mentioned embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0103] According to an embodiment of the present application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, optical cable, radio frequency signal, etc., or any suitable combination of the above.

[0104] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above-described embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A method for evaluating the output performance of a large language model, characterized in that: include: Generate structure-related instructions based on regular expressions corresponding to the target output format; The formulas corresponding to structure-related instructions are as follows: Among them, t i is the type of target output format, For its corresponding regular expression, the f function is implemented through a large language model; Generate task-related instructions based on the target application scenario; Merging the task-related instructions with the structure-related instructions to generate target test data; The target test data is used to test the large language model to be tested, and the output performance of the large language model to be tested is determined.

2. The method for evaluating the output performance of a large language model according to claim 1, characterized in that: The target output format includes at least one of a JSON format and an XML format; the generating structure-related instructions based on a regular expression corresponding to the target output format includes: Determine the regular expressions corresponding to each type of target output format; A corresponding structure-related instruction is generated based on each of the regular expressions.

3. The method for evaluating the output performance of a large language model according to claim 1, characterized in that: The formula corresponding to the target test data is as follows: P i =n(S i ,G i ); Where S i For structure-related instructions, G i For task-related instructions.

4. The method for evaluating the output performance of a large language model according to claim 2, characterized in that: The using the target test data to test the large language model to be tested to determine the output performance of the large language model to be tested includes: Inputting the target test data into the large language model to be tested to obtain target output data; Based on the comparison result between the target output data and the corresponding regular expression, the output performance of the large language model to be tested is determined.

5. The method for evaluating the output performance of a large language model according to claim 4, characterized in that: The step of determining the output performance of the large language model to be tested based on a comparison result between the target output data and the corresponding regular expression includes: When the target output format is a JSON format, determining a first regular expression corresponding to the JSON format; Based on the comparison result between the target output data and the corresponding first regular expression, the output performance of the large language model is determined.

6. The method for evaluating the output performance of a large language model according to claim 4, characterized in that: The step of determining the output performance of the large language model to be tested based on a comparison result between the target output data and the corresponding regular expression includes: When the target output format is XML format, determining a second regular expression corresponding to the XML format; Based on the comparison result between the target output data and the corresponding second regular expression, the output performance of the large language model to be tested is determined.

7. A device for evaluating the output performance of a large language model, characterized in that: include: A first generation module, a second generation module, a fusion module and a determination module; wherein, The first generation module is configured to generate structure-related instructions based on a regular expression corresponding to the target output format; the formula corresponding to the structure-related instructions is as follows: Among them, t i is the type of target output format, For its corresponding regular expression, the f function is implemented through a large language model; The second generating module is configured to generate task-related instructions based on the target application scenario; The fusion module is configured to fuse the task-related instructions with the structure-related instructions to generate target test data; The determination module is configured to test the large language model to be tested using the target test data to determine the output performance of the large language model to be tested.

8. An electronic device comprising a processor and a memory; the memory stores a computer program, wherein: When the computer program is executed by the processor, the computer program implements the method for evaluating the output performance of a large language model according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images