Model evaluation method and device, equipment, storage medium and program product
By acquiring and analyzing the output patterns and results of large language models in multiple evaluation tasks, the problem that existing evaluation methods cannot evaluate the potential capabilities of the model is solved, and the model optimization effect is improved.
Patent Information
- Application Number
- CN202411270508.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-13
AI Technical Summary
The existing large language model evaluation methods cannot effectively evaluate the potential capabilities of the model, resulting in poor late-stage optimization results.
By obtaining the target evaluation instruction set of multiple evaluation tasks, obtain the output mode and output results of the large language model for each task, analyze its semantic spatial transformation logic, and calculate the evaluation score to reasonably evaluate the model's ability.
It improves the logical analysis and optimization effect of the large language model on tasks, provides a reasonable evaluation of the existing and potential capabilities of the model, and improves the later optimization effect.
Smart Images
Figure CN119128461A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing technology, and in particular to a model evaluation method, device, equipment, storage medium and program product. Background Art
[0002] Large language models are in a rapid development stage, and various types of large language models are emerging. In order to highlight the capabilities of large language models, they are usually evaluated on multiple tasks, and the scoring and ranking on multiple tasks are used as the main means of evaluating the capabilities of large models. This evaluation method is a phenomenal evaluation method of ranking, which cannot make a reasonable evaluation of the potential capabilities of large language models, resulting in poor optimization effects of large language models in the later stage. Summary of the invention
[0003] The object of the present invention is to provide a model evaluation method, device, equipment, storage medium and program product to solve the problem that the optimization effect of the later large language model is poor when the existing model evaluation method is used.
[0004] In order to achieve the above object, in a first aspect, the present invention provides a model evaluation method, comprising:
[0005] Obtain target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets;
[0006] For each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, an output mode and a first output result of a large language model are obtained, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model;
[0007] Based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result.
[0008] In some embodiments, the step of obtaining a target evaluation instruction set for a plurality of evaluation tasks includes:
[0009] For each evaluation task, obtaining a first evaluation instruction set;
[0010] For each instruction in the first evaluation instruction set, input the instruction into the large language model to obtain a second output result;
[0011] The target evaluation instruction set is obtained by using the instructions corresponding to the second output result that represents the correct result.
[0012] In some embodiments, for each evaluation task, obtaining a first evaluation instruction set includes:
[0013] For each evaluation task, obtain the pre-built seed instructions;
[0014] The seed instruction is learned to generate a first evaluation instruction set corresponding to the evaluation task.
[0015] In some embodiments, obtaining the output mode and the first output result of the large language model according to the target evaluation instruction set corresponding to the evaluation task includes:
[0016] For each instruction of the target evaluation instruction set corresponding to the evaluation task, the instruction is input into the large language model, a word unit token output by each layer in the large language model and a confidence level of each layer output token are obtained, and the token output by each layer in the large language model is visualized; wherein the first output result of the large language model includes the confidence level of each layer output token in the large language model;
[0017] Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the evaluation task;
[0018] According to the obtained multiple blocks in the large language model that are strongly related to the evaluation task, an output mode of the large language model is determined.
[0019] In some embodiments, determining the output mode of the large language model according to the obtained multiple blocks strongly related to the evaluation task in the large language model includes:
[0020] Determine a target block based on multiple blocks in the large language model that are strongly related to the evaluation task, where the target block is a common block in the multiple blocks;
[0021] Based on the target block, the semantic space conversion logic of the large language model when solving the evaluation task is determined, and the semantic space conversion logic of the large language model when solving the evaluation task is used as the output mode of the large language model.
[0022] In some embodiments, the step of evaluating the large language model based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks to obtain the evaluation result includes:
[0023] For each evaluation task, using the layer number of the output mode of the large language model corresponding to the evaluation task in the large language model and the first output result, calculate the score corresponding to the large language model when solving the evaluation task;
[0024] Based on the calculated scores of the large language model when solving multiple evaluation tasks, an evaluation result of the large language model is obtained.
[0025] In a second aspect, the present invention further provides a model evaluation device, comprising:
[0026] An acquisition module is used to acquire target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets;
[0027] A processing module, configured to obtain, for each evaluation task, an output mode and a first output result of a large language model according to the target evaluation instruction set corresponding to the evaluation task, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model;
[0028] A model evaluation module is used to evaluate the large language model based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks to obtain an evaluation result.
[0029] In a third aspect, the present invention further provides a model evaluation device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, and the processor is used to perform the following operations:
[0030] Obtain target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets;
[0031] For each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, an output mode and a first output result of a large language model are obtained, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model;
[0032] Based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result.
[0033] In a fourth aspect, the present invention further provides a model evaluation device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the model evaluation method as described in the first aspect above is implemented.
[0034] In a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the model evaluation method described in the first aspect above.
[0035] In a sixth aspect, the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the model evaluation method as described in the first aspect above.
[0036] The above technical solution of the present invention has at least the following beneficial effects:
[0037] In an embodiment of the present invention, by obtaining target evaluation instruction sets of multiple evaluation tasks, different evaluation tasks correspond to different target evaluation instruction sets; then, for each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, the output mode and the first output result of the large language model are obtained, the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model; finally, based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result. In this way, through the above processing, the logic of the large language model when solving the task is analyzed, that is, the semantic space conversion logic of the large language model when solving the evaluation task is obtained, and the semantic space conversion logic of the large language model when solving the evaluation task is used to make a reasonable evaluation of the existing and potential capabilities of the large language model, thereby improving the optimization effect of the large language model in the later stage. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram showing a flow chart of a model evaluation method provided by an embodiment of the present invention;
[0039] Figure 2 A schematic diagram showing a module of a model evaluation device provided by an embodiment of the present invention;
[0040] Figure 3 A schematic diagram showing the hardware structure of a model evaluation device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0042] The essence of the big language model is the effective compression of the knowledge of the physical world. The knowledge of our physical world is complex, but at the same time logical. Human beings' understanding of the knowledge of the physical world is highly interpretable and logical. Therefore, the big language model also needs to have similar logic as humans in its semantic space when remembering, understanding and applying the knowledge generated by us humans. Only such a big language model can be understood and deeply applied by humans.
[0043] The existing evaluation methods of large language models cannot provide an objective assessment of the interpretability of large language models and the expected evolution of their future capabilities, which leads to poor optimization effects of large language models in the later stages.
[0044] In order to solve the above technical problems, a model training method, device, equipment, storage medium and program product are provided. The method and the device are based on the same application concept. Since the principles of solving the problem by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.
[0045] like Figure 1 FIG. 1 is a flow chart of a model evaluation method provided by an embodiment of the present invention, and the method may include:
[0046] Step 101, obtaining target evaluation instruction sets of multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets.
[0047] Among them, the evaluation task is a pre-constructed task. It can be a complete evaluation task designed by experts, which includes but is not limited to discriminative and generative tasks; the generated tasks include inductive summary and detailed description; the discriminative tasks include lexical, syntactic, classification, etc. In short, it is necessary to consider multiple dimensions of human knowledge and design relatively complete evaluation tasks. The tasks require the block capabilities of different levels of the large language model to solve, so that the capabilities of all levels of blocks of the large language model can be evaluated.
[0048] In an optional embodiment, this step 101, obtaining a target evaluation instruction set of multiple evaluation tasks, includes:
[0049] Step 1011: For each evaluation task, obtain a first evaluation instruction set.
[0050] In an optional embodiment, step 1011 includes:
[0051] 1) For each evaluation task, obtain the pre-built seed instructions.
[0052] The seed instructions include preset text related to the assessment task, preset instructions for the preset text, and an output format for the preset instructions.
[0053] It should be noted that the seed instruction can be an artificially constructed instruction to ensure the rigor of the domain task. Generally, the seed instruction includes multiple ones.
[0054] In one example, the task is a NER task, and the preset text is: The weather in Guangzhou is really hot. The preset instruction for the preset text is: Please extract the entities in the text. The output format of the preset instruction is JSON format, that is, {'entity': [entity name]}.
[0055] 2) Learning the seed instructions to generate a first evaluation instruction set corresponding to the evaluation task.
[0056] A more powerful large model (such as GPT4o) can be used to learn seed instructions and generate a large number of task instructions as the first evaluation instruction set corresponding to the evaluation task.
[0057] It should be noted that in order to ensure the rigor of the evaluation task, the instructions generated in step 2) may be manually reviewed, that is, after a large number of instructions are generated, they are manually reviewed to obtain a first set of evaluation instructions corresponding to the evaluation task.
[0058] Step 1012: for each instruction in the first evaluation instruction set, input the instruction into a large language model to obtain a second output result.
[0059] Here, the large language model is a base large language model, such as Qwen-14B-chat.
[0060] Step 1013, using the instructions corresponding to the second output result that represents the correct result, obtain the target evaluation instruction set.
[0061] That is to say, the instruction set corresponding to the second output result for which the large language model gives a correct answer is selected from the second output result as the target evaluation instruction set.
[0062] Step 102: For each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, obtain the output mode and the first output result of the large language model, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model.
[0063] In an optional embodiment, this step 102 includes:
[0064] Step 1021: for each instruction of the target evaluation instruction set corresponding to the evaluation task, input the instruction into the large language model, obtain the word-meta token output by each layer in the large language model and the confidence of each layer of output token, and visualize the token output by each layer in the large language model; wherein the first output result of the large language model includes the confidence of each layer of output token in the large language model.
[0065] For example, the instruction (prompt) for the NER task input to the large language model is: Please extract the entities in the text. \nText: The weather in Shanghai is really hot. \nPlease output in JSON format {'entity': [entity name]}.
[0066] Afterwards, the word-unit tokens output by each layer in the large language model are visualized, as shown in Table 1 below:
[0067] Table 1
[0068]
[0069]
[0070] Step 1022: Based on the visualized tokens, obtain a block in the large language model that is strongly related to the evaluation task.
[0071] Here, experts can analyze the visualized tokens to identify and judge the blocks in the large language model that are strongly related to the task. This block can be defined as a strong capability block for the task. For example, in the above example, when the token "Shanghai" is inferred, "things" is output at the 17th layer, "entities" are output at the 18th to 26th layers, "places" are output at the 27th to 29th layers, and "Shanghai" is output at the 30th to 40th layers. The reasoning process has a certain logic from abstract to concrete, so the 17th to 40th layers are defined as blocks that are strongly related to the task of this case (instruction) and can be explained.
[0072] Step 1023: Determine the output mode of the large language model according to the multiple blocks in the obtained large language model that are strongly related to the evaluation task.
[0073] In an optional embodiment, step 1023 includes:
[0074] (1) Based on multiple blocks in the large language model that are strongly related to the evaluation task, a target block is determined, where the target block is a common block among the multiple blocks.
[0075] (2) Based on the target block, determine the semantic space conversion logic of the large language model when solving the evaluation task, and use the semantic space conversion logic of the large language model when solving the evaluation task as the output mode of the large language model.
[0076] Here, after the instructions in the target evaluation instruction set are input into the large language model, multiple blocks can be obtained. The common block found from multiple blocks is the target block. That is, find the semantic space conversion logic when solving this evaluation task from multiple cases (instructions) as the output mode of the large language model. As in the above example, in the large language model, first find the semantic space of "things", then transfer to the semantic space of "entity", then to the semantic space of "location", and finally find "Shanghai". Therefore, the semantic space conversion logic of the large language model in the above example sentence is "things → entity → location → Shanghai". Since Shanghai is a location noun, it is output as an entity. This is our explanation of the internal reasoning of the large language model conforming to human logic. If many examples conform to this logical pattern, this pattern can be used as the semantic space measurement pattern of the large language model for the NER task.
[0077] Step 103, based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, evaluate the large language model to obtain an evaluation result.
[0078] Through the above processing, analyze the logic of the large language model when solving tasks, that is, obtain the semantic space conversion logic of the large language model when solving the evaluation task, and use the semantic space conversion logic of the large language model when solving the evaluation task to make a reasonable evaluation of the existing and potential capabilities of the large language model, so as to improve the optimization effect of the large language model in the later stage.
[0079] In an optional embodiment, the above step 103, based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, evaluate the large language model to obtain an evaluation result, including:
[0080] Step 1031, for each evaluation task, use the layer number where the output mode of the large language model corresponding to the evaluation task is located in the large language model and the first output result to calculate the score corresponding to the large language model when solving the evaluation task.
[0081] The semantic space conversion logic when the large language model solves the evaluation task can be designed, that is, the logicality of the large language model semantic space task, and the metric formula to calculate the score corresponding to the large language model when solving the evaluation task. For example, but not limited to the following formula:
[0082] V = (1 / the number of layers where the output pattern token is located in the large language model) * the confidence of the token output by the large language model; where V represents the logical evaluation score of the semantic space.
[0083] For example, the layer of the output mode token in the large language model can be the layer of things, entities, places, and Shanghai found in the NER task.
[0084] For the above example, the semantic space logicality metric score is calculated as:
[0085] East: (1 / 17)*0.01
[0086] Entity: (1 / 18)*0.011+(1 / 19)*0.03+…+(1 / 26)*0.083
[0087] Location: (1 / 27)*0.16+(1 / 28)*0.15+(1 / 29)*0.01
[0088] Shanghai: (1 / 30)*0.62+(1 / 31)*0.79+...+(1 / 40)*1.0
[0089] The total measurement score is: the score of "East and West" + the score of "Entity" + the score of "Place" + the score of "Shanghai".
[0090] For an evaluation task, each instruction in the first evaluation instruction set corresponding to it is input into the large language model, and the output mode token and the confidence of each layer of output token are obtained; the metric score corresponding to each instruction can be obtained through the above calculation. Then the average of the metric scores corresponding to all instructions is calculated, and the average is used as the score corresponding to the large language model when solving the evaluation task.
[0091] Step 1032: Obtain an evaluation result of the large language model based on the calculated scores corresponding to the large language model when solving multiple evaluation tasks.
[0092] Here, the sum of the scores corresponding to the large language model when solving multiple evaluation tasks can be calculated, and the obtained sum can be used as the evaluation result of the large language model.
[0093] It should be noted that if the score of the large language model is higher, it means that the large language model has better system logic in the task, and the large language model uses the ability of the lower layer to solve the task. The high layer of the large language model can accommodate higher-dimensional knowledge and reasoning ability in the future, that is, the potential ability of the large language model is better. If the score of the large language model is low, the large language model needs to be optimized. Since the semantic space conversion logic of the large language model when solving the evaluation task is used to make a reasonable evaluation of the existing and potential capabilities of the large language model, the optimization of the large language model based on this can improve the optimization effect.
[0094] like Figure 2 As shown, an embodiment of the present invention further provides a model evaluation device, the device comprising:
[0095] An acquisition module 201 is used to acquire target evaluation instruction sets of multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets;
[0096] A processing module 202 is used to obtain, for each evaluation task, an output mode and a first output result of a large language model according to the target evaluation instruction set corresponding to the evaluation task, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model;
[0097] The model evaluation module 203 is used to evaluate the large language model based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks to obtain an evaluation result.
[0098] Optionally, the acquisition module 201 includes:
[0099] A first acquisition unit, configured to acquire a first evaluation instruction set for each evaluation task;
[0100] A first processing unit is configured to input each instruction in the first evaluation instruction set into the large language model to obtain a second output result;
[0101] The second acquisition unit is used to obtain the target evaluation instruction set by using the instructions corresponding to the second output result that represents the correct result.
[0102] Optionally, the first acquiring unit is specifically configured to:
[0103] For each evaluation task, obtain the pre-built seed instructions;
[0104] The seed instruction is learned to generate a first evaluation instruction set corresponding to the evaluation task.
[0105] Optionally, the processing module 202 includes:
[0106] a second processing unit, for inputting each instruction of the target evaluation instruction set corresponding to the evaluation task into the large language model, obtaining word-unit tokens output by each layer in the large language model and confidences of output tokens of each layer, and visually displaying the tokens output by each layer in the large language model; wherein the first output result of the large language model includes the confidences of output tokens of each layer in the large language model;
[0107] A third processing unit is used to obtain a block in the large language model that is strongly related to the evaluation task based on the visually displayed token;
[0108] The fourth processing unit is used to determine the output mode of the large language model according to the multiple blocks in the obtained large language model that are strongly related to the evaluation task.
[0109] Optionally, the fourth processing unit is specifically configured to:
[0110] Determine a target block based on multiple blocks in the large language model that are strongly related to the evaluation task, where the target block is a common block in the multiple blocks;
[0111] Based on the target block, the semantic space conversion logic of the large language model when solving the evaluation task is determined, and the semantic space conversion logic of the large language model when solving the evaluation task is used as the output mode of the large language model.
[0112] Optionally, the model evaluation module 203 includes:
[0113] A first evaluation unit is used to calculate, for each evaluation task, a score corresponding to the large language model when solving the evaluation task by using the number of layers in the large language model where the output mode of the large language model corresponding to the evaluation task is located and a first output result;
[0114] The second evaluation unit is used to obtain an evaluation result of the large language model based on the calculated scores corresponding to the large language model when solving multiple evaluation tasks.
[0115] The model evaluation device of the embodiment of the present invention obtains target evaluation instruction sets of multiple evaluation tasks, and different evaluation tasks correspond to different target evaluation instruction sets; then, for each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, the output mode and the first output result of the large language model are obtained, the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model; finally, based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result. In this way, through the above processing, the logic of the large language model when solving the task is analyzed, that is, the semantic space conversion logic of the large language model when solving the evaluation task is obtained, and the semantic space conversion logic of the large language model when solving the evaluation task is used to make a reasonable evaluation of the existing and potential capabilities of the large language model, thereby improving the optimization effect of the large language model in the later stage.
[0116] In order to better achieve the above goals, Figure 3 As shown, an embodiment of the present invention further provides a model evaluation device, including a processor 300 and a transceiver 310, wherein the transceiver 310 receives and sends data under the control of the processor 300, and the processor 300 is used to perform the following process:
[0117] Obtain target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets;
[0118] For each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, an output mode and a first output result of a large language model are obtained, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model;
[0119] Based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result.
[0120] Optionally, the processor 300 is further configured to:
[0121] For each evaluation task, obtaining a first evaluation instruction set;
[0122] For each instruction in the first evaluation instruction set, input the instruction into the large language model to obtain a second output result;
[0123] The target evaluation instruction set is obtained by using the instructions corresponding to the second output result that represents the correct result.
[0124] Optionally, the processor 300 is further configured to:
[0125] For each evaluation task, obtain the pre-built seed instructions;
[0126] The seed instruction is learned to generate a first evaluation instruction set corresponding to the evaluation task.
[0127] Optionally, the processor 300 is further configured to:
[0128] For each instruction of the target evaluation instruction set corresponding to the evaluation task, the instruction is input into the large language model, a word unit token output by each layer in the large language model and a confidence level of each layer output token are obtained, and the token output by each layer in the large language model is visualized; wherein the first output result of the large language model includes the confidence level of each layer output token in the large language model;
[0129] Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the evaluation task;
[0130] According to the obtained multiple blocks in the large language model that are strongly related to the evaluation task, an output mode of the large language model is determined.
[0131] Optionally, the processor 300 is further configured to:
[0132] Determine a target block based on multiple blocks in the large language model that are strongly related to the evaluation task, where the target block is a common block in the multiple blocks;
[0133] Based on the target block, the semantic space conversion logic of the large language model when solving the evaluation task is determined, and the semantic space conversion logic of the large language model when solving the evaluation task is used as the output mode of the large language model.
[0134] Optionally, the processor 300 is further configured to:
[0135] For each evaluation task, using the layer number of the output mode of the large language model corresponding to the evaluation task in the large language model and the first output result, calculate the score corresponding to the large language model when solving the evaluation task;
[0136] Based on the calculated scores of the large language model when solving multiple evaluation tasks, an evaluation result of the large language model is obtained.
[0137] The model evaluation device of the embodiment of the present invention obtains target evaluation instruction sets of multiple evaluation tasks, and different evaluation tasks correspond to different target evaluation instruction sets; then, for each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, the output mode and the first output result of the large language model are obtained, the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model; finally, based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result. In this way, through the above processing, the logic of the large language model when solving the task is analyzed, that is, the semantic space conversion logic of the large language model when solving the evaluation task is obtained, and the semantic space conversion logic of the large language model when solving the evaluation task is used to make a reasonable evaluation of the existing and potential capabilities of the large language model, thereby improving the optimization effect of the large language model in the later stage.
[0138] An embodiment of the present invention also provides a model evaluation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the various processes in the model evaluation method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.
[0139] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the above-mentioned model evaluation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0140] An embodiment of the present invention further provides a computer program product, comprising computer instructions, which implement the steps in the model evaluation method described above when executed by a processor.
[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.
[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0145] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A model evaluation method, characterized in that: include: Obtain target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets; For each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, an output mode and a first output result of a large language model are obtained, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model; Based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result.
2. The method according to claim 1, characterized in that: The step of obtaining a target evaluation instruction set for multiple evaluation tasks includes: For each evaluation task, obtaining a first evaluation instruction set; For each instruction in the first evaluation instruction set, input the instruction into the large language model to obtain a second output result; The target evaluation instruction set is obtained by using the instructions corresponding to the second output result that represents the correct result.
3. The method according to claim 2, characterized in that The step of obtaining a first evaluation instruction set for each evaluation task includes: For each evaluation task, obtain the pre-built seed instructions; The seed instruction is learned to generate a first evaluation instruction set corresponding to the evaluation task.
4. The method according to claim 1, characterized in that: The obtaining, according to the target evaluation instruction set corresponding to the evaluation task, an output mode and a first output result of the large language model, comprises: For each instruction of the target evaluation instruction set corresponding to the evaluation task, the instruction is input into the large language model, a word unit token output by each layer in the large language model and a confidence level of each layer output token are obtained, and the token output by each layer in the large language model is visualized; wherein the first output result of the large language model includes the confidence level of each layer output token in the large language model; Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the evaluation task; According to the obtained multiple blocks in the large language model that are strongly related to the evaluation task, an output mode of the large language model is determined.
5. The method according to claim 4, characterized in that The step of determining the output mode of the large language model according to the obtained plurality of blocks strongly related to the evaluation task in the large language model comprises: Determine a target block based on multiple blocks in the large language model that are strongly related to the evaluation task, where the target block is a common block in the multiple blocks; Based on the target block, the semantic space conversion logic of the large language model when solving the evaluation task is determined, and the semantic space conversion logic of the large language model when solving the evaluation task is used as the output mode of the large language model.
6. The method according to claim 1 or 4, characterized in that: The step of evaluating the large language model based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks to obtain an evaluation result includes: For each evaluation task, using the layer number of the output mode of the large language model corresponding to the evaluation task in the large language model and the first output result, calculate the score corresponding to the large language model when solving the evaluation task; Based on the calculated scores of the large language model when solving multiple evaluation tasks, an evaluation result of the large language model is obtained.
7. A model evaluation device, characterized in that: include: An acquisition module is used to acquire target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets; A processing module, configured to obtain, for each evaluation task, an output mode and a first output result of a large language model according to the target evaluation instruction set corresponding to the evaluation task, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model; A model evaluation module is used to evaluate the large language model based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks to obtain an evaluation result.
8. A model evaluation device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, characterized in that: The processor is configured to perform the following operations: Obtain target evaluation instruction sets for multiple evaluation tasks, where different evaluation tasks correspond to different target evaluation instruction sets; For each evaluation task, according to the target evaluation instruction set corresponding to the evaluation task, an output mode and a first output result of a large language model are obtained, wherein the output mode of the large language model is used to characterize the semantic space conversion logic of the large language model when solving the evaluation task, and the first output result is determined by the target evaluation instruction set after passing through the large language model; Based on the output mode and the first output result of the large language model corresponding to the multiple evaluation tasks, the large language model is evaluated to obtain an evaluation result.
9. A model evaluation device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the model evaluation method according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the model evaluation method according to any one of claims 1 to 6 are implemented.
11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the model evaluation method according to any one of claims 1 to 6.