Instruction evaluation method and device, equipment, storage medium and program product
By evaluating the word sequence and output results of the large language model, the performance of instructions in the inference process is analyzed, the problem of how to evaluate the quality of instructions is solved, and the effective activation of the ability of the large language model is achieved.
Patent Information
- Application Number
- CN202411270506.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-13
AI Technical Summary
How to evaluate the pros and cons of instructions entered into a large language model to activate their capabilities.
By obtaining the word element sequence output from the first block of the large language model and the word element output from the last layer, the instructions are evaluated, their design advantages and disadvantages are analyzed in the inference process, and optimization is carried out based on the results or update model parameters. .
It realizes effective evaluation of the instruction design of large language model, provides directions to improve its internal logic, and promotes the activation of model capabilities.
Smart Images
Figure CN119128460A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to an instruction evaluation method, device, equipment, storage medium and program product. Background Art
[0002] At present, the application of large language models in domain tasks first considers the optimization of command prompt engineering. Designing the optimal prompt for different large language model characteristics can effectively activate the large language model capabilities, that is, the quality of the prompt plays an important role in this scenario. Therefore, in order to effectively activate the capabilities of the large language model, how to evaluate the quality of the prompt has become an urgent problem to be solved. Summary of the invention
[0003] The purpose of the present invention is to provide an instruction evaluation method, device, equipment, storage medium and program product to solve the problem of how to evaluate the quality of instructions input into a large language model.
[0004] In order to achieve the above object, in a first aspect, the present invention provides an instruction evaluation method, comprising:
[0005] Obtaining a first instruction for a first task;
[0006] Inputting the first instruction of the first task into the large language model to obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model;
[0007] Based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result.
[0008] In some embodiments, the first instruction is evaluated based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model to obtain an evaluation result, including:
[0009] When the word unit sequence output by the first block is correct and the word units output by the last layer of the large language model are correct, determining that the evaluation result is that the design of the first instruction is excellent;
[0010] When there are errors in the word-gram sequence output by the first block, but the word-grams output by the last layer of the large language model are correct, it is determined that the evaluation result is a design error of the first instruction.
[0011] In some embodiments, the method further comprises:
[0012] Get the instruction sets of multiple tasks. Different tasks correspond to different instruction sets.
[0013] For each task, input each instruction in the instruction set corresponding to the task into the large language model to obtain an output result;
[0014] Classify all the output results obtained corresponding to each task to obtain output results of the best category, and visualize the word units outputted at each layer in the large language model corresponding to the output results of the best category;
[0015] Based on the visually displayed word units, a block related to the task in the large language model is obtained.
[0016] In some embodiments, the obtaining of a set of instructions for multiple tasks includes:
[0017] For each task, get the pre-built seed instructions;
[0018] The seed instructions are learned to generate an instruction set corresponding to the task.
[0019] In some embodiments, when there is an error in the word-gram sequence output by the first block, but the word-gram output by the last layer of the large language model is correct, after determining that the evaluation result is a poor design of the first instruction, the method further includes:
[0020] The first instruction, the output result of the first instruction after passing through the large language model, and the standard answer corresponding to the first instruction are input into the preset large model, and the first instruction is optimized to obtain an optimized first instruction.
[0021] In some embodiments, when there is an error in the word-gram sequence output by the first block, but the word-gram output by the last layer of the large language model is correct, after determining that the evaluation result is a poor design of the first instruction, the method further includes:
[0022] Determining a target layer in the first block, the target layer being a layer corresponding to an erroneous word in the word sequence;
[0023] The network parameters of the target layer are updated, and an updated large language model is obtained based on the updated target layer.
[0024] In a second aspect, the present invention further provides an instruction evaluation device, comprising:
[0025] A first acquisition module, used to acquire a first instruction for a first task;
[0026] A first processing module, configured to input the first instruction of the first task into a large language model, and obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model;
[0027] An instruction evaluation module is used to evaluate the first instruction based on the word unit sequence output by the first block and the word units output by the last layer of the large language model to obtain an evaluation result.
[0028] In a third aspect, the present invention further provides an instruction evaluation device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, and the processor is used to perform the following operations:
[0029] Obtaining a first instruction for a first task;
[0030] Inputting the first instruction of the first task into the large language model to obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model;
[0031] Based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result.
[0032] In a fourth aspect, the present invention further provides an instruction evaluation device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the instruction evaluation method as described in the first aspect above is implemented.
[0033] In a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the instruction evaluation method described in the first aspect above.
[0034] In a sixth aspect, the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the instruction evaluation method described in the first aspect above.
[0035] The above technical solution of the present invention has at least the following beneficial effects:
[0036] In an embodiment of the present invention, a first instruction for a first task is obtained, and the first instruction of the first task is input into a large language model to obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block of the large language model is used to characterize the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model; finally, based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result. In this way, by analyzing the reasoning process of the first instruction in the large language model, that is, the word-gram sequence output by the first block of the large language model obtained after the first instruction passes through the large language model, the design advantages and disadvantages of the first instruction can be judged, which provides a clear improvement direction for the subsequent adaptation of the internal logic of the large language model, and can effectively activate the capabilities of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A schematic diagram showing a flow chart of an instruction evaluation method provided by an embodiment of the present invention;
[0038] Figure 2 A schematic diagram showing a module of an instruction evaluation device provided by an embodiment of the present invention;
[0039] Figure 3 A schematic diagram showing the hardware structure of an instruction evaluation device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0040] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0041] like Figure 1 FIG. 1 is a flow chart of an instruction evaluation method provided by an embodiment of the present invention. The method may include:
[0042] Step 101: Obtain a first instruction for a first task.
[0043] The first instruction is the instruction to be evaluated. It should be noted that the instruction described in this article refers to the prompt word prompt. The function of prompt is mainly to prompt the large language model with the context of input information and parameter information of the input model. A prompt can contain information such as instructions or questions transmitted to the large language model, and can also contain other detailed information, such as context, input, or examples.
[0044] The first task may be a specific task, where the specific meaning is to group together knowledge with the same human cognition, such as named entity recognition (NER), relation extraction (RE), event extraction (EE), etc.
[0045] For example, the first task is a NER task, and the first instruction includes: text: The weather in Guangzhou is really hot. The instruction for the text is: Please extract the entities in the text. The output format for the instruction is JSON format, that is, {'entity': [entity name]}.
[0046] Step 102: input the first instruction of the first task into the large language model to obtain the word sequence output by the first block of the large language model and the word sequence output by the last layer of the large language model; wherein the word sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model.
[0047] Here, different tasks correspond to different blocks in the large language model and have been obtained in advance. The blocks are strongly correlated with the tasks and can reflect the logical reasoning ability of the large language model in solving the tasks. Among them, the strong correlation between the blocks and the tasks means that there is an association relationship between the tasks and the blocks, and "strong correlation" means a high correlation, a strong correlation. That is, there is a strong association relationship between the tasks and the blocks. Among them, the acquisition of the blocks in the large language model can be referred to the subsequent embodiments.
[0048] It should be understood that the large language model is composed of stacked neural networks, wherein each layer of the neural network can be referred to as a layer, and the block includes at least one layer (neural network). After the instruction is input into the large language model, each layer of the large language model will output a word unit token.
[0049] Step 103: Based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result.
[0050] Here, by analyzing the reasoning process of the first instruction in the large language model, that is, the word sequence output by the first block of the large language model after the first instruction passes through the large language model, combined with the word elements output by the last layer of the large language model, the design advantages of the first instruction can be judged, which provides a clear improvement direction for the subsequent adaptation of the internal logic of the large language model, and can achieve effective activation of the capabilities of the large language model.
[0051] In an optional embodiment, the above step 103 includes:
[0052] Step 1031, when the word sequence output by the first block is correct and the word sequence output by the last layer of the large language model is correct, determine that the evaluation result is that the design of the first instruction is excellent.
[0053] Here, after the first instruction to be evaluated is input into the large language model, first check whether the result (i.e., the word unit output by the last layer of the large language model) is correct; if correct, then check whether the intermediate word unit sequence is output according to the reasoning logic of the first task (i.e., whether the first block corresponding to the first task outputs the word unit sequence). If the result and the reasoning logic are both correct, that is, the situation of step 1031 is met, then it is determined that the design of the first instruction is excellent, that is, the evaluation result is that the design of the first instruction is excellent.
[0054] Step 1032: When there are errors in the word-gram sequence output by the first block, but the word-gram output by the last layer of the large language model is correct, determine that the evaluation result is a design error of the first instruction.
[0055] Here, after the first instruction to be evaluated is input into the large language model, first check whether the result (i.e., the word unit output by the last layer of the large language model) is correct; if correct, then check whether the intermediate word unit sequence is output according to the reasoning logic of the first task (i.e., whether the first block corresponding to the first task outputs the word unit sequence). If the result is correct and the reasoning process is incorrect, that is, the situation of step 1032 is met, then the design difference of the first instruction is determined, that is, the evaluation result is the design difference of the first instruction.
[0056] This embodiment can interpretably evaluate the quality of instruction design of a large language model.
[0057] The following describes a specific process of obtaining blocks in a large language model through an embodiment.
[0058] In an optional embodiment, the method of the present invention further comprises:
[0059] Step a1, obtaining instruction sets of multiple tasks, where different tasks correspond to different instruction sets.
[0060] Among them, this step a1 may specifically include:
[0061] Step a11: for each task, obtain a pre-built seed instruction.
[0062] The tasks involved here are specific tasks, wherein the seed instructions include preset texts related to the tasks, preset instructions for the preset texts, and output formats for the preset instructions.
[0063] It should be noted that the seed instruction can be an instruction constructed manually to ensure the rigor of the task. Generally, there are multiple seed instructions.
[0064] In Example 1, the task is a NER task, and the preset text is: The weather in Shanghai is really hot. The preset instruction for the preset text is: Please extract the entities in the text. The output format for the preset instruction is JSON format, that is, {'entity': [entity name]}.
[0065] Step a12, learning the seed instructions and generating an instruction set corresponding to the task.
[0066] A more powerful large model (such as GPT4o) can be used to learn seed instructions and generate a large number of instructions as the instruction set corresponding to the task.
[0067] It should be noted that, in order to ensure the rigor of the task, the instructions generated in step a12 may be manually reviewed, that is, after a large number of instructions are generated, they are manually reviewed to obtain an instruction set corresponding to the task.
[0068] Step a2: for each task, input each instruction in the instruction set corresponding to the task into the large language model to obtain an output result.
[0069] Step a3, classifying all the output results corresponding to each task to obtain output results of the best category, and visually displaying the word units outputted at each layer in the large language model corresponding to the output results of the best category.
[0070] Specifically, all output results corresponding to each task may be classified based on at least one of user preference, the ability of the large language model to solve the task, and the output requirements of the large language model.
[0071] Here, the classification based on user preferences can be to set different classification standards for different tasks based on user preferences (or user portraits), and determine the category of output results based on the set classification standards, that is, which are output results classified as excellent, which are output results classified as medium, and which are output results classified as poor.
[0072] The classification based on the ability of a large language model to solve tasks can be based on the different abilities of a large language model to solve different tasks, setting different classification standards for different tasks, and determining the category of the output result based on the set classification standards.
[0073] The classification based on the output requirements of the large language model can be to classify the output results based on the fluency requirements, language style requirements, and security maintenance requirements of the large language model.
[0074] The above classification criteria can be quantified into scores, with different score ranges corresponding to different categories.
[0075] If the output result category corresponding to the above example 1 is excellent, that is, for the task is a NER task, the prompt input to the large model is: Please extract entities from the text. \nText: The weather in Shanghai is really hot. \nPlease output the JSON format {'entity': [entity name]}. After that, the word unit token output of each layer in the large language model is visualized, as shown in Table 1 below:
[0076]
[0077]
[0078] Step a4: based on the visually displayed word units, obtain the blocks related to the task in the large language model.
[0079] Here, based on the visualized word units obtained in step a3, the expert can obtain the reasoning logic hidden in the large language model when the large language model performs task reasoning through analysis, that is, the block in the large language model related to the task (the block reflects the logical reasoning ability of the large language model in solving the task). Then, through user input, the block in the large language model related to the task is obtained.
[0080] Experts can analyze the visualized tokens to identify and judge the blocks in the large language model that are strongly related to the task. This block can be defined as a strong capability block for the task. For example, in the above example Table 1, when the token of "Shanghai" is inferred, "things" is output at the 17th layer, "entity" is output at the 18th to 26th layers, "place" is output at the 27th to 29th layers, and "Shanghai" is output at the 30th to 40th layers. The reasoning process has a certain logic from abstract to concrete, so the 17th to 40th layers are defined as blocks that are strongly related to the task of this case (instruction).
[0081] Here, the token output by the large language model in the corresponding block of the corresponding task is used as the basis for evaluating other prompts. A good (excellent) prompt can effectively activate the reasoning logic of the large language model on the corresponding task. If the large language model does not output the corresponding reasoning logic (that is, the token output by the large language model in the corresponding block of the corresponding task is incorrect), although the answer (that is, the final output result of the large language model (the word unit output by the last layer of the large language model)) is correct, it may be an accidental event or the result of rote memorization by the large language model, and the prompt still cannot be used as a good prompt.
[0082] When the word unit sequence output by the first block is wrong, but the word unit output by the last layer of the large language model is correct, the evaluation result is determined to be that the design of the first instruction is poor, that is, the design of the first instruction is not good enough. At this time, the instruction details of the first instruction can be manually modified, and then the modified first instruction can be input into the large language model for secondary evaluation. Of course, it can also be modified by machine, and the specific implementation can be referred to the following embodiment.
[0083] In an optional embodiment, when there is an error in the word-gram sequence output by the first block, but the word-gram output by the last layer of the large language model is correct, after determining that the evaluation result is a design error of the first instruction, the method of the present invention further includes:
[0084] The first instruction, the output result of the first instruction after passing through the large language model, and the standard answer corresponding to the first instruction are input into the preset large model, and the first instruction is optimized to obtain an optimized first instruction.
[0085] Among them, the standard answer corresponding to the first instruction can be obtained in advance, and can be specifically obtained by querying from a pre-established database, that is, querying the first instruction from the database to obtain the standard answer corresponding to the first instruction. Here, the preset large model is a more powerful model used to optimize instructions. That is, the preset large language model is used to modify the instruction details of the first instruction. After that, the optimized first instruction can be input into the large language model for secondary evaluation until the token sequence output by the first block meets the reasoning logic of the first task.
[0086] In an optional embodiment, when there is an error in the word-gram sequence output by the first block, but the word-gram output by the last layer of the large language model is correct, after determining that the evaluation result is a design error of the first instruction, the method of the present invention further includes:
[0087] Determine a target layer in the first block, where the target layer is a layer corresponding to an erroneous word in the word sequence; update a network parameter of the target layer, and obtain an updated large language model based on the updated target layer.
[0088] That is, for the error token output by a certain layer in the first block, the network parameters of the layer can be adjusted (specifically, the loss function can be calculated based on the error token, and the network parameters of the layer can be adjusted based on the loss function), that is, the network parameters of the target layer are updated; then, the updated large language model is obtained based on the updated target layer. In this way, the large language model can be optimized.
[0089] like Figure 2 As shown, an embodiment of the present invention further provides an instruction evaluation device, the device comprising:
[0090] A first acquisition module 201, used to acquire a first instruction for a first task;
[0091] A first processing module 202 is used to input the first instruction of the first task into the large language model to obtain a word-gram sequence output by the first block of the large language model and a word-gram output by the last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model;
[0092] The instruction evaluation module 203 is used to evaluate the first instruction based on the word-unit sequence output by the first block and the word-unit output by the last layer of the large language model to obtain an evaluation result.
[0093] Optionally, the instruction evaluation module 203 includes:
[0094] A first evaluation unit, configured to determine that the evaluation result is that the design of the first instruction is excellent when the word unit sequence output by the first block is correct and the word units output by the last layer of the large language model are correct;
[0095] The second evaluation unit is used to determine that the evaluation result is a design error of the first instruction when there is an error in the word unit sequence output by the first block but the word units output by the last layer of the large language model are correct.
[0096] Optionally, the device in the embodiment of the present invention further includes:
[0097] The second acquisition module is used to acquire instruction sets of multiple tasks, where different tasks correspond to different instruction sets;
[0098] A second processing module is used for inputting each instruction in the instruction set corresponding to each task into the large language model to obtain an output result;
[0099] A third processing module is used to classify all output results obtained corresponding to each task, obtain output results of the excellent category, and visualize the word units outputted at each layer in the large language model corresponding to the output results of the excellent category;
[0100] The third acquisition module is used to obtain the block related to the task in the large language model based on the visually displayed word unit.
[0101] Optionally, the second acquisition module includes:
[0102] A first acquisition unit, configured to acquire a pre-built seed instruction for each task;
[0103] The second acquisition unit is used to learn the seed instruction and generate an instruction set corresponding to the task.
[0104] Optionally, the device of the embodiment of the present invention includes:
[0105] The fourth processing module is used to determine that the evaluation result is a design error of the first instruction when there is an error in the word unit sequence output by the first block but the word units output by the last layer of the large language model are correct, input the first instruction, the output result of the first instruction after passing through the large language model, and the standard answer corresponding to the first instruction into a preset large model, optimize the first instruction, and obtain an optimized first instruction.
[0106] Optionally, the device of the embodiment of the present invention includes:
[0107] a fifth processing module, configured to determine, when there is an error in the word-gram sequence output by the first block but the word-gram output by the last layer of the large language model is correct, a target layer in the first block after determining that the evaluation result is a design error of the first instruction, the target layer being a layer corresponding to the error word-gram in the word-gram sequence;
[0108] The sixth processing module is used to update the network parameters of the target layer and obtain an updated large language model based on the updated target layer.
[0109] The instruction evaluation device of the embodiment of the present invention obtains a first instruction for a first task, inputs the first instruction of the first task into a large language model, and obtains a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block of the large language model is used to characterize the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model; finally, based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result. In this way, by analyzing the reasoning process of the first instruction in the large language model, that is, the word-gram sequence output by the first block of the large language model obtained after the first instruction passes through the large language model, the design advantages and disadvantages of the first instruction are judged, which provides a clear improvement direction for the subsequent adaptation of the internal logic of the large language model, and can achieve effective activation of the capabilities of the large language model.
[0110] In order to better achieve the above goals, Figure 3As shown, an embodiment of the present invention further provides an instruction evaluation device, including a processor 300 and a transceiver 310, wherein the transceiver 310 receives and sends data under the control of the processor 300, and the processor 300 is used to perform the following process:
[0111] Obtaining a first instruction for a first task;
[0112] Inputting the first instruction of the first task into the large language model to obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model;
[0113] Based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result.
[0114] Optionally, the processor 300 is further configured to:
[0115] When the word unit sequence output by the first block is correct and the word units output by the last layer of the large language model are correct, determining that the evaluation result is that the design of the first instruction is excellent;
[0116] When there are errors in the word-gram sequence output by the first block, but the word-grams output by the last layer of the large language model are correct, it is determined that the evaluation result is a design error of the first instruction.
[0117] Optionally, the processor 300 is further configured to:
[0118] Get the instruction sets of multiple tasks. Different tasks correspond to different instruction sets.
[0119] For each task, input each instruction in the instruction set corresponding to the task into the large language model to obtain an output result;
[0120] Classify all the output results obtained corresponding to each task to obtain output results of the best category, and visualize the word units outputted at each layer in the large language model corresponding to the output results of the best category;
[0121] Based on the visually displayed word units, a block related to the task in the large language model is obtained.
[0122] Optionally, the processor 300 is further configured to:
[0123] For each task, get the pre-built seed instructions;
[0124] The seed instructions are learned to generate an instruction set corresponding to the task.
[0125] Optionally, the processor 300 is further configured to:
[0126] When there are errors in the word-gram sequence output by the first block, but the word-grams output by the last layer of the large language model are correct, after determining that the evaluation result is a design error of the first instruction, the first instruction, the output result of the first instruction after passing through the large language model, and the standard answer corresponding to the first instruction are input into the preset large model, and the first instruction is optimized to obtain an optimized first instruction.
[0127] Optionally, the processor 300 is further configured to:
[0128] When there is an error in the word-gram sequence output by the first block, but the word-gram output by the last layer of the large language model is correct, after determining that the evaluation result is a design error of the first instruction, determining a target layer in the first block, the target layer being a layer corresponding to the erroneous word-gram in the word-gram sequence;
[0129] The network parameters of the target layer are updated, and an updated large language model is obtained based on the updated target layer.
[0130] The instruction evaluation device of the embodiment of the present invention obtains a first instruction for a first task, inputs the first instruction of the first task into a large language model, and obtains a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block of the large language model is used to characterize the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model; finally, based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result. In this way, by analyzing the reasoning process of the first instruction in the large language model, that is, the word-gram sequence output by the first block of the large language model obtained after the first instruction passes through the large language model, the design advantages and disadvantages of the first instruction can be judged, which provides a clear improvement direction for the subsequent adaptation of the internal logic of the large language model, and can achieve effective activation of the capabilities of the large language model.
[0131] An embodiment of the present invention also provides an instruction evaluation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the various processes in the instruction evaluation method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.
[0132] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the instruction evaluation method embodiment described above is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0133] An embodiment of the present invention further provides a computer program product, including computer instructions, which implement the steps in the instruction evaluation method described above when executed by a processor.
[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.
[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0138] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for evaluating an instruction, characterized in that: include: Obtaining a first instruction for a first task; Inputting the first instruction of the first task into the large language model to obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model; Based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result.
2. The method according to claim 1, characterized in that The first instruction is evaluated based on the word-unit sequence output by the first block and the word-unit output by the last layer of the large language model to obtain an evaluation result, including: When the word unit sequence output by the first block is correct and the word units output by the last layer of the large language model are correct, determining that the evaluation result is that the design of the first instruction is excellent; When there are errors in the word-gram sequence output by the first block, but the word-grams output by the last layer of the large language model are correct, it is determined that the evaluation result is a design error of the first instruction.
3. The method according to claim 1, characterized in that The method further comprises: Get the instruction sets of multiple tasks. Different tasks correspond to different instruction sets. For each task, input each instruction in the instruction set corresponding to the task into the large language model to obtain an output result; Classify all the output results obtained corresponding to each task to obtain output results of the best category, and visualize the word units outputted at each layer in the large language model corresponding to the output results of the best category; Based on the visually displayed word units, a block related to the task in the large language model is obtained.
4. The method according to claim 3, characterized in that The instruction set for obtaining multiple tasks includes: For each task, get the pre-built seed instructions; The seed instructions are learned to generate an instruction set corresponding to the task.
5. The method according to claim 2, characterized in that: When there is an error in the word unit sequence output by the first block, but the word units output by the last layer of the large language model are correct, after determining that the evaluation result is a poor design of the first instruction, the method further includes: The first instruction, the output result of the first instruction after passing through the large language model, and the standard answer corresponding to the first instruction are input into the preset large model, and the first instruction is optimized to obtain an optimized first instruction.
6. The method according to claim 2, characterized in that When there is an error in the word unit sequence output by the first block, but the word units output by the last layer of the large language model are correct, after determining that the evaluation result is a poor design of the first instruction, the method further includes: Determining a target layer in the first block, the target layer being a layer corresponding to an erroneous word in the word sequence; The network parameters of the target layer are updated, and an updated large language model is obtained based on the updated target layer.
7. An instruction evaluation device, characterized in that: include: A first acquisition module, used to acquire a first instruction for a first task; A first processing module, configured to input the first instruction of the first task into a large language model, and obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model; An instruction evaluation module is used to evaluate the first instruction based on the word unit sequence output by the first block and the word units output by the last layer of the large language model to obtain an evaluation result.
8. An instruction evaluation device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, characterized in that: The processor is configured to perform the following operations: Obtaining a first instruction for a first task; Inputting the first instruction of the first task into the large language model to obtain a word-gram sequence output by a first block of the large language model and a word-gram output by a last layer of the large language model; wherein the word-gram sequence output by the first block is used to represent the reasoning logic of the large language model when solving the first task, and the first block includes at least one layer of the large language model; Based on the word-gram sequence output by the first block and the word-gram output by the last layer of the large language model, the first instruction is evaluated to obtain an evaluation result.
9. An instruction evaluation device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the instruction evaluation method according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the instruction evaluation method according to any one of claims 1 to 6 are implemented.
11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the instruction evaluation method according to any one of claims 1 to 6.
Citation Information
Cited By
Instruction compliance assessment method and device, electronic equipment and storage medium
CN121412618A