Construction method and device of high-quality complex instruction, equipment and medium

Abstract

The invention relates to a construction method and device of a high-quality complex instruction, equipment and a medium, and belongs to the technical field of large language models.The method comprises the steps that a high-quality instruction template is set, and a reference instruction is constructed according to the high-quality instruction template; the high-quality instruction template comprises an instruction structure, a constraint type and a capability type; inputting the high-quality instruction template and the reference instruction into a large language model for adjustment, and generating a to-be-evaluated instruction; inputting the high-quality instruction template, an evaluation standard and a to-be-evaluated instruction into a large language model, and calculating a total score of the to-be-evaluated instruction; and if the total score is smaller than a preset threshold value, performing iterative optimization on the to-be-evaluated instruction until the total score is greater than the preset threshold value, and outputting the to-be-evaluated instruction as a high-quality complex instruction. According to the method, the high-quality complex instruction is constructed by using the large language model, and the construction efficiency and quality of the high-quality complex instruction are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and in particular relates to a method, device, equipment and medium for constructing high-quality complex instructions. Background Art

[0002] In the field of large language models, constructing instructions is a job that requires both skill and art. It requires people to have a deep understanding of the principles of large model response generation, be familiar with the model's capability boundaries, have high logical reasoning ability, the ability to analyze and solve problems, and have a high level of linguistic knowledge. They should be able to understand the impact of instructions on model responses from the quality of model responses, and then optimize their own prompt strategies.

[0003] Humans construct and evaluate instructions. The instruction construction process is tedious and detailed, requiring the ability to decompose tasks and sufficient understanding of the model by the person constructing the instructions. However, the construction process lacks inspiration from scenario-based tasks, which makes it difficult to advance in practice. Instruction construction is not a purely creative task, but it also places high demands on people, resulting in a slow instruction construction process. However, the various prompt construction tutorials guided by the existing prompt engineering only provide broad and abstract instructions on how to use what methods and what frameworks to construct prompt words. There is a lack of exploration and thinking on how each element in these frameworks affects the quality of the response at what level, resulting in technical problems of low efficiency and low quality in the existing instruction construction methods. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method, apparatus, device and medium for constructing high-quality complex instructions, so as to solve one or more of the above problems existing in the prior art.

[0005] The object of the present invention is achieved in that:

[0006] A first aspect of the present invention provides a method for constructing a high-quality complex instruction, comprising:

[0007] Setting a high-quality instruction template, and constructing a reference instruction according to the high-quality instruction template; the high-quality instruction template includes an instruction structure, a constraint type, and a capability type;

[0008] Inputting the high-quality instruction template and the reference instruction into a large language model for adjustment to generate an instruction to be evaluated;

[0009] Inputting the high-quality instruction template, evaluation criteria and the instruction to be evaluated into a large language model, and calculating the total score of the instruction to be evaluated;

[0010] If the total score is less than a preset threshold, the instruction to be evaluated is iteratively optimized until the total score is greater than the preset threshold, and the instruction to be evaluated is output as a high-quality complex instruction.

[0011] Furthermore, the instruction structure includes a main task, task knowledge, constraints, text materials, demonstration cases and output format.

[0012] Furthermore, the constraint types include expected constraints and unexpected constraints. The expected constraints include personality specificity, logical judgment, execution steps and format limitations. The unexpected constraints include content themes, default constraints, powerlessness and ideal abstraction.

[0013] Furthermore, the ability types include basic ability, general ability, marginal ability and beyond ability.

[0014] Furthermore, the high-quality instruction template and reference instruction are input into a large language model for adjustment to generate an instruction to be evaluated, including: the large language model identifies and marks the constraint type of the constraint condition in the reference instruction according to the high-quality instruction template; modifies or adds the constraint type to be personalized, logically judged and format-limited constraint conditions according to the task knowledge; and outputs the instruction after the constraint modification or addition as the instruction to be evaluated.

[0015] Furthermore, the step of inputting the high-quality instruction template, the evaluation criteria and the instruction to be evaluated into a large language model and calculating the total score of the instruction to be evaluated includes: extracting, by the large language model, the main task and the constraint conditions of the instruction to be evaluated according to the high-quality instruction template, and identifying the capability type of the main task, and the constraint type and capability type of the constraint conditions; scoring the evaluability of the instruction to be evaluated according to the main task, the capability type of the main task and the constraint conditions and the evaluation criteria to obtain an evaluability score; scoring the complexity of the instruction to be evaluated according to the constraint type of the constraint conditions and the evaluation criteria to obtain a complexity score; and calculating a weighted average of the evaluability score and the complexity score as the total score.

[0016] Furthermore, the evaluation criteria include: an evaluability evaluation criteria, specifically including: a 100-point system, if the ability type of the main task and the constraint condition is an instruction that exceeds the ability, the evaluability score is 0, if the ability type of the main task and the constraint condition is an instruction that barely has the ability, the evaluability score is 0-30 points, and if the ability type of the main task and the constraint condition is an instruction that has general ability, the evaluability score is 50-80 points; a complexity evaluation criteria, specifically including: a 100-point system, if the constraint type of the constraint condition is an instruction in which the proportion of non-expected constraints is higher than expected constraints, the complexity score is 0-40 points, and if the constraint type of the constraint condition is an instruction in which the proportion of expected constraints is higher than non-expected constraints, the complexity score is 15-80 points based on the number of individual constraints and logical judgments.

[0017] A second aspect of the present invention provides a high-quality complex instruction construction device, comprising:

[0018] A reference instruction construction module, used to set a high-quality instruction template and construct a reference instruction according to the high-quality instruction template; the high-quality instruction template includes an instruction structure, a constraint type and a capability type;

[0019] An instruction adjustment module, used for inputting the high-quality instruction template and the reference instruction into a large language model for adjustment to generate an instruction to be evaluated;

[0020] An instruction evaluation module, used for inputting the high-quality instruction template, evaluation criteria and the instruction to be evaluated into a large language model for evaluation, and obtaining a total score of the instruction to be evaluated;

[0021] The high-quality instruction output module is used to iteratively optimize the instruction to be evaluated if the total score is less than a preset threshold, until the total score is greater than the preset threshold, and output the instruction to be evaluated as a high-quality complex instruction.

[0022] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for constructing high-quality complex instructions described in any embodiment is implemented.

[0023] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for constructing high-quality complex instructions described in any one of the embodiments.

[0024] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0025] The method for constructing high-quality complex instructions provided by the present invention designs a high-quality instruction template including an instruction structure, a constraint type, and a capability type, constructs a reference instruction according to the high-quality instruction template, and then uses a large language model to adjust the reference instruction according to the high-quality instruction template and the reference instruction to generate instructions to be evaluated, scores and evaluates the instructions to be evaluated to optimize the instructions, and uses the instructions that meet the evaluation criteria as high-quality complex instructions, thereby improving the construction efficiency and quality of high-quality complex instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0027] Figure 1A flowchart of a method for constructing high-quality complex instructions provided in Embodiment 1 of the present invention;

[0028] Figure 2 A schematic diagram of a construction device for high-quality complex instructions provided in Embodiment 2 of the present invention;

[0029] Figure 3 This is a schematic diagram of the electronic device architecture provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. It should be noted that, in the absence of conflict, the embodiments and features in the embodiments disclosed in this disclosure can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] Example 1

[0032] A specific embodiment of the present invention, as Figure 1 As shown, a method for constructing high-quality complex instructions is disclosed, including:

[0033] S1. Set a high-quality instruction template, and construct a reference instruction according to the high-quality instruction template; the high-quality instruction template includes an instruction structure, a constraint type, and a capability type.

[0034] In this embodiment, the instruction structure includes a main task, task knowledge, constraints, text materials, demonstration cases and output format, which are as follows:

[0035] (I) The main task can reflect the main intention of the instruction. The main task must be something that the model can do within its capabilities. It must be specific enough and cannot be a high-demand task that only humans or human experts can complete.

[0036] (II) Task knowledge is the knowledge extracted from the knowledge base to complete the main task. Some main tasks may require some background knowledge or relevant information. With this background knowledge or information, the model can better complete the main task.

[0037] (III) Constraints represent the details of completing the main task. Constraints should be sufficiently detailed and specific, closely related to the main task, personalized, logical, and have output format restrictions. Constraints should not contradict each other.

[0038] (IV) Text materials are text objects that need to be processed according to various specific requirements in the instructions. They must match the main task and constraints. For example, if the main task is to extract names from text, then names must appear in the text.

[0039] (V) Demonstration cases are cases that may appear in some complex high-quality complex instructions. They are demonstration references used in instructions to illustrate how to complete the main task according to the constraints;

[0040] (VI) Output format is the specific format requirement of the instruction for the model output, which must be consistent with the purpose of the main task. There may also be format-related demonstration cases.

[0041] In this embodiment, the constraint types include expected constraints and unexpected constraints. The expected constraints include personality specificity, logical judgment, execution steps and format limitations. The unexpected constraints include content theme, default constraints, powerlessness and ideal abstraction.

[0042] Specifically, expected constraints refer to the types of constraints that must be present in an instruction, and unexpected constraints refer to the types of constraints that are not present in an instruction;

[0043] Among the expected constraints, "personality and specificity" means that if the constraint conditions are sufficiently personalized and specific, the restrictions on the response are relatively high and have nothing to do with format restrictions, the response must provide a relatively fixed response method in accordance with such constraints; logical judgment refers to the constraints that require logic to judge, such as conditional statements, selection statements and negative statements; execution steps refer to the constraints that are specific execution steps on how to complete the main task, with a clear sequence of steps, the constraints are interrelated, and the main intention of the constraints is a series of actions; format restrictions refer to the constraints that are specific output format restrictions that are not related to the word limit.

[0044] Among non-expected constraints, content theme refers to constraints that are about how to write and what content should be written; default constraints refer to constraints that pursue the basic standard for answering any questions, such as fluent and logical language, or the characteristics of the constraints themselves are already reflected in the main task; powerlessness refers to constraints that are difficult for the model to implement, involving online searches, such as literature searches, drawing, multimodality, poetry creation, current affairs information searches, etc., or involving tasks that are too grand, abstract, professional, and time-consuming and laborious for humans to complete, such as writing risk management plans, which require human experts to do; ideal abstraction refers to constraints that are relatively abstract, and the language expression of the constraints is not clear enough, resulting in an inability to understand how the task is performed, or the constraints are actually a series of highly artistic and technical goals that are too difficult to achieve, such as poetry must rhyme and writing content must have depth.

[0045] In this embodiment, the capability types include basic capability, general capability, marginal capability and beyond capability.

[0046] Specifically, the capability type is determined based on the main task and constraints. Basic capability refers to the basic requirements that the model must meet, which generally exist in the default constraints, such as being able to generate fluent, natural, and grammatically error-free responses; general capability refers to tasks that the model can do but has a certain probability of making mistakes, such as text formatting, general text translation, summarization, topic classification, information extraction, sentiment analysis, and evaluation and scoring; barely capable refers to tasks that the model can barely complete but with a high probability of completing tasks that do not meet the requirements, such as poetry and novel writing, professional technical document writing, professional paper writing, professional translation, etc.; beyond capability refers to things that the model cannot do, including: drawing, online searching, generating audio, generating video, outputting files, sending files, etc.

[0047] It should be noted that the reference instructions provided in this embodiment can be manually constructed or generated through a pre-trained model, and this embodiment does not limit this.

[0048] Exemplarily, the constructed reference instructions are:

[0049] "Now you are a scenario salesperson. Please simulate and construct some dialogues between customers and salespeople according to the following conditions:

[0050] Scenario setting: Focus on the electronic products industry, especially the promotion of smart bracelets.

[0051] Key points for conversation: Factors that customers should focus on include product features, price inquiries, function demonstrations, applicable groups, warranty services, and comparisons with other brands.

[0052] Round requirements:

[0053] Intention to purchase group: There are at least 10 rounds of communication between customers and sales. After fully understanding the product, customers express their intention to buy, for example: "This smart bracelet is really good, I decided to buy it."

[0054] No intention to buy group: At least 7 rounds of dialogue, the mismatch is reflected by the product not meeting the customer's secondary needs or features, such as appearance design, lack of a specific function, etc. The customer directly expresses the reason for not buying, such as: "The appearance of this bracelet is not the type I like, I'd better consider other products."

[0055] Please construct these two sets of dialogues respectively and directly output what the different roles say. Note that one question and one answer is one round. Use turn plus square brackets with serial numbers to mark the rounds, such as turn[1]. In the dialogue, make sure that the salesperson's promotion is natural and smooth, and the customer's questions and feedback are also in line with the actual purchase scenario. "

[0056] S2. Input the high-quality instruction template and the reference instruction into the large language model for adjustment to generate an instruction to be evaluated.

[0057] In this embodiment, step S2 specifically includes the following steps:

[0058] S201, the large language model identifies the constraint type of the constraint condition in the reference instruction according to the high-quality instruction template and marks it;

[0059] S202, modifying or adding constraint conditions of a specific personality, logical judgment and format limitation according to the task knowledge;

[0060] S203: Output the instruction after the constraint modification or addition as the instruction to be evaluated.

[0061] Exemplarily, the following instructions are designed according to the high-quality instruction template and step S1 and input into the large language model for adjustment:

[0062] "You are a command format modification tool. You need to identify and rewrite a series of human-computer interaction commands, focusing on the constraints on the reply output format in the commands, and fully considering the restrictions on the main tasks and constraints in the reference commands. Rewrite or add a format constraint that reasonably matches the main task of the reference command. However, the rewritten or added format constraint will cause the output effect of the reference command to be completely different. The specific steps are as follows:

[0063] 1. Identify the content related to format restrictions in the instruction and mark it with <format restriction>; whether rewriting or adding new format constraints, they need to meet personalized and specific requirements. It is best to use the judgment condition modification of "if XX, then XX", and the modified format constraints must conform to the scenarios of the main tasks and constraint conditions of the reference instruction, and the matching must be reasonable;

[0064] If there is no <format restriction> related content in the reference instruction, please output "The reference instruction has no format-related constraints, which can be added." first, and then add the new format restriction constraints; if there are <format restriction> type constraints in the reference instruction, please output "The reference instruction has format-related constraints, which can be modified." first, and then you need to directly modify the constraints related to the format restriction to make the format output of the reference instruction different from the original one;

[0065] 2. Output the judgment modification content in step 1 separately, and separate it from the content of the complete output instruction in step 3 by using "== format to limit the judgment modification==".

[0066] 3. Keep the rest of the reference instruction unchanged, and reasonably add the part with added or modified format constraints to the reference instruction. The part with added or modified format constraints needs to be highlighted in bold, and then use "===instruction to be evaluated===" plus a line break, and then output the complete rewritten instruction.

[0067] The following are the reference instructions to be adjusted:

[0068] [Reference instructions for step S1 example]

[0069] ”

[0070] The output of the large language model is:

[0071] "The reference instructions have format constraints and can be modified.

[0072] ==Format limitation judgment modification==

[0073] The format of the reference instruction is limited to 'The conversation turn is marked with turn and square brackets with serial numbers'. Modify it to 'Each conversation turn is marked with TurnNumber followed by a serial number. If the customer expresses a purchase intention, it is indicated by capital letters.'

[0074] ===Instructions to be evaluated===

[0075] Now you are a scenario salesperson, please simulate and construct some dialogues between customers and salespersons according to the following conditions:

[0076] Scenario setting: Focus on the electronic products industry, especially the promotion of smart bracelets.

[0077] Key points for conversation: Factors that customers should focus on include product features, price inquiries, function demonstrations, applicable groups, warranty services, and comparisons with other brands.

[0078] Round requirements:

[0079] Intention to purchase group: There are at least 10 rounds of communication between customers and sales. After fully understanding the product, customers express their intention to buy, for example: "This smart bracelet is really good, I decided to buy it."

[0080] No intention to buy group: At least 7 rounds of dialogue, the mismatch is reflected by the product not meeting the customer's secondary needs or features, such as appearance design, lack of a specific function, etc. The customer directly expresses the reason for not buying, such as: "The appearance of this bracelet is not the type I like, I'd better consider other products."

[0081] Please construct these two groups of dialogues separately and directly output what different roles say. Note that a question and an answer is a round, which is marked with **TurnNumber followed by a sequence number. If the customer expresses a purchase intention, it is indicated by capital letters. In the dialogue, make sure that the salesperson's recommendations are natural and smooth, and that the customer's questions and feedback are consistent with the actual purchase scenario.

[0082] ”

[0083] S3. Input the high-quality instruction template, evaluation criteria and the instruction to be evaluated into a large language model, and calculate the total score of the instruction to be evaluated.

[0084] Specifically, step S3 includes the following steps:

[0085] S301, the large language model extracts the main task and constraint conditions of the instruction to be evaluated according to the high-quality instruction template, and identifies the capability type of the main task, and the constraint type and capability type of the constraint conditions;

[0086] S302, scoring the evaluability of the instruction to be evaluated according to the main task, the capability type of the main task and the constraint condition, and the evaluation criteria to obtain an evaluability score;

[0087] S303, scoring the complexity of the instruction to be evaluated according to the constraint type of the constraint condition and the evaluation standard to obtain a complexity score;

[0088] In this embodiment, the evaluation criteria include:

[0089] The evaluability evaluation criteria include: out of 100 points, if the ability type of the main task and constraint condition is an instruction that exceeds the ability, the evaluability score is 0, if the ability type of the main task and constraint condition is an instruction that is barely capable, the evaluability score is 0-30 points, and if the ability type of the main task and constraint condition is an instruction that is of average ability, the evaluability score is 50-80 points;

[0090] The complexity assessment criteria include: a 100-point system, for instructions where the constraint type of the constraint condition is that the proportion of unexpected constraints is higher than the expected constraints, the complexity score is 0-40 points; for instructions where the constraint type of the constraint condition is that the proportion of expected constraints is higher than the unexpected constraints, the complexity score is 15-80 points based on the number of individual constraints and logical judgments.

[0091] Exemplarily, the following instructions are designed based on the high-quality instruction template, the evaluation criteria, and the instructions to be evaluated generated in step S2 and input into the large language model for evaluation:

[0092] "As the god of high-quality complex instructions, you need to refer to the following high-quality complex instruction knowledge base to complete the analysis and evaluation of instructions. The specific details are as follows:

[0093] 1. Instruction knowledge base to refer to:

[0094] 1. The components and definitions of the directive are as follows:

[0095] [Main Task]: This is a task that reflects the main intention of the instruction. The main task must be something that the model can do. It must be specific enough and cannot be a high-demand task that only humans or human experts can complete. The main task and various constraints must be reasonable, that is, as a model, you are capable of executing it in place, and the constraints can effectively serve the main task. If the main task is too broad, abstract, and difficult to achieve, the main task is unreasonable.

[0096] [Task Knowledge]: Similar to the knowledge base of this instruction, some main tasks may require some background knowledge or relevant information. With this background knowledge or information, you, as a model, can better complete the main task;

[0097] [Constraints]: These are the various detailed constraints related to completing the main task. They are details about how to complete the main task. The constraints must be detailed and specific enough, closely related to the main task, personalized, logical, and have output format restrictions.

[0098] [Text material]: This is the text object that needs to be processed according to various specific requirements in this instruction. It must match the main task and constraints. For example, if the main task is to extract the names of people in the text, then the names must appear in the text.

[0099] [Demonstration Case]: It is a case that may appear in some complex instructions. It is a demonstration reference used in the instructions to illustrate how to complete the main task according to the constraints;

[0100] [Output Format]: This is the specific format requirement of the instruction for the model output. It must be consistent with the purpose of the main task. There may also be format-related demonstration cases.

[0101] 2. The constraint type of the instruction is:

[0102] Unexpected constraint types: Unwanted constraint types in instructions, including:

[0103] a. <Content Writing>: If the constraint is about how to write and what content should be written, it belongs to <Content Writing> constraint, unless the content-related constraint is specific to a fixed expression, for example, the output must be "I don't understand!" In this case, it is a <Personality Specific> type constraint. Any other broad topic-related content constraints only belong to <Content Writing>;

[0104] b. <Default constraint>: If the standard pursued by the constraint itself is the basic standard for you to answer any question, such as fluent and logical language, or the characteristics of the constraint itself are already reflected in the main task, then it is (default constraint);

[0105] c. <Nothing can be done>: If the constraint is difficult for the model to implement, and involves online search, such as literature search, drawing, multimodality, poetry creation, current affairs information search, etc., or involves tasks that are too grand, abstract, professional, and time-consuming and labor-intensive for humans to complete, such as writing risk management plans, which requires human experts to do, then it belongs to the <Nothing can be done> constraint;

[0106] d. <Ideal Abstract>: If the constraint is relatively abstract, the language of the constraint is not clear enough, making it difficult to understand how the task is performed; or the constraint is actually a series of highly artistic and highly technical goals that are too difficult to achieve, such as poetry must rhyme or writing content must have depth, then it is an <Ideal Abstract> constraint;

[0107] Expected constraints: The types of constraints that must be present in the instruction, including:

[0108] e. <Personalized Specific>: If the constraint is sufficiently personalized and specific, the restrictions on the response are relatively high, and it is not related to the personalized format. The response must provide a relatively fixed response method according to this type of constraint, then it is <Personalized Specific>;

[0109] f. <Logical judgment>: If the language form of the constraint is a conditional statement, which is used to express that a specific condition requires a specific thing to be done, it is marked as <Logical judgment>. Logical judgment does not refer to any problem that needs to be thought about. It only matters whether there are expressions related to conditional statements in the text, such as: 'If A, then B', 'If not A, then not B';

[0110] g. <Execution steps>: If the constraint is not about the characteristics of the response, but about the specific execution steps of how to complete the main task, there is a clear sequence, the constraints are interrelated, and the main intention part of the constraint is a series of actions, then this constraint is an <Execution steps> constraint;

[0111] h. <Format restriction>: If the constraint is about a specific output format restriction that is not related to the word count, it is marked as <Format restriction>. Note that the word count requirement is not considered a personal format constraint. If there is only a word count requirement, this constraint is ignored.

[0112] 3. The capability type of the instruction is:

[0113] a. Basic capabilities: The model can generate fluent, natural, and grammatically correct responses. The model must meet the requirements of basic constraints within its capabilities; this involves the main task and various constraints;

[0114] b. General capabilities: refers to tasks that the model can do but has a certain probability of error, such as text format processing, general text translation, summary, topic classification, information extraction, sentiment analysis, and evaluation and scoring;

[0115] c. Barely capable: refers to tasks that the model can barely complete but with a high probability of not meeting the quality requirements, such as poetry and novel writing, professional technical document writing, professional paper writing, professional translation, etc.

[0116] d. Beyond capabilities: The model cannot do things such as drawing, searching online, generating audio and video. The model needs to be online to search and verify timeliness information, which is also beyond the model's capabilities; writing poems, essays, or doing anything that requires professionals to spend a lot of time to complete; involving the main task and various constraints;

[0117] 4. Instruction quality assessment

[0118] a. The model has a moderate degree of executableness. Avoid having a model executable score that is too high or too low. If the model completes a task that is too simple or too difficult, it will lose its evaluation value.

[0119] b. The higher the human-evaluable score, the better. Avoid tasks that are too difficult to evaluate, such as writing, tasks that require expert knowledge, and tasks that require a lot of searching.

[0120] 2. Instruction Quality Assessment

[0121] 1. According to the knowledge base definition, mark the instruction structure type, constraint type and capability type at the same time: the structure type is marked with [] at the beginning, and the constraint type and capability mark are marked with <> at the end: If there is an unexpected constraint cd in the instruction, add a mark to the main task or the corresponding constraint mark: <Make it difficult for me! >; if there is no unexpected constraint of cd type, simply mark it as <Excellent capability>;

[0122] 2. Rate the instructions

[0123] A. Instruction evaluability score: 100 points. First, pay attention to whether there are CD-type capability-related constraints in the instruction. If there are, you can only score below 30 points. The lower the score, the more difficult it is to implement. Here are some scoring standards for reference. You must refer to these benchmarks to score! You need to comprehensively score whether you can implement it and whether it is easy for humans to evaluate. The score is higher than 70 points if both are easy.

[0124] For example, novel, poetry and essay writing are all 0-point tasks, and it is difficult for ordinary people to evaluate the quality of your essays and poems; essay rewriting and polishing, or instructions involving a large number of professional terms and a large number of professional field-related requirements, such as writing essays involving knowledge of different disciplines, these tasks require human experts to complete, and naturally they also need human experts to evaluate! Therefore, the difficulty of such tasks must be less than 30 points! The general level of translation and proofreading is a 50-point system, which can generally be evaluated, and data formatting is 80 points, which is relatively evaluable; you can briefly evaluate the executable degree of the main task and constraints and give a score;

[0125] B. Instruction complexity score: 100 points. The complexity of the instruction only focuses on the number of expected type constraints marked in the first step. It has nothing to do with the number of other non-expected type constraints, the difficulty of the task itself, and whether creative thinking is required. If and only if there are enough of the four types of constraints efgh in the expected constraints, the higher the instruction complexity score, the total score is 100 points; Note: This dimension is different from evaluability, and does not need to consider the difficulty of the task. The weight of (logical judgment) is heavier than (specific personality); at least 3 (specific personality) constraints must be met to reach more than 50 points; if there are more than 2 (logical judgment) constraints, you can reach 70 points. Specifically:

[0126] a. If the constraints are mainly (ideal abstraction), (powerlessness), (default constraint) and (content writing), the complexity of the instruction is relatively low, basically below 40 points;

[0127] b. If the (content writing) constraint is dominant, no matter how many (content writing) constraints there are, as long as there are not enough (personality specific) constraints, the instruction complexity score will be very low, generally less than 20 points; note that instruction complexity is only related to (personality specific) and (logical judgment);

[0128] c. Finally, count the number of constraints of each type and then score them according to the above criteria;

[0129] 3. Be careful not to output the reply to the command. Just mark the command, briefly analyze the reason for the score, and give the final score.

[0130] 4. The format for outputting the total score separately is: "Evaluability score: 80" "Complexity score: 65"

[0131] 5. Final score: 100 points, use the average of the evaluability score and the complexity score, where the evaluability score weight is 0.4, the complexity score weight is 0.6, retain one decimal place, the calculation formula is: total instruction score = "evaluability score" 0.4 + "complexity score" 0.6, for example, the total instruction score of the above case is: 80*0.4+65*0.6=71, the output total score format is: total score of instructions to be evaluated: 71.0.

[0132] The following are the instructions to be evaluated:

[0133] [Instructions to be evaluated in step S2 example]

[0134] ”

[0135] The output of the large language model is:

[0136] “

[0137] [Main task]: Imitate and construct some conversations between customers and sales, which can be used in the sales scenarios of smart bracelet products. <Excellent ability>

[0138] [Constraint 1]: The factors that customers should pay attention to include product features, price inquiries, function demonstrations, applicable groups, warranty services, and comparisons with other brands. <Content writing>

[0139] [Constraint 2]: Intention to purchase group: at least 10 rounds of communication between customers and sales, after fully understanding the product, customers express their intention to purchase. <Personality specific>

[0140] [Constraint 3]: No intention to buy group: At least 7 rounds of dialogue, the mismatch is reflected by the product not meeting the customer's secondary needs or features. <Personality specific>

[0141] [Constraint 4]: Note that one question and one answer is one round, marked with a Turn Number followed by a sequence number. If the customer expresses a purchase intention, it is indicated in capital letters. <Personalized format><Conditional judgment><Excellent ability>

[0142] 2. Rate the instructions:

[0143] A. Instruction evaluability score:

[0144] The main task of this instruction is to imitate and construct dialogues, which is a task within the scope of basic capabilities. All constraints are also clear and specific requirements, including no less than 10 rounds and 7 rounds of dialogues and format requirements such as TurnNumber sequence marking method, which can also be easily evaluated by humans. There is no content that requires human experts to complete or evaluate, so it is highly evaluable.

[0145] Score: 80 points.

[0146] B. Instruction complexity score:

[0147] This instruction has a total of 4 constraints, including 2 individual specific constraints (constraint 2 and constraint 3), 1 conditional constraint (constraint 4), and 1 format restriction constraint related to the format (constraint 4). These constraints are very clear and specific, detailing the specific requirements of the reply, and fall within the category of expected type constraints. Therefore, overall, the complexity of the instruction is relatively high.

[0148] Score: 70 points.

[0149] 4. The format for outputting the total score separately is:

[0150] 『Evaluability score: 80』

[0151] Complexity score: 70

[0152] 5. Final score:

[0153] Total score of instruction = "evaluability score" 0.4 + "complexity score" 0.6 = 80*0.4 + 70*0.6 = 74.0

[0154] Total instruction score: 74.0

[0155] ”

[0156] S4. If the total score is less than a preset threshold, iteratively optimize the instruction to be evaluated until the total score is greater than the preset threshold, and output the instruction to be evaluated as a high-quality complex instruction.

[0157] Specifically, the total score of the instruction to be evaluated obtained in step S3 is compared with the preset threshold. If the total score is less than the preset threshold, it means that the current instruction to be evaluated has not yet reached the standard of a high-quality complex instruction and needs to be optimized. The optimization may be to further modify the constraints or adjust the instruction template, etc. The optimized instruction to be evaluated continues to calculate its total score until the total score is greater than the preset threshold, indicating that the instruction is sufficient to meet the standard of a high-quality complex instruction and is output.

[0158] Compared with the prior art, the method for constructing high-quality complex instructions provided in this embodiment designs a high-quality instruction template including an instruction structure, a constraint type, and a capability type, constructs a reference instruction based on the high-quality instruction template, and then uses a large language model to adjust the reference instruction based on the high-quality instruction template and the reference instruction to generate instructions to be evaluated, scores and evaluates the instructions to be evaluated to optimize the instructions, and uses instructions that meet the evaluation criteria as high-quality complex instructions, thereby improving the construction efficiency and quality of high-quality complex instructions.

[0159] Example 2

[0160] This embodiment provides a high-quality complex instruction construction device, such as Figure 2 As shown, including:

[0161] A reference instruction construction module, used to set a high-quality instruction template and construct a reference instruction according to the high-quality instruction template; the high-quality instruction template includes an instruction structure, a constraint type and a capability type;

[0162] An instruction adjustment module, used for inputting the high-quality instruction template and the reference instruction into a large language model for adjustment to generate an instruction to be evaluated;

[0163] An instruction evaluation module, used for inputting the high-quality instruction template, evaluation criteria and the instruction to be evaluated into a large language model for evaluation, and obtaining a total score of the instruction to be evaluated;

[0164] The high-quality instruction output module is used to iteratively optimize the instruction to be evaluated if the total score is less than a preset threshold, until the total score is greater than the preset threshold, and output the instruction to be evaluated as a high-quality complex instruction.

[0165] Example 3

[0166] This embodiment provides an electronic device, such as Figure 3 As shown, it includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, it implements the construction method of high-quality complex instructions as described in any of the above embodiments.

[0167] Example 4

[0168] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for constructing high-quality complex instructions as described in any of the above embodiments is implemented.

[0169] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0171] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0172] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing high-quality complex instructions, characterized in that: include: Setting a high-quality instruction template, and constructing a reference instruction according to the high-quality instruction template; the high-quality instruction template includes an instruction structure, a constraint type, and a capability type; Inputting the high-quality instruction template and the reference instruction into a large language model for adjustment to generate an instruction to be evaluated; Inputting the high-quality instruction template, evaluation criteria and the instruction to be evaluated into a large language model, and calculating the total score of the instruction to be evaluated; If the total score is less than a preset threshold, the instruction to be evaluated is iteratively optimized until the total score is greater than the preset threshold, and the instruction to be evaluated is output as a high-quality complex instruction.

2. The method for constructing high-quality complex instructions according to claim 1, characterized in that: The instruction structure includes main tasks, task knowledge, constraints, text materials, demonstration cases and output formats.

3. The method for constructing high-quality complex instructions according to claim 2, characterized in that: The constraint types include expected constraints and unexpected constraints. The expected constraints include personality specificity, logical judgment, execution steps and format limitations. The unexpected constraints include content themes, default constraints, powerlessness and ideal abstraction.

4. The method for constructing high-quality complex instructions according to claim 1, characterized in that: The ability types include basic ability, general ability, marginal ability and beyond ability.

5. The method for constructing high-quality complex instructions according to claim 2, characterized in that: The step of inputting the high-quality instruction template and the reference instruction into a large language model for adjustment to generate an instruction to be evaluated includes: The large language model identifies and marks the constraint type of the constraint condition in the reference instruction according to the high-quality instruction template; Modify or add constraint conditions of personality-specific, logical judgment and format-limited types according to the task knowledge; The instruction after the constraint modification or addition is output as the instruction to be evaluated.

6. The method for constructing high-quality complex instructions according to claim 3, characterized in that: The step of inputting the high-quality instruction template, the evaluation criteria, and the instruction to be evaluated into a large language model, and calculating the total score of the instruction to be evaluated includes: The large language model extracts the main task and constraint conditions of the instruction to be evaluated according to the high-quality instruction template, and identifies the capability type of the main task, and the constraint type and capability type of the constraint conditions; Scoring the evaluability of the instruction to be evaluated according to the main task, the capability type of the main task and the constraint condition, and the evaluation criteria to obtain an evaluability score; Scoring the complexity of the instruction to be evaluated according to the constraint type of the constraint condition and the evaluation criteria to obtain a complexity score; A weighted average of the evaluability score and the complexity score is calculated as the total score.

7. The method for constructing high-quality complex instructions according to claim 6, characterized in that: The evaluation criteria include: The evaluability evaluation criteria include: out of 100 points, if the ability type of the main task and constraint condition is an instruction that exceeds the ability, the evaluability score is 0, if the ability type of the main task and constraint condition is an instruction that is barely capable, the evaluability score is 0-30 points, and if the ability type of the main task and constraint condition is an instruction that is of average ability, the evaluability score is 50-80 points; The complexity assessment criteria include: a 100-point system, for instructions where the constraint type of the constraint condition is that the proportion of unexpected constraints is higher than the expected constraints, the complexity score is 0-40 points; for instructions where the constraint type of the constraint condition is that the proportion of expected constraints is higher than the unexpected constraints, the complexity score is 15-80 points based on the number of individual constraints and logical judgments.

8. A high-quality complex instruction construction device, characterized in that: The device comprises: A reference instruction construction module, used to set a high-quality instruction template and construct a reference instruction according to the high-quality instruction template; the high-quality instruction template includes an instruction structure, a constraint type and a capability type; An instruction adjustment module, used for inputting the high-quality instruction template and the reference instruction into a large language model for adjustment to generate an instruction to be evaluated; An instruction evaluation module, used for inputting the high-quality instruction template, evaluation criteria and the instruction to be evaluated into a large language model for evaluation to obtain a total score of the instruction to be evaluated; The high-quality instruction output module is used to iteratively optimize the instruction to be evaluated if the total score is less than a preset threshold, until the total score is greater than the preset threshold, and output the instruction to be evaluated as a high-quality complex instruction.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for constructing high-quality complex instructions as described in any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method for constructing high-quality complex instructions as described in any one of claims 1-7 is implemented.

Images