A prompt template generation method, device, electronic device and storage medium

Abstract

The present invention discloses a prompt template generation method, device, electronic device and storage medium. The method comprises: classifying samples in an instruction sample library to obtain a positive sample data set and a negative sample data set; extracting triples from the query question, background knowledge and thinking chain included in each positive sample in the positive sample data set, and constructing a knowledge network based on the extracted triples, wherein the triples include task event entities, knowledge entities and entity relationships; clustering task patterns on the knowledge network to construct a task pattern library containing the corresponding relationship between task event types and task patterns; generating a prompt template corresponding to each positive sample, clustering each prompt template according to the corresponding relationship to obtain a clustered prompt template, and constructing a prompt template library, so that the answer model matches the corresponding prompt template in the prompt template library based on the query request. The answer model standardizes the form of using prompts in the query process, which facilitates the continuous optimization and management of prompts.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for generating prompt templates. Background Technology

[0002] With the rapid development of generative artificial intelligence, represented by Large Language Models (LLM), large models are widely used in many fields such as information retrieval, intelligent customer service, copywriting generation, and design assistance.

[0003] Large-scale models trained on massive general-purpose corpora can leverage "external knowledge" through contextual learning. For example, Retrieval-Augmented Generation (RAG) can use the current input content as keywords to retrieve relevant information from external knowledge bases (such as the internet, databases, and knowledge graphs), and then add this information to the prompt to improve model performance. However, this method is a "temporary" use of knowledge and is difficult to continuously optimize and manage. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for generating prompt templates, which can standardize the way prompts are used in the query process by the answer model, and facilitate continuous optimization and management of prompts.

[0005] In a first aspect, embodiments of the present invention provide a method for generating a prompt template, including:

[0006] The samples in the instruction sample library are classified to obtain a positive sample dataset and a negative sample dataset. Each sample in the instruction sample library consists of a query question, background knowledge, thought chain, and answer.

[0007] Triples are extracted from the query question, background knowledge, and thought chain included in each positive sample in the positive sample dataset, and a knowledge network is constructed based on the extracted triples. The triples include task event entities, knowledge entities, and entity relations.

[0008] The knowledge network is clustered by task patterns to construct a task pattern library containing the correspondence between task event types and task patterns;

[0009] Generate a prompt template corresponding to each positive sample, and cluster the prompt templates according to the correspondence to obtain clustered prompt templates. Construct a prompt template library so that the answer model can match the corresponding prompt template in the prompt template library based on the query request.

[0010] Secondly, embodiments of the present invention provide a prompt template generation device, comprising:

[0011] The classification module is used to classify the samples in the instruction sample library to obtain positive sample datasets and negative sample datasets. Each sample in the instruction sample library consists of a query question, background knowledge, thought chain, and answer.

[0012] The knowledge network construction module is used to extract triples from the query question, background knowledge and thinking chain included in each positive sample in the positive sample dataset, and to construct a knowledge network based on the extracted triples. The triples include task event entities, knowledge entities and entity relations.

[0013] The task pattern library construction module is used to perform task pattern clustering on the knowledge network and construct a task pattern library containing the correspondence between task event types and task patterns.

[0014] The prompt template library construction module is used to generate a prompt template corresponding to each positive sample, and to cluster each prompt template according to the correspondence to obtain clustered prompt templates, thereby constructing a prompt template library so that the answer model can match the corresponding prompt template in the prompt template library based on the query request.

[0015] Thirdly, embodiments of the present invention provide an electronic device, including:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0020] The technical solution of this invention classifies samples in the instruction sample library to obtain a positive sample dataset. Each sample consists of a query question, background knowledge, a thought process, and an answer. Triples (task event entity, knowledge entity, and entity relation) are extracted from the positive samples in the positive sample dataset to construct a knowledge network. Task pattern clustering is performed on the knowledge network to construct a task pattern library containing the correspondence between task event types and task patterns. A prompt template library is constructed, containing prompt templates corresponding to each positive sample, as well as clustered prompt templates obtained by clustering each prompt template according to the correspondence. This solution, through the construction of the prompt template library, enables the answer model to match the corresponding prompt template in the prompt template library based on the query request, standardizing the way the answer model utilizes prompts during the query process and facilitating continuous optimization and management of prompts.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a prompt template generation method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a prompt template generation method provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram illustrating the construction of a large-scale event knowledge system according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram illustrating the updating and prompting optimization of a large-scale event knowledge system according to Embodiment 3 of the present invention;

[0027] Figure 5 This is a schematic diagram of a prompt template generation device according to Embodiment 4 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] In this invention, within the context of large model prompts, events can be understood as specific queries or requests submitted by users, and the responses generated by the large model. These events encompass the user's intent, the contextual information of the query, the model's understanding of the query, and the final answer. Each event can consist of several key elements, including trigger words (which may be keywords), event type (which may be a query-specific pattern, intent classification, etc.), and arguments (i.e., the participants related to task execution, constituting the knowledge set required for task execution). Furthermore, events may have various relationships, such as temporal, causal, and conditional relationships. These relationships are manifested in the large model prompts as logical chains between different prompts; the output of one prompt may become the input of another, or one prompt may trigger the generation of another related prompt. This event definition and relationship framework helps the large model more comprehensively capture and reason about the complexity of user interactions.

[0032] This invention proposes a method for generating prompt templates, which can construct a knowledge network, a task pattern library, and a prompt template library based on the above event definition and relationship framework, so as to realize the generation and management of prompt templates.

[0033] Example 1

[0034] Figure 1This is a flowchart of a prompt template generation method according to Embodiment 1 of the present invention. This embodiment is applicable to the generation of prompt templates. The method can be executed by a prompt template generation device, which can be implemented in software and / or hardware and integrated into an electronic device. Furthermore, the electronic device includes, but is not limited to, computers, laptops, etc.

[0035] like Figure 1 As shown, the method includes:

[0036] S110. Classify the samples in the instruction sample library to obtain a positive sample dataset and a negative sample dataset. Each sample in the instruction sample library consists of a query question, background knowledge, thought chain, and answer.

[0037] The instruction sample library can be a library containing multiple samples. The samples can be determined within the context of prompts from a larger model, and each sample can consist of a query question, background knowledge, thought process, and answer.

[0038] The query question can be a question or instruction posed by a user to perform a specific query, also known as a query or instruction, and can be in the form of natural language. Background knowledge can be the knowledge needed by the large model to respond to the query question, i.e., additional information that helps understand the query question, such as external knowledge related to the query question, also known as Background Knowledge (abbreviated as bg_kg). The Chain of Thought (CoT) can be an ordered reasoning process that connects background knowledge and the query question, demonstrating how to gradually deduce the answer from existing information. The answer can be the large model's response to the query question, also known as an answer. Therefore, a sample can be denoted as S = {Query|Instruction, bg_kg, CoT, Answer}.

[0039] In this step, a large model can be used to reason about samples in the instruction sample library to determine the perplexity of each sample. Samples with low perplexity are selected as positive samples to obtain a positive sample dataset, and samples with high perplexity are selected as negative samples to obtain a negative sample dataset. The perplexity of a sample reflects the level of uncertainty of the large model when processing query questions. If the large model can successfully understand the query question, effectively utilize background knowledge, reasonably construct a thought process chain, and accurately provide an answer, then the perplexity of this sample should be very low, indicating that the large model has high fluency and low error rate in understanding and generating this content.

[0040] S120. Extract triples from the query question, background knowledge, and thought chain included in each positive sample in the positive sample dataset, and construct a knowledge network based on the extracted triples. The triples include task event entities, knowledge entities, and entity relations.

[0041] The generative task of the large model, Task = {Prompt->LLM->Answer}, can include three parts: prompt triggering, model generation, and result output (i.e., answer output). From the perspective of event analysis, the prompt can be considered to contain the task's trigger words, the event type, and the event's arguments. The task's trigger words and event type can be obtained from the query questions in the above samples, while the event arguments can be understood as the background knowledge and thought chain in the above samples. Based on the above analysis, this step can extract triples from the query questions, background knowledge, and thought chains included in the samples to construct the knowledge network.

[0042] A triple consists of a task event entity, a knowledge entity, and an entity relation. The task event entity can be the entity corresponding to a task event, which can be understood as a task to be executed based on the user's intent and needs, also known as a TaskEvent. The knowledge entity can be the entity corresponding to the relevant knowledge involved in the execution of the task event, also known as Knowledge. The entity relation can be the relationship between the task event entity and the knowledge entity, also known as a Relation, and there are no restrictions on the entity relation. That is, a triple can be represented as [TaskEvent]-[Relation]-[Knowledge].

[0043] For each positive sample in the positive sample dataset, triples are extracted from the query questions included in that positive sample. Specifically, Query or Instruction are typical trigger words for task events. Natural language processing techniques can be used to analyze the text of Query or Instruction to identify the user's intent and needs. Using intent recognition and slot filling, key task events and related knowledge entities are extracted by analyzing the structure and content of sentences. By associating this information, triples of task event entities, knowledge entities, and entity relationships are formed.

[0044] Triples are extracted from the background knowledge included in the positive sample. Specifically, Background Knowledge (bg_kg) is a typical knowledge type argument, and knowledge entities can be extracted using argument detection methods, such as rule-based methods, graph neural network (GNN) based methods, and large model-based methods. Then, relation extraction methods are used to extract the entity relations between the obtained knowledge entities and the previously extracted task event entities, and construct new triples of task event entities, knowledge entities, and entity relations.

[0045] Extract triples from the thought chain included in the positive sample. Specifically, the thought chain can be regarded as a subtask or sub-event of generating task events. Therefore, for each step of the thought chain, the methods for extracting triples from the query question and from background knowledge described above can be used to extract and output the triples [subTaskEvent]-[Relation]-[Knowledge] corresponding to the sub-event. It should be noted that there is a temporal relationship between sub-events, which also belong to the temporal trigger word type; the relationship between [TaskEvent] and [subTaskEvent] is a subordinate relationship.

[0046] After extracting triples from the query question, background knowledge, and thought chain included in each positive sample using the above method, the extracted triples need to undergo event knowledge fusion processing to remove or integrate triples with similar expressions, that is, to align entities in different triples.

[0047] For TaskEvent, Relation, and Knowledge in the triples, similarity matching and standardization are required separately to determine whether different TaskEvent entities point to the same task event. Specifically, this can be achieved by comparing name similarity based on string similarity matching; it can also be based on contextual semantic similarity matching to analyze the semantic environment of TaskEvent, Relation, and Knowledge to determine if they are the same task event; or it can utilize existing entity mapping tables or external knowledge bases. When processing Relation, if different triples use different relation terms, they need to be normalized to the same relation. The normalization process can be implemented by establishing a thesaurus or a predefined relation mapping table to ensure consistency in relation expression within the knowledge network. Furthermore, the fusion of the Knowledge component can be handled using similar methods, which will not be elaborated upon here.

[0048] After performing event knowledge fusion processing on the extracted triples, a knowledge network can be constructed. Specifically, an empty graph structure is initialized using a graph database. For each TaskEvent and Knowledge in the triples, they can be added as nodes to the graph structure. For example, the node creation command of the graph database can be used to create task event entity nodes corresponding to TaskEvent and knowledge entity nodes corresponding to Knowledge. Each node can have a unique identifier. The Relation in the triples is added as a directed edge to the graph structure, connecting the corresponding task event entity node and knowledge entity node. For example, edges can be added in the graph structure by specifying the connection relationship between task event entity nodes and knowledge entity nodes. The graph structure obtained through the above processing is the knowledge network, or knowledge graph, in this embodiment of the invention.

[0049] S130. Perform task pattern clustering on the knowledge network to construct a task pattern library containing the correspondence between task event types and task patterns.

[0050] In this step, task pattern clustering can be performed based on the knowledge network constructed above. Task pattern clustering can be understood as the process of grouping similar task event patterns into one category, that is, grouping similar task event patterns together to represent the same task event type. Here, task event type can be the type of task event, and task pattern can be the pattern corresponding to the task event.

[0051] In one embodiment, task pattern clustering of the knowledge network includes: clustering the knowledge network according to task patterns using a method of extracting common subgraphs or a graph-based clustering method.

[0052] This involves extracting common subgraphs from the knowledge network using frequent subgraph mining and graph neural network subgraph matching techniques. Common subgraphs are recurring patterns in the knowledge network, defined as task patterns (or TaskSchemas) in this step. Alternatively, graph-based clustering methods can be used to aggregate scattered task events and knowledge from the knowledge network into a single task pattern, forming a complete task pattern. After task pattern clustering, a one-to-one correspondence is established between task event types and task patterns, denoted as [TaskEvent]-[TaskSchema].

[0053] A task pattern library is built by combining different task patterns. This library can contain a one-to-one correspondence between multiple different task event types and task patterns. The task pattern library can support query services for TaskEvent or TaskSchema, which are existing methods and will not be elaborated upon here.

[0054] S140. Generate a prompt template corresponding to each positive sample, and cluster each prompt template according to the correspondence to obtain a clustered prompt template, and construct a prompt template library so that the answer model can match the corresponding prompt template in the prompt template library based on the query request.

[0055] The prompt template corresponding to a positive sample can be understood as a structured prompt form extracted from the positive sample. The prompt template standardizes the structure of the prompt corresponding to the positive sample and the specific content that the structure should contain. Clustered prompt templates can be obtained by clustering prompt templates based on the correspondence between task event types and task patterns.

[0056] In this step, a hint engineering suitable for large models can be constructed. This hint engineering indicates the generation of corresponding hint templates based on positive samples and specifies the structure and content of the hint templates. The large model is then used to generate a hint template for each positive sample based on the constructed hint engineering.

[0057] When generating a prompt template for each positive sample, the prompt templates of positive samples belonging to the same task mode can be clustered to obtain clustered prompt templates according to the correspondence between task event types and task modes indicated in the task mode library.

[0058] A prompt template library is constructed based on prompt templates and clustered prompt templates, meaning both prompt templates and clustered prompt templates are stored in the prompt template library. The subsequent answer model can match the prompt template corresponding to the query question in the prompt template library, or match the clustered prompt template corresponding to the query question, based on the matched template. A structured prompt corresponding to the query request is generated based on the generated prompt, and the answer to the query question is output based on the generated prompt. The answer model can be any model used to generate the answer to the query question; it can be a large model, and this is not limited here.

[0059] The technical solution of this invention classifies samples in the instruction sample library to obtain a positive sample dataset. Each sample consists of a query question, background knowledge, a thought process, and an answer. Triples (task event entity, knowledge entity, and entity relation) are extracted from the positive samples in the positive sample dataset to construct a knowledge network. Task pattern clustering is performed on the knowledge network to construct a task pattern library containing the correspondence between task event types and task patterns. A prompt template library is constructed, containing prompt templates corresponding to each positive sample, as well as clustered prompt templates obtained by clustering each prompt template according to the correspondence. This solution, through the construction of the prompt template library, enables the answer model to match the corresponding prompt template in the prompt template library based on the query request, standardizing the way the answer model utilizes prompts during the query process and facilitating continuous optimization and management of prompts.

[0060] Example 2

[0061] Figure 2 This is a flowchart of a prompt template generation method according to Embodiment 2 of the present invention. This embodiment is based on Embodiment 1 above, further refining the classification of samples in the instruction sample library to obtain positive sample datasets and negative sample datasets; and generating prompt templates corresponding to each positive sample, and clustering each prompt template according to the correspondence to obtain clustered prompt templates, thereby constructing a further refinement of the prompt template library.

[0062] like Figure 2 As shown, the method includes:

[0063] S111. In the instruction sample library, samples with perplexity lower than the perplexity threshold are identified as positive samples, and those with perplexity higher than the perplexity threshold are identified as negative samples.

[0064] In this step, a large model can be used to infer the perplexity of samples in the instruction sample library, determining the perplexity of each sample. Samples with perplexity below the perplexity threshold are identified as positive samples, and samples with perplexity at or above the perplexity threshold are identified as negative samples. The perplexity threshold can be pre-set based on experience and is not limited here; for example, a perplexity threshold of 3 is acceptable. The choice of the perplexity threshold can be adjusted based on experimental results to ensure classification accuracy and dataset validity.

[0065] S112. The set of positive samples is taken as the positive sample dataset, and the set of negative samples is taken as the negative sample dataset.

[0066] S120. Extract triples from the query question, background knowledge, and thought chain included in each positive sample in the positive sample dataset, and construct a knowledge network based on the extracted triples. The triples include task event entities, knowledge entities, and entity relations.

[0067] S130. Perform task pattern clustering on the knowledge network to construct a task pattern library containing the correspondence between task event types and task patterns.

[0068] S141. Generate a model using a prompt template. Based on the prompt engineering, take each positive sample as input and output a prompt template corresponding to each positive sample. The prompt template includes the query question of the positive sample, parameters related to the query, and context information required for the query.

[0069] The prompt template generation model can be a model used to generate prompt templates corresponding to positive samples, or it can be a large model.

[0070] In this step, a prompt project suitable for the prompt template generation model can be constructed. This prompt project can be as follows:

[0071] "Please act as a prompt template extraction engineer to generate structured prompt templates based on the input content;"

[0072] Input: Positive samples;

[0073] The output prompt template must contain the following elements:

[0074] Main query: The main query requirement corresponding to the input content, i.e., the query problem of the positive sample;

[0075] Related parameters: Parameters related to the query;

[0076] Context information: The context information required for the query.

[0077] Example: If the input is "Query sales data for product X", the output prompt template should be:

[0078] {

[0079] "query": "Query sales data for product X";

[0080] "parameters": {"time range":"past year","region":"nationwide"};

[0081] "context": "Sales performance analysis of product X".

[0082] }”

[0083] By constructing the aforementioned prompting project, the prompt template generation model can take each positive sample as input and output a prompt template corresponding to each positive sample. The output format of the prompt template is an object {query, parameters, context}. This format ensures that the prompts generated by the prompt template have a complete and easily processed structure when passed to a large model, so as to be efficiently called and executed in subsequent inference tasks.

[0084] S142. Determine the task mode corresponding to each positive sample according to the correspondence, and cluster the prompt templates of positive samples belonging to the same task mode to obtain cluster prompt templates.

[0085] In this step, the task pattern corresponding to each positive sample can be determined by following the correspondence between task event types and task patterns indicated in the task pattern library and combining the task events corresponding to each positive sample. Clustering of prompt templates is then performed on the prompt templates of positive samples belonging to the same task pattern to obtain clustered prompt templates. The output format of the clustered prompt templates is consistent with the output format of the prompt templates in S141.

[0086] S143. Construct a prompt template library based on the prompt template and the clustering prompt template, so that the answer model can match the corresponding prompt template in the prompt template library based on the query request.

[0087] In this step, you can design and create the structure of the prompt template library, so that the database table in the prompt template library contains the following fields: template_id, template_name, query_structure, parameters_structure, and context_structure. Insert each prompt template and each cluster prompt template into the database table according to the above fields. For example, each prompt template or cluster prompt template can be a row in the database table, and each row includes the above multiple fields.

[0088] Wherein, template_id can be the identifier corresponding to the template, and each hint template or cluster hint template has a unique identifier; template_name can be the name corresponding to the template; query_structure can be the query structure, used to store the main query in the hint template or cluster hint template; parameters_structure can be the parameter structure, used to store the relevant parameters in the hint template or cluster hint template; context_structure can be the context structure, used to store the context information in the hint template or cluster hint template.

[0089] During application, the answer model can match the corresponding prompt template in the prompt template library based on the query question corresponding to the query request. If the match is successful, a structured prompt corresponding to the query request is generated based on the prompt template. If the match fails, the model can match the corresponding clustered prompt template in the prompt template library based on the query question, and generate a structured prompt corresponding to the query request based on the clustered prompt template. The answer to the query question is then output based on the generated prompt.

[0090] In one embodiment, other methods can also be used to manage the prompt template library, such as graph databases, knowledge graphs, etc., which are not limited here.

[0091] The technical solution of this invention determines the prompt template corresponding to each positive sample through a prompt template generation model, clusters the prompt templates corresponding to positive samples belonging to the same task mode to obtain clustered prompt templates, and constructs a prompt template library based on the prompt templates and clustered prompt templates. The prompt templates and clustered prompt templates standardize the structured prompt format, and the generation of these prompt formats ensures that the reasoning ability of the response model can be fully utilized, thereby improving the accuracy, relevance, and efficiency of reasoning and response.

[0092] In one embodiment, the method further includes:

[0093] For any negative sample in the negative sample dataset, the query question, background knowledge, and thought process included in the negative sample are taken as suggestions to be optimized, the answers included in the negative sample are taken as actual answers, and the expected answer of the negative sample is determined.

[0094] The attribution analysis agent takes the prompt to be optimized, the actual answer, and the expected answer as input, and outputs the error type and error cause corresponding to the negative sample. The error type includes query question error, background knowledge error, and thought chain error.

[0095] For any negative sample in the negative sample dataset, separate the prompt to be optimized (corresponding to Query|Instruction, bg_kg, CoT in the negative sample) and the actual answer (corresponding to Answer in the negative sample), and determine the expected answer of the negative sample. Here, the prompt to be optimized can be the prompt to be optimized corresponding to the negative sample; the actual answer can be the actual answer in the negative sample to the prompt to be optimized; and the expected answer can be the expected answer corresponding to the negative sample.

[0096] By constructing an attribution analysis agent, attribution analysis is performed on the inference errors of a large model. The attribution analysis agent can be an agent that performs attribution analysis on the difference between the expected and actual answers corresponding to negative samples. The attribution analysis agent can use an open-source large model or fine-tuning the large model using instructions, and the method of fine-tuning is not limited.

[0097] The prompt format for attribution analysis agents can be as follows:

[0098] "Please analyze the difference between the expected and actual answers corresponding to the negative samples, identify the possible root causes, classification error types, and error reasons."

[0099] Input format:

[0100] Optimization suggestions in negative samples;

[0101] The actual answers in the negative samples;

[0102] The expected response to a negative sample.

[0103] Output format:

[0104] {

[0105] "error_type":"string",

[0106] "root_cause":"string",

[0107] }

[0108] error_type indicates the error type corresponding to the negative sample, and root_cause indicates the error cause corresponding to the negative sample.

[0109] Your execution steps are as follows:

[0110] Analyze the input: Compare the differences between the provided suggestions for improvement, the actual answers, and the expected answers;

[0111] Identify error types: Determine the nature of the discrepancies and classify the errors (search question error, background knowledge error, and thought chain error);

[0112] Determine the cause of the error: Explain why this difference occurred based on the identified type of error.

[0113] The error type can be the type of error corresponding to negative samples, including query error, background knowledge error, and thought chain error. Query error can be caused by problems with the expression or structure of the query, denoted as Query Error, which the agent determines by analyzing the explicitness and correctness of the query. Background knowledge error can be an error that occurs when using a background knowledge graph, denoted as bg_kg Error, which the agent determines by verifying whether the background knowledge used in the reasoning is correct. Thought chain error can be caused by a break or inconsistency in the logical chain during the reasoning process, denoted as CoTError, which the agent determines by checking the coherence of the reasoning steps and the internal logic.

[0114] In one embodiment, the method further includes:

[0115] The intelligent agent is optimized by providing prompts, based on the error type and the cause of the error, and the optimized prompts are output.

[0116] By constructing a prompt optimization agent, the prompts to be optimized are improved based on the error type and cause corresponding to negative samples, resulting in optimized prompts. The optimized prompts are essentially the improved prompts from the original prompts. The prompt optimization agent can be any agent that optimizes the original prompts. It can use a large open-source model or a large model fine-tuned by instructions; the method of instruction fine-tuning is not limited.

[0117] The prompt format for optimizing the agent can be as follows:

[0118] "Please help optimize a flawed suggestion so that it can generate correct inference results."

[0119] The specific content of this prompt is: "Prompt for optimization."

[0120] The known errors are: the error type (error_type) and the error cause (root_cause) corresponding to the negative samples;

[0121] Please optimize the prompts for the known errors so that they can accurately describe the task and generate accurate results. Output the optimized prompts.

[0122] In one embodiment, the method further includes:

[0123] The optimized prompt is input into the answer model, and the corresponding optimized answer is output.

[0124] If the optimized answer is incorrect, then continue to perform attribution analysis and prompt optimization on the optimized prompt;

[0125] If the optimized answer is correct, the optimized prompt and the optimized answer are used as new positive samples, and the knowledge network, the task pattern library and the prompt template library are updated based on the new positive samples.

[0126] The optimized prompt is input into the answer model, the optimized answer is output and the corresponding perplexity is calculated; if the optimized answer is incorrect, that is, the corresponding perplexity is not lower than the perplexity threshold, then the attribution analysis and prompt optimization are continued for the optimized prompt until the answer generated based on the further optimized prompt is correct; if the optimized answer is correct, that is, the corresponding perplexity is lower than the perplexity threshold, it means that the optimization is successful, then the optimized prompt and optimized answer are used as new positive samples, and operations S120-S143 are performed based on the new positive samples to update the knowledge network, task pattern library and prompt template library.

[0127] Example 3

[0128] This embodiment is an exemplary description of the above embodiment. The content of this embodiment can be described in two stages: stage one is the construction of the large-scale event knowledge system, and stage two is the updating and optimization of the large-scale event knowledge system.

[0129] Figure 3 This is a schematic diagram illustrating the construction of a large-scale event knowledge system according to Embodiment 3 of the present invention, specifically the implementation of Phase 1. The large-scale event knowledge system may include a knowledge network, a task pattern library, and a prompt template library. Figure 3 As shown, it may include the following steps:

[0130] 1. Classify the samples in the instruction sample library based on perplexity. Samples with perplexity below the perplexity threshold are identified as positive samples, and those above the perplexity threshold are identified as negative samples.

[0131] 2. Extract triples from the query question, background knowledge, and thought chain included in each positive sample. Triples include task event entities, knowledge entities, and entity relations. That is, triples can be represented as [TaskEvent]-[Relation]-[Knowledge]. Perform event knowledge fusion processing on the extracted triples to remove or integrate triples with similar expressions. Construct a knowledge network based on the triples after event knowledge fusion processing.

[0132] 3. Based on the constructed knowledge network, cluster task patterns, grouping similar task event patterns into the same group to represent the same task event type; construct a task pattern library, which contains a one-to-one correspondence between multiple different task event types and task patterns.

[0133] 4. Generate a prompt template for each positive sample using the prompt template generation model, and cluster the prompt templates of positive samples belonging to the same task pattern according to the correspondence between task event types and task patterns in the task pattern library to obtain clustered prompt templates; build a prompt template library based on the prompt templates and clustered prompt templates. Both prompt templates and clustered prompt templates can include the main query, related parameters, and context information.

[0134] against Figure 3 The negative samples shown can be processed in stage two. Figure 4 This is a schematic diagram illustrating the updating and prompting optimization of a large-scale event knowledge system according to Embodiment 3 of the present invention, specifically the implementation of Phase Two. For example... Figure 4 As shown, it may include the following steps:

[0135] 1. For any negative sample, the query question, background knowledge, and thought process included in the negative sample are used as suggestions for optimization. The suggestions for optimization are input into the basic large model (which can be understood as the answer model) to obtain the actual answer and determine the expected answer of the negative sample.

[0136] 2. Using the attribution analysis agent, the system takes the prompts to be optimized, the actual answers, and the expected answers as inputs, and outputs the error type and error cause corresponding to the negative samples.

[0137] 3. Optimize the agent by providing prompts, based on the error type and cause, and output the optimized prompts.

[0138] 4. Input the optimized prompts into the basic large model and output the corresponding optimized answers. If the optimized answers are incorrect, continue to perform attribution analysis and prompt optimization on the optimized prompts. If the optimized answers are correct, use the optimized prompts and optimized answers as new positive samples, and update the knowledge network, task pattern library, and prompt template library based on the new positive samples, that is, update the large model event knowledge system.

[0139] The first stage described above, through the construction of a large-scale model event knowledge system, provides support and assurance for the high-quality execution of large-scale model generation tasks, and can provide optimal knowledge support, optimal task modes, and prompt words for each type of task event.

[0140] Phase Two, described above, further enhances the functionality of the large model's event knowledge system through updates, enabling effective sample utilization. The introduction of attribution analysis and prompt optimization agents automatically evolves incorrect prompts into correct ones, improving the flexibility of large model applications. By employing a multi-agent-assisted knowledge evolution approach, the large model's knowledge acquisition and application capabilities in handling complex tasks are effectively optimized, significantly improving its performance in new and complex reasoning tasks.

[0141] Furthermore, in practical applications, the complete process of parsing, optimizing, and reasoning about user input prompts can be achieved by integrating the prompt optimization in Phase 2 with the large-scale model event knowledge system in Phase 1. Specifically, this includes: first, parsing the user input prompts to extract key information and structured data; second, matching prompt tasks based on the large-scale model event knowledge system, and optimizing the original prompts using a task pattern library and a prompt template library based on the matching relationships; and finally, using the optimized prompts for reasoning calculations to generate the final reasoning result.

[0142] The technical solution of this invention proposes a prompt optimization method based on a knowledge network constructed from events and knowledge. Through the collaboration of multiple agents, the content of the large-scale event knowledge system is automatically adjusted to adapt to the needs of different tasks. A prompt optimization strategy based on knowledge evolution is also proposed. This strategy can generate positive and negative samples based on task execution results to continuously optimize the large-scale event knowledge system, ensuring its adaptability to new domains and complex tasks. Using the knowledge network as a medium, prompt evolution for different tasks is completed through several historical interactions. The optimal "solution" set for completing the task is actively managed, and the final result is standardized and improved based on "historical successful experience."

[0143] Based on the above technical solution, several improvements can be made:

[0144] 1. The intelligent agent can access data in the large model event knowledge system through application programming interface (API) calls, and utilize the knowledge network, task pattern library and prompt template library of the large model event knowledge system to increase the effectiveness of optimization.

[0145] 2. By introducing an adaptive learning algorithm, the usage frequency and task adaptability of knowledge items in the large model event knowledge system are analyzed. Knowledge items with low usage frequency or unsuitable for the current task are automatically marked as candidate update items. During the actual task execution, knowledge items are dynamically updated based on real-time evaluation results, invalid knowledge is eliminated and new relevant knowledge is automatically introduced to improve the efficiency and practicality of the large model event knowledge system.

[0146] 3. A multi-layered knowledge interaction mechanism is added to better support complex reasoning tasks. The large-scale model event knowledge system is divided into three layers: a core knowledge layer, a related knowledge layer, and an auxiliary knowledge layer. Information is transferred and mutually verified between different layers through intelligent agents. During task execution, relevant knowledge is first invoked from the core knowledge layer. If uncertainties or high complexity are detected, information from the related and auxiliary knowledge layers is gradually introduced to achieve more accurate reasoning results and a higher task completion rate.

[0147] Example 4

[0148] Figure 5 This is a schematic diagram of a prompt template generation device according to Embodiment 4 of the present invention. This embodiment is applicable to situations involving the generation of prompt templates. Figure 5 As shown, the specific structure of the device includes:

[0149] The classification module 51 is used to classify the samples in the instruction sample library to obtain a positive sample dataset and a negative sample dataset. Each sample in the instruction sample library consists of a query question, background knowledge, thought chain, and answer.

[0150] The knowledge network construction module 52 is used to extract triples from the query question, background knowledge and thinking chain included in each positive sample in the positive sample dataset, and to construct a knowledge network based on the extracted triples. The triples include task event entities, knowledge entities and entity relations.

[0151] The task pattern library construction module 53 is used to perform task pattern clustering on the knowledge network and construct a task pattern library containing the correspondence between task event types and task patterns.

[0152] The prompt template library construction module 54 is used to generate a prompt template corresponding to each positive sample, and to cluster each prompt template according to the correspondence to obtain a clustered prompt template, thereby constructing a prompt template library so that the answer model can match the corresponding prompt template in the prompt template library based on the query request.

[0153] The prompt template generation device provided in this embodiment classifies samples in the instruction sample library using a classification module to obtain positive sample datasets and negative sample datasets. Each sample in the instruction sample library consists of a query question, background knowledge, a thought chain, and an answer. A knowledge network construction module extracts triples from the query question, background knowledge, and thought chain of each positive sample in the positive sample dataset and constructs a knowledge network based on the extracted triples. Each triple includes a task event entity, a knowledge entity, and entity relations. A task pattern library construction module performs task pattern clustering on the knowledge network to construct a task pattern library containing the correspondence between task event types and task patterns. A prompt template library construction module generates a prompt template corresponding to each positive sample and clusters each prompt template according to the correspondence to obtain clustered prompt templates, thus constructing a prompt template library. This allows the answer model to match the corresponding prompt template in the prompt template library based on the query request. This solution, through the construction of the prompt template library, enables the answer model to match the corresponding prompt template in the prompt template library based on the query request, standardizing the way the answer model uses prompts during the query process and facilitating continuous optimization and management of prompts.

[0154] Furthermore, the classification module 51 is specifically used for:

[0155] Samples in the instruction sample library with a perplexity level below the perplexity threshold are identified as positive samples, and those with a perplexity level above the perplexity threshold are identified as negative samples.

[0156] The set of positive samples is used as the positive sample dataset, and the set of negative samples is used as the negative sample dataset.

[0157] Furthermore, the task mode library building module 53 is specifically used for:

[0158] The knowledge network is clustered according to task patterns using a common subgraph extraction method or a graph-based clustering method.

[0159] Furthermore, the template library building module 54 is specifically used for:

[0160] The model is generated by using prompt templates. Based on the prompt engineering, each positive sample is taken as input and the prompt template corresponding to each positive sample is output. The prompt template includes the query question of the positive sample, the parameters related to the query, and the context information required for the query.

[0161] The task mode corresponding to each positive sample is determined according to the correspondence, and the prompt templates of positive samples belonging to the same task mode are clustered to obtain clustered prompt templates.

[0162] A prompt template library is constructed based on the prompt template and the clustering prompt template.

[0163] Furthermore, the device also includes:

[0164] The determination module is used to, for any negative sample in the negative sample dataset, take the query question, background knowledge and thought chain included in the negative sample as suggestions to be optimized, take the answers included in the negative sample as actual answers, and determine the expected answer of the negative sample.

[0165] The error analysis module is used to output the error type and error cause corresponding to the negative sample by taking the prompt to be optimized, the actual answer and the expected answer as input through the attribution analysis agent. The error type includes query question error, background knowledge error and thinking chain error.

[0166] Furthermore, the device also includes:

[0167] The prompt optimization module is used to optimize the prompt to be optimized based on the error type and the error cause by using the prompt optimization agent, and output the optimized prompt.

[0168] Furthermore, the device also includes:

[0169] The output module is used to take the optimized prompt input response model and output the corresponding optimized response.

[0170] The optimization module is used to continue attribution analysis and prompt optimization on the optimized prompt if the optimized answer is incorrect.

[0171] An update module is used to, if the optimized answer is correct, treat the optimized prompt and the optimized answer as new positive samples, and update the knowledge network, the task pattern library, and the prompt template library based on the new positive samples.

[0172] The prompt template generation device provided in this embodiment of the invention can execute the prompt template generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0173] Example 5

[0174] Figure 6This is a schematic diagram of the structure of an electronic device implementing embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0175] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0176] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the prompt template generation method.

[0178] In some embodiments, the prompt template generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the prompt template generation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the prompt template generation method by any other suitable means (e.g., by means of firmware).

[0179] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0183] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0184] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0185] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0186] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating a prompt template, characterized in that: include: Classifying samples in the instruction sample library to obtain a positive sample data set and a negative sample data set, wherein each sample in the instruction sample library consists of a query question, background knowledge, a thinking chain, and an answer; Extracting triples from the query question, background knowledge, and thinking chain included in each positive sample in the positive sample data set, and constructing a knowledge network based on the extracted triples, wherein the triples include task event entities, knowledge entities, and entity relationships; Performing task pattern clustering on the knowledge network to construct a task pattern library including the correspondence between task event types and task patterns; Generate a prompt template corresponding to each positive sample, cluster the prompt templates according to the corresponding relationship to obtain clustered prompt templates, and build a prompt template library so that the answer model matches the corresponding prompt template in the prompt template library based on the query request.

2. The method according to claim 1, characterized in that: Classify the samples in the instruction sample library to obtain positive sample data sets and negative sample data sets, including: Determine the samples in the instruction sample library whose perplexity is lower than the perplexity threshold as positive samples, otherwise determine them as negative samples; The set of positive samples is taken as a positive sample data set, and the set of negative samples is taken as a negative sample data set.

3. The method according to claim 1, characterized in that The knowledge network is clustered into task patterns, including: The knowledge network is clustered into task patterns by using a common subgraph extraction method or a graph-based clustering method.

4. The method according to claim 1, characterized in that: Generate a prompt template corresponding to each positive sample, and cluster the prompt templates according to the corresponding relationship to obtain clustered prompt templates, and build a prompt template library, including: Generate a model through a prompt template, take each positive sample as input based on the prompt project, and output a prompt template corresponding to each positive sample, wherein the prompt template includes the query question of the positive sample, parameters related to the query, and context information required for the query; Determine the task mode corresponding to each positive sample according to the corresponding relationship, and cluster the prompt templates of the positive samples belonging to the same task mode to obtain a cluster prompt template; A prompt template library is constructed based on the prompt template and the cluster prompt template.

5. The method according to claim 1, characterized in that Also includes: For any negative sample in the negative sample data set, the query question, background knowledge and thinking chain included in the negative sample are used as prompts to be optimized, the answer included in the negative sample is used as the actual answer, and the expected answer of the negative sample is determined; Through the attribution analysis agent, the prompt to be optimized, the actual answer and the expected answer are taken as input, and the error type and error cause corresponding to the negative sample are output, and the error type includes query question error, background knowledge error and thinking chain error.

6. The method according to claim 5, characterized in that Also includes: By means of the prompt optimization agent, the prompt to be optimized is optimized based on the error type and the error cause, and the optimized prompt is output.

7. The method according to claim 6, characterized in that Also includes: Inputting the optimized prompt into the answer model and outputting the corresponding optimized answer; If the optimized answer is wrong, continue to perform attribution analysis and prompt optimization on the optimized prompt; If the optimized answer is correct, the optimized prompt and the optimized answer are used as new positive samples, and the knowledge network, the task pattern library and the prompt template library are updated based on the new positive samples.

8. A prompt template generating device, characterized in that: include: A classification module is used to classify samples in the instruction sample library to obtain a positive sample data set and a negative sample data set, wherein each sample in the instruction sample library consists of a query question, background knowledge, a thinking chain and an answer; A knowledge network construction module, used to extract triples from the query question, background knowledge and thinking chain included in each positive sample in the positive sample data set, and construct a knowledge network based on the extracted triples, wherein the triples include task event entities, knowledge entities and entity relationships; A task pattern library construction module is used to cluster the task patterns of the knowledge network and construct a task pattern library containing the correspondence between task event types and task patterns; The prompt template library construction module is used to generate a prompt template corresponding to each positive sample, cluster the prompt templates according to the corresponding relationship to obtain clustered prompt templates, and construct a prompt template library so that the answer model matches the corresponding prompt template in the prompt template library based on the query request.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images