Training data generation method and platform, electronic equipment and medium

Abstract

One or more embodiments of the present specification provide a training data generation method, platform, electronic device and medium, the method comprising: generating training data through a data generation agent, the training data comprising Prompt or Response in a dialogue; evaluating the training data through an evaluation agent, and outputting evaluation feedback data; the data generation agent and the evaluation agent are realized through a generative language model; under the condition that training data is determined to be unqualified based on the evaluation feedback data, returning the evaluation feedback data to the data generation agent; and the data generation agent regenerates training data according to the evaluation feedback data until the evaluation agent determines that the training data meets a data qualification condition.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of artificial intelligence technology, and in particular, to a method, platform, electronic device, and medium for generating training data. Background Art

[0002] In the training of a large model, the training data used may be dialogue data, which may include prompts and responses. During the training process, a large number of different prompts can be used to train the model to understand various types of inputs. Responses are answers or outputs generated by the model based on prompts. During the training process, they can be used to make the model generate more and more accurate responses that better meet user needs.

[0003] For the training data of the above-mentioned large model, the relevant technology generally uses manual annotation to manually design the dialogue data used for model training, such as designing prompts or responses. However, the disadvantages of this data generation method are that the labor cost is high and the data generation efficiency is low; in addition, the quality of manually generated data is limited by the ability of the annotation personnel, which may produce poor quality training data. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method, platform, electronic device, and medium for generating training data to improve the generation efficiency and quality of training data.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of an embodiment of this specification, a method for generating training data is provided, the method comprising:

[0007] Generate an agent through data to generate training data, wherein the training data includes prompts or responses in the dialogue;

[0008] The training data is evaluated by a critical agent, and evaluation feedback data is output; the data generation agent and the critical agent are implemented by a generative language model;

[0009] In the case where it is determined based on the evaluation feedback data that the training data is unqualified, returning the evaluation feedback data to the data generating agent;

[0010] The data generating agent regenerates the training data according to the evaluation feedback data until the critical agent determines that the training data meets the data qualification conditions.

[0011] According to a second aspect of the embodiments of this specification, a training data generation platform is provided, the platform comprising: a data generation agent and a critical agent; the data generation agent and the critical agent are implemented by a generative language model;

[0012] The data generating agent is used to: generate training data through the data generating agent, the training data including prompts or responses in the dialogue; and regenerate the training data according to the evaluation feedback data returned by the critical agent, until the critical agent determines that the generated training data meets the data qualification condition;

[0013] The critical agent is used to: evaluate the training data and output evaluation feedback data; and if the training data is determined to be unqualified based on the evaluation feedback data output by the critical agent, return the evaluation feedback data to the data generating agent.

[0014] According to a third aspect of the embodiments of this specification, an electronic device is provided, including:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor implements the method described in any embodiment of this specification by running the executable instructions.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the method described in any embodiment of this specification is implemented.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method described in any embodiment of this specification is implemented.

[0020] The training data generation method, platform, electronic device and medium of the embodiments of this specification generate training data through a data generation agent, thereby improving the data generation efficiency; and also evaluate the generated training data through a critical agent. When the training data is unqualified, the data generation agent can regenerate the training data based on the evaluation feedback data output by the critical agent, thereby improving the training data generation efficiency and ensuring the training data generation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or related technologies, the drawings required for use in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 It is a schematic diagram of a training data generation process provided by an exemplary embodiment.

[0023] Figure 2 It is an architectural diagram of a training data generation platform provided by an exemplary embodiment.

[0024] Figure 3 An exemplary embodiment provides a process for a data generation agent and a critical agent to generate training data.

[0025] Figure 4 It shows the generation process of Prompt / Response.

[0026] Figure 5A This is a method for generating training data provided by an exemplary embodiment.

[0027] Figure 5B This is a method for generating training data provided by an exemplary embodiment.

[0028] Figure 6 It is an optimized way of prompting an intelligent agent provided by an exemplary embodiment.

[0029] Figure 7 It is a schematic diagram of the composition of a response agent provided by an exemplary embodiment.

[0030] Figure 8 It is an optimized schematic diagram of a critical agent provided by an exemplary embodiment.

[0031] Fig. 9 It is an architectural diagram of a training data generation platform provided by an exemplary embodiment.

[0032] Fig.10 is a schematic diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0034] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0035] Figure 1 FIG. 1 is a schematic diagram of a flow chart of training data generation provided by an exemplary embodiment. Figure 1 As shown, the generation of training data used for large model training generally goes through a labeling stage 11, a quality inspection stage 12, and an acceptance stage 13.

[0036] The annotation stage 11 may generate training data, for example, a Prompt or a Response.

[0037] The quality inspection stage 12 can be used to perform quality inspection on the training data generated in the annotation stage to check whether there are errors in the generated training data, for example, whether the context of the dialogue data is appropriate, whether the data format is wrong, etc.

[0038] After the quality inspection stage determines that it is qualified, it can enter the acceptance stage 13 and store the training data for subsequent large model training.

[0039] The embodiments of the present specification provide a training data generation platform, which can be used to execute the above-mentioned training data generation process, that is, through the generation platform, training data can be generated, and quality inspection / acceptance and other processing can be performed on the training data.

[0040] In this embodiment, intelligent agents are introduced in the process of generating training data. For example, these intelligent agents can be implemented through a generative language model (for example, a Large Language Model, LLM for short) to automatically generate training data through the intelligent agent instead of manually generating data.

[0041] See also Figure 2 The hint, Figure 2 1 is an architecture diagram of a training data generation platform provided by an exemplary embodiment, wherein the platform may include: a data generation agent 21 and a critical agent 22, both of which may be implemented by a generative language model. Furthermore, the data generation agent 21 and the critical agent 22 may be applied to the annotation stage in the data generation process, i.e., used to generate training data.

[0042] Specifically, Figure 3 The process of generating training data through the above-mentioned data generation agent and critical agent is illustrated, wherein the process can be used to generate prompts in training data or to generate responses.

[0043] In step 300, an agent is generated through data to generate training data.

[0044] Take the training data as Prompt or Response as an example: Figure 4 The hint, Figure 4 The generation process of Prompt / Response is shown in FIG. The data generation agent may include: Prompt Agent and Response Agent.

[0045] like Figure 4 As shown, when the training data is dialogue data including prompts and responses, a large model can be trained by using multiple sets of dialogue data. The following is an example of a set of dialogue data:

[0046] {Prompt:Hi, I'd like to learn how to use my new fitness tracker.

[0047] Response:Hello! Glad to help you. First, make sure your fitness tracker is fully charged. Then, download the corresponding mobile app and pair the tracker with your phone via Bluetooth.

[0048] Prompt: I have completed these steps, what should I do next?

[0049] Response: Next, you can open the app and create an account. In the settings, enter your personal information, such as age, weight, and height, so that the tracker can calculate your activity data more accurately.

[0050] Prompt: I have set up my profile, can I start exercising now?

[0051] Response: Of course you can. Select the type of exercise you want to do on your tracker, such as running, walking or cycling. When you start your exercise, make sure the tracker is in active mode and it will automatically record your exercise data.

[0052] Prompt: How can I view my exercise data after the exercise?

[0053] Response: After your workout, sync your tracker data to the mobile app. On the main interface of the app, you will see a summary showing your exercise time, distance, heart rate, calories burned, and more.

[0054] Prompt: Great, thank you so much for your help!

[0055] Response: You're welcome! Feel free to come back if you have any other questions or need more help. Have fun with your workout!}

[0056] From the above conversation data, we can see that this conversation process can include multiple sets of "Prompt / Response". Figure 4 As shown, when generating dialogue data, Prompt-41 may be generated first, and then based on the Prompt-41, the corresponding Response-42 may be generated. The "Prompt-41, Response-42" is a set of "Prompt / Response", such as the above "Prompt: Hello, I would like to know how to use my new fitness tracker. Response: Hello! I'm happy to help you. First, make sure your fitness tracker is fully charged. Then, download the corresponding mobile app and pair the tracker with your phone via Bluetooth".

[0057] After generating a set of "Prompt / Response", you can Figure 4 As shown, the judgment step 43 is executed to judge whether the conversation has ended. If the conversation has ended, the conversation data in the above example can be obtained, and the quality inspection stage can be continued. If the conversation has not ended, it can return to continue to generate the next set of "Prompt / Response". Among them, when judging whether the conversation has ended, it can be determined based on the number of loops. For example, if five sets of "Prompt / Response" have been generated, it is considered that the length of the conversation is sufficient and the loop ends; or it can also be automatically determined by the model whether it can be ended. For example, the model can determine whether the current conversation data has ended based on the context of the generated conversation data.

[0058] The generation of a set of "Prompt / Response" can be as follows:

[0059] For example, the prompt agent (Prompt Agent) 43 may generate a Prompt, and the response agent (Response Agent) 44 may be responsible for generating a Response.

[0060] Specifically, for the prompt agent 43, its input and output are as follows:

[0061] Input: system prompt + chat history

[0062] Output: Prompt.

[0063] When the prompt agent 43 generates a prompt for the first time, the input may be only the system prompt. When the second loop is executed, for example, a set of "Prompt / Response" is generated first, and the conversation has not ended, when the second set of "Prompt / Response" is generated, the system prompt and chat history may be included.

[0064] The conversation history chat history can be the "Prompt / Response" data that has been generated in this round of conversation. For example, for the "Prompt: I have set up my personal information, can I start exercising now?" in the above example, the generation of this Prompt, and its corresponding input Prompt Agent's chat history, can include all the "Prompt / Response" data before the Prompt. For the data generation agent, the conversation history helps the data generation agent understand the context of the conversation, so as to output more accurate and coherent training data.

[0065] The input system prompt is a guiding instruction to the agent, which defines the role the agent should play, the rules it should follow, or the tasks it should perform. It can be used to guide the agent to provide appropriate responses according to the user's needs and to guide the agent on how to respond. For example, the system prompt can include "You are an expert in generating questions. You should generate questions based on historical information, and please generate more diverse question information." For another example, if you want to generate a prompt of a specific category, you can use the system prompt to instruct the agent.

[0066] System prompt and chat history are usually used in combination. The agent can first set the way and style of its response based on the system prompt, then use the chat history to understand the context of the conversation and output the prompt based on this. For example, if the system prompt is "You are a doctor", the data generation agent can try to generate the prompt from the professional perspective of a doctor and refer to the previous conversation history to ensure the consistency and coherence of the context.

[0067] Specifically, for the response agent 44, its input and output are as follows:

[0068] Input: system prompt + chat history + prompt

[0069] Output: Response.

[0070] The working principle of the response agent 44 is similar to that of the prompt agent and will not be described in detail.

[0071] This embodiment does not limit the specific implementation structure of the prompt agent 43 and the response agent 44. For example, the prompt agent 43 may be implemented by LLM (Large Language Model), and the prompt agent 43 may also generate N (N is a natural number) prompts first, and then perform diversity verification on the N prompts to verify whether they are diverse compared with the existing prompts, and select the prompt that meets the diversity condition as the final output prompt of the prompt agent 43.

[0072] In step 302, the training data is evaluated by a critical agent, and evaluation feedback data is output.

[0073] Please continue to see Figure 4 As shown, the critical agent may include a first critical agent 45 and a second critical agent 46 .

[0074] Among them, the first critical agent 45 can be used to evaluate the Prompt output by the prompt agent 43, for example, it can evaluate whether the Prompt deviates from the topic, whether it is clear, whether the grammatical spelling is wrong, etc. In addition, the first critical agent 45 can output evaluation feedback data. If the output evaluation feedback data is empty, it means that there is no problem with the Prompt and no further adjustment is required. The response agent 44 can continue to be input to generate the corresponding response. If the evaluation feedback data is not empty, it means that the Prompt output by the prompt agent 43 is unqualified and there are still areas that need to be adjusted.

[0075] Similarly, the second critical agent 46 can be used to evaluate the Response output by the response agent 44, for example, whether the Response accurately answers the question in the Prompt, whether the Response is logically consistent, whether there are any contradictions, whether the text is fluent, whether the sentence grammar is correct, whether appropriate terms are used, etc. In addition, the second critical agent 46 can output evaluation feedback data. If the output evaluation feedback data is empty, it means that there is no problem with the Response and no further adjustment is required, thereby obtaining a set of Prompt / Response, and it can continue to determine whether to end the conversation. If the evaluation feedback data is not empty, it means that the Response output by the response agent 44 is unqualified and there are still areas that need to be adjusted.

[0076] In step 304, when it is determined that the training data is unqualified based on the evaluation feedback data, the evaluation feedback data is returned to the data generating agent.

[0077] If the training data is determined to be unqualified based on the evaluation feedback data output by the critical agent, for example, the second critical agent 46 outputs the evaluation feedback data "The Response text is not fluent enough and uses inappropriate terms", the evaluation feedback data can be returned to the data generating agent.

[0078] like Figure 4 As shown, the evaluation feedback data output by the first critical agent 45 can be returned to the prompting agent 43. The evaluation feedback data output by the second critical agent 46 can be returned to the responding agent 44.

[0079] In addition, for the critical agent, its input and output are as follows:

[0080] Input: system prompt + Prompt / Response

[0081] Output: evaluation feedback data (critiques).

[0082] Similarly, the system prompt input into the critical agent can be used to guide the critical agent on how to evaluate the prompt / response. For example, the system prompt can be used to indicate which standards the critical agent should use when evaluating. These standards may include accuracy, logic, completeness, etc., and the scope of evaluation that the critical agent focuses on can be limited, such as the quality, style, structure or other aspects of the training data.

[0083] In step 306, the data generation agent regenerates the training data based on the evaluation feedback data until the critical agent determines that the training data meets the data qualification conditions.

[0084] In this step, the data generation agent can regenerate the training data based on the evaluation feedback data fed back by the critical agent. The regenerated training data will continue to be evaluated by the critical agent. If it is unqualified, it will continue to be regenerated based on the new evaluation feedback data until the critical agent determines that the output training data meets the data qualification conditions. For example, when the data qualification conditions are met, the evaluation feedback data output by the critical agent can be empty, or it can be other content that can indicate that the training data is qualified.

[0085] As shown below, when it is necessary to regenerate training data based on the evaluation feedback data critiques of the critical agent, the input of the prompt agent 43 and the response agent 44 will increase critiques. Critiques is the evaluation feedback of the training data output by the data generation agent, and these evaluation feedback data help the data generation agent to optimize and generate better training data.

[0086] Specifically, for the prompt agent 43, its input and output are as follows:

[0087] Input: system prompt + chat history + evaluation feedback data (critiques)

[0088] Output: Prompt.

[0089] For the response agent 44, the input and output are as follows:

[0090] Input: system prompt + chat history + prompt + evaluation feedback data (critiques)

[0091] Output: Response.

[0092] Let's take an example of regenerating training data based on evaluation feedback data critiques:

[0093] For example, suppose a Prompt is initially generated. After being evaluated by the critical agent, the following evaluation feedback data is given: "Prompt does not specify in what context sales are made, and does not provide a way to deal with users asking questions that cannot be answered. Prompt should pay more attention to user needs rather than just selling products."

[0094] By adding the above evaluation feedback data to the input of the Prompt Agent, the Prompt Agent can regenerate a prompt based on the critiques and other information (e.g., system prompt, chat history). In the regenerated prompt, the application scenario of the chatbot (online electronics store) is clarified, the importance of solving user problems is emphasized, and the sales suggestions are based on user needs. It also provides the steps to be taken when the robot cannot answer the question. In this way, the prompt is optimized.

[0095] In addition, from Figure 4 It can also be seen that the generation of training data in the annotation stage includes the following steps:

[0096] 1) Prompt generation;

[0097] 2) Evaluation of Prompt;

[0098] 3) Generation of Response;

[0099] 4) Evaluation of Response.

[0100] In the above four steps, the manager who generates the training data can choose to use the manual method or the intelligent agent method for each step. For example, Figure 4 As shown, all four links adopt the agent approach. For example, Prompt is generated by Prompt Agent, the first critical agent is responsible for evaluating Prompt, Response is generated by Response Agent, and the second critical agent is responsible for evaluating Response.

[0101] As mentioned above, optionally, in actual implementation, the user can independently choose to adopt a manual approach or an intelligent agent approach for each link. Figure 5A and Figure 5B Two other methods are exemplified. In both methods, training data is generated by a data generation agent and the training data is evaluated by a critical agent. The only difference is that manual methods are used in some links of these two methods.

[0102] For example, Figure 5AAs shown, the Prompt output by the prompting agent can be evaluated manually. If the evaluation is qualified, the Prompt is directly input into the responding agent to generate the corresponding response. If the evaluation is unqualified, the Prompt output by the prompting agent can be manually adjusted, and the adjusted Prompt is input into the responding agent.

[0103] For example, Figure 5B As shown, a Prompt can be generated manually and input into a response agent to generate a corresponding response.

[0104] Other methods are not listed in detail. For example, the generation and evaluation of Prompt can be done through an intelligent agent, while the generation or evaluation of Response can be done manually.

[0105] This embodiment does not limit the number of agents used in the generation / evaluation of Prompts and the generation / evaluation of Responses, and multiple agents may be used. For example, if a large number of Prompts are generated in the previous steps, multiple Response Agents may be used in the generation of Responses for faster processing, which can improve efficiency.

[0106] In addition, the specific implementation structure of the agent used in each link is not limited in this embodiment. For example, some agents can use LLM, some agents can use GPT (Generative Pre-Trained Transformer), etc.

[0107] The training data generation method of this embodiment generates training data through a data generation agent, thereby improving the data generation efficiency; and also evaluates the generated training data through a critical agent. When the training data is unqualified, the data generation agent can regenerate the training data based on the evaluation feedback data output by the critical agent, thereby improving the training data generation efficiency and ensuring the training data generation quality.

[0108] Furthermore, as described above, for the data generating agent, the training data output by the data generating agent can be supervised and evaluated by the critical agent, and better training data can be regenerated based on the evaluation feedback data obtained by the critical agent evaluating the training data.

[0109] See also Figure 6As shown in the example, when the data generating agent is a prompting agent, the input of the prompting agent includes: system prompt + chat history + evaluation feedback data (critiques), and the output is a prompt. Then, the optimization of the prompting agent may include:

[0110] On the one hand, the prompt agent can optimize the prompt according to the evaluation feedback data critiques. On the other hand, the prompt agent can also optimize the system prompt according to the evaluation feedback data critiques.

[0111] For example, the system prompt can be optimized based on multiple rounds of critiques. For example, after multiple rounds of evaluation feedback output by the critical agent, it can be found that the generated prompts do not guide the user to the next step of operation. In this case, the system prompt can be optimized so that the subsequent prompt agent adds prompts for the user's next operation when outputting the prompt.

[0112] That is, the prompt agent can optimize the system prompt according to the evaluation feedback data, and regenerate the Prompt in the training data based on the evaluation feedback data and the optimized system prompt. By optimizing the system prompt based on the evaluation feedback data, the quality of the generated Prompt can be further improved.

[0113] Similarly, the responding agent can also optimize the system prompt of the responding agent based on the evaluation feedback data obtained by the critical agent to further improve the quality of the generated Response, which will not be elaborated in detail.

[0114] In another embodiment, when the data generating agent is a responding agent, multiple sub-agents may collaborate with each other to generate a final Response.

[0115] Specifically, see Figure 7 As shown in the figure, the response agent may include: a planning sub-agent (PlanningAgent) 71, a tool sub-agent (Tool Agent) 72, a storage sub-agent (Memory Agent) 73 and an answer sub-agent (Answer Agent) 74.

[0116] The planning sub-agent 71 can generate an execution plan based on the input data of the response agent, and call the tool sub-agent 72 or the storage sub-agent 73 in sequence according to the execution plan to obtain information for generating a response.

[0117] For example, the planning sub-agent 71 can generate an action queue (Action Queue), which specifies a series of actions to be executed in sequence. The next operation to be performed can be determined based on this queue, and each action in the queue may involve the use of an agent. These actions may require scheduling the tool sub-agent 72 to execute, or calling the storage sub-agent 73 to execute. For example, the action queue specifies that Tool Agent, Memory Agent, Tool Agent, etc. should be used in sequence. That is, in the action queue, Tool Agent and Memory Agent may be called multiple times, and each call is to obtain necessary information or perform a specific task.

[0118] The tool sub-agent 72 is responsible for processing the interaction with external tools (such as API, database or other services), generating commands to call the tools, executing the commands, and obtaining the returned information. The storage sub-agent 73 is responsible for managing the memory or knowledge base, parsing the current context, generating commands to call the memory, executing the commands, and obtaining the returned information.

[0119] When all necessary actions in the action queue are executed, the answer sub-agent 74 can generate a response Response based on the acquired information.

[0120] Here are some examples:

[0121] Assume that the response agent receives input including:

[0122] System Prompt (need to generate travel suggestions) + critiques (previous suggestions were not specific enough) + conversation history + prompt (user's question: I'm going to Paris tomorrow, can you recommend some attractions?)

[0123] The response agent can generate a response in the following way:

[0124] 1) Planning Agent generates an action queue: The action queue may be: [Memory (check user preferences), Planning, Tools (call travel API), Planning, Memory (get historical recommendations), Planning, Answer].

[0125] 2) Execute the action queue, including:

[0126] Memory Agent: Checks user preferences and finds that users like historical and cultural attractions.

[0127] Tools Agent: Calls the tourism API to obtain a list of tourist attractions in Paris.

[0128] Memory Agent: Get historical recommendations and find out that the user was previously interested in the Louvre.

[0129] Answer Agent: Combines the above information and calls LLM to generate the answer.

[0130] 3) Finally, the response generated by the Answer Agent can be as follows: "Considering your interest in history and culture, I recommend you to visit the Louvre and the Palace of Versailles in Paris. Have a nice trip!"

[0131] As shown above, in this way, different agents collaborate to generate a personalized response to the user's question. Each agent performs its specific task and finally merges the results to generate a complete answer. This way of multi-agent collaborative response generation is equivalent to disassembling the response generation task, which helps to generate higher quality training data.

[0132] In another embodiment, as described above, the evaluation feedback data output by the critical agent helps the auxiliary data generation agent to generate better training data. This embodiment also provides two ways for the critical agent to generate evaluation feedback data in order to obtain more accurate evaluation feedback data.

[0133] Approach 1: Detailed decomposition of a single critical agent

[0134] In this method, the critical agent may include multiple sub-agents, and these multiple sub-agents can be used to evaluate different data aspects of the training data respectively to obtain sub-evaluation feedback data corresponding to the different data aspects.

[0135] For example, taking the evaluation of Prompt as an example, when evaluating Prompt, the critical agent can evaluate from the following aspects: "relevance of content", "clear and coherent expression", "appropriate language style", etc., then multiple sub-agents can be set up to evaluate different aspects. For example, the relevance of the context can be checked through the relevance sub-agent, the clarity of Prompt's expression can be checked through the expression sub-agent, and the language style of Prompt can be checked through the style sub-agent, etc. Each sub-agent can output feedback data for the corresponding evaluation aspect, which can be called sub-evaluation feedback data. The sum of these sub-evaluation feedback data is the evaluation feedback data of the critical agent.

[0136] For another example, the response agent can be divided into multiple sub-agents such as "planning sub-agent, tool sub-agent, storage sub-agent and answer sub-agent", and multiple sub-agents can be set up to critically evaluate each of the above sub-agents. For example, a planning critical sub-agent (Planning Critique Agent) is set up to evaluate whether the output of the Planning Agent is appropriate. A tool critical sub-agent (Tools Critique Agent) is set up to evaluate whether the output of the Tools Agent is appropriate. Others are not listed here.

[0137] That is, each sub-agent included in the response agent can be evaluated by the corresponding critical sub-agent when executing actions respectively. If qualified, it will enter the next action, otherwise, it will re-execute the action. For example, the Critiques output by the critical sub-agent Planning Critique Agent can be returned to the PlanningAgent, so that the Planning Agent can output a better action queue. This can ensure that the training data output in the end is better.

[0138] It is understandable that in this stage-by-stage annotation method, the output evaluation of each action can be performed by either a critical sub-agent or a manual evaluation. After human confirmation, the next step can be carried out. For example, after humans confirm that the generated action queue is OK, the call to the Tools Agent will be continued. If the manual evaluation finds that there is a problem, it can be manually modified before continuing to the next action. For example, if the manual evaluation of the action queue output by the Planning Agent finds that there are deficiencies, the action queue can be adjusted before continuing to the next action.

[0139] In addition to the above-mentioned breakpoint annotation method, an end-to-end annotation method can also be used. In this method, the critical agent is no longer disassembled, and the response output by the responding agent is directly evaluated by a single critical agent. It can be understood that the critical agent in this case can not only evaluate the Response finally output by the responding agent, but also include the evaluation of the generation process of the Response. For example, if an error occurs in calling the tool agent during the generation of the Response, these can all be included in the evaluation feedback data.

[0140] Method 2: More critical debates with intelligent agents

[0141] In this method, there can be multiple critical agents, and the training data can be evaluated by multiple critical agents respectively to obtain multiple evaluation results. Through agent debate, the final evaluation feedback data is determined based on the multiple evaluation results. This method can be called multi-agent debate. Through debate and collaboration between multiple agents, the accuracy and comprehensiveness of the output evaluation feedback data are improved.

[0142] For example, suppose we have a generated prompt: "How should my country deal with the increasingly serious air pollution problem?" and the corresponding response: "The government should increase penalties on polluting companies and promote clean energy to reduce pollutant emissions."

[0143] The critical agents participating in the debate may include: affirmative agents (supporting the viewpoint of response) and negative agents (opposing the viewpoint of response). The affirmative agents may include critical agents A and B, and the negative agents may include critical agents C and D. The views of these critical agents are as follows:

[0144] Critical Agent A: If the government increases penalties on polluting companies, it can encourage companies to reduce pollutant emissions and help improve air quality.

[0145] Critical Agent B: Promoting clean energy is a long-term strategy that can gradually replace traditional energy, reduce pollutant emissions, and play a positive role in environmental protection.

[0146] Critical Agent C: Relying solely on punishing enterprises may lead to higher costs for enterprises, which in turn affects economic development. The government should adopt more moderate incentives.

[0147] Critical Agent D: The promotion of clean energy requires a large amount of capital investment and technological research and development, which is difficult to achieve in the short term and may lead to insufficient energy supply.

[0148] Among them, in the multi-agent debate, critical agents A and C debated on the impact of penalties on enterprises, and critical agents B and D debated on the feasibility of promoting clean energy. After the debate, it was considered that the views of the affirmative agent were more convincing, the suggestions in the response were feasible to a certain extent, and the concerns raised by the negative agent were reasonable. It was suggested that both economic development and environmental protection should be taken into account during the implementation process.

[0149] As mentioned above, through this multi-agent debate method, the response can be comprehensively evaluated, thereby improving the quality and reliability of the response.

[0150] As can be seen from the embodiments described above, the critical agent Critique Agent can help the data generation agent generate better training data. If the Critique Agent can accurately evaluate and criticize the outputs of the Prompt Agent and the Response Agent, the need for human supervision will be reduced, and humans can focus more on the optimization of the Critique Agent instead of directly supervising every detail of the Prompt Agent and the Response Agent.

[0151] The following methods can be used to optimize the critical agent itself:

[0152] See also Figure 8 As shown in the figure, the input of the critical agent can include: system prompt, and training data (such as Prompt or Response).

[0153] For the evaluation feedback data Critiques output by the critical agent, the human 81 can evaluate the evaluation feedback data to obtain evaluation supervision data, and return the evaluation supervision data to the critical agent as input, so that the critical agent can re-output the evaluation feedback data based on the evaluation supervision data and the training data. This cycle is repeated until the critical agent outputs qualified evaluation feedback data Critiques.

[0154] Among them, when humans evaluate the evaluation feedback data Critiques output by the critical intelligent agent Critique Agent, they can include the following aspects: the relevance of Critiques to the context, whether Critiques accurately points out the shortcomings of the training data, whether Critiques provides improvement suggestions or solutions, etc.

[0155] In this way, the critical agent can improve its own capabilities based on human evaluation supervision data to generate more accurate evaluation feedback data.

[0156] In addition, if Figure 8As shown in the figure, the input of the Critique Agent also includes the system prompt SystemPrompt. Similar to the data generation agent, the Critique Agent can also optimize the system prompt of the input Critique Agent based on the above-mentioned evaluation supervision data. For example, if the evaluation supervision data points out that the Critiques lack improvement suggestions, then the system prompt of the Critique Agent can be added with guidance such as "Please give improvement suggestions".

[0157] In some other embodiments, as mentioned above, some links can replace the agent with artificial, for example, Figure 5A In the method shown, although the prompt is generated by the prompt agent, there is no closed-loop feedback formed by the evaluation of the critical agent, and the ability of the prompt agent itself cannot be improved. In this case, the prompt agent can be trained separately according to the human modification and editing of the prompt. For example, the prompt agent can be trained by reinforcement learning to improve the ability of the data generation agent. The ability improvement of the response agent is similar, so it will not be described in detail.

[0158] Also, see Figure 2 In the generation platform shown, in addition to setting an agent in the annotation stage, an agent can also be set in the quality inspection stage, which is called a quality inspection agent. The quality inspection agent is used to evaluate the quality inspection results of the training data generated in the annotation stage to obtain quality inspection feedback data. When the training data is determined to be qualified based on the quality inspection feedback data, the training data is input into the data acceptance module to enter the acceptance stage.

[0159] See also Fig. 9 As shown in the figure, humans can perform quality inspection on the training data generated in the annotation phase to obtain quality inspection results. For example, the quality inspection results may include: whether the response accurately answers the question in the prompt; whether the response has grammatical errors or spelling errors; whether the tool tool call in the response generation process is correct, and the contextual relevance of prompt / response in the entire conversation data process, etc.

[0160] The quality inspection results can be entered into Fig. 9The quality inspection agent 91 in the data processing unit can help check whether there are problems with the human quality inspection results and output quality inspection feedback data. If the quality inspection feedback data indicates that there are problems with the human quality inspection results, it can be fed back to the human, and the human re-checks and generates new quality inspection results. If the quality inspection feedback data indicates that there are no problems with the human quality inspection results and the training data is qualified, the training data can be input into the data acceptance module to enter the acceptance stage. If there are no problems with the human quality inspection results, but it indicates that the training data is unqualified, the training data can be regenerated.

[0161] By setting up a quality inspection agent in the quality inspection stage, it can assist in checking the quality inspection results of humans and help improve the quality of training data.

[0162] In addition, the capabilities of the quality inspection agent can also be improved. Similar to the above-mentioned embodiment, for example, humans can check the quality inspection feedback data output by the quality inspection agent 91 for evaluation. If deficiencies are found, the feedback information used to describe the deficiencies can be returned as input to the quality inspection agent, so that the quality inspection agent can regenerate better quality inspection feedback data accordingly.

[0163] The implementation process of the functions and effects of each module in the above system is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0164] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0165] like Fig.10 As shown, Fig.10 A hardware structure diagram of an electronic device in which the navigation broadcast device of an embodiment of the specification is located is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050.

[0166] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0167] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store operating systems and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0168] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0169] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0170] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0171] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0172] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method of any of the above embodiments is implemented.

[0173] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0174] The embodiments of the present specification also provide a computer program, which is used to implement the method described in any embodiment of the present specification when executed.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0178] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for generating training data, characterized in that: The method comprises: Generate an agent through data to generate training data, wherein the training data includes prompts or responses in the dialogue; The training data is evaluated by a critical agent, and evaluation feedback data is output; the data generation agent and the critical agent are implemented by a generative language model; In the case where it is determined based on the evaluation feedback data that the training data is unqualified, returning the evaluation feedback data to the data generating agent; The data generating agent regenerates the training data according to the evaluation feedback data until the critical agent determines that the training data meets the data qualification conditions.

2. The method according to claim 1, characterized in that The step of generating training data by generating an intelligent agent through data includes: generating prompts in the training data by a prompting intelligent agent; generating responses in the training data by a responding intelligent agent; The evaluating of the training data by the critical agent includes: By means of a first critical agent, the prompt output by the prompt agent is evaluated; In the case where the prompt is determined to be qualified based on the evaluation feedback data output by the first critical agent, the prompt is input into the responding agent so that the responding agent obtains a corresponding response according to the prompt; The response output by the responding agent is evaluated by the second critical agent, and when the response is determined to be qualified based on the evaluation feedback data output by the second critical agent, a set of prompts and responses included in the training data is obtained.

3. The method according to claim 1, characterized in that The generating training data by the data generating agent includes: taking the system prompt as the input of the data generating agent to obtain the training data output by the data generating agent; The method further includes: the data generating agent optimizing the system prompt based on evaluation feedback data obtained by the critical agent evaluating the training data; The data generation agent regenerates the training data according to the evaluation feedback data, including: the data generation agent regenerates the training data according to the evaluation feedback data and the optimized system prompt.

4. The method according to claim 1, characterized in that: The critical agent comprises: a plurality of sub-agents, wherein the plurality of sub-agents are respectively used to evaluate different data aspects of the training data; The evaluating of the training data by the critical agent includes: Different data aspects of the training data are evaluated respectively through the multiple sub-agents to obtain sub-evaluation feedback data corresponding to the different data aspects.

5. The method according to claim 1, characterized in that The number of the critical intelligent agents is multiple; The evaluating of the training data by the critical agent includes: Through multiple critical agents, the training data is evaluated separately to obtain multiple evaluation results; The multiple critical agents determine final evaluation feedback data based on the multiple evaluation results through agent debate.

6. The method according to claim 1, characterized in that After evaluating the training data by the critical agent and outputting evaluation feedback data, the method further includes: The critical agent receives evaluation supervision data, wherein the evaluation supervision data is an evaluation of the evaluation feedback data output by the critical agent; The critical agent re-outputs evaluation feedback data based on the evaluation supervision data and the training data.

7. The method according to claim 6, characterized in that The step of evaluating the training data and outputting evaluation feedback data by the critical agent includes: taking a system prompt as an input of the critical agent, and obtaining the evaluation feedback data according to the input system prompt and the training data; The method further comprises: The critical agent optimizes system prompts input into the critical agent based on the evaluation supervision data.

8. The method according to claim 1, characterized in that After the critical agent determines that the training data meets the data qualification condition, the method further includes: Inputting the quality inspection result of the training data into the quality inspection agent to obtain the quality inspection feedback data output by the quality inspection agent; When it is determined that the training data is qualified based on the quality inspection feedback data, the training data is input into a data acceptance module.

9. The method according to claim 1, characterized in that: In the case where the training data is a response Response, the data generation agent includes: a planning sub-agent, a tool sub-agent, a storage sub-agent and an answer sub-agent; The step of generating an intelligent agent through data and generating training data includes: The planning sub-agent generates an execution plan based on the input data of the agent, and calls the tool sub-agent or the storage sub-agent in sequence according to the execution plan to obtain information for generating a response; The answer sub-agent generates a response based on the information.

10. A training data generation platform, characterized in that: The platform includes: a data generation agent and a critical agent; the data generation agent and the critical agent are implemented by a generative language model; The data generating agent is used to: generate training data through the data generating agent, the training data including prompts or responses in the dialogue; and regenerate the training data according to the evaluation feedback data returned by the critical agent, until the critical agent determines that the generated training data meets the data qualification condition; The critical agent is used to: evaluate the training data and output evaluation feedback data; and if the training data is determined to be unqualified based on the evaluation feedback data output by the critical agent, return the evaluation feedback data to the data generating agent.

11. The platform according to claim 10, characterized in that The data generating agent includes: a prompting agent and a responding agent; The critical agent includes: a first critical agent and a second critical agent; The prompt agent is used to generate prompts in the training data; the response agent is used to generate responses in the training data; The first critical agent is used to evaluate the prompt output by the prompt agent; the second critical agent is used to evaluate the response output by the response agent.

12. The platform according to claim 10, characterized in that The data generates an intelligent agent and is also used to: Optimizing system prompts based on the evaluation feedback data output by the critical agent, wherein the system prompts serve as inputs to the data generating agent; And regenerate training data according to the optimized system prompts and the evaluation feedback data.

13. The platform according to claim 10, characterized in that The critical agent is also used to: receive evaluation supervision data, which is an evaluation of the evaluation feedback data output by the critical agent; and re-output the evaluation feedback data based on the evaluation supervision data and the training data.

14. The platform according to claim 10, characterized in that The platform also includes: a quality inspection agent; The quality inspection agent is used to evaluate the quality inspection results of the training data to obtain quality inspection feedback data.

15. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

16. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images