Double-agent training method and device, equipment, storage medium and program product

Abstract

The application relates to the field of artificial intelligence, and discloses a double-agent training method, device, equipment, storage medium and program product. The method comprises the following steps: obtaining question information; using a first agent to generate a plurality of candidate questions corresponding to the question information; screening a to-be-solved question from the plurality of candidate questions; using a second agent to generate first answer information corresponding to the to-be-solved question; updating first parameters of the first agent and second parameters of the second agent based on the matching degree of the to-be-solved question and the first answer information; wherein the first parameters and the second parameters are used to indicate that the to-be-solved question matches the first answer information. The application can solve the problem that model training depends on manual annotation and cannot be iteratively optimized in a closed loop.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a training method, apparatus, device, storage medium, and program product for dual intelligent agents. Background Technology

[0002] As the application of large language models continues to expand in agent-based tasks such as complex question answering, open-domain retrieval, and multi-step reasoning, tool-enhanced large language models have acquired the ability to access search engines and external knowledge bases. Currently, scenarios such as intelligent question answering and question bank generation often revolve around a single agent. The sources of questions rely on manual construction, and automatically generated questions lack effective filtering. The training signals for the problem-solving process typically only use the correctness of the final answer as a reward, ignoring multi-round search behavior and the quality of the reasoning chain, making it difficult to simultaneously ensure both question quality and answer accuracy. Furthermore, the question-generating and answering stages are independent of each other, making it difficult to form a closed-loop optimization. Summary of the Invention

[0003] This invention provides a dual-agent training method to solve the problem that a single agent cannot simultaneously ensure the quality of question generation and the quality of answering.

[0004] In a first aspect, the present invention provides a training method for two agents, comprising: Obtain the question information; Based on the first intelligent agent, multiple candidate questions corresponding to the question information are generated; The target question is selected from the plurality of candidate questions; Based on the second intelligent agent, generate target answer information corresponding to the target question; Based on the matching degree between the target question and the target answer information, update the first parameter of the first agent and the second parameter of the second agent; The first parameter and the second parameter are used to indicate that the target question matches the target answer information.

[0005] The dual-agent training method provided in this invention utilizes a first agent to generate candidate questions based on question information, thus achieving automatic question generation by the first agent. Subsequently, questions are selected from the candidate questions generated by the first agent for the second agent to answer. The first parameter of the first agent and the second parameter of the second agent are jointly optimized based on the matching degree between question and answer information, achieving collaborative training for both agents. Therefore, by having the first agent automatically generate questions based on question information, and the second agent automatically answering these questions, a closed-loop training process of question generation and answering is formed, effectively improving the adaptability of questions and answers. Bidirectional parameter updates based on question-answer matching degree enable mutual supervision and collaborative iteration between the first and second agents, eliminating the need for extensive manual data annotation, reducing training costs, enriching question diversity, realizing a self-game training closed loop for dual agents, and improving the question generation quality and reasoning ability of the agents.

[0006] In one optional implementation, a first intelligent agent is used to generate multiple candidate questions corresponding to the question information, including: Obtain the chained knowledge data corresponding to the question information, and using the first intelligent agent, generate multiple candidate questions based on the knowledge context corresponding to the chained knowledge data; and / or, Obtain the seed information corresponding to the question information, and use the first intelligent agent to generate multiple candidate questions based on the knowledge context corresponding to the seed information.

[0007] The dual-agent training method provided in this invention involves the first agent generating candidate questions based on the knowledge context corresponding to chained knowledge data. This allows candidate questions to span multiple knowledge nodes by incorporating the knowledge context, significantly enhancing the diversity and complexity of the candidate questions. The first agent can also utilize the knowledge context corresponding to seed information to generate candidate questions, ensuring that the generated candidate questions are semantically related but have different perspectives. This enriches the training samples while limiting the randomness of the candidate questions.

[0008] In one optional implementation, selecting the problem to be solved from the plurality of candidate problems includes: Perform format detection on each of the candidate problems to obtain the format detection results; Perform answer leakage detection on each of the candidate questions to obtain the answer leakage detection results; Perform dependency detection on each of the candidate questions to obtain the dependency detection results; The quality of each candidate question is tested to obtain the quality test results. Based on one or more of the format detection results, answer leakage detection results, dependency detection results, and quality detection results, multiple candidate questions are filtered to obtain the unsolved questions.

[0009] The dual-agent training method provided in this invention performs multi-dimensional detection on multiple candidate questions and filters the candidate questions by combining the multi-dimensional detection results to obtain the questions to be solved. It can effectively eliminate invalid questions with format errors, direct leaks of answers, questions that can be answered without searching, and questions of low quality, thereby significantly improving the usability and training value of the questions to be solved. It ensures that the questions to be solved have good parsability, search necessity, and reasonable difficulty, and reduces the interference of invalid samples on the training of the second agent.

[0010] In one optional implementation, a second intelligent agent is used to generate first answer information corresponding to the problem to be solved, including: The second intelligent agent is used to analyze the problem to be solved and generate first search information corresponding to the problem to be solved. Based on the first search information, retrieve second answer information that matches the problem to be solved; If the second answer information matches the problem to be solved, then the second answer information is determined as the first answer information for the problem to be solved; If the second answer information does not match the problem to be solved, then the second search information for the problem to be solved is generated, and the first answer information for the problem to be solved is retrieved based on the second search information.

[0011] The dual-agent training method provided in this invention involves a second agent analyzing the problem to be solved to determine corresponding first search information. This first search information is then used as a benchmark to retrieve second answer information, improving the accuracy and rationality of the answer content. Simultaneously, it effectively utilizes search services, avoiding excessive reliance on internal knowledge for the solution path of the second answer information, reducing the deviation of the second agent's autonomously generated answers, and ensuring that the problem-solving solution aligns with the agent's question-and-answer application scenario. When the second answer information retrieved by the second agent does not match the problem to be solved, further searching can be performed based on the second search information. This ensures that the second agent has the ability to adjust its search strategy and conduct multiple rounds of searching and verification, improving the success rate of solving complex or multi-hop problems and the reliability of the second agent's answer information generation.

[0012] In one optional implementation, a second intelligent agent is used to generate first answer information corresponding to the problem to be solved, including: Get the number of retrieval rounds in which the first answer information was retrieved; If the number of retrieval rounds reaches a preset threshold, then a third answer is generated.

[0013] The dual-agent training method provided in this invention obtains the number of retrieval rounds, stops the retrieval of the first answer information when the number of retrieval rounds reaches a preset threshold, and generates the third answer information. This can effectively control the computational overhead and response time of the second agent's solution process and avoid the second agent from stalling in reasoning due to getting stuck in invalid searches or repeated attempts.

[0014] In one optional implementation, based on the matching degree between the problem to be solved and the first answer information, the first parameter of the first agent and the second parameter of the second agent are updated, including: Obtain the specified answer information corresponding to the problem to be solved; Based on the matching degree between the specified answer information and the first answer information, the question-generating reward information and the question-solving reward information are determined; Based on the question-generating reward information and the question-solving reward information, update the first parameter of the first agent and the second parameter of the second agent.

[0015] The dual-agent training method provided in this invention combines the matching degree between specified answer information and first answer information to determine the corresponding question-generating reward information and problem-solving reward information. The parameters of the first agent and the second agent are then updated using the question-generating reward information and problem-solving reward information, thereby achieving collaborative optimization for the first agent and the second agent, and making the question-generating ability of the first agent and the problem-solving ability of the second agent mutually compatible.

[0016] In one optional implementation, based on the matching degree between the specified answer information and the first answer information, the question-generating reward information and the question-solving reward information are determined, including: The reward is determined based on the matching degree between the specified answer information and the first answer information; Obtain the question generation trajectory corresponding to the candidate question and the solution trajectory corresponding to the first answer information, and determine the process consistency reward based on the consistency between the question generation trajectory and the solution trajectory; Based on the format detection results of the candidate questions, a format reward is determined. The difficulty reward is determined based on the success rate of solving the same problem. The question-setting reward information includes the format reward and the difficulty reward; the problem-solving reward information includes the result reward and the process consistency reward.

[0017] The dual-agent training method provided in this invention utilizes result rewards to improve the accuracy of the second agent's answers, process consistency rewards to enhance the interpretability and credibility of the answers, format rewards to control the first agent's generation of formatted questions, and difficulty rewards to encourage the first agent to continuously generate questions with training value, rather than overly easy or unsolvable questions. This makes the reward signals during the dual-agent training process denser, effectively constraining the effectiveness of questions, answer consistency, retrieval value, and difficulty distribution during training.

[0018] In one optional implementation, updating the first parameter of the first agent and the second parameter of the second agent based on the question-generating reward information and the question-solving reward information includes: Following a preset order, the first parameter of the first agent is updated based on the question-generating reward information, and the second parameter of the second agent is updated based on the question-solving reward information; or, Based on the fusion result of the question-generating reward information and the question-solving reward information, the first parameter of the first agent and the second parameter of the second agent are jointly updated.

[0019] The dual-agent training method provided in this invention employs a sequential update approach, updating the first parameter of the first agent using question-generating reward information and the second parameter of the second agent using problem-solving reward information. This allows the first and second agents to achieve differentiated optimizations under the independent drive of their respective reward signals. The joint update approach combines the fusion results of question-generating and problem-solving reward information to simultaneously update the first parameter of the first agent and the second parameter of the second agent, thus simultaneously addressing optimization objectives in both question-generating and problem-solving directions and promoting the collaborative evolution of the two agents.

[0020] In a second aspect, the present invention provides a dual-agent training device, comprising: The information acquisition module is used to acquire question information; The question generation module is used to generate multiple candidate questions corresponding to the question information using the first intelligent agent; The problem filtering module is used to filter out the problems to be solved from the plurality of candidate problems; The answer generation module is used to generate first answer information corresponding to the problem to be solved using a second intelligent agent; The parameter update module updates the first parameter of the first agent and the second parameter of the second agent based on the matching degree between the problem to be solved and the first answer information; wherein, the first parameter and the second parameter are used to indicate that the problem to be solved matches the first answer information.

[0021] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the dual-agent training method of the first aspect or any corresponding embodiment described above.

[0022] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the dual-agent training method of the first aspect or any corresponding embodiment thereof.

[0023] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the dual-agent training method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the first process of a dual-agent training method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a second process for a dual-agent training method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the third process of the dual-agent training method according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating a specific implementation of an embodiment of the present invention; Figure 6 This is a structural block diagram of a dual-agent training device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] As an optional application scenario of this invention, such as Figure 1 As shown, the dual-agent training system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0030] The server 103 houses a training controller for executing a dual-agent reinforcement learning training process, specifically iteratively updating the parameters of the first and second agents based on self-game theory. The server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. The computer 101 and mobile terminal 102 can provide relevant input information for the training process. The trained agent model can be deployed on the computer 101 or mobile terminal 102 to perform tasks such as question-answering reasoning, search and problem-solving, or question generation. The network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), wide area networks (WANs), mobile communication networks, and combinations thereof.

[0031] In current scenarios such as intelligent question answering and question bank generation, a single intelligent agent struggles to balance question quality and answer accuracy. The question generation and answering processes are independent, failing to form a closed-loop optimization mechanism. Current solutions often employ independent models to complete question generation and answering tasks separately, with model parameters iterated solely based on labeled data. This results in poor matching between question content and answers, insufficient question diversity, and the inability of the two models to supervise each other or co-evolve. Furthermore, manually labeling samples is costly and inefficient, and relying solely on static datasets for training makes it difficult to continuously improve the rationality of question generation and the accuracy of answers.

[0032] Based on this, the present invention utilizes a dual-agent collaborative training architecture, in which the first agent automatically generates questions with search value and appropriate difficulty, and the second agent searches for and solves the questions generated by the first agent. Thus, through two-way feedback between question generation and answering, the parameters of the two agents are jointly optimized, thereby achieving the effect of a self-game training closed loop.

[0033] According to an embodiment of the present invention, a training method for a dual agent is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides a dual-agent training method, which can be used on the aforementioned server or on a computing cluster composed of the aforementioned servers. The first agent model or the second agent model after training can be deployed on a computer or mobile terminal.

[0035] Figure 2 This is a flowchart of a dual-agent training method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the question information.

[0036] The question-generating information is used to define the direction and content of the generated questions. This information may include the knowledge points to be tested, the question type, the difficulty level, the scope of the questions, and text materials, such as an existing question, an answer, a descriptive hint, or a set of related knowledge fragments. The specific content of the question-generating information is not specifically limited here.

[0037] Specifically, the question information can be extracted by the server from a publicly available database; or it can be received by the server from an external source, as described above. The server communicates with terminal devices such as computers and mobile terminals, and the question information can be edited through the interactive interface provided by the terminal device. Accordingly, the server can obtain the question information.

[0038] Of course, there are other ways to obtain the question information, and no specific restrictions are made here.

[0039] Step S202: Using the first intelligent agent, generate multiple candidate questions corresponding to the question information.

[0040] The first intelligent agent, acting as the question-generating agent, can be trained based on a large language model architecture.

[0041] The server calls the training controller to input the question information into the first intelligent agent. Using the language generation capability of the first intelligent agent, it generates multiple question texts with different expressions, perspectives, or difficulties around the knowledge domain defined by the question information. These question texts are the multiple candidate questions.

[0042] In a specific example, if the question information is "Scientist X - Award - Nobel Prize in Physics - Research Field - Theory M", then the first agent can generate candidate questions such as "Which scientist who won the Nobel Prize in Physics twice participated in proposing Theory M?" and "What is the other Nobel Prize contribution of scientist X who proposed Theory M and won the Nobel Prize in Physics?"

[0043] During the generation of candidate questions, the first agent simultaneously generates thought text, search requests, candidate questions, candidate answers, and the corresponding entity chains or evidence fragments, which can be stored as a question generation trajectory. By generating multiple candidate questions at once, richer training materials can be obtained, providing selection space for subsequent question screening.

[0044] Step S203: Select the problem to be solved from multiple candidate problems.

[0045] The unsolved problems are candidate problems that meet the criteria of setting the question rules, having standardized expression, and being logically complete; that is, target problems used to participate in subsequent problem-solving training.

[0046] Specifically, the server calls the training controller to detect and filter multiple candidate questions generated by the first agent, and determines whether each candidate question meets the criteria such as question setting rules, standardized expression and logical completeness. Candidate questions that meet the criteria are identified as problems to be solved.

[0047] In some optional implementations, a feedback question pool can be set up to store historically valid questions. If there are insufficient questions to be solved later, historically valid questions can be drawn from the feedback question pool as replacement questions. If the samples in the feedback question pool are insufficient, placeholder samples can also be used temporarily as questions to be solved, but a penalty must be imposed on the placeholder samples to prevent them from participating in the agent's effective parameter updates. This implementation step can effectively filter out low-value and invalid questions, thereby ensuring that each question participating in problem-solving training has answerability and training value, and reducing the interference of invalid samples on the training process.

[0048] Step S204: Using the second intelligent agent, generate the first answer information corresponding to the problem to be solved.

[0049] The second agent, acting as the problem-solving agent, can be trained based on a large language model architecture. Specifically, the second agent and the first agent can be implemented on the same large language model architecture through different role prompts, or two models with independent or partially shared parameters can serve as the first and second agents respectively.

[0050] The problem to be solved serves as the input to the second intelligent agent, which autonomously plans the solution path and outputs text that can serve as the answer to the problem to be solved. This text is the first answer information corresponding to the problem to be solved.

[0051] The search terms, retrieval results, reasoning steps, and answer text generated by the second agent during the problem-solving process can be stored as solution trajectories. The same problem can be sampled multiple times in parallel by the second agent to obtain multiple solution trajectories. This stage can employ asynchronous unfolding, parallel sampling, or multiple-trial mechanisms to improve training throughput and reward estimation stability. This approach enables efficient use of computational resources during training, avoiding the blockage of overall training progress due to single solution failures or delays.

[0052] Step S205: Based on the matching degree between the problem to be solved and the first answer information, update the first parameter of the first agent and the second parameter of the second agent; wherein the first parameter and the second parameter are used to indicate that the problem to be solved matches the first answer information.

[0053] The first parameter is the model parameter of the first agent, used to make the questions generated by the first agent more diverse and complex. The second parameter is the model parameter of the second agent, used to enable the second agent to answer the questions more accurately and completely.

[0054] After generating the first answer information, the training controller on the server can retrieve the question generation trajectory corresponding to the problem to be solved and compare it with the solution trajectory corresponding to the first answer information to determine the degree of matching. For example, it can compare the similarity of the search text in the question generation trajectory and the solution trajectory, and the similarity of the preset answer text and the generated answer text. Based on the degree of matching, the corresponding reward information is determined; the higher the degree of matching, the greater the positive reward. Then, based on the reward information corresponding to the degree of matching, the first parameters of the first agent and the second parameters of the second agent are updated, and this process is repeated until the model capabilities of the two agents are mutually adapted.

[0055] The dual-agent training method provided in this invention utilizes a first agent to generate candidate questions based on question information, thus achieving automatic question generation by the first agent. Subsequently, questions are selected from the candidate questions generated by the first agent for the second agent to answer. The first parameter of the first agent and the second parameter of the second agent are jointly optimized based on the matching degree between question and answer information, achieving collaborative training for both agents. Therefore, by having the first agent automatically generate questions based on question information, and the second agent automatically answering these questions, a closed-loop training process of question generation and answering is formed, effectively improving the adaptability of questions and answers. Bidirectional parameter updates based on question-answer matching degree enable mutual supervision and collaborative iteration between the first and second agents, eliminating the need for extensive manual data annotation, reducing training costs, enriching question diversity, realizing a self-game training closed loop for dual agents, and improving the question generation quality and reasoning ability of the agents.

[0056] This embodiment provides a dual-agent training method, which can be used on the aforementioned server or on a computing cluster composed of the aforementioned servers. The first agent model or the second agent model after training can be deployed on a computer or mobile terminal.

[0057] Figure 3 This is a flowchart of a dual-agent training method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain the question information. For details, please refer to [link / reference]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0058] Step S302: Using the first intelligent agent, generate multiple candidate questions corresponding to the question information.

[0059] Specifically, step S302 includes: Step S3021: Obtain the chain-like knowledge data corresponding to the question information.

[0060] Chained knowledge data is knowledge data with a chain structure, such as knowledge chains, entity relationship chains, knowledge graph paths, or offline chained data. It is usually composed of multiple entities and their relationships connected sequentially (e.g., in the form of "entity A - relationship 1 - entity B - relationship 2 - entity C").

[0061] Specifically, chained knowledge data can be obtained from offline knowledge graphs, path data extracted based on entity relationships or manually organized, or document paragraph chains, event timelines, causal chains, step chains, or paper citation chains. The method of obtaining chained knowledge data is not specifically limited here, as long as it can provide knowledge evidence to support the generation of candidate questions.

[0062] Step S3022: Using the first intelligent agent, generate multiple candidate questions based on the knowledge context corresponding to the chained knowledge data.

[0063] The chained knowledge data is input into the first intelligent agent, which understands the knowledge context corresponding to the chained knowledge data and filters out entities that are not suitable as answers or termination nodes (such as entities that are too short, year entities, list-type entities, ambiguous entities, or entities that are easily exposed in the question), thereby reducing the risk of answer leakage and improving the quality of candidate questions.

[0064] Chained knowledge data can provide clear multi-hop reasoning paths for question generation, expand the question space, increase the complexity and search necessity of candidate questions, and effectively avoid generating shallow questions that can be answered by single-step common sense or simple table lookup.

[0065] The dual-agent training method provided in this invention involves the first agent generating candidate questions based on the knowledge context corresponding to chained knowledge data. This allows the candidate questions to span multiple knowledge nodes by combining the knowledge context, significantly enhancing the diversity and complexity of the candidate questions.

[0066] Specifically, step S302 may further include: Step S3023: Obtain the seed information corresponding to the question information.

[0067] Seed information is the core foundation of the question information, including seed questions, seed answers, or seed hints. Specifically, seed information can be existing questions, corresponding answers, or a descriptive hint extracted from the question information; alternatively, it can be generated by semantically deconstructing the question information, extracting keywords, and then structurally generating the question keywords.

[0068] Step S3024: Using the first intelligent agent, generate multiple candidate questions based on the knowledge context corresponding to the seed information.

[0069] The seed information is input into the first intelligent agent, which performs semantic understanding on the knowledge context corresponding to the seed information. Based on the seed information and its knowledge context, it generates multiple candidate questions that are semantically related but have different expressions and perspectives. This allows the limited seed samples to be used to automatically expand the space of existing questions, enrich the diversity of training data, and at the same time maintain consistency with the original seeds in terms of knowledge and logic.

[0070] The dual-agent training method provided in this embodiment of the invention involves the first agent generating candidate questions using the knowledge context corresponding to the seed information. This ensures that the generated candidate questions are semantically related but have different perspectives, thereby enriching the training samples while limiting the randomness of the candidate questions.

[0071] Step S303: Select the problem to be solved from the multiple candidate problems.

[0072] Specifically, step S303 includes: Step S3031: Perform format detection on each candidate problem to obtain the format detection results.

[0073] Format checking is used to verify whether candidate questions meet preset format constraints. Preset format constraints are predefined question formats, which may include constraints such as question label constraints, sentence structure constraints, length constraints, and invalid information constraints.

[0074] Specifically, each candidate question is checked sequentially to determine whether it contains a specified question tag, whether it is an interrogative sentence or a task instruction, whether it is too short or too long, whether it contains a preset question tag, and whether there are invalid messages such as empty questions, duplicate questions, garbled characters, truncated tags, duplicate tags, or obviously invalid characters, so as to obtain the format detection results for each candidate question.

[0075] Step S3032: Perform answer leakage detection on each candidate question to obtain the answer leakage detection results.

[0076] Answer leakage detection is used to check whether the questions contain content related to the first answer. Specifically, it sequentially checks each candidate question for synonyms, capitalization variations, parenthesis-removed aliases, and directly copyable fragments of the first answer information or answer entity, obtaining answer leakage detection results for each candidate question. These results are then used to prevent the leakage of the first answer information.

[0077] If the first answer information appears directly in a candidate question, it will be marked as a leak and penalized with a formatting penalty or discarded.

[0078] Step S3033: Perform dependency detection on each candidate question to obtain the dependency detection results.

[0079] Dependency detection is used to check whether candidate questions are generated solely from templates without relying on search materials or retrieved information.

[0080] Specifically, it sequentially determines whether the entities, relationships, or constraints in each candidate question require external search or multi-step reasoning to answer, and whether each candidate question can be supported by retrieval fragments, knowledge chains, or evidence documents, thereby obtaining dependency detection results for each candidate question.

[0081] Step S3034: The quality of each candidate question is checked to obtain the quality check results.

[0082] Quality checks are used to score the quality of candidate questions. Scoring dimensions can include link fidelity, multi-hop, single focus, semantic clarity, answerability, and training value.

[0083] Specifically, a pre-trained quality assessment model can be used to evaluate the quality of each candidate question across multiple scoring dimensions, thereby obtaining the quality detection results for each candidate question.

[0084] The quality assessment model can be trained based on a convolutional neural network, or it can be trained based on a convolutional neural network and a gated recurrent unit, or it can be trained bidirectionally based on a transformer encoder. There are no specific limitations here, as long as it can achieve multi-dimensional quality assessment.

[0085] Step S3035: Based on one or more of the following: format detection results, answer leakage detection results, dependency detection results, and quality detection results, multiple candidate questions are filtered to obtain the questions to be solved.

[0086] Using one or more of the following methods—format detection results, answer leakage detection results, and dependency detection results—multiple candidate questions are screened to remove those with format errors, invalid questions, questions with no search value, and unsolvable questions. Then, from the remaining candidate questions, several candidate questions with higher scores are selected as the questions to be solved based on the quality detection results.

[0087] It should be noted that candidate problems with minor formatting issues can be treated as unsolved problems but with reduced formatting rewards. For candidate problems with serious errors or failed extractions, feedback problem pools or seed problems are used instead to maintain training batch size and training stability.

[0088] The dual-agent training method provided in this invention performs multi-dimensional detection on multiple candidate questions and filters the candidate questions by combining the multi-dimensional detection results to obtain the questions to be solved. It can effectively eliminate invalid questions with format errors, direct leaks of answers, questions that can be answered without searching, and questions of low quality, thereby significantly improving the usability and training value of the questions to be solved. It ensures that the questions to be solved have good parsability, search necessity, and reasonable difficulty, and reduces the interference of invalid samples on the training of the second agent.

[0089] Step S304: Using the second intelligent agent, generate first answer information corresponding to the problem to be solved. See details below. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0090] Step S305: Based on the matching degree between the problem to be solved and the first answer information, update the first parameters of the first agent and the second parameters of the second agent. For details, please refer to [link to details]. Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0091] The dual-agent training method provided in this invention utilizes a first agent to generate candidate questions based on question information, thus achieving automatic question generation by the first agent. Subsequently, questions are selected from the candidate questions generated by the first agent for the second agent to answer. The first parameter of the first agent and the second parameter of the second agent are jointly optimized based on the matching degree between question and answer information, achieving collaborative training for both agents. Therefore, by having the first agent automatically generate questions based on question information, and the second agent automatically answering these questions, a closed-loop training process of question generation and answering is formed, effectively improving the adaptability of questions and answers. Bidirectional parameter updates based on question-answer matching degree enable mutual supervision and collaborative iteration between the first and second agents, eliminating the need for extensive manual data annotation, reducing training costs, enriching question diversity, realizing a self-game training closed loop for dual agents, and improving the question generation quality and reasoning ability of the agents.

[0092] This embodiment provides a dual-agent training method, which can be used on the aforementioned server or on a computing cluster composed of the aforementioned servers. The first agent model or the second agent model after training can be deployed on a computer or mobile terminal.

[0093] Figure 4 This is a flowchart of a dual-agent training method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps: Step S301: Obtain the question information. For details, please refer to [link / reference]. Figure 2Step S201 of the illustrated embodiment will not be described again here.

[0094] Step S302: Using the first intelligent agent, generate multiple candidate questions corresponding to the question information. See details below. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0095] Step S303: Select the problem to be solved from the plurality of candidate problems. For details, please refer to [link to relevant documentation]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0096] Step S304: Using the second intelligent agent, generate first answer information corresponding to the problem to be solved.

[0097] Specifically, step S304 includes: Step S3041: Use the second intelligent agent to analyze the problem to be solved and generate the first search information corresponding to the problem to be solved.

[0098] The second intelligent agent performs semantic parsing on the problem to be solved, identifies key entities, relational constraints and unknown information points in the problem, and generates search terms or query statements required for retrieval based on these information. The search terms or query statements are the first search information.

[0099] Step S3042: Based on the first search information, retrieve the second answer information that matches the problem to be solved.

[0100] The server calls the training controller to control the second agent to call the retrieval service to retrieve the first search information and obtain the search results such as document fragments, knowledge fragments or search result summaries returned by the retrieval service. The second agent can then perform reasoning based on these search results to obtain second answer information that can serve as the answer to the problem to be solved.

[0101] The retrieval service can be a vector retrieval service, a document retrieval service, or an internet search agent; it can also be a knowledge base, an educational knowledge graph, a professional literature database, or a web search system.

[0102] Step S3043: If the second answer information matches the problem to be solved, then the second answer information is determined as the first answer information for the problem to be solved.

[0103] The second answer information is matched with the question to be solved to determine whether the second answer information can answer the question. If the second answer information contains the key facts required for the question to be solved, or is semantically consistent with the question to be solved, then the second answer information is determined to be a successful match with the question to be solved. At this time, the second answer information can be determined as the first answer information and output to end this round of solving.

[0104] Step S3044: If the second answer information does not match the problem to be solved, then continue to generate the second search information for the problem to be solved.

[0105] If the second answer information does not contain the key facts required to solve the problem, or is semantically inconsistent with the problem, then the second answer information is deemed unsuccessful in matching the problem. In this case, it is necessary to regenerate new second search information for the problem and use this second search information as the search information for a new round of retrieval to obtain supplementary evidence from different angles or more precise directions to solve the problem.

[0106] Step S3045: Based on the second search information, retrieve the first answer information for the problem to be solved.

[0107] The server invokes the training controller to control the second agent to again call the retrieval service to retrieve the second search information and attempt to solve the problem again. This process of "generation-retrieval-reasoning-regeneration-re-retrieval-re-reasoning" continues until content matching the problem is deduced and output as the first answer.

[0108] The dual-agent training method provided in this invention involves a second agent analyzing the problem to be solved to determine corresponding first search information. This first search information is then used as a benchmark to retrieve second answer information, improving the accuracy and rationality of the answer content. Simultaneously, it effectively utilizes search services, avoiding over-reliance on internal knowledge for the solution path of the second answer information, reducing the bias in the second agent's autonomously generated answers, and ensuring that the problem-solving approach aligns with the agent's question-and-answer application scenario.

[0109] When the second answer information obtained by the second agent does not match the problem to be solved, the search can continue based on the second search information, ensuring that the second agent has the ability to adjust the search strategy and carry out multiple rounds of search and verification, thereby improving the success rate of the second agent in solving complex or multi-hop problems and the reliability of the second agent's answer information generation.

[0110] In some alternative implementations, the above method may further include: Step a1: Obtain the number of retrieval rounds in which the first answer information was retrieved.

[0111] Step a2: If the number of retrieval rounds reaches the preset round threshold, then generate the third answer information.

[0112] The number of retrieval rounds is the total number of retrievals performed by the second agent from invoking the retrieval service to obtaining the first answer information; the preset round threshold is the maximum number of retrieval rounds set in advance to avoid the waste of system resources and training blockage caused by unlimited search; the third answer information is the fallback response output by the second agent.

[0113] Specifically, each time the second agent performs a solution process, the training controller in the server synchronously obtains a retrieval round number. Then, it compares this retrieval round number with a preset round number threshold to determine whether the retrieval round number has reached the preset round number threshold.

[0114] If the first matching answer is not obtained after reaching a preset threshold number of retrieval rounds, the second agent is controlled to terminate the solution process and return the third answer. Specifically, the third answer can be generated based on existing evidence or can be a preset default answer, such as "This retrieval failed" or "This problem cannot be solved".

[0115] The dual-agent training method provided in this invention obtains the number of retrieval rounds, stops the retrieval of the first answer information when the number of retrieval rounds reaches a preset threshold, and generates the third answer information. This can effectively control the computational overhead and response time of the second agent's solution process and avoid the second agent from stalling in reasoning due to getting stuck in invalid searches or repeated attempts.

[0116] Step S305: Based on the matching degree between the problem to be solved and the first answer information, update the first parameters of the first agent and the second parameters of the second agent.

[0117] Specifically, step S305 includes: Step S3051: Obtain the specified answer information corresponding to the problem to be solved.

[0118] The training controller in the server can retrieve the problem generation trajectory corresponding to the stored unsolved problem and extract the specified answer information corresponding to the unsolved problem from the problem generation trajectory.

[0119] Step S3052: Based on the matching degree between the specified answer information and the first answer information, determine the question-generating reward information and the question-solving reward information.

[0120] The question-generating reward information serves as an incentive signal characterizing the quality of the generated questions; that is, this reward information is generated based on the matching degree and is used as an incentive signal fed back to the first agent. The problem-solving reward information serves as an incentive signal characterizing the quality of the answered questions; that is, this reward information is generated based on the matching degree and is used as an incentive signal fed back to the second agent.

[0121] Specifically, by comparing the semantic and content similarity between the specified answer information and the first answer information, a quantitative matching score is obtained. Based on this matching score, the question-generating reward information for the first agent and the question-solving reward information for the second agent are generated respectively. Among them, a high matching score generates a positive reward; a low matching score generates a negative reward.

[0122] In some optional implementations, step S3052 above includes: Step b1: Determine the result reward based on the matching degree between the specified answer information and the first answer information.

[0123] The resulting reward, R_ans, can be determined by an exact match, a normalized soft match, or a match evaluation model between the first answer and the specified answer. For example, a correct answer earns R_ans = 1 point, an incorrect answer earns R_ans = 0 points, and acceptable aliases, numerical approximations, or natural language answers are evaluated by a soft score between 0 and 1, output by the review model.

[0124] The matching evaluation model here can be trained based on a large language model, a deep learning model, or a neural network, and can be used to evaluate the matching degree of answer information using methods such as rule matching and semantic matching.

[0125] Step b2: Obtain the question generation trajectory corresponding to the candidate question and the solution trajectory corresponding to the first answer information, and determine the process consistency reward based on the consistency between the question generation trajectory and the solution trajectory.

[0126] The process consistency reward R_proc can be obtained based on the consistency between the search path, intermediate entity chain, evidence chain, or intermediate reasoning steps and the problem generation trajectory. For example, if the evidence fragments retrieved by the solution trajectory cover key entities and relations in the knowledge chain, the reward score is increased; if the final answer is correct but the search evidence is irrelevant to the question, or the answer is incorrect but some relevant evidence is retrieved, a partial score is given according to the coverage ratio; if the final answer is incorrect, a small process reward can also be given based on partially correct search evidence or entity chains to avoid completely suppressing search behavior.

[0127] Step b3: Determine the format reward based on the format detection results of the candidate questions.

[0128] The format reward R_fmt is derived based on the success rate of question extraction, the validity of the question, and the completeness of the format. For example, a positive reward is given when a candidate question is correctly extracted without empty questions, duplicate labels, leaked answers, or illegal characters; a negative reward is given when extraction fails, labels are missing, or the answer is directly leaked.

[0129] Step b4: Determine the difficulty reward based on the success rate of solving the same problem.

[0130] The difficulty reward R_diff is estimated by back-evaluating the success rate of the second agent's multiple solutions to the same problem. It is used to encourage problems with training value rather than overly easy or unsolvable problems. Specifically, if the second agent's final answer to problem q in its i-th solution is a_i, the standard or reference answer is a*, the number of samples for the same problem is N, and the correct answer indicator function is I(a_i, a*), then the success rate s(q) can be calculated using the formula s(q) = N^{-1}Σ_i I(a_i, a*). In one optional implementation, an exponential function R_diff = exp(-(s(q)-τ)^2 / (2σ^2)) can be used, where τ is the target success rate and σ is the width parameter. In another optional implementation, a multimodal function R_diff = Σ_j w_j•exp(-(s(q)-τ_j)^2 / (2σ_j^2)) can be used to reward reasonable proportions of easy, medium, and difficult problems respectively.

[0131] Outcome rewards and process consistency rewards primarily affect the second agent, enabling it to learn how to search for and integrate evidence and output the correct answer. Question generation rewards are not simply based on "the more failed attempts, the better," but rather on constructing a target difficulty range based on the success rate, making the first agent more inclined to generate questions with training value. Format rewards and difficulty rewards primarily affect the first agent, ensuring that the difficulty distribution of generated questions remains within the target range and that the format is standardized. By jointly modeling outcome rewards, process consistency rewards, format rewards, and difficulty rewards, the problem of reward sparsity during training can be significantly alleviated.

[0132] The dual-agent training method provided in this invention utilizes result rewards to improve the accuracy of the second agent's answers, process consistency rewards to enhance the interpretability and credibility of the answers, format rewards to control the first agent's generation of formatted questions, and difficulty rewards to encourage the first agent to continuously generate questions with training value, rather than questions that are too easy or unsolvable. This makes the reward signals in the dual-agent training process denser, effectively constraining the effectiveness of questions, answer consistency, retrieval value, and difficulty distribution during the dual-agent training process.

[0133] Step S3053: Based on the question-generating reward information and the question-solving reward information, update the first parameter of the first agent and the second parameter of the second agent.

[0134] The reward information for generating and solving problems is written into the terminal words or trajectory-level fields of the corresponding trajectories. Then, the model parameters of the first and second agents are updated by selecting the same or different reinforcement learning algorithms, such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). To prevent policy drift, update constraints can be introduced to limit the update direction, such as Kullback-Leibler divergence constraint (KL), pruning objective function, entropy regularization, and gradient pruning.

[0135] In some optional implementations, step S3053 above includes: Following a preset order, the first parameter of the first agent is updated based on the question-generating reward information, and the second parameter of the second agent is updated based on the question-solving reward information.

[0136] The first agent can form a relative advantage within the group based on multiple candidate problems under the same knowledge chain, thereby performing relative optimization within the group; the second agent completes policy updates based on multiple solution trajectories under the same problem. The preset order can be to update the first agent first and then the second agent, or to update the second agent first and then the first agent, or to update alternately according to the training steps.

[0137] The dual-agent training method provided in this embodiment of the invention adopts a sequential update approach, which updates the first parameter of the first agent using the question-generating reward information and updates the second parameter of the second agent using the question-solving reward information, so that the first agent and the second agent can achieve differential optimization under the independent drive of their respective reward signals.

[0138] In some optional implementations, step S3053 above includes: Based on the fusion results of the question-generating reward information and the question-solving reward information, the first parameters of the first agent and the second parameters of the second agent are jointly updated.

[0139] The total reward can be updated by weighting and combining the four types of rewards. For example, the reward R_solver for a single problem-solving path can be represented as R_solver = λ_1•R_ans + λ_2•R_proc + λ_3•R_search + λ_4•R_fmt, where R_search represents effective search behavior or evidence usage. The problem-generating reward R_proposer can be represented as R_proposer = α•R_fmt + β•R_diff + γ•R_quality - δ•R_leak, where R_quality represents the quality review score, and R_leak represents the answer disclosure penalty, both obtained during the selection of unsolved problems. The above weights can be pre-configured by the training controller in the server and can be adjusted during the training phase. The parameters of the two agents are then updated using the combined reward method.

[0140] The first and second agents employ different optimization algorithms and update strategies to adapt to different model sizes, hardware resources, and training objectives. A joint update approach is used, combining the fusion results of question-generating and problem-solving rewards to simultaneously update the first parameter of the first agent and the second parameter of the second agent. This approach can simultaneously address the optimization objectives in both question-generating and problem-solving directions, promoting the collaborative evolution of the two agents.

[0141] The dual-agent training method provided in this invention combines the matching degree between specified answer information and first answer information to determine the corresponding question-generating reward information and problem-solving reward information. The parameters of the first agent and the second agent are then updated using the question-generating reward information and problem-solving reward information, thereby achieving collaborative optimization for the first agent and the second agent, and making the question-generating ability of the first agent and the problem-solving ability of the second agent mutually compatible.

[0142] To better illustrate the technical solution and beneficial effects of the present invention, the embodiments of the present invention are described below in conjunction with specific application scenarios. It should be noted that these specific application embodiments are only for helping to understand the present invention and are not intended to limit the scope of protection of the present invention; those skilled in the art can make appropriate adjustments or combinations to the embodiments without departing from the principles of the present invention.

[0143] like Figure 5As shown, the offline knowledge chain "Movie A—Director—Director's Name B—Directed Works—Movie C—Lead Actor—Lead Actor's Name D" is obtained and sent to the question-generating agent. After receiving the knowledge chain, the question-generating agent can call the retrieval service to generate four candidate questions that require traversing multiple relationships to answer. Candidate question 1 is "Who is the lead actor in movie C directed by director B?"; Candidate question 2 is "Which biographical film is directed by the director of movie A and stars actor D?" The answer to this question is movie C, which requires first identifying the director of movie A and then reasoning along the director-works relationship and the lead actor relationship; Candidate question 3 is "Who directed movie A?"; Candidate question 4 confuses the director and lead actor relationships.

[0144] Of course, candidate questions can also be collected from a pool of candidate questions, such as... Figure 5 As shown.

[0145] While generating candidate questions, the question generation trajectory corresponding to the candidate questions can be stored through a data management component deployed on the server.

[0146] Next, a problem to be solved is selected from multiple candidate problems and sent to the problem-solving agent. If the output of the problem-generating agent includes the thought process, tool call records, and the final question, the problem to be solved can be detected through the problem generation trajectory. Candidate problem 1 has a high risk of answer leakage because it exposes a key intermediate entity; candidate problem 2 requires two-hop reasoning and does not directly expose the final answer, so it scores higher; candidate problem 3 is too simple, so the difficulty reward is low; candidate problem 4 confuses the relationship between the director and the lead actor, so the evidence consistency score is low. Alternatively, candidates with format errors, answer leakage, or inconsistent evidence can be filtered out first, and then the top few candidates can be selected from the remaining candidates according to their quality scores. One or more better problems can then be randomly or selected according to their scores to enter the subsequent solving stage.

[0147] The problem-solving agent performs multiple rounds of searching, evidence reading, and reasoning within a limited number of rounds and outputs the first answer information. The same problem can be sampled multiple times in parallel to estimate the success rate of solving the problem. The data management module records multiple problem-solving trajectories generated during this process. The parameter update service retrieves the problem to be solved, the first answer information, the problem generation trajectory, and the problem-solving trajectory to calculate the reward.

[0148] like Figure 5As shown, for example, M candidate questions are generated from the same knowledge chain. For the selected unsolved question q, the problem-solving agent samples N solution trajectories for q. If N=4 and 2 of the answers are correct, then s(q)=0.5. If the target success rate τ=0.4 and σ=0.15, the question can obtain a higher difficulty reward; if all 4 are correct, the training controller considers the question too easy and reduces the difficulty reward; if all 4 fail and the process evidence is inconsistent, then the controller considers the question potentially unsolvable and reduces the reward. Furthermore, the target success rate τ can be dynamically changed with the training phase, biasing towards answerable and stable questions in the early stages of training, and gradually biasing towards more challenging medium-difficulty questions in the later stages. That is, the problem-solving reward is calculated from the solution trajectory, and the problem-generating reward is calculated from the question-generating trajectory. The fused reward information is determined from the solution gradient corresponding to the solution reward and the question-generating gradient corresponding to the question-generating reward. Using the fused reward information, the model parameters of the problem-generating agent and the problem-solving agent are jointly updated, and the PPO policy gradient algorithm is updated.

[0149] To improve training stability, a feedback question pool is included in the data management module. This pool stores questions from the past training process that have passed screening, have valid answers, and can form a solution trajectory. When a candidate question fails to be extracted, has an incorrect format, has a leaked answer, or is of insufficient quality in a training step, the training controller can extract valid historical questions from the feedback question pool as replacement samples. The samples in the feedback question pool can also be managed hierarchically based on their historical solution success rate. For example, the training controller can prioritize extracting questions with a success rate within the target difficulty range to reduce the interference of overly easy or unsolvable questions on the training.

[0150] In an alternative embodiment, the training entry point can be a Ray cluster started by a training script, with configuration files specifying the model path, training data, validation data, retrieval tool configuration, Structured Generation Language (SGLang) inference parameters, Fully Sharded Data Parallel (FSDP) training parameters, reward manager, and self-game parameters. In each training step, the training controller generates a problem-generating trajectory, extracts problems, generates a solution trajectory, asynchronously calculates rewards, and calculates the probabilities of the old policy and the reference policy.

[0151] In another application example, you can also retrieve the seed question contained in the seed information: Who directed movie A; seed answer: Director B; seed hint: Generate a question that requires searching to answer based on the director and representative works.

[0152] After receiving the input, the first intelligent agent does not simply rewrite the original question, but can call the retrieval service to generate the following candidate trajectory: first, it retrieves the director of movie A, obtaining director B; then it retrieves other works and award information of the director; finally, it outputs the candidate questions "Which other movie with dreams or time structure as the core narrative element was directed by the director of movie A?" or "Which biographical film did the director of movie A later receive an Oscar nomination or award for Best Director?" It should be noted that candidate questions are surrounded by preset labels, such as <question> Candidate question text< / question> .

[0153] Each candidate question is checked to determine whether the target answer, such as "Director B," directly appears in the candidate question; if it does, it is judged as an answer leak and the format reward is reduced or the candidate question is discarded.

[0154] The selected problems are sent to a problem-solving agent, which performs multiple rounds of searching, evidence reading, and reasoning within a limited number of rounds, and outputs the first answer. The same problem can be sampled multiple times in parallel to estimate the success rate of solving it, and the data management module records the multiple problem-solving trajectories generated during this process.

[0155] Specifically, the training controller in the server first caches the old policy probabilities and reference policy probabilities of the question-generating trajectories, then generates solution trajectories and calculates rewards; subsequently, it constructs question-generating batches and solution batches. The question-generating batch calculates its advantage based on the relative performance of candidate questions under the same seed information, thereby learning to generate questions with higher training value. The solution batch calculates its advantage based on the relative rewards of multiple solution trajectories under the same problem, thereby learning which search behaviors, evidence integration methods, and answer expression methods are more likely to obtain high rewards. The training controller then updates the question-generating and solution agents according to a preset order. The training controller records the format pass rate, answer leakage rate, number of search rounds, success rate, mean reward, and reward variance during the training process for subsequent diagnosis and experimental explanation.

[0156] For example, Table 1 presents the main test results of this implementation scheme on four datasets: Natural Questions (NQ), Hotpot Question Answering (HotpotQA), 2Wiki Multi-hop QA (2Wiki), and Multi-Step Question Answering (MuSiQue). This figure illustrates the types of metrics that can be recorded and compared when implementing this method, including but not limited to model size, training data source, number of training steps, number of question samples, number of solution samples, average question-generating reward, average solution reward, validation set accuracy, average number of search rounds, format pass rate, answer leakage filtering rate, and standard deviation or fluctuation range across multiple runs.

[0157] Table 1 Test results for multiple datasets

[0158] It should be noted that the specific values ​​of the above indicators will vary depending on factors such as model structure, dataset, and training parameters. Therefore, the data shown in Table 1 should not be interpreted as a limitation that all models or datasets will necessarily achieve a fixed improvement, but rather as an illustration that the technical effect of this invention can be verified through reproducible experimental logs. Compared to training methods that only use static question-answering data or only optimize the accuracy of the final answer, this method, through candidate question selection, search trajectory rewards, difficulty rewards, and feedback question pool mechanisms, makes the training samples closer to the current capability boundary of the model, thereby helping to improve the success rate of multi-round searches, reduce the proportion of invalid questions, and improve training stability.

[0159] This embodiment also provides a dual-agent training device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0160] This embodiment provides a dual-agent training device, such as... Figure 6 As shown, it includes: The information acquisition module 601 is used to acquire question information.

[0161] The question generation module 602 is used to generate multiple candidate questions corresponding to the question information using the first intelligent agent.

[0162] In some alternative implementations, the problem generation module 602 includes: The first acquisition unit is used to acquire the chain of knowledge data corresponding to the question information.

[0163] The first generation unit is used to generate multiple candidate questions based on the knowledge context corresponding to the chained knowledge data, using the first intelligent agent.

[0164] In some alternative implementations, the problem generation module 602 includes: The second acquisition unit is used to acquire the seed information corresponding to the question information.

[0165] The second generation unit is used to generate multiple candidate questions based on the knowledge context corresponding to the seed information, using the first intelligent agent.

[0166] Problem filtering module 603 is used to filter out unsolved problems from multiple candidate problems.

[0167] In some alternative implementations, the problem screening module 603 includes: The format detection unit is used to perform format detection on each candidate problem and obtain the format detection result.

[0168] The answer leakage detection unit is used to detect answer leakage for each candidate question and obtain the answer leakage detection results.

[0169] The dependency detection unit is used to search for dependencies in each candidate question and obtain the dependency detection results.

[0170] The quality inspection unit is used to inspect the quality of each candidate problem and obtain the quality inspection results.

[0171] The filtering unit is used to filter multiple candidate questions based on one or more of the following: format detection results, answer leakage detection results, dependency detection results, and quality detection results, to obtain the questions to be solved.

[0172] The answer generation module 604 is used to generate first answer information corresponding to the problem to be solved by utilizing a second intelligent agent.

[0173] In some alternative implementations, the answer generation module 604 includes: The first problem-solving unit is used to analyze the problem to be solved by the second intelligent agent and generate the first search information corresponding to the problem to be solved.

[0174] The second problem-solving unit is used to retrieve second answer information that matches the problem to be solved, based on the first search information.

[0175] The third problem-solving unit is used to determine whether the second answer information matches the problem to be solved. If they match, the second answer information is determined as the first answer information for the problem to be solved.

[0176] The fourth problem-solving unit is used to generate second search information for the problem to be solved when the third problem-solving unit determines that the second answer information does not match the problem to be solved.

[0177] The fifth problem-solving unit is used to retrieve the first answer information for the problem to be solved based on the second search information.

[0178] The sixth problem-solving unit is used to obtain the number of retrieval rounds in which the first answer information was retrieved.

[0179] The seventh problem-solving unit is used to generate the third answer information when the number of retrieval rounds obtained in the sixth problem-solving unit reaches a preset round threshold.

[0180] The parameter update module 605 is used to update the first parameters of the first agent and the second parameters of the second agent based on the matching degree between the problem to be solved and the first answer information.

[0181] In some alternative implementations, the parameter update module 605 includes: The acquisition unit is used to obtain the specified answer information corresponding to the problem to be solved.

[0182] The reward unit is used to determine the question-generating reward information and the question-solving reward information based on the matching degree between the specified answer information and the first answer information.

[0183] In some alternative implementations, the reward unit includes: The result reward subunit is used to determine the result reward based on the matching degree between the specified answer information and the first answer information.

[0184] The process reward subunit is used to obtain the question generation trajectory corresponding to the candidate question and the solution trajectory corresponding to the first answer information, and to determine the process consistency reward based on the consistency between the question generation trajectory and the solution trajectory.

[0185] The format reward subunit is used to determine the format reward based on the format detection results of the candidate questions.

[0186] The difficulty reward sub-unit is used to determine the difficulty reward based on the success rate of solving the same problem.

[0187] An update unit is used to update the first parameters of the first agent and the second parameters of the second agent based on the question-generating reward information and the question-solving reward information.

[0188] In some optional implementations, the updating unit includes: The sequential update subunit is used to update the first parameters of the first agent based on the question-generating reward information and the second parameters of the second agent based on the question-solving reward information in a preset order.

[0189] In some optional implementations, the updating unit includes: The fusion update subunit is used to jointly update the first parameters of the first agent and the second parameters of the second agent based on the fusion result of the question-generating reward information and the question-solving reward information.

[0190] The dual-agent training device provided in this embodiment of the invention can execute the dual-agent training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0191] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0192] The following is a detailed reference. Figure 7 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in ROM 702 or a program loaded from memory 708 into RAM 703. RAM 703 also stores various programs and data required for the operation of the electronic device. The processor 701, ROM 702, and RAM 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704; wherein ROM is a read-only memory and RAM is a random access memory.

[0193] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0194] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a memory 708, or installed from a ROM 702. When the computer program is executed by the processor 701, it performs the functions defined in the dual-agent training method of the embodiments of the present invention.

[0195] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0196] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the dual-agent training method shown in the above embodiments is implemented.

[0197] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0198] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A training method for two agents, characterized in that, The method includes: Obtain the question information; Using a first intelligent agent, generate multiple candidate questions corresponding to the question information; The problem to be solved is selected from the plurality of candidate problems; Using a second intelligent agent, generate first answer information corresponding to the problem to be solved; Based on the matching degree between the problem to be solved and the first answer information, update the first parameter of the first agent and the second parameter of the second agent; The first parameter and the second parameter are used to indicate that the problem to be solved matches the first answer information.

2. The training method according to claim 1, characterized in that, The process of generating multiple candidate questions corresponding to the question information using a first intelligent agent includes: Obtain the chained knowledge data corresponding to the question information, and using the first intelligent agent, generate multiple candidate questions based on the knowledge context corresponding to the chained knowledge data; and / or, Obtain the seed information corresponding to the question information, and use the first intelligent agent to generate multiple candidate questions based on the knowledge context corresponding to the seed information.

3. The training method according to claim 1, characterized in that, The step of selecting the problem to be solved from the plurality of candidate problems includes: Perform format detection on each of the candidate problems to obtain the format detection results; Perform answer leakage detection on each of the candidate questions to obtain the answer leakage detection results; Perform dependency detection on each of the candidate questions to obtain the dependency detection results; The quality of each candidate question is tested to obtain the quality test results. Based on one or more of the format detection results, answer leakage detection results, dependency detection results, and quality detection results, multiple candidate questions are filtered to obtain the unsolved questions.

4. The training method according to any one of claims 1 to 3, characterized in that, The step of using a second intelligent agent to generate first answer information corresponding to the problem to be solved includes: The second intelligent agent is used to analyze the problem to be solved and generate first search information corresponding to the problem to be solved. Based on the first search information, retrieve second answer information that matches the problem to be solved; If the second answer information matches the problem to be solved, then the second answer information is determined as the first answer information for the problem to be solved; If the second answer information does not match the problem to be solved, then the second search information for the problem to be solved is generated, and the first answer information for the problem to be solved is retrieved based on the second search information.

5. The training method according to claim 4, characterized in that, The step of using a second intelligent agent to generate first answer information corresponding to the problem to be solved includes: Get the number of retrieval rounds in which the first answer information was retrieved; If the number of retrieval rounds reaches a preset threshold, then a third answer is generated.

6. The training method according to claim 1, characterized in that, The step of updating the first parameter of the first agent and the second parameter of the second agent based on the matching degree between the problem to be solved and the first answer information includes: Obtain the specified answer information corresponding to the problem to be solved; Based on the matching degree between the specified answer information and the first answer information, the question-generating reward information and the question-solving reward information are determined; Based on the question-generating reward information and the question-solving reward information, update the first parameter of the first agent and the second parameter of the second agent.

7. The training method according to claim 6, characterized in that, The step of determining the question-generating reward information and the question-solving reward information based on the matching degree between the specified answer information and the first answer information includes: The reward is determined based on the matching degree between the specified answer information and the first answer information; Obtain the question generation trajectory corresponding to the candidate question and the solution trajectory corresponding to the first answer information, and determine the process consistency reward based on the consistency between the question generation trajectory and the solution trajectory; Based on the format detection results of the candidate questions, a format reward is determined. The difficulty reward is determined based on the success rate of solving the same problem. The question-setting reward information includes the format reward and the difficulty reward; the problem-solving reward information includes the result reward and the process consistency reward.

8. The training method according to claim 6 or 7, characterized in that, The step of updating the first parameter of the first agent and the second parameter of the second agent based on the question-generating reward information and the question-solving reward information includes: Following a preset order, the first parameter of the first agent is updated based on the question-generating reward information, and the second parameter of the second agent is updated based on the question-solving reward information; or, Based on the fusion result of the question-generating reward information and the question-solving reward information, the first parameter of the first agent and the second parameter of the second agent are jointly updated.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the dual-agent training method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the dual-agent training method according to any one of claims 1 to 8.

Images