Information reasoning method and device

Abstract

The present invention relates to the technical field of large-scale language models, and in particular to an information reasoning method, apparatus, computer equipment, and storage medium. The method utilizes a multi-agent debate framework to promote collaboration through task debates between agents, and learns the team joint actions of an agent decision network to guide the updating of individual agent strategies. This enables each agent to incrementally contribute to task solutions when faced with complex tasks, thereby improving the accuracy and efficiency of the agent decision network's reasoning.

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale language models, and in particular to an information reasoning method, apparatus, computer equipment, and storage medium. Background Art

[0002] In recent years, large language models (LLMs) have demonstrated remarkable capabilities in language generation, comprehension, and few-shot learning. Recent work has focused on exploring how to improve the factual accuracy and reasoning capabilities of large models. Techniques such as hint learning, thought chaining, and self-consistency can improve performance without adjusting model parameters. Furthermore, state-of-the-art large model training techniques, such as human-feedback reinforcement learning, can significantly enhance these capabilities.

[0003] Supervised fine-tuning (SFT) is a widely adopted and effective technique for improving the capabilities of large models. This approach enhances the ability of large models to complete complex tasks. Compared to SFT methods, reinforcement learning techniques such as proximal policy optimization (PPO) and direct preference optimization (DPO) offer new tuning strategies for large models. However, these methods focus solely on tuning a single large-model agent. This single large-model agent is susceptible to model bias and cognitive degradation during reasoning. Furthermore, due to a lack of external feedback, it struggles to form diverse insights into the problem, limiting its ability to handle complex tasks. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide an information reasoning method, apparatus, computer equipment and storage medium, which utilizes a multi-agent debate framework to promote collaboration through task debates between agents, learns the team joint actions of the agent decision network to guide the update of individual agent strategies, and enables each agent to incrementally contribute to the task solution when facing complex tasks, thereby improving the accuracy and efficiency of the reasoning of the agent decision network.

[0005] In a first aspect, an embodiment of the present application provides an information reasoning method, wherein an agent decision network includes a plurality of agents; the method comprises the following steps:

[0006] Obtaining state space information of multiple agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot;

[0007] According to the state space information of several agents and the corresponding current time slot, the action space information of several agents in the current time slot is obtained; according to the action space information of several agents and the corresponding current time slot, task debate is performed to obtain the state space information of several agents in the next time slot;

[0008] Reward calculation is performed based on the action space information of the multiple agents in the current time slot to obtain reward information of the multiple agents in the current time slot; the state space information, action space information, and reward information of the current time slot of the same agent are combined to construct a training information combination of the multiple agents in the current time slot;

[0009] Performing task debate based on the action space information of the multiple agents and the corresponding current time slot to obtain state space information of the multiple agents in the next time slot; repeatedly constructing training information combinations based on the state space information of the multiple agents in the next time slot to obtain training information combinations for multiple time slots; and updating the multiple agents in the agent decision network based on the training information combinations for the multiple time slots;

[0010] Obtain text information of the problem to be processed, input the text information of the problem to be processed into several updated agents in the agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain answer text information output by several updated agents in the last number of iterations, and use the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

[0011] In a second aspect, an embodiment of the present application provides an information reasoning device based on an agent decision network, comprising:

[0012] A data acquisition module is used to obtain state space information of multiple agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot;

[0013] The task debate model is used to obtain the action space information of several agents in the current time slot based on the state space information of several agents and the corresponding current time slot;

[0014] A training information combination construction module is used to calculate rewards based on the action space information of the multiple agents in the current time slot to obtain reward information of the multiple agents in the current time slot; and to combine the state space information, action space information, and reward information of the same agent in the current time slot to construct a training information combination of the multiple agents in the current time slot;

[0015] An agent updating module is configured to perform task debate based on the action space information of the multiple agents and the corresponding current time slot to obtain the state space information of the multiple agents in the next time slot; repeatedly construct training information combinations based on the state space information of the multiple agents in the next time slot to obtain training information combinations for multiple time slots; and update the multiple agents in the agent decision network based on the training information combinations for the multiple time slots;

[0016] The information reasoning module is used to obtain the text information of the problem to be processed, input the text information of the problem to be processed into several updated intelligent agents in the intelligent agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain the answer text information output by several updated intelligent agents in the last number of iterations, and use the answer text information with the highest frequency as the reasoning result of the text information of the problem to be processed.

[0017] In a third aspect, an embodiment of the present application provides a computer device comprising: a processor, a memory, and a computer program stored on the memory and executable on the processor; when the computer program is executed by the processor, the steps of the information reasoning method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the information reasoning method as described in the first aspect are implemented.

[0019] In an embodiment of the present application, an information reasoning method, apparatus, computer device, and storage medium are provided, which utilize a multi-agent debate framework to promote collaboration through task debates between agents, and learn the team joint actions of the agent decision network to guide the update of individual agent strategies, so that each agent can incrementally contribute to the task solution when faced with complex tasks, thereby improving the accuracy and efficiency of the reasoning of the agent decision network.

[0020] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of the information reasoning method provided in the first embodiment of the present application;

[0022] Figure 2 This is a flowchart of S2 in the information reasoning method provided in the first embodiment of the present application;

[0023] Figure 3 This is a flowchart of S3 in the information reasoning method provided in the first embodiment of the present application;

[0024] Figure 4 This is a flowchart of S4 in the information reasoning method provided in the first embodiment of the present application;

[0025] Figure 5 This is a flowchart of S42 in the information reasoning method provided in the first embodiment of the present application;

[0026] Figure 6 This is a flowchart of S42 in the information reasoning method provided in the first embodiment of the present application;

[0027] Figure 7 This is a flowchart of S5 in the information reasoning method provided in the third embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of an information reasoning device based on an agent decision network provided in the fourth embodiment of the present application;

[0029] Figure 9 A schematic structural diagram of a computer device provided in the fifth embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination."

[0033] The executor of the information reasoning method is the reasoning device of the information reasoning method (hereinafter referred to as the reasoning device). The reasoning device can be implemented by software and / or hardware, and the information reasoning method can be implemented by software and / or hardware. The reasoning device can be composed of two or more physical entities, or it can be composed of one physical entity. The hardware pointed to by the reasoning device essentially refers to computer equipment. For example, the reasoning device can be a computer, a mobile phone, a tablet or an interactive tablet. In an optional embodiment, the reasoning device can specifically be a server, or a server cluster composed of multiple computer devices.

[0034] See also Figure 1 , Figure 1 This is a flowchart of the information reasoning method provided in the first embodiment of the present application, the method comprising the following steps:

[0035] S1: Obtain the state space information of several agents in the current time slot of the agent decision network.

[0036] The agent decision network includes a plurality of agents. In the agent decision network, each agent will perform multiple rounds of debate and collaboration on the same task, and each round serves as a time slot.

[0037] In this embodiment, the reasoning device obtains the state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes the question text information and the agent conversation history information from the initial time slot to the current time slot. The agent conversation history information is the text information record generated by the task debate between the agent and other agents, which is recorded as s t ={u 0 ,u 1 ,...,u t-1}, s t is the state space information of the agent in the tth time slot, u t-1 is the agent conversation history information of the t-1th time slot, u 0 is the agent conversation history information of the initial time slot.

[0038] S2: According to the state space information of several agents and the corresponding current time slot, the action space information of several agents in the current time slot is obtained.

[0039] In this embodiment, the reasoning device obtains the action space information of several agents in the current time slot based on the state space information of several agents and the corresponding current time slot, wherein the action space information is the answer text information generated by the agent based on the state space information, and the action space information includes several position index tokens, which represent the characters corresponding to the nth position index in the answer text information, recorded as is the action space information of the ith agent in the tth time slot, w n The token indexed at the nth position indicates the character corresponding to the nth position index in the answer text information.

[0040] The agent includes an actor network; the actor network is used to receive the state space information of the agent in the current time slot and generate corresponding action space information. Figure 2 , Figure 2 The flowchart of S2 in the information reasoning method provided in the first embodiment of the present application includes step S21, which is as follows:

[0041] S21: Input the state space information of the multiple agents in the current time slot into the actor network in the corresponding agent respectively, make action decisions according to the preset strategy, and obtain the action space information of the multiple agents in the current time slot.

[0042] The actor network is a model built on the basis of the large language model (LLM) architecture.

[0043] In this embodiment, the inference device inputs the state space information of several agents in the current time slot into the actor network in the corresponding agent respectively. The actor network perceives the agent session history information in the state space information, makes action decisions according to the preset strategy, and obtains the action space information of several agents in the current time slot.

[0044] S3: Calculate rewards based on the action space information of the multiple agents in the current time slot to obtain reward information of the multiple agents in the current time slot; combine the state space information, action space information and reward information of the current time slot of the same agent to construct a training information combination of the multiple agents in the current time slot.

[0045] In this embodiment, the inference device calculates rewards based on the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot; the state space information, action space information and reward information of the current time slot of the same agent are combined to construct a training information combination of the multiple agents in the current time slot.

[0046] See also Figure 3 , Figure 3 The flowchart of S3 in the information reasoning method provided in the first embodiment of the present application includes steps S31 to S32, which are specifically as follows:

[0047] S31: Obtain standard answer information of several intelligent agents.

[0048] In this embodiment, the inference device obtains standard answer information of several of the intelligent agents, wherein the standard answer information is answer text information corresponding to the question text information in the corresponding state space information.

[0049] S32: Obtain reward information of the multiple agents in the current time slot based on the action space information, standard answer information and a preset reward calculation algorithm of the multiple agents in the current time slot.

[0050] The reward calculation algorithm is:

[0051]

[0052] Where, is the reward information of the ith agent in the tth time slot, ε is a preset minimum number, is the action space information of the i-th agent in the t-th time slot, GT i | is the standard answer information of the i-th agent.

[0053] In this embodiment, the reasoning device obtains the reward information of the multiple agents in the current time slot based on the action space information, standard answer information and a preset reward calculation algorithm of the multiple agents in the current time slot.

[0054] S4: Conduct task debate based on several intelligent agents and the corresponding action space information of the current time slot to obtain the state space information of several intelligent agents in the next time slot; repeatedly construct training information combinations based on the state space information of several intelligent agents in the next time slot to obtain training information combinations for several time slots; update several intelligent agents in the intelligent agent decision network based on the training information combinations for the several time slots.

[0055] In this embodiment, the reasoning device performs task debate based on several agents and the corresponding action space information of the current time slot to obtain the state space information of several agents in the next time slot.

[0056] The inference device repeatedly constructs a training information combination based on the state space information of the multiple agents in the next time slot to obtain the training information combination of the multiple time slots. The inference device updates the multiple agents in the agent decision network based on the training information combination of the multiple time slots.

[0057] The agent also includes a critic network; the critic network is used to evaluate the value of the agent in task debate. Figure 4 , Figure 4 The flowchart of S4 in the information reasoning method provided in the first embodiment of the present application includes steps S41 to S42, which are specifically as follows:

[0058] S41: Input the state space information of several agents in the training information combination of several time slots into the critic network of the corresponding agents respectively, and obtain the individual value parameters of several agents in several time slots according to the preset first value calculation algorithm.

[0059] The first value calculation algorithm is:

[0060]

[0061] Where, is the individual value parameter of the ith agent in the tth time slot, V ω () is the critic network function, is the state space information of the i-th agent in the t-th time slot.

[0062] In this embodiment, the inference device inputs the state space information of several agents in the training information combination of several time slots into the critic network of the corresponding agents, and obtains the individual value parameters of several agents in several time slots according to the preset first value calculation algorithm.

[0063] S42: performing loss calculation based on the training information combination of several time slots and the individual value parameters of several agents, and updating the actor networks of several agents in the agent decision network based on the obtained loss values.

[0064] In this embodiment, the inference device calculates the loss based on the combination of the training information of several time slots and the individual value parameters of several agents, and updates the actor network of several agents in the agent decision network according to the obtained loss value.

[0065] See also Figure 5 , Figure 5 The flowchart of S42 in the information reasoning method provided in the first embodiment of the present application includes steps S421 to S422, which are specifically as follows:

[0066] S421: Obtain supervised fine-tuning loss values ​​for the plurality of agents based on the standard answer information of the plurality of agents, the state space information and action space information of the plurality of agents in the training information combination of the last time slot, and a preset supervised fine-tuning loss algorithm, wherein the supervised fine-tuning loss algorithm is:

[0067]

[0068] Where, L SFTi is the supervised fine-tuning loss value of the i-th agent, M is the number of tokens in the action space information, () is the function corresponding to the strategy, y i is the standard answer information of the i-th agent, is the token indexed by the jth position in the action space information of the i-th agent in the last time slot, is the state space information of the i-th agent in the last time slot.

[0069] In this embodiment, the inference device obtains the supervised fine-tuning loss values ​​of several of the intelligent agents based on the standard answer information of several of the intelligent agents, the state space information, action space information of several of the intelligent agents in the training information combination of the last time slot, and the preset supervised fine-tuning loss algorithm.

[0070] S422: Obtain a first loss value based on the supervised fine-tuning loss values ​​of the several agents, the individual value parameters of the several agents in the last time slot, and the preset first loss algorithm, and update the actor network of the several agents in the agent decision network based on the first loss value.

[0071] The first loss algorithm is:

[0072]

[0073] Where, L a is the first loss value, N is the number of agents in the agent decision network, is the individual value parameter of the i-th agent in the last time slot.

[0074] In this embodiment, the inference device obtains a first loss value based on the standard answer information of several of the agents, the state space information, action space information, individual value parameters of several agents in the training information combination of the last time slot, and a preset first loss algorithm, and updates the actor network of several of the agents in the agent decision network based on the first loss value.

[0075] The goal of the actor network is to maximize the output of its critic network. At the same time, in order to ensure that the actor network retains its ability to solve specific tasks, the inference device combines the supervised fine-tuning loss related to the downstream task to calculate a first loss value, updates the actor networks of several agents in the agent decision network, optimizes the agents with the goal of individual optimality, and improves the accuracy and efficiency of task debate.

[0076] In an optional embodiment, the agent decision network further includes a hybrid network, which is used to ensure consistency among multiple agents. Figure 6 , Figure 6The flowchart of S42 in the information reasoning method provided in the second embodiment of the present application includes steps S423 to S425, which are specifically as follows:

[0077] S423: Combine the individual value parameters of several intelligent agents in the same time slot to construct a first individual value parameter set of the intelligent agent decision network of several time slots; compare the individual value parameters of several intelligent agents in the same time slot to obtain the maximum individual value parameters of several time slots, replace the individual value parameters of several intelligent agents in several time slots with the maximum individual value parameters of the corresponding time slots, combine the individual value parameters of several intelligent agents in the same time slot after replacement, and construct a second individual value parameter set of the intelligent agent decision network of several time slots.

[0078] In this embodiment, the reasoning device combines the individual value parameters of several intelligent agents in the same time slot to construct the first individual value parameter set of the intelligent agent decision network of the several time slots.

[0079] The reasoning device compares the individual value parameters of several intelligent agents in the same time slot, obtains the maximum individual value parameters of several time slots, replaces the individual value parameters of several intelligent agents in several time slots with the maximum individual value parameters of the corresponding time slots, combines the replaced individual value parameters of several intelligent agents in the same time slot, and constructs a second individual value parameter set of the intelligent agent decision network of several time slots.

[0080] S424: The first body value parameter set and the second body value parameter set of the intelligent agent decision network of several time slots are respectively used as the input parameter set of the hybrid network, and the global value parameter is calculated according to the input parameter set to obtain the first global value parameter and the second global value parameter of the intelligent agent decision network of several time slots.

[0081] The hybrid network consists of several fully connected layers whose parameters are determined by the hypernetwork. The hypernetwork input is a natural language text H describing the problem to be solved. To map H to the same vector space, H is first input to the critic network to obtain its vector representation H. This vector is then used with a linear layer to calculate the parameters w and b of the hybrid network. The hybrid network ensures consistency between each agent and all agents in the agent decision network, ensuring that each agent effectively contributes to the agent decision network's team task.

[0082] In this embodiment, the inference device uses the first body value parameter set and the second body value parameter set of the intelligent agent decision network of several time slots as the input parameter set of the hybrid network, performs global value parameter calculation based on the input parameter set, and obtains the first global value parameter and the second global value parameter of the intelligent agent decision network of several time slots to promote the coordinated optimization of the entire intelligent agent decision network.

[0083] In an optional embodiment, the inference device constructs a consistency constraint, calculates a global value parameter based on the input parameter set and the consistency constraint, and obtains a first global value parameter and a second global value parameter of the agent decision network for a plurality of time slots, so as to ensure that the obtained first global value parameter is proportional to the individual value parameters of a plurality of agents in the plurality of time slots, and the obtained second global value parameter is proportional to the maximum individual value parameter of the corresponding time slot, so as to indicate that when the individual value parameter of each agent reaches its maximum individual value, the obtained first global value parameter and the obtained second global value parameter also reach their maximum individual value, wherein the consistency constraint is:

[0084]

[0085] Where Q tot is the global value parameter, Q i is the individual value parameter of the i-th agent.

[0086] S425: Accumulate the reward information of several agents in the same time slot to obtain the cumulative reward information of the agent decision network for several time slots; obtain the second loss value based on the cumulative reward information of the agent decision network for several time slots, the first global value parameter and the second global value parameter and the preset second loss algorithm, and update the actor network of several agents in the agent decision network based on the first loss value and the second loss value.

[0087] In this embodiment, the inference device accumulates the reward information of several agents in the same time slot to obtain the cumulative reward information of the agent decision network in several time slots.

[0088] The inference device obtains a second loss value based on the accumulated reward information of the agent decision network in several time slots, the first global value parameter, the second global value parameter, and a preset second loss algorithm, and updates the actor networks of the several agents in the agent decision network based on the first loss value and the second loss value, wherein the second loss algorithm is:

[0089]

[0090] Where, L mix is, T is the total number of time slots, R t is the cumulative reward information of the agent decision network at the tth time slot, γ is the discount factor, is the first global value parameter of the agent decision network at the t-th time slot, is the second global value parameter of the agent decision network at the tth time slot.

[0091] By introducing the loss calculated by the hybrid network to update the agent's actor network, it promotes coordination between multiple agents within the agent decision network to promote global optimization and ensure that the action space information generated by each agent contributes optimally and consistently to solving the problem, thereby ensuring that each agent contributes incrementally to the task solution and improving the accuracy and efficiency of the task debate of the agent decision network.

[0092] S5: Obtain the text information of the problem to be processed, input the text information of the problem to be processed into several updated intelligent agents in the intelligent agent decision network respectively, iterate repeatedly according to the preset number of iterations, obtain the answer text information output by several updated intelligent agents in the last number of iterations, and use the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

[0093] In this embodiment, the inference device obtains text information of the question to be processed, wherein the text information of the question to be processed may be input by a user or obtained from a preset database.

[0094] The inference device inputs the text information of the problem to be processed into several updated intelligent agents in the intelligent agent decision network respectively, and iterates repeatedly according to the preset number of iterations to obtain the answer text information output by several updated intelligent agents for the last number of iterations, and takes the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

[0095] Utilizing a multi-agent debate framework, we promote collaboration through task debates between agents, and learn the team joint actions of the agent decision network to guide the update of individual agent strategies. This enables each agent to incrementally contribute to the task solution when faced with complex tasks, thereby improving the accuracy and efficiency of the agent decision network's reasoning.

[0096] In an optional embodiment, the step S:6 is further included: training the critic network of the plurality of agents; the step S6 is performed before the step S1, see Figure 7 , Figure 7The flowchart of S6 in the information reasoning method provided in the third embodiment of the present application includes steps S61 to S63, which are specifically as follows:

[0097] S61: Obtain state space information and reward information of several agents in several sample time slots. Based on each sample time slot, accumulate the reward information of several sample time slots before the sample time slot of the same agent to obtain the cumulative reward information of several agents in several sample time slots.

[0098] In this embodiment, the inference device obtains the state space information and reward information of several intelligent agents for several sample time slots, and based on each sample time slot, accumulates the reward information of several sample time slots before the sample time slot of the same intelligent agent to obtain the cumulative reward information of several intelligent agents for several sample time slots.

[0099] S62: Input the state space information and cumulative reward information of several agents in several sample time slots into the critic network of the corresponding agents respectively, and obtain the individual value parameters of several agents in several sample time slots according to the preset second value calculation algorithm.

[0100] In order to better evaluate the status of the actor network, in this embodiment, the critic network to be trained of the agent is built based on the same large language model architecture as the actor network, and a linear layer is added after the last layer of the large language model.

[0101] In this embodiment, the inference device inputs the state space information and accumulated reward information of several agents in several sample time slots into the critic network of the corresponding agents, and obtains the individual value parameters of the several agents in the several sample time slots according to a preset second value calculation algorithm, wherein the second value calculation algorithm is:

[0102]

[0103] Where, is the individual value parameter of the ith agent in the tth sample time slot, E[] is the calculation expectation function, is the cumulative reward information of the ith agent in the tth sample time slot, s is the state space information, is the state space information of the i-th agent in the t-th sample time slot.

[0104] S63: Obtain a third loss value based on the individual value parameters, reward information and a preset third loss algorithm of several agents in several sample time slots, and train the critic network of the corresponding agent based on the third loss value.

[0105] The third loss algorithm is:

[0106]

[0107] Where, is the third loss value, are the parameters of the critic network, is the reward information of the ith agent in the tth sample time slot, is the individual value parameter of the i-th agent in the t+1-th time slot.

[0108] In this embodiment, the inference device obtains a third loss value based on the individual value parameters, reward information and a preset third loss algorithm of several intelligent agents in several sample time slots, and trains the critic network of the corresponding intelligent agent based on the third loss value.

[0109] The third loss value is calculated using the temporal mean square error, and the critic network of the corresponding intelligent agent is trained based on the third loss values ​​of several of the intelligent agents, thereby improving the accuracy of the critic network in evaluating the value of the intelligent agent in conducting task debates, thereby improving the accuracy of the update of the intelligent agent's actor network.

[0110] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of an information reasoning device based on an agent decision network provided in the fourth embodiment of the present application. The device can implement all or part of the information reasoning device based on the agent decision network through software, hardware, or a combination of both. The device 8 includes:

[0111] A data acquisition module 81 is configured to obtain state space information of multiple agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot;

[0112] The task debate model 82 is used to obtain action space information of the plurality of agents in the current time slot based on the state space information of the plurality of agents and the corresponding current time slot;

[0113] The training information combination construction module 83 is configured to calculate rewards based on the action space information of the multiple agents in the current time slot to obtain reward information for the multiple agents in the current time slot; and to combine the state space information, action space information, and reward information of the same agent in the current time slot to construct a training information combination for the multiple agents in the current time slot.

[0114] The agent updating module 84 is configured to cause the inference device to perform task debate based on the action space information of the multiple agents and the corresponding current time slot to obtain the state space information of the multiple agents in the next time slot; repeatedly construct training information combinations based on the state space information of the multiple agents in the next time slot to obtain training information combinations for multiple time slots; and update the multiple agents in the agent decision network based on the training information combinations for the multiple time slots;

[0115] The information reasoning module 85 is used to obtain the text information of the problem to be processed, input the text information of the problem to be processed into the updated several intelligent agents in the intelligent agent decision network respectively, and iterate repeatedly according to the preset number of iterations to obtain the answer text information output by the updated several intelligent agents of the last iteration number, and use the answer text information with the highest frequency as the reasoning result of the text information of the problem to be processed.

[0116] In an embodiment of the present application, the state space information of several agents in the current time slot of the agent decision network is obtained through a data acquisition module, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot; the action space information of several agents in the current time slot is obtained based on the state space information of several agents and the corresponding current time slot through a task debate model; the reward calculation is performed based on the action space information of several agents in the current time slot through a training information combination construction module to obtain the reward information of several agents in the current time slot; the state space information, action space information and reward information of the current time slot of the same agent are combined to construct a training information combination of several agents in the current time slot; the training information combination of several agents in the current time slot is constructed based on the state space information of several agents and the reward information of the same agent. The agents and their corresponding action space information for the current time slot conduct task debates to obtain state space information for several agents in the next time slot. Based on the state space information for the agents in the next time slot, training information combinations are repeatedly constructed to obtain training information combinations for several time slots. Based on the training information combinations for the several time slots, several agents in the agent decision network are updated. Through an information reasoning module, the problem text information to be processed is obtained and input into the updated agents in the agent decision network. Repeated iterations are performed according to a preset number of iterations to obtain answer text information output by the updated agents in the last iteration. The answer text information with the highest frequency is used as the reasoning result for the problem text information to be processed. A multi-agent debate framework is used to promote collaboration through task debates between agents. The team-coordinated actions of the agent decision network are learned to guide the update of individual agent strategies. This allows each agent to incrementally contribute to the solution of complex tasks, thereby improving the accuracy and efficiency of the agent decision network's reasoning.

[0117] Please refer to Figure 9 , Figure 9 The structural diagram of the computer device provided for the fifth embodiment of the present application is as follows: the computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device can store a plurality of instructions, which are suitable for being loaded by the processor 91 and executing the method steps shown in the first to third embodiments above. For the specific execution process, please refer to the specific description shown in the first to third embodiments, which will not be repeated here.

[0118] The processor 91 may include one or more processing cores. The processor 91 utilizes various interfaces and circuits to connect to various components within the server. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 92 and accessing data within the memory 92, the processor 91 performs various functions and processes data of the information reasoning device 8 based on the agent decision network. Optionally, the processor 91 may be implemented in the form of at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 91 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 91 and may be implemented as a separate chip.

[0119] Among them, the memory 92 may include a random access memory 92 (Random Access Memory, RAM), and may also include a read-only memory 92 (Read-Only Memory). Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 92 may also optionally be at least one storage device located away from the aforementioned processor 91.

[0120] An embodiment of the present application also provides a storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing the method steps shown in the first to third embodiments above. The specific execution process can be found in the specific descriptions shown in the first to third embodiments, which will not be repeated here.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0122] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0123] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraint algorithm of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0124] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0127] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form.

[0128] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications of the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims of the present invention and equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. An information reasoning method, wherein the agent decision network includes a plurality of agents; characterized in that: The method comprises the following steps: Obtaining state space information of multiple agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot; Obtaining action space information of the multiple agents in the current time slot according to the multiple agents and the corresponding state space information of the current time slot, wherein the action space information is answer text information generated by the agents based on the state space information; Reward calculation is performed based on the action space information of the multiple agents in the current time slot to obtain reward information of the multiple agents in the current time slot; the state space information, action space information, and reward information of the current time slot of the same agent are combined to construct a training information combination of the multiple agents in the current time slot; Performing task debate based on the action space information of the multiple agents and the corresponding current time slot to obtain state space information of the multiple agents in the next time slot; repeatedly constructing training information combinations based on the state space information of the multiple agents in the next time slot to obtain training information combinations for multiple time slots; and updating the multiple agents in the agent decision network based on the training information combinations for the multiple time slots; Obtain text information of the problem to be processed, input the text information of the problem to be processed into several updated agents in the agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain answer text information output by several updated agents in the last number of iterations, and use the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

2. The information inference method according to claim 1, characterized in that: The agent includes an actor network; the actor network is used to receive the state space information of the agent in the current time slot and generate corresponding action space information; The method of obtaining action space information of several agents in the current time slot based on the state space information of several agents and the corresponding current time slot includes the following steps: The state space information of the multiple agents in the current time slot is respectively input into the actor network in the corresponding agent, and action decisions are made according to the preset strategy to obtain the action space information of the multiple agents in the current time slot.

3. The information inference method according to claim 2, characterized in that: The reward calculation is performed based on the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot, including the steps of: Obtaining standard answer information of a plurality of the intelligent agents, wherein the standard answer information is answer text information corresponding to the question text information in the corresponding state space information; According to the action space information, standard answer information and preset reward calculation algorithm of the multiple agents in the current time slot, the reward information of the multiple agents in the current time slot is obtained, wherein the reward calculation algorithm is: Where r i t is the reward information of the ith agent in the tth time slot, ε is a preset minimum number, is the action space information of the i-th agent in the t-th time slot, GT i is the standard answer information of the i-th agent.

4. The information inference method according to claim 3, characterized in that: The agent also includes a critic network; the critic network is used to evaluate the value of the agent in performing task debate; The updating of the plurality of agents in the agent decision network according to the combination of the training information of the plurality of time slots comprises the steps of: The state space information of several agents in the training information combination of several time slots is respectively input into the critic network of the corresponding agent, and the individual value parameters of several agents in the several time slots are obtained according to a preset first value calculation algorithm, wherein the first value calculation algorithm is: Where, is the individual value parameter of the ith agent in the tth time slot, V ω () is the critic network function, is the state space information of the i-th agent in the t-th time slot; Loss calculation is performed based on the training information combination of several time slots and the individual value parameters of several agents, and the actor networks of several agents in the agent decision network are updated based on the obtained loss values.

5. The information inference method according to claim 4, characterized in that: The action space information includes several position index tokens, which represent the character corresponding to the nth position index in the answer text information; The loss calculation is performed based on the combination of the training information of the plurality of time slots and the individual value parameters of the plurality of agents, and the actor networks of the plurality of agents in the agent decision network are updated according to the obtained loss values, including the steps of: According to the standard answer information of the plurality of agents, the state space information and action space information of the plurality of agents in the training information combination of the last time slot, and a preset supervised fine-tuning loss algorithm, the supervised fine-tuning loss values ​​of the plurality of agents are obtained, wherein the supervised fine-tuning loss algorithm is: Where, L SFTi is the supervised fine-tuning loss value of the i-th agent, M is the number of tokens in the action space information, is the function corresponding to the strategy, y i is the standard answer information of the i-th agent, is the token indexed by the jth position in the action space information of the i-th agent in the last time slot, is the state space information of the i-th agent in the last time slot; A first loss value is obtained based on the supervised fine-tuning loss values ​​of the plurality of agents, the individual value parameters of the plurality of agents in the last time slot, and a preset first loss algorithm. The actor networks of the plurality of agents in the agent decision network are updated based on the first loss value, wherein the first loss algorithm is: Where, L a is the first loss value, N is the number of agents in the agent decision network, is the individual value parameter of the i-th agent in the last time slot.

6. The information inference method according to claim 5, characterized in that: The agent decision network further includes a hybrid network, wherein the hybrid network is used to ensure consistency among the multiple agents; The method further includes the steps of performing loss calculation based on the combination of the training information of the plurality of time slots and the individual value parameters of the plurality of agents, and updating the actor networks of the plurality of agents in the agent decision network according to the obtained loss values: The individual value parameters of several intelligent agents in the same time slot are combined to construct a first set of individual value parameters of the intelligent agent decision network of the several time slots; the individual value parameters of several intelligent agents in the same time slot are compared to obtain the maximum individual value parameters of the several time slots, the individual value parameters of several intelligent agents in the several time slots are replaced with the maximum individual value parameters of the corresponding time slots, and the individual value parameters of several intelligent agents in the same time slot after replacement are combined to construct a second set of individual value parameters of the intelligent agent decision network of the several time slots; The first and second agent value parameter sets of the agent decision network of the plurality of time slots are respectively used as input parameter sets of the hybrid network, and a global value parameter is calculated based on the input parameter sets to obtain the first and second global value parameters of the agent decision network of the plurality of time slots; The reward information of several agents in the same time slot is accumulated to obtain the cumulative reward information of the agent decision network in the several time slots; a second loss value is obtained according to the cumulative reward information of the agent decision network in the several time slots, the first global value parameter and the second global value parameter and a preset second loss algorithm; and the actor network of the several agents in the agent decision network is updated according to the first loss value and the second loss value, wherein the second loss algorithm is: Where, L mix is the second loss value, T is the total number of time slots, R t is the cumulative reward information of the agent decision network at the tth time slot, γ is the discount factor, is the first global value parameter of the agent decision network at the t-th time slot, is the second global value parameter of the agent decision network at the tth time slot.

7. The information inference method according to claim 6, characterized in that: The method further comprises the steps of: training a critic network of a plurality of said intelligent agents; said training of the critic network of a plurality of said intelligent agents comprises the steps of: Obtaining state space information and reward information of a plurality of agents for a plurality of sample time slots, and accumulating reward information of a plurality of sample time slots preceding the sample time slot of the same agent for each sample time slot to obtain cumulative reward information of the plurality of agents for the plurality of sample time slots; The state space information and accumulated reward information of several agents in several sample time slots are respectively input into the critic network of the corresponding agents, and the individual value parameters of the several agents in several sample time slots are obtained according to the preset second value calculation algorithm, wherein the second value calculation algorithm is: Where, is the individual value parameter of the ith agent in the tth sample time slot, E[] is the calculation expectation function, is the cumulative reward information of the ith agent in the tth sample time slot, s is the state space information, is the state space information of the i-th agent in the t-th sample time slot; According to the individual value parameters, reward information and a preset third loss algorithm of a plurality of agents in a plurality of sample time slots, a third loss value is obtained, and the critic network of the corresponding agent is trained according to the third loss value, wherein the third loss algorithm is: Where, is the third loss value, are the parameters of the critic network, is the reward information of the ith agent in the tth sample time slot, is the individual value parameter of the i-th agent in the t+1-th time slot.

8. An information reasoning device, wherein the agent decision network includes a plurality of agents, characterized in that: include: A data acquisition module is used to obtain state space information of multiple agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot; A task debate model is configured to obtain action space information of the plurality of agents in the current time slot based on the state space information of the plurality of agents and the corresponding current time slot, wherein the action space information is answer text information generated by the agents based on the state space information; A training information combination construction module is used to calculate rewards based on the action space information of the multiple agents in the current time slot to obtain reward information of the multiple agents in the current time slot; and to combine the state space information, action space information, and reward information of the same agent in the current time slot to construct a training information combination of the multiple agents in the current time slot; An agent updating module is configured to perform task debate based on the action space information of the multiple agents and the corresponding current time slot to obtain the state space information of the multiple agents in the next time slot; repeatedly construct training information combinations based on the state space information of the multiple agents in the next time slot to obtain training information combinations for multiple time slots; and update the multiple agents in the agent decision network based on the training information combinations for the multiple time slots; The information reasoning module is used to obtain the text information of the problem to be processed, input the text information of the problem to be processed into several updated intelligent agents in the intelligent agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain the answer text information output by several updated intelligent agents in the last number of iterations, and use the answer text information with the highest frequency as the reasoning result of the text information of the problem to be processed.

9. A computer device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the information reasoning method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the information reasoning method according to any one of claims 1 to 7 are implemented.

Images