Information reasoning method and device

Abstract

The invention relates to the technical field of large-scale language models, in particular to an information reasoning method and device, computer equipment and a storage medium, which utilize a multi-agent debate framework, promote cooperation through task debate among agents, learn team joint actions of an agent decision network to guide updating of a single agent strategy, and improve the strategy updating efficiency. Therefore, each agent can make an incremental contribution to a task solution when facing a complex task, and the reasoning accuracy and efficiency of the agent decision network are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale language models, and in particular to an information reasoning method, device, computer equipment and storage medium. Background Art

[0002] In recent years, large language models (LLMs) have demonstrated extraordinary language generation, understanding, and few-shot learning capabilities. A large amount of recent work has focused on exploring how to improve the factual accuracy and reasoning capabilities of large models. On the one hand, using techniques such as prompt learning, thought chaining, or self-consistency can improve the performance of large models without adjusting the parameters of large models. On the other hand, the latest large model training techniques such as human feedback reinforcement learning can also significantly improve the capabilities of large models.

[0003] Supervised fine-tuning (SFT) is a widely used and effective technique to improve the capabilities of large models. This method enhances the ability of large models to complete complex tasks. Compared with the SFT method, reinforcement learning techniques such as proximal policy optimization (PPO) and direct preference optimization (DPO) provide new tuning strategies for large models. However, the above methods only focus on tuning a single large model agent, and a single large model agent is easily affected by problems such as model bias and thinking degradation during the reasoning process. It is difficult to form multiple insights into the problem due to the lack of external feedback, and thus has limited ability to face complex tasks. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide an information reasoning method, apparatus, computer equipment and storage medium, which utilize a multi-agent debate framework to promote collaboration through task debates between agents, learn the team joint actions of the agent decision network to guide the update of a single agent strategy, and enable each agent to incrementally contribute to the task solution when faced with complex tasks, thereby improving the accuracy and efficiency of the reasoning of the agent decision network.

[0005] In a first aspect, an embodiment of the present application provides an information reasoning method, wherein an agent decision network includes a plurality of agents; the method comprises the following steps:

[0006] Obtaining state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot;

[0007] According to the state space information of several agents and the corresponding current time slot, the action space information of several agents in the current time slot is obtained, and according to the action space information of several agents and the corresponding current time slot, the task debate is performed to obtain the state space information of several agents in the next time slot;

[0008] Reward calculation is performed according to the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot; the state space information, action space information and reward information of the current time slot of the same agent are combined to construct the training information combination of the multiple agents in the current time slot;

[0009] Performing task debate according to the action space information of the several agents and the corresponding current time slot to obtain the state space information of the several agents in the next time slot; repeatedly constructing the training information combination according to the state space information of the several agents in the next time slot to obtain the training information combination of the several time slots; updating the several agents in the agent decision network according to the training information combination of the several time slots;

[0010] Obtain text information of the problem to be processed, input the text information of the problem to be processed into several updated agents in the agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain answer text information output by several updated agents at the last iteration number, and use the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

[0011] In a second aspect, an embodiment of the present application provides an information reasoning device based on an agent decision network, comprising:

[0012] A data acquisition module, used to obtain state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot;

[0013] The task debate model is used to obtain the action space information of several agents in the current time slot according to the state space information of several agents and the corresponding current time slot;

[0014] A training information combination construction module is used to calculate rewards according to the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot; combine the state space information, action space information and reward information of the current time slot of the same agent to construct a training information combination of the multiple agents in the current time slot;

[0015] The agent updating module is used to perform task debate according to the action space information of the several agents and the corresponding current time slot to obtain the state space information of the several agents in the next time slot; repeatedly construct the training information combination according to the state space information of the several agents in the next time slot to obtain the training information combination of the several time slots; and update the several agents in the agent decision network according to the training information combination of the several time slots;

[0016] The information reasoning module is used to obtain the text information of the problem to be processed, input the text information of the problem to be processed into the updated several agents in the agent decision network respectively, iterate repeatedly according to the preset number of iterations, obtain the answer text information output by the updated several agents at the last iteration number, and use the answer text information with the highest frequency as the reasoning result of the text information of the problem to be processed.

[0017] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the information reasoning method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the information reasoning method described in the first aspect are implemented.

[0019] In an embodiment of the present application, an information reasoning method, apparatus, computer device and storage medium are provided, which utilize a multi-agent debate framework to promote collaboration through task debates between agents, and learn the team joint actions of the agent decision network to guide the update of the strategy of a single agent, so that each agent can incrementally contribute to the task solution when faced with complex tasks, thereby improving the accuracy and efficiency of the reasoning of the agent decision network.

[0020] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of the information reasoning method provided in the first embodiment of the present application;

[0022] Figure 2 A schematic diagram of the process of S2 in the information reasoning method provided in the first embodiment of the present application;

[0023] Figure 3 A schematic diagram of the process of S3 in the information reasoning method provided in the first embodiment of the present application;

[0024] Figure 4 A schematic diagram of the process of S4 in the information reasoning method provided in the first embodiment of the present application;

[0025] Figure 5 A schematic diagram of the process of S42 in the information reasoning method provided in the first embodiment of the present application;

[0026] Figure 6 A schematic diagram of the process of S42 in the information reasoning method provided in the first embodiment of the present application;

[0027] Figure 7 A schematic diagram of the process of S5 in the information reasoning method provided in the third embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of an information reasoning device based on an agent decision network provided in the fourth embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a computer device provided in the fifth embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0033] The executor of the information reasoning method is the reasoning device of the information reasoning method (hereinafter referred to as the reasoning device). The reasoning device can be implemented by software and / or hardware, and the information reasoning method can be implemented by software and / or hardware. The reasoning device can be composed of two or more physical entities, or it can be composed of one physical entity. The hardware pointed to by the reasoning device essentially refers to computer equipment. For example, the reasoning device can be a computer, a mobile phone, a tablet or an interactive tablet. In an optional embodiment, the reasoning device can specifically be a server, or a server cluster composed of multiple computer devices.

[0034] See also Figure 1 , Figure 1 The present invention provides a flowchart of an information reasoning method according to the first embodiment of the present invention. The method comprises the following steps:

[0035] S1: Obtain the state space information of several agents in the current time slot of the agent decision network.

[0036] The agent decision network includes a plurality of agents. In the agent decision network, each agent will perform multiple rounds of debate and collaboration on the same task, and each round serves as a time slot.

[0037] In this embodiment, the reasoning device obtains the state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes the question text information and the agent conversation history information from the initial time slot to the current time slot, and the agent conversation history information is the text information record generated by the agent debating with other agents on the task, which is recorded as s t = {u 0 ,u 1 ,...,u t-1},s t is the state space information of the agent in the tth time slot, u t-1 is the agent conversation history information of the t-1th time slot, u 0 is the agent conversation history information of the initial time slot.

[0038] S2: According to the state space information of the several agents and the corresponding current time slot, the action space information of the several agents in the current time slot is obtained.

[0039] In this embodiment, the reasoning device obtains the action space information of several agents in the current time slot according to the state space information of several agents and the corresponding current time slot, wherein the action space information is the answer text information generated by the agent based on the state space information, and the action space information includes several position index tokens, which represent the characters corresponding to the nth position index in the answer text information, and are recorded as is the action space information of the ith agent in the tth time slot, w n The token indexed at the nth position indicates the character corresponding to the index at the nth position in the answer text information.

[0040] The agent includes an actor network; the actor network is used to receive the state space information of the agent in the current time slot and generate corresponding action space information. Figure 2 , Figure 2 The flowchart of S2 in the information reasoning method provided in the first embodiment of the present application includes step S21, which is as follows:

[0041] S21: respectively inputting the state space information of the several agents in the current time slot into the actor network in the corresponding agent, making action decisions according to a preset strategy, and obtaining the action space information of the several agents in the current time slot.

[0042] The actor network is a model built on the basis of the large language model (LLM) architecture.

[0043] In this embodiment, the inference device inputs the state space information of several agents in the current time slot into the actor network in the corresponding agent respectively. The actor network perceives the agent session history information in the state space information, makes action decisions according to preset strategies, and obtains the action space information of several agents in the current time slot.

[0044] S3: Calculate rewards based on the action space information of the multiple agents in the current time slot to obtain reward information of the multiple agents in the current time slot; combine the state space information, action space information and reward information of the current time slot of the same agent to construct a training information combination of the multiple agents in the current time slot.

[0045] In this embodiment, the inference device calculates rewards based on the action space information of the several agents in the current time slot to obtain the reward information of the several agents in the current time slot; combines the state space information, action space information and reward information of the current time slot of the same agent to construct a training information combination of the several agents in the current time slot.

[0046] See also Figure 3 , Figure 3 The flowchart of S3 in the information reasoning method provided in the first embodiment of the present application includes steps S31 to S32, which are as follows:

[0047] S31: Obtain standard answer information of several of the intelligent agents.

[0048] In this embodiment, the reasoning device obtains standard answer information of several of the intelligent agents, wherein the standard answer information is answer text information corresponding to the question text information in the corresponding state space information.

[0049] S32: Obtain reward information of several intelligent agents in the current time slot according to the action space information of several intelligent agents in the current time slot, standard answer information and a preset reward calculation algorithm.

[0050] The reward calculation algorithm is:

[0051]

[0052] In the formula, is the reward information of the ith agent in the tth time slot, ε is a preset minimum number, is the action space information of the ith agent in the tth time slot, GT i | is the standard answer information of the i-th agent.

[0053] In this embodiment, the reasoning device obtains the reward information of the multiple agents in the current time slot according to the action space information of the multiple agents in the current time slot, the standard answer information and the preset reward calculation algorithm.

[0054] S4: Conduct task debate based on several agents and the corresponding action space information of the current time slot to obtain the state space information of several agents in the next time slot; repeatedly construct training information combinations based on the state space information of several agents in the next time slot to obtain training information combinations of several time slots; update several agents in the agent decision network based on the training information combinations of the several time slots.

[0055] In this embodiment, the reasoning device performs task debate based on the action space information of the multiple agents and the corresponding current time slot to obtain the state space information of the multiple agents in the next time slot.

[0056] The inference device repeatedly constructs the training information combination according to the state space information of the several agents in the next time slot to obtain the training information combination of the several time slots. The inference device updates the several agents in the agent decision network according to the training information combination of the several time slots.

[0057] The agent also includes a critic network; the critic network is used to evaluate the value of the agent in performing task debate. Figure 4 , Figure 4 The flowchart of S4 in the information reasoning method provided in the first embodiment of the present application includes steps S41 to S42, which are as follows:

[0058] S41: respectively inputting the state space information of several agents in the training information combination of several time slots into the critic network of the corresponding agent, and obtaining the individual value parameters of several agents in several time slots according to the preset first value calculation algorithm.

[0059] The first value calculation algorithm is:

[0060]

[0061] In the formula, is the individual value parameter of the ith agent in the tth time slot, V ω () is the critic network function, is the state space information of the i-th agent in the t-th time slot.

[0062] In this embodiment, the inference device inputs the state space information of several agents in the training information combination of several time slots into the critic network of the corresponding agent, and obtains the individual value parameters of several agents in several time slots according to the preset first value calculation algorithm.

[0063] S42: Calculate the loss based on the training information combination of several time slots and the individual value parameters of several agents, and update the actor network of several agents in the agent decision network according to the obtained loss value.

[0064] In this embodiment, the reasoning device calculates the loss based on the combination of the training information of several time slots and the individual value parameters of several agents, and updates the actor network of several agents in the agent decision network according to the obtained loss value.

[0065] See also Figure 5 , Figure 5 The flowchart of S42 in the information reasoning method provided in the first embodiment of the present application includes steps S421 to S422, which are specifically as follows:

[0066] S421: Obtain supervised fine-tuning loss values ​​of the plurality of agents according to the standard answer information of the plurality of agents, the state space information and action space information of the plurality of agents in the training information combination of the last time slot, and a preset supervised fine-tuning loss algorithm, wherein the supervised fine-tuning loss algorithm is:

[0067]

[0068] Where, L SFTi is the supervised fine-tuning loss value of the ith agent, M is the number of tokens in the action space information, () is the function corresponding to the strategy, y i is the standard answer information of the ith agent, is the token indexed by the jth position in the action space information of the ith agent in the last time slot, is the state space information of the i-th agent in the last time slot.

[0069] In this embodiment, the reasoning device obtains the supervised fine-tuning loss values ​​of several of the agents based on the standard answer information of several of the agents, the state space information, action space information of several of the agents in the training information combination of the last time slot, and a preset supervised fine-tuning loss algorithm.

[0070] S422: Obtain a first loss value based on the supervised fine-tuning loss values ​​of the several agents, the individual value parameters of the several agents in the last time slot, and a preset first loss algorithm, and update the actor network of the several agents in the agent decision network based on the first loss value.

[0071] The first loss algorithm is:

[0072]

[0073] Where, L a is the first loss value, N is the number of agents in the agent decision network, is the individual value parameter of the i-th agent in the last time slot.

[0074] In this embodiment, the reasoning device obtains a first loss value based on the standard answer information of several of the agents, the state space information, action space information, individual value parameters of several agents in the training information combination of the last time slot, and a preset first loss algorithm, and updates the actor network of several of the agents in the agent decision network based on the first loss value.

[0075] The goal of the actor network is to maximize the output of its critic network. At the same time, in order to ensure that the actor network retains its ability to solve specific tasks, the inference device combines the supervised fine-tuning loss related to the downstream task to calculate a first loss value, updates the actor networks of several of the agents in the agent decision network, optimizes the agents with the goal of individual optimality, and improves the accuracy and efficiency of task debate.

[0076] In an optional embodiment, the agent decision network further comprises a hybrid network, and the hybrid network is used to ensure consistency among several agents. Figure 6 , Figure 6The flowchart of S42 in the information reasoning method provided in the second embodiment of the present application includes steps S423 to S425, which are specifically as follows:

[0077] S423: Combine the individual value parameters of several intelligent agents in the same time slot to construct a first individual value parameter set of the intelligent agent decision network of several time slots; compare the individual value parameters of several intelligent agents in the same time slot to obtain the maximum individual value parameters of several time slots, replace the individual value parameters of several intelligent agents in several time slots with the maximum individual value parameters of the corresponding time slots, combine the individual value parameters of several intelligent agents in the same time slot after replacement, and construct a second individual value parameter set of the intelligent agent decision network of several time slots.

[0078] In this embodiment, the reasoning device combines the individual value parameters of several intelligent agents in the same time slot to construct the first individual value parameter set of the intelligent agent decision network of the several time slots.

[0079] The reasoning device compares the individual value parameters of several intelligent agents in the same time slot, obtains the maximum individual value parameters of several time slots, replaces the individual value parameters of several intelligent agents in several time slots with the maximum individual value parameters of the corresponding time slots, combines the individual value parameters of several intelligent agents in the same time slot after replacement, and constructs a second individual value parameter set of the intelligent agent decision network of several time slots.

[0080] S424: The first individual value parameter set and the second individual value parameter set of the intelligent agent decision network of several time slots are respectively used as the input parameter set of the hybrid network, and the global value parameter is calculated according to the input parameter set to obtain the first global value parameter and the second global value parameter of the intelligent agent decision network of several time slots.

[0081] The hybrid network includes several fully connected layers whose parameters are determined by the hypernetwork. The input of the hypernetwork is a natural language text H describing the problem to be solved. In order to map to the same vector space, H is first input into the critic network to obtain its vector representation H. This vector and a linear layer are then used to calculate the parameters w and b of the hybrid network. The consistency between each agent and all agents in the agent decision network is ensured by the hybrid network, ensuring that each agent effectively contributes to the team task of the agent decision network.

[0082] In this embodiment, the inference device uses the first individual value parameter set and the second individual value parameter set of the intelligent agent decision network of several time slots as the input parameter set of the hybrid network, performs global value parameter calculation based on the input parameter set, and obtains the first global value parameter and the second global value parameter of the intelligent agent decision network of several time slots to promote the coordinated optimization of the entire intelligent agent decision network.

[0083] In an optional embodiment, the reasoning device constructs a consistency constraint condition, performs a global value parameter calculation according to the input parameter set and the consistency constraint condition, obtains a first global value parameter and a second global value parameter of the agent decision network of several time slots, so as to ensure that the obtained first global value parameter is proportional to the individual value parameters of several agents in several time slots, and the obtained second global value parameter is proportional to the maximum individual value parameter of the corresponding time slot, so as to indicate that when the individual value parameter of each agent reaches its maximum individual value, the obtained first global value parameter and the second global value parameter also reach their maximum individual value, wherein the consistency constraint condition is:

[0084]

[0085] In the formula, Q tot is the global value parameter, Q i is the individual value parameter of the ith agent.

[0086] S425: Accumulate the reward information of several agents in the same time slot to obtain the cumulative reward information of the agent decision network for several time slots; obtain the second loss value according to the cumulative reward information of the agent decision network for several time slots, the first global value parameter and the second global value parameter and the preset second loss algorithm, and update the actor network of several agents in the agent decision network according to the first loss value and the second loss value.

[0087] In this embodiment, the inference device accumulates the reward information of several agents in the same time slot to obtain the cumulative reward information of the agent decision network in several time slots.

[0088] The inference device obtains a second loss value according to the accumulated reward information of the agent decision network in several time slots, the first global value parameter, the second global value parameter, and a preset second loss algorithm, and updates the actor networks of several agents in the agent decision network according to the first loss value and the second loss value, wherein the second loss algorithm is:

[0089]

[0090] Where, L mix is, T is the total number of time slots, R t is the cumulative reward information of the agent decision network at the tth time slot, γ is the discount factor, is the first global value parameter of the agent decision network at the tth time slot, is the second global value parameter of the agent decision network at the tth time slot.

[0091] By introducing the loss calculated by the hybrid network to update the agent's actor network, the coordination between multiple agents within the agent decision network is promoted to promote the global optimum, ensuring that the action space information generated by each agent contributes optimally and consistently to solving the problem, thereby ensuring that each agent contributes incrementally to the task solution and improving the accuracy and efficiency of the task debate of the agent decision network.

[0092] S5: Obtain the text information of the problem to be processed, input the text information of the problem to be processed into several updated agents in the agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain the answer text information output by several updated agents of the last iteration number, and use the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

[0093] In this embodiment, the reasoning device obtains text information of the problem to be processed, wherein the text information of the problem to be processed may be input by a user or obtained from a preset database.

[0094] The inference device inputs the text information of the problem to be processed into the updated several agents in the agent decision network respectively, and iterates repeatedly according to the preset number of iterations to obtain the answer text information output by the updated several agents for the last number of iterations, and takes the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

[0095] Utilizing the multi-agent debate framework, we promote collaboration through task debates between agents and learn the team joint actions of the agent decision network to guide the update of individual agent strategies. This enables each agent to incrementally contribute to the task solution when faced with complex tasks, thereby improving the accuracy and efficiency of the agent decision network's reasoning.

[0096] In an optional embodiment, the step S:6 is further included: training the critic network of the plurality of agents; the step S6 is performed before the step S1, see Figure 7 , Figure 7The flowchart of S6 in the information reasoning method provided in the third embodiment of the present application includes steps S61 to S63, which are as follows:

[0097] S61: Obtain state space information and reward information of several agents in several sample time slots. Based on each sample time slot, accumulate the reward information of several sample time slots before the sample time slot of the same agent to obtain the cumulative reward information of several agents in several sample time slots.

[0098] In this embodiment, the inference device obtains the state space information and reward information of several agents in several sample time slots, and based on each sample time slot, accumulates the reward information of several sample time slots before the sample time slot of the same agent to obtain the cumulative reward information of several agents in several sample time slots.

[0099] S62: Input the state space information and accumulated reward information of several agents in several sample time slots into the critic network of the corresponding agents respectively, and obtain the individual value parameters of several agents in several sample time slots according to the preset second value calculation algorithm.

[0100] In order to better evaluate the status of the actor network, in this embodiment, the critic network to be trained of the agent is a model built based on the same large language model architecture as the actor network, and a linear layer is added after the last layer of the large language model.

[0101] In this embodiment, the inference device inputs the state space information and accumulated reward information of several agents in several sample time slots into the critic network of the corresponding agent, and obtains the individual value parameters of several agents in several sample time slots according to the preset second value calculation algorithm, wherein the second value calculation algorithm is:

[0102]

[0103] In the formula, is the individual value parameter of the ith agent in the tth sample time slot, E[] is the calculation expectation function, is the cumulative reward information of the ith agent in the tth sample time slot, s is the state space information, is the state space information of the ith agent in the tth sample time slot.

[0104] S63: Obtain a third loss value according to individual value parameters of several agents in several sample time slots, reward information and a preset third loss algorithm, and train the critic network of the corresponding agent according to the third loss value.

[0105] The third loss algorithm is:

[0106]

[0107] In the formula, is the third loss value, are the parameters of the critic network, is the reward information of the ith agent in the tth sample time slot, is the individual value parameter of the i-th agent in the t+1-th time slot.

[0108] In this embodiment, the inference device obtains a third loss value based on the individual value parameters of several agents in several sample time slots, reward information and a preset third loss algorithm, and trains the critic network of the corresponding agent based on the third loss value.

[0109] The third loss value is calculated using the temporal mean square error, and the critic network of the corresponding agent is trained based on the third loss values ​​of several of the agents, thereby improving the accuracy of the critic network's value of commenting on the agent's task debate, thereby improving the accuracy of the agent's actor network update.

[0110] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of an information reasoning device based on an agent decision network provided in the fourth embodiment of the present application. The device can implement all or part of the information reasoning device based on an agent decision network through software, hardware, or a combination of both. The device 8 includes:

[0111] A data acquisition module 81 is used to obtain state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot;

[0112] The task debate model 82 is used to obtain the action space information of the plurality of agents in the current time slot according to the state space information of the plurality of agents and the corresponding current time slot;

[0113] The training information combination construction module 83 is used to calculate the reward according to the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot; combine the state space information, action space information and reward information of the current time slot of the same agent to construct the training information combination of the multiple agents in the current time slot;

[0114] The agent updating module 84 is used for the reasoning device to perform task debate according to the action space information of the agents and the corresponding current time slot to obtain the state space information of the agents in the next time slot; repeatedly construct the training information combination according to the state space information of the agents in the next time slot to obtain the training information combination of the several time slots; and update the several agents in the agent decision network according to the training information combination of the several time slots;

[0115] The information reasoning module 85 is used to obtain the text information of the problem to be processed, input the text information of the problem to be processed into the updated several agents in the agent decision network respectively, iterate repeatedly according to the preset number of iterations, obtain the answer text information output by the updated several agents of the last iteration number, and use the answer text information with the highest frequency as the reasoning result of the text information of the problem to be processed.

[0116] In an embodiment of the present application, the state space information of several agents in the current time slot of the agent decision network is obtained through a data acquisition module, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot; the action space information of several agents in the current time slot is obtained according to the state space information of several agents and the corresponding current time slot through a task debate model; the reward calculation is performed according to the action space information of several agents in the current time slot through a training information combination construction module to obtain the reward information of several agents in the current time slot; the state space information, action space information and reward information of the current time slot of the same agent are combined to construct a training information combination of several agents in the current time slot; and the training information combination of several agents in the current time slot is constructed according to the agent update module. The agent and the corresponding action space information of the current time slot are used to conduct task debate to obtain the state space information of several agents in the next time slot; according to the state space information of several agents in the next time slot, the training information combination is repeatedly constructed to obtain the training information combination of several time slots; according to the training information combination of several time slots, several agents in the agent decision network are updated; through the information reasoning module, the text information of the problem to be processed is obtained, and the text information of the problem to be processed is respectively input into the updated several agents in the agent decision network, and it is repeatedly iterated according to the preset number of iterations to obtain the answer text information output by the updated several agents in the last iteration number, and the answer text information with the highest frequency is used as the reasoning result of the text information of the problem to be processed. The multi-agent debate framework is used to promote collaboration through task debate between agents, and the team joint action of the learning agent decision network is used to guide the update of the strategy of a single agent, so that each agent can incrementally contribute to the task solution when facing complex tasks, and improve the accuracy and efficiency of the reasoning of the agent decision network.

[0117] Please refer to Fig. 9 , Fig. 9 A structural diagram of a computer device provided for the fifth embodiment of the present application, wherein the computer device 9 comprises: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device may store a plurality of instructions, and the instructions are suitable for being loaded by the processor 91 and executing the method steps shown in the first to third embodiments above. The specific execution process may refer to the specific description shown in the first to third embodiments, and will not be described in detail here.

[0118] Among them, the processor 91 may include one or more processing cores. The processor 91 uses various interfaces and lines to connect various parts in the server, and executes various functions and processes data of the information reasoning device 8 based on the intelligent agent decision network by running or executing instructions, programs, code sets or instruction sets stored in the memory 92, and calling the data in the memory 92. Optionally, the processor 91 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 91 can integrate one or more combinations of a central processing unit 91 (Central Processing Unit, CPU), a graphics processor 91 (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 91, and it can be implemented by a single chip.

[0119] Among them, the memory 92 may include a random access memory 92 (Random Access Memory, RAM), and may also include a read-only memory 92 (Read-Only Memory). Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 92 may also be optionally at least one storage device located away from the aforementioned processor 91.

[0120] An embodiment of the present application also provides a storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing the method steps shown in the first to third embodiments above. The specific execution process can refer to the specific description shown in the first to third embodiments, which will not be repeated here.

[0121] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0122] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraint algorithm of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0124] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0128] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. An information reasoning method, wherein the agent decision network includes a plurality of agents; characterized in that: The method comprises the following steps: Obtaining state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot; According to the state space information of the multiple agents and the corresponding current time slot, the action space information of the multiple agents in the current time slot is obtained, wherein the action space information is the answer text information generated by the agent based on the state space information; Reward calculation is performed according to the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot; the state space information, action space information and reward information of the current time slot of the same agent are combined to construct the training information combination of the multiple agents in the current time slot; Performing task debate according to the action space information of the several agents and the corresponding current time slot to obtain the state space information of the several agents in the next time slot; repeatedly constructing the training information combination according to the state space information of the several agents in the next time slot to obtain the training information combination of the several time slots; updating the several agents in the agent decision network according to the training information combination of the several time slots; Obtain text information of the problem to be processed, input the text information of the problem to be processed into several updated agents in the agent decision network respectively, iterate repeatedly according to a preset number of iterations, obtain answer text information output by several updated agents at the last iteration number, and use the answer text information with the highest frequency as the inference result of the text information of the problem to be processed.

2. The information reasoning method according to claim 1, characterized in that: The agent includes an actor network; the actor network is used to receive state space information of the agent in the current time slot and generate corresponding action space information; The method of obtaining the action space information of the plurality of agents in the current time slot according to the state space information of the plurality of agents and the corresponding current time slot comprises the steps of: The state space information of the several agents in the current time slot is respectively input into the actor network in the corresponding agent, and action decisions are made according to a preset strategy to obtain the action space information of the several agents in the current time slot.

3. The information reasoning method according to claim 2, characterized in that: The step of performing reward calculation according to the action space information of the plurality of agents in the current time slot to obtain the reward information of the plurality of agents in the current time slot comprises the following steps: Obtaining standard answer information of a plurality of the intelligent agents, wherein the standard answer information is answer text information corresponding to the question text information in the corresponding state space information; According to the action space information, standard answer information and preset reward calculation algorithm of the multiple agents in the current time slot, the reward information of the multiple agents in the current time slot is obtained, wherein the reward calculation algorithm is: In the formula, r it is the reward information of the ith agent in the tth time slot, ε is a preset minimum number, is the action space information of the ith agent in the tth time slot, GT i | is the standard answer information of the i-th agent.

4. The information reasoning method according to claim 3, characterized in that: The agent also includes a critic network; the critic network is used to evaluate the value of the agent in performing task debate; The updating of the plurality of agents in the agent decision network according to the combination of the training information of the plurality of time slots comprises the steps of: The state space information of several agents in the training information combination of several time slots is respectively input into the critic network of the corresponding agent, and the individual value parameters of several agents in several time slots are obtained according to the preset first value calculation algorithm, wherein the first value calculation algorithm is: In the formula, is the individual value parameter of the ith agent in the tth time slot, V ω () is the critic network function, is the state space information of the ith agent in the tth time slot; The loss is calculated based on the training information combination of several time slots and the individual value parameters of several agents, and the actor networks of several agents in the agent decision network are updated according to the obtained loss values.

5. The information reasoning method according to claim 4, characterized in that: The action space information includes a number of position index tokens, indicating the character corresponding to the nth position index in the answer text information; The loss calculation is performed according to the combination of the training information of the plurality of time slots and the individual value parameters of the plurality of agents, and the actor networks of the plurality of agents in the agent decision network are updated according to the obtained loss values, including the steps of: According to the standard answer information of the several agents, the state space information, the action space information of the several agents in the training information combination of the last time slot and the preset supervised fine-tuning loss algorithm, the supervised fine-tuning loss values ​​of the several agents are obtained, wherein the supervised fine-tuning loss algorithm is: Where, L SFTi is the supervised fine-tuning loss value of the ith agent, M is the number of tokens in the action space information, is the function corresponding to the strategy, y i is the standard answer information of the ith agent, is the token indexed by the jth position in the action space information of the ith agent in the last time slot, is the state space information of the i-th agent in the last time slot; According to the supervised fine-tuning loss values ​​of the plurality of agents, the individual value parameters of the plurality of agents in the last time slot and a preset first loss algorithm, a first loss value is obtained, and according to the first loss value, the actor networks of the plurality of agents in the agent decision network are updated, wherein the first loss algorithm is: Where, L a is the first loss value, N is the number of agents in the agent decision network, is the individual value parameter of the i-th agent in the last time slot.

6. The information reasoning method according to claim 5, characterized in that: The agent decision network also includes a hybrid network, which is used to ensure consistency among several agents. The method of performing loss calculation based on the training information combination of several time slots and the individual value parameters of several agents, and updating the actor networks of several agents in the agent decision network according to the obtained loss values, further includes the steps of: The individual value parameters of several intelligent agents in the same time slot are combined to construct a first individual value parameter set of the intelligent agent decision network of several time slots; the individual value parameters of several intelligent agents in the same time slot are compared to obtain the maximum individual value parameters of several time slots, the individual value parameters of several intelligent agents in several time slots are replaced with the maximum individual value parameters of the corresponding time slots, and the individual value parameters of several intelligent agents in the same time slot after replacement are combined to construct a second individual value parameter set of the intelligent agent decision network of several time slots; The first body value parameter set and the second body value parameter set of the agent decision network of several time slots are respectively used as the input parameter set of the hybrid network, and the global value parameter is calculated according to the input parameter set to obtain the first global value parameter and the second global value parameter of the agent decision network of several time slots; The reward information of several agents in the same time slot is accumulated to obtain the cumulative reward information of the agent decision network in several time slots; the second loss value is obtained according to the cumulative reward information of the agent decision network in several time slots, the first global value parameter and the second global value parameter and the preset second loss algorithm, and the actor network of several agents in the agent decision network is updated according to the first loss value and the second loss value, wherein the second loss algorithm is: Where, L mix is, T is the total number of time slots, R t is the cumulative reward information of the agent decision network at the tth time slot, γ is the discount factor, is the first global value parameter of the agent decision network at the tth time slot, is the second global value parameter of the agent decision network at the tth time slot.

7. The information reasoning method according to claim 6, characterized in that: The method further comprises the steps of: training the critic networks of the plurality of said agents; the training of the critic networks of the plurality of said agents comprises the steps of: Obtaining state space information and reward information of several agents at several sample time slots, and accumulating reward information of several sample time slots before the sample time slot of the same agent based on each sample time slot to obtain cumulative reward information of several agents at several sample time slots; The state space information and accumulated reward information of several agents in several sample time slots are respectively input into the critic network of the corresponding agent, and the individual value parameters of several agents in several sample time slots are obtained according to the preset second value calculation algorithm, wherein the second value calculation algorithm is: In the formula, is the individual value parameter of the ith agent in the tth sample time slot, E[] is the calculation expectation function, is the cumulative reward information of the ith agent in the tth sample time slot, s is the state space information, is the state space information of the ith agent in the tth sample time slot; According to the individual value parameters, reward information and a preset third loss algorithm of several agents in several sample time slots, a third loss value is obtained, and according to the third loss value, the critic network of the corresponding agent is trained, wherein the third loss algorithm is: In the formula, is the third loss value, are the parameters of the critic network, is the reward information of the ith agent in the tth sample time slot, is the individual value parameter of the i-th agent in the t+1-th time slot.

8. An information reasoning device based on an agent decision network, wherein the agent decision network includes a plurality of agents, characterized in that: include: A data acquisition module, used to obtain state space information of several agents in the current time slot of the agent decision network, wherein the state space information includes question text information and agent conversation history information from the initial time slot to the current time slot; The task debate model is used to obtain the action space information of several agents in the current time slot according to the state space information of several agents and the corresponding current time slot, wherein the action space information is the answer text information generated by the agent based on the state space information; A training information combination construction module is used to calculate rewards according to the action space information of the multiple agents in the current time slot to obtain the reward information of the multiple agents in the current time slot; combine the state space information, action space information and reward information of the current time slot of the same agent to construct a training information combination of the multiple agents in the current time slot; The agent updating module is used to perform task debate according to the action space information of the several agents and the corresponding current time slot to obtain the state space information of the several agents in the next time slot; repeatedly construct the training information combination according to the state space information of the several agents in the next time slot to obtain the training information combination of the several time slots; and update the several agents in the agent decision network according to the training information combination of the several time slots; The information reasoning module is used to obtain the text information of the problem to be processed, input the text information of the problem to be processed into the updated several agents in the agent decision network respectively, iterate repeatedly according to the preset number of iterations, obtain the answer text information output by the updated several agents at the last iteration number, and use the answer text information with the highest frequency as the reasoning result of the text information of the problem to be processed.

9. A computer device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the information reasoning method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the information reasoning method according to any one of claims 1 to 7 are implemented.

Images