Multi-agent reasoning framework construction method based on meta-cognitive thought

Abstract

The invention discloses a multi-agent reasoning framework construction method based on a meta-cognitive thought, which is characterized in that a multi-agent reasoning framework is adopted to simulate a human brain thinking model, and the method comprises the following steps: firstly, constructing a meta-cognitive agent training data set, and training an open source large language model based on a thinking injection training normal form; the method comprises the following steps of: constructing an open-source large language model to become a meta-cognitive agent with thinking ability, constructing a central agent training data set, training the open-source large language model based on a supervised training normal form to accurately evaluate the difficulty degree of a task, and finally, taking the open-source large language model which is not additionally trained as a cognitive agent to accurately evaluate the task difficulty degree. A multi-agent reasoning framework capable of effectively simulating a human meta-cognitive thinking model is constructed by combining a meta-cognitive agent and a central agent. Compared with the prior art, the method has the capability of improving the model to solve complex problems, and effectively relieves the two limitations of hindering the reasoning capability of the large language model, lacking a serial calculation mode and lacking the'thinking 'capability.

Description

Technical Field

[0001] The present invention relates to the field of metacognition technology, and in particular to a method for constructing a multi-agent reasoning framework based on metacognition ideas. Background Art

[0002] Large Language Models (LLMs) are natural language processing (NLP) models based on deep learning technology. They usually use billions to hundreds of billions of parameters and are trained through large-scale text data. They can generate, understand and process human language and have strong language generation and semantic understanding capabilities.

[0003] At present, the reasoning ability of the large language models used (such as ChatGPT and Llama, etc.) is insufficient, which is specifically manifested in the difficulty in solving problems with more complex logic such as mathematics. There are two main reasons for this: 1) Lack of serial computing mode. Large language models are all based on the Transformer architecture, and they always follow the parallel computing mode during the reasoning process. In contrast, the human brain usually activates the parallel conduction mode of neurons when solving simple problems, that is, fast thinking, and activates the serial conduction mode of neurons when solving complex problems, that is, slow thinking. Therefore, the lack of serial computing mode in large language models limits their ability to deal with complex problems; 2) Lack of "thinking" ability. Large language models are trained using large-scale human corpora. Although these corpora contain rich knowledge, humans usually do not write down their thinking process. Therefore, large language models trained using these corpora usually lack the ability to "think" deeply.

[0004] Existing solutions for enhancing the reasoning ability of large language models are mainly divided into the following two types: 1) Prompt word-based methods, such as Chain of Thought and Tree of Thought, which guide the large language model to output intermediate reasoning steps through prompt engineering. Although these methods simulate the thinking process of the human brain when solving complex problems, they do not change the parallel computing mode of the large language model, limiting the upper limit of its reasoning ability; 2) Multi-agent-based multi-step reasoning frameworks, such as the CAMEL framework, which improve reasoning ability through multi-model collaborative computing. This type of method introduces a serial computing mode through multi-model collaboration, but a single model lacks the "thinking" ability, making it difficult to fully utilize the advantages of the serial computing mode. Summary of the invention

[0005] The purpose of the present invention is to provide a method for constructing a multi-agent reasoning framework based on metacognitive thinking to address the lack of reasoning ability of existing large language models. The multi-agent reasoning framework constructed by metacognitive thinking is used to realize the serial computing mode of the large language model, effectively simulating the thinking mode used by humans to solve complex problems. In this thinking mode, humans will introspect, evaluate and self-regulate, thereby improving their performance. Combining metacognitive thinking to construct a multi-agent reasoning framework can effectively alleviate the two major limitations that hinder the reasoning ability of large language models: 1) lack of serial computing mode; 2) lack of "thinking" ability, thereby comprehensively improving the reasoning ability of large language models. The method collects training data for constructing a metacognitive agent, and trains a large language model through a thinking injection training paradigm to obtain a metacognitive agent, then collects training data for constructing a central agent, and trains a large language model through a supervised fine-tuning training paradigm to obtain a central agent, which is used to control when to activate the metacognitive agent. Finally, a multi-agent reasoning framework is constructed in combination with metacognitive thinking. The method is simple, effective, and greatly improves the ability of the model to solve complex problems. It has a wide range of application scenarios and application value.

[0006] The specific technical solution to achieve the purpose of the present invention is: a method for constructing a multi-agent reasoning framework combining metacognitive ideas, which is characterized by adopting a multi-agent reasoning framework to enhance the reasoning ability of a large language model to simulate the thinking mode of the human brain. The method includes the following specific steps:

[0007] Step 1: Build a metacognitive agent training dataset

[0008] First, design and think about injecting prompt words (Prompt) t , which is used to stimulate the thinking process of the large language model. Then P t User command prompt words Perform splicing to get input prompt And input into the large language model to obtain the output result, which is specifically expressed as follows (a):

[0009] LLM0(x i )={z i ,y i} (a).

[0010] Among them, LLM0 represents a naive large language model without additional training, and the z in the output result is i and i Represent the thinking process and the answer respectively. Then, a judge model is used to judge the output y i The quality of is specifically expressed as follows (b):

[0011] Judge(y i)=label,label∈{pos,neg} (b).

[0012] Among them, Judge represents the judge model, which is used to judge y i The quality of y is determined by the label label, label = pos / neg represents the current y i is a positive sample / negative sample.

[0013] Finally, the metacognitive agent training dataset D meta It is expressed by the following formula (c):

[0014]

[0015] in, and Represent the positive and negative samples in the answer respectively, D meta Represents a metacognitive agent training dataset.

[0016] Step 2: Acquire metacognitive agents through thinking infusion training paradigm

[0017] Using the Direct Preference Optimization algorithm, the D obtained in step 1 meta Train an LLM0 on it to get the metacognitive agent Agent meta ,The training loss function is expressed as follows (d):

[0018]

[0019] Among them, π=LLM0 represents the large language model to be trained.

[0020] By minimizing the above loss function, we can obtain the "slow thinking" process Agent shown in the following formula (e): meta :

[0021]

[0022] Among them, Agent meta To simulate the "slow thinking" process of the human brain and to deal with complex problems, it will implicitly generate the thinking process while generating the answer, which makes the Agent meta More computing power will be allocated to complex problems, effectively alleviating the limitation of lack of "thinking" ability mentioned in the background technology, as shown in the following formula (f): meta (x):

[0023] Agent meta (x) = {z latent,y} (f).

[0024] Among them, x, z latent and y represent the input, implicit thinking process, and answer output respectively.

[0025] Step 3: Build a central agent training dataset

[0026] To ensure the generalization of the central agent, we first collect data samples from multiple fields (mathematics, logical reasoning, general knowledge, medicine, and law) and process them into the form of question-answer pairs as shown in the following formula (g):

[0027] Multi_data = {Q i ,A i} (g).

[0028] Then, the data samples are divided into two categories, simple and difficult, according to the difficulty of the problem, and labeled, as shown in the following formula (h):

[0029] D hub = {Q i ,A i ,R i} (h).

[0030] Among them, D hub represents the central agent training dataset, R i ∈{easy,hard} represents the difficulty label of the problem.

[0031] Step 4: Obtaining the central agent based on the supervised fine-tuning training paradigm

[0032] Use the supervised fine-tuning training paradigm for the classification task to obtain D in step 3 hub Train an LLM0 to get the central agent Agent hub , the training loss function is expressed as follows (i):

[0033]

[0034] Among them, π = LLM0 represents the large language model to be trained, R and Represent the true difficulty label and the predicted difficulty label respectively.

[0035] By minimizing the above loss function, we can get the central agent Agent hub , as shown in the following formula (j):

[0036]

[0037] in, Indicates the search for The smallest π, Agent hub Used to determine the difficulty of the input question to determine whether to enable the Agent obtained in step 2 meta , specifically as shown in the following formula (k):

[0038]

[0039] Step 5: Construct a multi-agent reasoning framework based on metacognitive thinking

[0040] Inspired by human metacognition, this multi-agent reasoning framework is composed of cognitive agents. cog , the metacognitive agent obtained in step 2 meta The central agent obtained in step 4 hub The three parts are composed together, which are specifically expressed as follows (1):

[0041] MetalInf=(Agent cog ,Agent meta ,Agent hub ) (l).

[0042] MetaInf represents a multi-agent reasoning framework based on metacognitive thinking. cog This is done by a regular large language model LLM0 that has not received any additional training.

[0043] In MetaInf, Agent cog Used to handle simple tasks, Agent meta To handle complex tasks, this multi-agent reasoning framework uses Agent hub To analyze the difficulty level of the task, so as to activate the Agent when receiving a simple task cog , when receiving a difficult task, activate an agent with stronger reasoning ability but greater resource consumption meta Therefore, MetaInf has two working modes: 1) simple task processing; 2) complex task processing.

[0044] When Agent hub When the current task is judged to be simple, MetaInf will activate the simple task processing mode, that is, only use Agent cog Solve the task, as shown in the following formulas (m) to (n):

[0045] Agent hub (Q) = easy (m);

[0046] O=Agentcog (Q) (n).

[0047] Among them, Q represents the query of the current task, and O represents the output result of MetaInf.

[0048] When Agent hub When the current task is judged to be difficult, MetaInf will activate the difficult task processing mode, that is, first use Agent cog Generate intermediate results and then use Agent meta , combined with optimization instructions to optimize the intermediate results to obtain the final output, as shown in the following formulas (o) to (q):

[0049] Agent hub (Q) = hard (o);

[0050] S=Agent cog (Q) (p);

[0051] O=Agent meta (P refine ,S) (q).

[0052] Among them, S represents the intermediate result, P refine This working mode introduces a serial computing mode, which effectively alleviates the limitation of the lack of a serial computing mode mentioned in the background technology.

[0053] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0054] 1) The present invention can effectively alleviate the two major limitations that hinder the reasoning ability of large language models: the lack of serial computing mode and the lack of "thinking" ability.

[0055] 2) The present invention uses a novel thinking-injection training paradigm to enable the large language model to have thinking ability, effectively alleviating the limitation of lack of "thinking" ability.

[0056] 3) Based on a large language model with thinking ability and combined with human metacognitive ideas, a multi-agent reasoning framework is constructed and a serial computing mode is introduced to effectively alleviate the limitations of the lack of a serial computing mode.

[0057] 4) The present invention combines the metacognitive concept in cognitive neuroscience to simultaneously solve the above two major limitations.

[0058] Comprehensively improve the reasoning capabilities of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of a multi-agent reasoning framework based on metacognitive thinking proposed by the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0061] Example 1

[0062] See also Figure 1 ,The construction of a multi-agent reasoning framework based on metacognitive thinking specifically includes the following steps:

[0063] Step 1: Build a metacognitive agent training dataset

[0064] The thought injection prompts and user command prompts are written by human experts. The data used can be obtained from the public datasets TriviaQA and AlpacaEval. LLM0 and Judge can be any open source large language model. First, design the thought injection prompt (Prompt) P t , which is used to stimulate the thinking process of the large language model. Then P t User command prompt words Perform splicing to get input prompt And input into the large language model to obtain the output result, which is specifically expressed as follows (a):

[0065] LLM0(x i )={z i ,y i} (a).

[0066] Among them, LLM0 represents an ordinary large language model without additional training, and the z in the output result i and i Represent the thinking process and the answer respectively. Then, a judge model is used to judge the output y i The quality of is expressed as follows (b):

[0067] Judge(y i )=label,label∈{pos,neg} (b).

[0068] Among them, Judge represents the judge model, which is used to judge y i The quality of y is determined by the label label, label = pos / neg represents the current y i is a positive sample / negative sample.

[0069] Finally, the metacognitive agent training dataset D metaIt is expressed by the following formula (c):

[0070]

[0071] in, and Represent the positive and negative samples D in the answer respectively meta Represents a metacognitive agent training dataset.

[0072] Step 2: Acquire metacognitive agents through thinking infusion training paradigm

[0073] The LLM0 used can be any open source large language model. Using the Direct Preference Optimization algorithm, the D obtained in step 1 is meta Train an LLM0 on it to get the metacognitive agent Agent meta , the training loss function is expressed as follows (d):

[0074]

[0075] Where π = LLM0 represents the large language model to be trained. By minimizing the above loss function, we can get Agent meta , as shown in the following formula (e):

[0076]

[0077] Agent meta Simulates the "slow thinking" process of the human brain to deal with complex problems. It generates the thinking process implicitly while generating the answer, which makes the Agent meta More computing power will be allocated to complex problems, effectively alleviating the limitation of the lack of "thinking" ability mentioned in the background technology, as shown in the following formula (f):

[0078] Agent meta (x) = {z latent ,y} (f).

[0079] Among them, x, z latent and y represent the input, implicit thinking process, and answer output respectively.

[0080] Step 3: Build a central agent training dataset

[0081] The data used can be obtained from the public datasets TriviaQA and AlpacaEval, and LLM0 can be any open source large language model. To ensure the generalization of the central agent, data samples are first collected from multiple fields (mathematics, logical reasoning, general knowledge, medical, and law) and processed into question-answer pairs, as shown in the following formula (g):

[0082] Multi_data = {Q i ,A i} (g).

[0083] Then, the data samples are divided into two categories, simple and difficult, according to the difficulty of the problem, and labeled, as shown in the following formula (h):

[0084] D hub = {Q i ,A i ,R i} (h).

[0085] Among them, D hub represents the central agent training dataset, R i ∈{easy,hard} represents the difficulty label of the problem.

[0086] Step 4: Obtaining the central agent based on the supervised fine-tuning training paradigm

[0087] The LLM0 used can be any open source large language model, and the D obtained in step 3 using the supervised fine-tuning training paradigm for classification tasks hub Train an LLM0 to get the central agent Agent hub , the training loss function is expressed as follows (i):

[0088]

[0089] Among them, π = LLM0 represents the large language model to be trained, R and Represent the true difficulty label and the predicted difficulty label respectively.

[0090] By minimizing the above loss function, we can get Agent hub , as shown in the following formula (j):

[0091]

[0092] Agent hub Used to determine the difficulty of the input question to determine whether to enable the Agent obtained in step 2 meta , specifically as shown in the following formula (k):

[0093]

[0094] Step 5: Construct a multi-agent reasoning framework based on metacognitive thinking

[0095] Using P refine Written by human experts and inspired by human metacognitive thinking, this multi-agent reasoning framework is composed of cognitive agents cog , the metacognitive agent obtained in step 2 meta The central agent obtained in step 4 hub The three parts are composed together, which are specifically expressed as follows (1):

[0096] MetaInf=(Agent cog ,Agent meta ,Agent hub ) (l).

[0097] MetaInf represents a multi-agent reasoning framework based on metacognitive thinking. cog This is done by a regular large language model LLM0 that has not received any additional training.

[0098] In MetaInf, Agent cog Used to handle simple tasks, Agent meta To handle complex tasks, this multi-agent reasoning framework uses Agent hub To analyze the difficulty level of the task, so as to activate the Agent when receiving a simple task cog , when receiving a difficult task, activate an agent with stronger reasoning ability but greater resource consumption meta Therefore, MetaInf has two working modes: 1) simple task processing; 2) complex task processing.

[0099] When Agent hub When the current task is judged to be simple, MetaInf will activate the simple task processing mode, that is, only use Agent cog Solve the task, as shown in the following formulas (m) to (n):

[0100] Agent hub (Q) = easy (m);

[0101] O=Agent cog (Q) (n).

[0102] Among them, Q represents the query of the current task, and O represents the output result of MetaInf.

[0103] When Agent hub When the current task is judged to be difficult, MetaInf will activate the difficult task processing mode, that is, first use Agent cog Generate intermediate results and then use Agent meta Combine the optimization instructions to optimize the intermediate results to get the final output, as shown in the following formulas (o) to (q):

[0104] Agent hub (Q) = hard (o);

[0105] S=Agent cog (Q) (p);

[0106] O=Agent meta (P refine ,S) (q).

[0107] Among them, S represents the intermediate result, P refine This working mode introduces a serial computing mode, which effectively alleviates the limitation of the lack of a serial computing mode mentioned in the background technology.

[0108] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for constructing a multi-agent reasoning framework based on metacognitive thinking, characterized in that: The multi-agent reasoning architecture is used to enhance the reasoning ability of the large language model to simulate the thinking mode of the human brain. The method specifically includes the following steps: Step 1: Build a metacognitive agent training dataset 1-1: Injecting thoughts into prompt words P and user command prompt words Perform splicing and get input prompt Input it into the large language model and obtain the output result shown in the following formula (a): LLM0(x i )={z i ,y i } (a); Among them, LLM0 is a common large language model without additional training; z i and i The thinking process and the answer respectively; 1-2: Use the judge model shown in the following formula (b) to judge the output result y i The pros and cons: Judge(y i )=label,label∈{pos,neg} (b); Among them, Judge is the judgment y i Good or bad represents the judge model, and generates a label for it, label = pos / neg represents the current y i is a positive sample / negative sample; 1-3: Metacognitive Agent Training Dataset D meta It is expressed by the following formula (c): in, and are the positive and negative samples in the answer respectively; Step 2: Acquire metacognitive agents through thought-injection training paradigm 2-1: Using the direct preference optimization algorithm, in the metacognitive agent training dataset D meta The metacognitive agent Agent is trained as shown in the following formula (d) meta The loss function L is: Among them, π=LLM0 is the large language model to be trained; 2-2: Minimize the above loss function L and obtain the "slow thinking" agent shown in the following formula (e) meta : in, It means to find the π that makes L the smallest; The "slow thinking" agent meta While generating the answer, it implicitly generates the following thinking process Agent represented by formula (f) meta (x): Agent meta (x)={z latent ,y} (f); Among them, x, z latent and y represent the input, implicit thinking process, and answer output respectively; Step 3: Build a central agent training dataset 3-1: Collect data samples from multiple fields such as mathematics, logical reasoning, general knowledge, medicine and law, and process them into question-answer pairs Multi_data using the following formula (g): Multi_data={Q i ,A i } (g); Among them, Q i , and A i are questions and answers respectively; 3-2: Divide the data samples into two categories: simple and difficult according to the difficulty of the problem, and label them with R using the following formula (h): i : D hub ={Q i ,A i ,R i } (h); Among them, D hub is the central agent training dataset; R i ∈{easy,hard} represents the difficulty label of the problem; Step 4: Obtain the central agent through supervised fine-tuning training paradigm 4-1: Using the supervised fine-tuning training paradigm for classification tasks on the central agent training dataset D hub Train LLM0 to get the central agent Agent hub , the training uses the loss function L represented by the following formula (i) π : Among them, π = LLM0 is the large language model to be trained; R and They are the real difficulty label and the predicted difficulty label respectively; 4-2: By minimizing the loss function L π , we get the central agent Agent represented by the following formula (j): hub : 4-3: Using the Central Agent hub The agent that determines the difficulty of the input question and is represented by the following formula (k) hub (Q) Determine whether to enable "Slow Thinking": Step 5: Construct a multi-agent reasoning framework based on metacognitive thinking The multi-agent reasoning framework MetaInf is represented by the cognitive agent Agent as follows (l): cog , Metacognitive Agent meta and the central agent hub Built with: MetaInf=(Agent cog ,Agent meta ,Agent hub ) (l); The multi-agent reasoning framework MetaInf has two working modes for processing simple tasks and complex tasks. The cognitive agent cog Acts as a cognitive agent for simple tasks for the untrained large language model LLM0; The metacognitive agent meta Intelligent agents for handling complex tasks; The central agent hub To introduce the serial computing mode, the Agent is activated when a simple task is received. cog , activate Agent when receiving difficult tasks meta , the specific processing is as follows: 5-1: Agent hub When the current task is judged to be simple, MetaInf activates the simple task processing mode, that is, only using Agent cog Solve the task, which is specifically expressed by the following (m) to (n) formulas: Agent hub (Q)=easy (m); O=Agent cog (Q) (n); Among them, Q is the query of the current task; O is the output result of MetaInf; 5-2: Being an Agent hub When the current task is judged to be difficult, MetaInf activates the difficult task processing mode, that is, first use Agent cog Generate intermediate results and then use Agent meta The intermediate results are optimized by combining the optimization instructions to obtain the final output, which is specifically expressed by the following formulas (o) to (q): Agent hub (Q)=hard (o); S=Agent cog (Q) (p); O=Agent meta (P refine ,S) (q); Among them, S is the intermediate result; P refine ,S is the optimization instruction.

Images