Multi-agent task planning method for high-order thinking development under mixed game

Abstract

The invention discloses a multi-agent task planning method for high-order thinking development under a mixed game, and belongs to the technical field of task coupling planning, and the method comprises the steps: S1, carrying out the multi-agent reinforcement learning based on two-stage intention sharing under the guidance of the high-order thinking of a learner; s2, multi-agent task coupling planning for higher-order thinking of learners under the heuristic of the mixed game, specifically comprising the steps of S21, constructing a multi-agent model under the heuristic of the mixed game; s22, multi-agent model strategy optimization oriented to high-order thinking under the heuristic of the mixed game; s23, multi-agent task coupling planning for higher-order thinking of the learner under the heuristic of the mixed game; s3, high-order thinking of the learner based on a multi-agent migration strategy is developed; according to the multi-agent task planning method for high-order thinking development under the mixed game, solid support and powerful promotion are provided for physical development of high-order thinking of learners.

Description

Technical Field

[0001] The present invention relates to the technical field of task coupling planning, and in particular to a multi-agent task planning method for developing high-order thinking under a hybrid game. Background Art

[0002] In today's rapidly developing intelligent era, the cultivation of high-order thinking skills is crucial for learners. High-order thinking, which covers core elements such as critical thinking, creative thinking and problem-solving skills, is the key for learners to adapt to complex and changing environments, solve nonlinear problems and achieve innovative breakthroughs. It is not only related to personal cognitive development, but also has a profound impact on social progress and technological innovation. Therefore, exploring effective methods for cultivating high-order thinking has become an important issue that needs to be urgently addressed in the field of education.

[0003] As an important branch of artificial intelligence, multi-agent reinforcement learning has shown great potential in solving complex tasks and optimizing strategies due to its strong adaptability and flexibility. In a multi-agent system, each agent explores the environment together and optimizes its own strategy through mutual cooperation or competition to maximize the overall benefits. This mechanism not only conforms to the complexity of multi-agent interaction in the real world, but also provides an ideal experimental platform for simulating and training high-order thinking abilities.

[0004] As an important form of game theory, hybrid games allow agents to maintain a certain degree of randomness in their strategy selection, thereby increasing the complexity and uncertainty of the game. Under the framework of hybrid games, agents need to comprehensively consider the possible behavior of their opponents, the allocation of their own resources, and the trade-off between long-term interests and short-term gains, and formulate more flexible and robust strategies. This strategy formulation process is highly consistent with critical analysis and creative decision-making in higher-order thinking, and provides a new perspective and path for the cultivation of higher-order thinking.

[0005] Based on the above background, the present invention proposes a multi-agent task planning method for the development of high-order thinking under hybrid games. The method aims to build a multi-agent system and simulate a hybrid game environment, so that learners can continuously exercise and improve their high-order thinking ability in the process of participating in multi-agent interaction.

[0006] In addition, the method also focuses on the application of multi-agent transfer strategies to promote the embodied development of learners' high-level thinking. The transfer strategy allows the agent to effectively transfer the knowledge and skills learned in one task to another related but different task, thereby accelerating the learning process and improving learning efficiency. This transfer ability has important practical significance and application value for learners to quickly call on existing knowledge and flexibly adjust strategies when facing new situations and new problems. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-agent task planning method for the development of high-order thinking under a hybrid game, so as to solve the problems existing in the above-mentioned background technology.

[0008] To achieve the above object, the present invention provides a multi-agent task planning method for high-order thinking development under a hybrid game, comprising the following steps: S1. Multi-agent reinforcement learning based on two-stage intention sharing under the guidance of learners' high-order thinking; S2. Multi-agent task coupling planning for learners’ high-order thinking inspired by hybrid games, including: S21. Construction of multi-agent model inspired by hybrid game; S22. Strategy optimization of multi-agent model for high-level thinking inspired by hybrid games; S23, Multi-agent task coupling planning for learners' high-order thinking inspired by hybrid games; S3. Embodied development of learners’ higher-order thinking based on multi-agent transfer strategy.

[0009] Preferably, step S1 specifically includes: S11. Acquisition and processing of data related to learners' high-order thinking based on multi-agents, specifically: through advanced language recognition, image recognition and natural language processing technologies, real-time capture of learners' thinking characteristics in high-order thinking in-depth discussions such as text, including but not limited to logical reasoning paths in images, critical thinking expressions in text, and innovative viewpoints, etc. The capture formula is: (1) in, Represents a dataset of high-level thinking and in-depth discussion content. Represents the capture function, Indicates speech recognition, represents image recognition, Represents the output of the natural language processing module; After preprocessing, the captured high-order thinking data is input into the multi-agent system for collaborative processing. The multi-agent system conducts in-depth mining and analysis of the high-order thinking data through its internal communication and collaboration mechanism, and extracts key features that can reflect the learner's high-order thinking ability. The formula is: (2) in, Represents higher-order thinking eigenvector geometry, Represents the processing function, Represent the communication and cooperation mechanism of multi-agent system; S12. Multi-agent reinforcement learning based on two-stage intention sharing.

[0010] Preferably, step S12 specifically includes: Each agent Each maintains a value network First, the high-level thinking in-depth discussion content dataset is analyzed independently. Generate a preliminary intention action , the specific formula is: (3) Then, the first round of intention sharing begins, and the agent sends its intended action Broadcast to all other agents in the system. Assume that the total number of agents in the system is N. After the first round of intention sharing is completed, each agent not only , and also combined with the intention action information received from other agents , assess the importance of this information to one’s current decision; The V value is calculated to measure the potential impact of each received intention action information on its decision-making process. Specifically, for a given action k, the agent Provide Agent Action information, agent The maximum value of V when taking different optimal actions and minimum value , the specific formula is: (4) (5) In order to establish a unified quantitative standard, the maximum and minimum values ​​are normalized. The specific formula is: (6) If the importance of an agent's intentional action information to another agent's decision exceeds a preset threshold , then the agent will be included in the latter's dependent object set; Then, we enter the second round of intention sharing. In this stage, each agent will send its set of dependent objects to all other agents in the system. Through this round of information exchange, the agents can build a more comprehensive dependency graph, which is a directed graph that is used to intuitively display the dependency relationships between agents.

[0011] Preferably, in step S12, a circular dependency removal algorithm is used to detect and eliminate circular dependencies in the dependency graph, specifically: Initialize all nodes , set visited(r)=false and onStack(r)=false; visited(r) is a status flag used to indicate whether the node r has been visited. visited(r) = false means that in the initialization phase, it is assumed that all nodes have not been visited. onStack(r) is also a status flag used to indicate whether the node r is currently in the DFS stack. onStack(r) = false means that in the initialization phase, it is assumed that all nodes are not in the stack. Depth-first search (DFS), for each unvisited node r, does the following: Will Mark as visited, set visited(r)=true; Will Push into DFS stack, set onStack(r)=true; for For each adjacent node w of (i.e., there is a directed edge from r to w), if w has not been visited, recursively perform a depth-first search on w. If w has been visited and is still in the stack (i.e., visited(w)=true and onStack(r)=true), a circular dependency is detected; Will Pop from the stack and set onStack(r)=false; Circular dependency elimination, once a circular dependency is detected, the dependencies are adjusted to ensure that the resulting graph is a directed acyclic graph; After obtaining the acyclic dependency graph, the decision-making process of the agent will follow the following rules: If an agent is dependent on other agents, it cannot make a new decision, and its final action will directly adopt its initially generated intention action On the contrary, if an agent is not dependent on any agent, it will make a new decision based on the intention action information of the dependent agent and its own local observations to optimize its action selection.

[0012] Preferably, step S21 specifically includes: Assume that the learning process of other agents has an impact on the agent The formula for the impact is: (7) in, Representing an Agent The policy update amount, Representing an Agent The learning rate, Representing an Agent The objective function is Representing an Agent strategy, represents the estimated value of other agents’ strategies, Indicates the gradient operation of the strategy update of agent i; In order to more accurately capture the mutual influence between agents, based on the mean field theory, the average action of the surrounding agents is used as input to influence and optimize the strategy learning of the agent itself. The strategy update formula is: (8) in, Representing an Agent The strategy before the update, Representing an Agent The updated strategy, Representing an Agent The state-action value function, represents the average action of other agents; In order to improve the effectiveness of the agent's strategy in a non-stationary environment, a deep Bayesian strategy reuse method is combined, a neural network is used as a value function approximator, and a distillation strategy network is introduced to achieve efficient strategy learning and reuse, and higher cumulative rewards and convergence performance are achieved in multiple random game scenarios. Among them, the update formula of the value function approximator is: (9) in, is the parameter of the value function approximator, s is the current state, is the immediate reward for the agent; Finally, ensure that the agent's algorithm can reach the Nash equilibrium, that is, the optimal response based on the strategies of all agents. The formula is: (10) in, Representing an Agent strategy, In addition to the intelligent agent The joint strategy of other agents, Representing an Agent The utility value of represents the optimal response strategy that satisfies all agents.

[0013] Preferably, step S22 specifically includes: We optimize the strategy of the multi-agent model inspired by hybrid games. Through the gradient-based optimization strategy, we achieve efficient joint optimization of the agent strategy and the model in each training round. In order to achieve the above optimization goals, we first transform the strategy function and helper functions Parameterized, respectively and , then, the gradient-based method is used to update these parameters. The specific update rule formula is: (11) (12) in, represents the objective function of expected cumulative reward, Represents the objective function of the policy parameters The gradient of Represents the objective function for auxiliary parameters The gradient of represents the update step size of the policy parameters, Indicates the update step size of auxiliary parameters; During the training process, the state and action of the agent at each moment are evaluated through the neural network value function. First, the learner's high-level thinking data sample tasks are preliminarily processed, and then the current state of all tasks is input. and actions , using the neural network value function Output the evaluation value at this moment; define a loss function , which is used to measure the difference between the output of the neural network value function and the actual reward. Subsequently, the gradient descent method is used to update the parameters of the neural network value function. , the specific update rule formula is: (13) Among them, ξ is the update step size of the neural network value function parameters; By constructing an optimized objective function to maximize the weighted sum of the expected cumulative internal and external rewards, the objective function formula is: (14) in, Indicates that the agent is in state Next action Internal rewards obtained when Represented by the auxiliary function External rewards generated, represents the weight coefficient that regulates the relative importance of internal and external rewards, and T represents the time step of the training round; The above training process is repeated continuously until the task faced by the agent reaches the predetermined indicators.

[0014] Preferably, step S23 specifically includes: First, set up a multi-task coupling relationship matrix, and use the outputs of a multi-task strategy network of a certain agent to form a multi-task coupling relationship matrix. Suppose the strategy network of task R has output, task The policy network has outputs, and p, q = 1, 2, ..., n, n is the total number of tasks; then construct a Multi-task coupling relationship matrix ,in , , if the oth output of task R is equal to task There is a coupling relationship between the lth output of , the corresponding element of the output is 1, and the rest are 0; Then, each agent forms a multi-agent coupling relationship matrix. According to the actual application situation, it is judged whether there is a collaborative relationship between the agents. This is shown in the multi-agent coupling relationship matrix, which reflects the collaborative relationship between the multi-agents and supports the input reconstruction of the evaluation network of each task of each agent. If there are N agents in the agent cluster, an N×N multi-agent coupling relationship matrix C is constructed, where 1≤i≤N, 1≤j≤N. If there is a collaborative relationship between agent i and agent j, the corresponding output element is 1, and the rest are 0. According to the multi-task coupling relationship matrix and the multi-agent coupling relationship matrix C, through matrix operations, from Extract the information closely related to the current task of the current agent from C and input it into the evaluation network of the policy network. The formula is: (15) Among them, I represents the input vector of the evaluation network, represents the element-by-element multiplication operation of the multi-task coupling relationship matrix and the multi-agent coupling relationship matrix, and W is the weight matrix, which is used to adjust the influence of each element on the evaluation network input; After receiving the input vector I, the evaluation network processes it and outputs the evaluation result of the current task. According to the evaluation result, combined with the current state and task goal of the intelligent agent, a heuristic search task coupling planning algorithm is adopted based on the greedy strategy to generate a task plan that meets actual needs.

[0015] Preferably, the heuristic search task coupling planning algorithm is specifically: Initialization, set the initial state of the agent to , the task target is G, the current time is t=0, and the task planning is an empty set ; Evaluation function, defines the evaluation function f(S, P, G), which is used to evaluate the difficulty or quality of achieving the task goal G under a given state S and task plan P. The formula is: (16) in, Indicates the distance from the current state S to the task target G, represents the cost of task planning P, represents the execution time of task plan P, , , Both represent weight coefficients, which are used to adjust the relative importance of each indicator in the evaluation function; Heuristic search: From the current state Start by generating a set of all next actions ; For each action , after executing the action, the new state is obtained , and update the mission plan ; Calculate the evaluation function , and select the action that makes the evaluation function value optimal as the optimal action for the current step; Repeat the above steps until the task goal G is reached or no new actions can be generated; Output task planning, the final task planning P is from the initial state To the optimal solution of task goal G.

[0016] Preferably, step S3 specifically includes: S31. The setting of the theoretical framework of multi-agent migration strategy is as follows: First, set a series of source tasks and initialize them as mutually orthogonal vector sets ,in Representative The vector representation of each source task is processed one by one using the orthogonalization algorithm to obtain the orthogonalized task representation set. ; Next, search for the task representation that is most similar to the new task in the orthogonalized task representation set, denoted as ; Then, initialize a task representation vector z and fix the source task representation y to build a parameterized forward model that can predict the transfer function and the loss of the reward function Y(v,c), where v and Represent the current state and the next state respectively, and c represents the executed action; the task representation vector z is updated by the gradient descent optimization method so that the new task can be retained in the task space; then, the task representation is further learned by modeling the transition function and the reward function; in order to handle the interactive and non-interactive actions in the task, a population-invariant network structure is designed. For non-interactive actions, the traditional deep network is directly applied to calculate its Q value. The specific formula is: (17) Indicates the Q value between the current state v and the executed action c in a non-interactive environment; Represents the functional relationship between the current state v and the executed action c in a non-interactive environment; For interactive actions, a shared network is used , the observation part related to the corresponding entity And the connection z represented by the task is taken as input, and the Q value estimation of the corresponding interaction action is output, the formula is: (18) Indicates the Q value between the current state v and the executed action c in an interactive environment; Finally, the calculated Q value is combined with the task representation and input into the hybrid network to generate new tasks and perform task coupling planning. The hybrid network can comprehensively consider the correlation between tasks and realize the effective transfer and utilization of knowledge. S32. The embodied development of learners’ higher-order thinking based on multi-agent transfer strategies, specifically: Based on the multi-agent transfer strategy, through the synergy between agents, new tasks are efficiently generated and optimized, and then the deep transfer and integration of knowledge are promoted through the planning comparison between the source task and the new task. This process not only strengthens the learner's mastery of existing knowledge, but also shows significant results in promoting the embodied development of learners' higher-order thinking. The source task planning set is defined as , the new task planning set is ; In order to quantify the transfer potential between the source task and the new task, a transfer evaluation model based on task feature similarity is adopted, and the formula is: (19) in, Represents the first A mission plan, represents the hth task plan in the new task plan set, Indicates The extraction function of the task feature maps the task planning to the feature space, Represents the similarity measure between features, which is used to calculate the similarity between two feature vectors. Indicates The weight of a feature reflects the importance of the feature in the migration evaluation, and d represents the total number of features, that is, the number of task features considered.

[0017] The model evaluates the migration potential between the source task plan and the new task plan by calculating the weighted sum of their similarities in various feature spaces.

[0018] Preferably, the orthogonalization algorithm in step S31 is specifically: a. Initialize the orthogonal vector set to an empty set; b. For each original vector , calculate its inner product with each vector in the orthogonalized vector set to obtain the projection component; c. From the original vector Subtract all projection components from the vector to obtain the orthogonalized vector ; d. Add to the set of orthogonal vectors; e. Repeat steps b to d until all original vectors are orthogonalized.

[0019] Therefore, the present invention adopts the above-mentioned multi-agent task planning method for the development of high-order thinking under a mixed game, and the high-level thinking characteristics in the collected data related to the learners' high-order thinking show a significant upward trend compared with the previous ones. In particular, in terms of key features reflecting high-order thinking abilities such as problem solving, critical thinking and innovation, learners' performance has been significantly improved. This achievement provides solid support and strong impetus for the embodied development of learners' high-order thinking.

[0020] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a multi-agent task planning method for developing high-order thinking under a hybrid game of the present invention; Figure 2 A schematic diagram of multi-agent reinforcement learning based on two-stage intention sharing of the present invention; Figure 3 This is a schematic diagram of multi-agent task coupling planning for learners' high-order thinking inspired by the hybrid game of the present invention. DETAILED DESCRIPTION

[0022] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] like Figure 1 As shown in the figure, a multi-agent task planning method for the development of high-order thinking under hybrid games: Learners' higher-order thinking: refers to the advanced thinking ability that learners demonstrate in the process of cognitive development, which goes beyond basic cognitive skills (such as memory, understanding, etc.). It covers multiple dimensions such as critical thinking, creative thinking, problem-solving ability, logical reasoning, metacognitive monitoring, and self-regulation. It is the ability of learners to comprehensively use existing knowledge, skills, and strategies to conduct in-depth analysis, comprehensive judgment, creative solutions, and innovative thinking when facing complex and unstructured problems. Specifically, learners' higher-order thinking is reflected in the following aspects: (1) Critical thinking: Learners are able to think independently, identify, analyze and evaluate information, form their own opinions and judgments, and not blindly accept the opinions or information of others.

[0024] (2) Creative thinking: Learners are able to break through the traditional thinking framework and propose novel and unique ideas, methods and solutions, demonstrating strong innovation ability and imagination.

[0025] (3) Problem-solving ability: When faced with complex problems, learners are able to identify the nature of the problem, develop solutions, and effectively implement and evaluate the effectiveness of the solutions to achieve effective problem solving.

[0026] (4) Logical reasoning: Learners can reason and deduce based on known information and logical rules to reach reasonable conclusions and judgments.

[0027] (5) Metacognitive monitoring: Learners are able to monitor and regulate their own cognitive processes, clarify their learning goals and strategies, and reflect on and adjust their learning behaviors in a timely manner.

[0028] (6) Self-regulation: Learners can flexibly adjust their mindset and behavior according to their own learning situation and changes in the external environment, and maintain a positive learning attitude and efficient learning state.

[0029] Multi-agent hybrid game: refers to a system that includes multiple agents with autonomous decision-making capabilities. Each agent adopts a variety of action plans including pure strategies and hybrid strategies in the strategy space according to certain rules and constraints. In this process, the agent not only needs to consider the direct consequences of its own actions, but also needs to predict and respond to the possible reactions of other agents and the potential impact of these reactions on its own interests. Through the dynamic interactive process of mutual competition and cooperation, a complex game framework is designed to maximize the interests of each party.

[0030] The following steps are involved: S1. Multi-agent reinforcement learning based on two-stage intention sharing under the guidance of learners' high-order thinking; S11. Acquisition and processing of relevant data of learners' higher-order thinking based on multi-agents; Through advanced language recognition, image recognition and natural language processing technologies, learners’ thinking characteristics in high-level thinking in-depth discussions such as text are captured in real time, including but not limited to logical reasoning paths in images, critical thinking expressions in text, and innovative viewpoints. The capture formula is: (1) in, Represents a dataset of high-level thinking and in-depth discussion content. Represents the capture function, Indicates speech recognition, represents image recognition, Represents the output of the natural language processing module; After preprocessing, the captured high-order thinking data is input into the multi-agent system for collaborative processing. The multi-agent system conducts in-depth mining and analysis of the high-order thinking data through its internal communication and collaboration mechanism, and extracts key features that can reflect the learner's high-order thinking ability. The formula is: (2) in, Represents higher-order thinking eigenvector geometry, Represents the processing function, Represent the communication and cooperation mechanism of multi-agent system; S12, Multi-agent reinforcement learning based on two-stage intention sharing; This embodiment uses a two-stage intention sharing mechanism to reinforce multi-agent learning, aiming to optimize the collaborative decision-making process of the multi-agent system. Figure 2 Each agent Each maintains a value network First, the high-level thinking in-depth discussion content dataset is analyzed independently. Generate a preliminary intention action , the specific formula is: (3) Then, the first round of intention sharing begins, and the agent sends its intended action Broadcast to all other agents in the system (assuming that the total number of agents in the system is N); after the first round of intention sharing is completed, each agent not only , and also combined with the intention action information received from other agents , assess the importance of this information to one’s current decision; The V value is calculated to measure the potential impact of each received intention action information on its decision-making process. Specifically, for a given action k, the agent Provide Agent Action information, agent The maximum value of V when taking different optimal actions and minimum value , the specific formula is: (4) (5) In order to establish a unified quantitative standard, the maximum and minimum values ​​are normalized. The specific formula is: (6) If the importance of an agent's intentional action information to another agent's decision exceeds a preset threshold , then the agent will be included in the latter's dependent object set; Then, we enter the second round of intention sharing. In this stage, each agent will send its set of dependent objects to all other agents in the system. Through this round of information exchange, the agents can build a more comprehensive dependency graph, which is a directed graph that is used to intuitively display the dependency relationships between agents.

[0031] However, it should be noted that there may be directed loops in the initially constructed dependency graph, i.e., circular dependencies. To solve this problem, this embodiment uses an algorithm to detect and eliminate circular dependencies in the dependency graph, thereby ensuring that the final dependency graph is a directed acyclic graph, as follows: 1. Initialization, for all nodes , set visited(r)=false and onStack(r)=false; visited(r) is a status flag used to indicate whether the node r has been visited. visited(r) = false means that in the initialization phase, it is assumed that all nodes have not been visited. onStack(r) is also a status flag used to indicate whether the node r is currently in the DFS stack. onStack(r) = false means that in the initialization phase, it is assumed that all nodes are not in the stack; 2. Depth-first search (DFS), for each unvisited node r, do the following: Will Mark as visited, set visited(r)=true; Will Push into DFS stack, set onStack(r)=true; for For each adjacent node w of (i.e., there is a directed edge from r to w), if w has not been visited, recursively perform DFS on w. If w has been visited and is still in the stack (i.e., visited(w)=true and onStack(r)=true), a circular dependency is detected; Will Pop from the stack and set onStack(r)=false; 3. Circular dependency elimination: Once a circular dependency is detected, the dependency relationship is adjusted to ensure that the resulting graph is a directed acyclic graph; After obtaining the acyclic dependency graph, the decision-making process of the agent will follow the following rules: If an agent is dependent on other agents, it cannot make a new decision, and its final action will directly adopt its initially generated intention action On the contrary, if an agent is not dependent on any agent, it will make a new decision based on the intention action information of the dependent agent and its own local observations to optimize its action selection.

[0032] S2, Multi-agent task coupling planning for learners' high-order thinking inspired by hybrid games; S21. Construction of multi-agent model inspired by hybrid game; In a multi-agent system, each agent tries to maximize its benefits by optimizing its own strategy. However, due to the mutual influence between agents, traditional single-agent reinforcement learning methods often fail to achieve ideal results. Therefore, this embodiment constructs a multi-agent model inspired by hybrid games, aiming to achieve efficient strategy updates through mutual learning and prediction between agents. Assume that the learning process of other agents has an impact on the agent. The formula for the impact is: (7) in, Representing an Agent The policy update amount, Representing an Agent The learning rate, Representing an Agent The objective function is Representing an Agent strategy, represents the estimated value of other agents’ strategies, Indicates the gradient operation of the strategy update of agent i; In order to more accurately capture the mutual influence between agents, based on the mean field theory, the average action of the surrounding agents is used as input to influence and optimize the strategy learning of the agent itself. The strategy update formula is: (8) in, Representing an Agent The strategy before the update, Representing an Agent The updated strategy, Representing an Agent The state-action value function, represents the average action of other agents; In order to improve the effectiveness of the agent's strategy in a non-stationary environment, a deep Bayesian strategy reuse method is combined, a neural network is used as a value function approximator, and a distillation strategy network is introduced to achieve efficient strategy learning and reuse, and higher cumulative rewards and convergence performance are achieved in multiple random game scenarios. Among them, the update formula of the value function approximator is: (9) in, is the parameter of the value function approximator, s is the current state, is the immediate reward for the agent; Finally, ensure that the agent's algorithm can reach the Nash equilibrium, that is, the optimal response based on the strategies of all agents. The formula is: (10) in, Representing an Agent strategy, In addition to the intelligent agent The joint strategy of other agents, Representing an Agent The utility value of represents the optimal response strategy that satisfies all agents.

[0033] S22. Strategy optimization of multi-agent model for high-level thinking inspired by hybrid games; We optimize the strategy of the multi-agent model inspired by hybrid games. Through the gradient-based optimization strategy, we achieve efficient joint optimization of the agent strategy and the model in each training round. In order to achieve the above optimization goals, we first transform the strategy function and helper functions Parameterized, respectively and , then, the gradient-based method is used to update these parameters. The specific update rule formula is: (11) (12) in, represents the objective function of expected cumulative reward, Represents the objective function of the policy parameters The gradient of Represents the objective function for auxiliary parameters The gradient of represents the update step size of the policy parameters, Indicates the update step size of auxiliary parameters; During the training process, the state and action of the agent at each moment are evaluated through the neural network value function. First, the learner's high-level thinking data sample tasks are preliminarily processed, and then the current state of all tasks is input. and actions , using the neural network value function Output the evaluation value at this moment; in order to continuously optimize the neural network value function, define a loss function , which is used to measure the difference between the output of the neural network value function and the actual reward. Subsequently, the gradient descent method is used to update the parameters of the neural network value function. , the specific update rule formula is: (13) Among them, ξ is the update step size of the neural network value function parameters; By constructing an optimized objective function to maximize the weighted sum of the expected cumulative internal and external rewards, the objective function formula is: (14) in, Indicates that the agent is in state Next action Internal rewards obtained when Represented by the auxiliary function External rewards generated, represents the weight coefficient that regulates the relative importance of internal and external rewards, and T represents the time step of the training round; The above training process is repeated continuously until the task faced by the agent reaches the predetermined indicators.

[0034] S23, Multi-agent task coupling planning for learners' high-order thinking inspired by hybrid games; In this embodiment, in a multi-agent model inspired by hybrid games, coupling planning is performed on learners' high-order thinking data tasks. By constructing a multi-task coupling relationship matrix and a multi-agent coupling relationship matrix, the coupling relationship between each agent and its tasks is accurately described, providing strong support for collaborative decision-making. Figure 3 shown.

[0035] First, a multi-task coupling relationship matrix is ​​set up. The multi-task coupling relationship matrix is ​​formed by the various outputs of the multi-task strategy network of a certain agent, so as to show the coupling relationship between the outputs, so as to reflect the coupling relationship between the multi-tasks faced by the current agent and support the input reconstruction of the evaluation network of each task of each agent. Assume that the strategy network of task R has output, task The policy network has outputs, and p, q = 1, 2, ..., n, n is the total number of tasks; then construct a Multi-task coupling relationship matrix ,in , , if the oth output of task R is equal to task There is a coupling relationship between the lth output of , the corresponding element of the output is 1, and the rest are 0; Then, each agent forms a multi-agent coupling relationship matrix. According to the actual application situation, it is judged whether there is a collaborative relationship between the agents. This is shown in the multi-agent coupling relationship matrix, which reflects the collaborative relationship between the multi-agents and supports the input reconstruction of the evaluation network of each task of each agent. If there are N agents in the agent cluster, an N×N multi-agent coupling relationship matrix C is constructed, where 1≤i≤N, 1≤j≤N. If there is a collaborative relationship between agent i and agent j, the corresponding output element is 1, and the rest are 0. According to the multi-task coupling relationship matrix and the multi-agent coupling relationship matrix C, through matrix operations, from Extract the information closely related to the current task of the current agent from C and input it into the evaluation network of the policy network. The formula is: (15) Among them, I represents the input vector of the evaluation network, represents the element-by-element multiplication operation of the multi-task coupling relationship matrix and the multi-agent coupling relationship matrix, and W is the weight matrix, which is used to adjust the influence of each element on the evaluation network input; After receiving the input vector I, the evaluation network processes it and outputs the evaluation result of the current task. According to the evaluation result, combined with the current state and task goal of the intelligent agent, a heuristic search task coupling planning algorithm is adopted based on the greedy strategy to generate a task plan that meets actual needs.

[0036] The specific heuristic search task coupling planning algorithm is as follows: 1. Initialization, set the initial state of the agent to , the task target is G, the current time is t=0, and the task planning is an empty set ; 2. Evaluation function, define the evaluation function f(S, P, G), which is used to evaluate the difficulty or quality of achieving the task goal G under a given state S and task plan P. The formula is: (16) in, Indicates the distance from the current state S to the task target G, represents the cost of task planning P, represents the execution time of task plan P, , , Both represent weight coefficients, which are used to adjust the relative importance of each indicator in the evaluation function; 3. Heuristic Search: From the current state Start by generating a set of all next actions ; For each action , after executing the action, the new state is obtained , and update the mission plan ; Calculate the evaluation function , and select the action that makes the evaluation function value optimal as the optimal action for the current step; Repeat the above steps until the task goal G is reached or no new actions can be generated; 4. Output the task plan. The final task plan P is obtained from the initial state To the optimal solution of task goal G.

[0037] S3, embodied development of learners’ higher-order thinking based on multi-agent transfer strategies; S31. Setting up the theoretical framework of multi-agent migration strategy; This embodiment transfers knowledge based on the multi-agent transfer strategy for learners' high-level thinking task coupling planning. First, a series of source tasks are set and initialized as mutually orthogonal vector sets. ,in Representative The vector representation of the source tasks is processed one by one using the orthogonalization algorithm as follows to obtain the orthogonalized task representation set. ; 1. Initialize the orthogonal vector set to an empty set; 2. For each original vector , calculate its inner product with each vector in the orthogonalized vector set to obtain the projection component; 3. From the original vector Subtract all projection components from the vector to obtain the orthogonalized vector ; 4. Add to the set of orthogonal vectors; 5. Repeat steps 2 to 4 until all original vectors are orthogonalized.

[0038] Next, search for the task representation that is most similar to the new task in the orthogonalized task representation set, denoted as ; Then, initialize a task representation vector z and fix the source task representation y to build a parameterized forward model that can predict the transfer function And the reward function The loss of v and Represent the current state and the next state respectively, and c represents the executed action; the task representation vector z is updated by the gradient descent optimization method so that the new task can be retained in the task space; then, the task representation is further learned by modeling the transition function and the reward function; in order to handle the interactive and non-interactive actions in the task, a population-invariant network structure is designed. For non-interactive actions, the traditional deep network is directly applied to calculate its Q value. The specific formula is: (17) Indicates the Q value between the current state v and the executed action c in a non-interactive environment; Represents the functional relationship between the current state v and the executed action c in a non-interactive environment; For interactive actions, a shared network is used , the observation part related to the corresponding entity And the connection z represented by the task is taken as input, and the Q value estimation of the corresponding interaction action is output, the formula is: (18) Indicates the Q value between the current state v and the executed action c in an interactive environment; Finally, the calculated Q value is combined with the task representation and input into the hybrid network to generate new tasks and perform task coupling planning. The hybrid network can comprehensively consider the correlation between tasks and realize the effective transfer and utilization of knowledge. S32, embodied development of learners’ higher-order thinking based on multi-agent transfer strategy; This embodiment realizes knowledge transfer through planning and comparison of source tasks and new tasks, and promotes the embodied development of learners' high-order thinking, which is specifically manifested in the improvement of key features that can reflect learners' high-order thinking ability.

[0039] Based on the multi-agent transfer strategy, through the synergy between agents, new tasks are efficiently generated and optimized, and then the deep transfer and integration of knowledge are promoted through the planning comparison between the source task and the new task. This process not only strengthens the learner's mastery of existing knowledge, but also shows significant results in promoting the embodied development of learners' higher-order thinking. The source task planning set is defined as , the new task planning set is ; In order to quantify the transfer potential between the source task and the new task, a transfer evaluation model based on task feature similarity is adopted, and the formula is: (19) in, Represents the first A mission plan, represents the hth task plan in the new task plan set, Indicates The extraction function of the task feature maps the task planning to the feature space, Represents the similarity measure between features, which is used to calculate the similarity between two feature vectors. Indicates The weight of a feature reflects the importance of the feature in the migration evaluation, and d represents the total number of features, that is, the number of task features considered.

[0040] The model evaluates the migration potential between the source task plan and the new task plan by calculating the weighted sum of their similarities in various feature spaces.

[0041] Therefore, the present invention adopts the above-mentioned multi-agent task planning method for the development of high-order thinking under a mixed game, and the high-level thinking characteristics in the collected data related to the learners' high-order thinking show a significant upward trend compared with the previous ones. In particular, in the key features reflecting high-order thinking ability such as problem solving, critical thinking and innovation ability, the learners' performance has been significantly improved, providing solid support and strong impetus for the embodied development of learners' high-order thinking.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-agent task planning method for the development of high-order thinking under hybrid games, characterized in that: The following steps are involved: S1. Multi-agent reinforcement learning based on two-stage intention sharing under the guidance of learners' high-order thinking; S2. Multi-agent task coupling planning for learners’ high-order thinking inspired by hybrid games, including: S21. Construction of multi-agent model inspired by hybrid game; S22. Strategy optimization of multi-agent model for high-level thinking inspired by hybrid games; S23, Multi-agent task coupling planning for learners' high-order thinking inspired by hybrid games; S3. Embodied development of learners’ higher-order thinking based on multi-agent transfer strategy.

2. According to claim 1, a multi-agent task planning method for the development of high-order thinking under a hybrid game is characterized in that: Step S1 specifically includes: S11. Acquisition and processing of relevant data of learners’ high-order thinking based on multi-agents, specifically: real-time capture of learners’ thinking characteristics in high-order thinking in-depth discussions, the capture formula is: （1） in, Represents a dataset of high-level thinking and in-depth discussion content. Represents the capture function, Indicates speech recognition, represents image recognition, Represents the output of the natural language processing module; After preprocessing, the captured high-order thinking data is input into the multi-agent system for collaborative processing. The multi-agent system conducts in-depth mining and analysis of the high-order thinking data through its internal communication and collaboration mechanism, and extracts the key features that reflect the learner's high-order thinking ability. The formula is: （2） in, Represents higher-order thinking eigenvector geometry, Represents the processing function, Represent the communication and cooperation mechanism of multi-agent system; S12. Multi-agent reinforcement learning based on two-stage intention sharing.

3. According to claim 2, a multi-agent task planning method for the development of high-order thinking under a hybrid game is characterized in that: Step S12 specifically includes: Each agent Each maintains a value network First, the high-level thinking in-depth discussion content dataset is analyzed independently. Generate a preliminary intention action , the specific formula is: （3） Then, the first round of intention sharing begins, and the agent sends its intended action Broadcast to all other agents in the system. Assume that the total number of agents in the system is N. After the first round of intention sharing is completed, each agent not only , and also combined with the intention action information received from other agents , assess the importance of this information to one’s current decision; The V value is calculated to measure the potential impact of each received intention action information on its decision-making process. Specifically, for a given action k, the agent Provide Agent Action information, agent The maximum value of V when taking different optimal actions and minimum value , the specific formula is: （4） （5） The maximum and minimum values ​​are normalized, and the specific formula is: （6） If the importance of an agent's intentional action information to another agent's decision exceeds a preset threshold , then the agent will be included in the latter's dependent object set; Then, the second round of intention sharing begins. In this phase, each agent sends its set of dependent objects to all other agents in the system. Through this round of information exchange, the agents build a dependency graph.

4. According to claim 3, a multi-agent task planning method for the development of high-order thinking under a hybrid game is characterized in that: In step S12, a circular dependency removal algorithm is used to detect and eliminate circular dependencies in the dependency graph, specifically: Initialize all nodes , set visited(r)=false and onStack(r)=false; visited(r) is a status flag used to indicate whether the node r has been visited. visited(r) = false means that in the initialization phase, it is assumed that all nodes have not been visited. onStack(r) is also a status flag used to indicate whether the node r is currently in the DFS stack. onStack(r) = false means that in the initialization phase, it is assumed that all nodes are not in the stack. Depth-first search, for each unvisited node r, does the following: Will Mark as visited, set visited(r)=true; Will Push into DFS stack, set onStack(r)=true; for For each adjacent node w of , if w has not been visited, recursively perform a depth-first search on w. If w has been visited and is still in the stack, a circular dependency is detected; Will Pop from the stack and set onStack(r)=false; Circular dependency elimination, once a circular dependency is detected, the dependencies are adjusted to ensure that the resulting graph is a directed acyclic graph; After obtaining the acyclic dependency graph, the decision-making process of the agent will follow the following rules: If an agent is dependent on other agents, it cannot make a new decision, and its final action will directly adopt its initially generated intention action ; On the contrary, if an agent is not dependent on any agent, it will make a new decision based on the intention action information of the dependent agent and its own local observations to optimize its action selection.

5. The multi-agent task planning method for developing high-order thinking under a hybrid game according to claim 1, characterized in that: Step S21 specifically includes: Assume that the learning process of other agents has an impact on the agent The formula for the impact is: （7） in, Representing an Agent The policy update amount, Representing an Agent The learning rate, Representing an Agent The objective function is Representing an Agent strategy, represents the estimated value of other agents’ strategies, Indicates the gradient operation of the strategy update of agent i; Based on the mean field theory, the average action of the surrounding intelligent agents is used as input to influence and optimize the strategy learning of the intelligent agent. The strategy update formula is: （8） in, Representing an Agent The strategy before the update, Representing an Agent The updated strategy, Representing an Agent The state-action value function, represents the average action of other agents; Combined with the deep Bayesian strategy reuse method, a neural network is used as the value function approximator, and a distillation strategy network is introduced to achieve efficient strategy learning and reuse. The update formula of the value function approximator is: （9） in, is the parameter of the value function approximator, s is the current state, is the immediate reward for the agent; Finally, ensure that the agent's algorithm receives Nash equilibrium, the formula is: （10） in, Representing an Agent strategy, In addition to the intelligent agent The joint strategy of other agents, Representing an Agent The utility value of represents the optimal response strategy that satisfies all agents.

6. The multi-agent task planning method for developing high-order thinking under a hybrid game according to claim 1, characterized in that: Step S22 specifically includes: First, the policy function and helper functions Parameterized, respectively and , then, the gradient-based method is used to update these parameters. The specific update rule formula is: （11） （12） in, represents the objective function of expected cumulative reward, Represents the objective function of the policy parameters The gradient of Represents the objective function for auxiliary parameters The gradient of represents the update step size of the policy parameters, Indicates the update step size of auxiliary parameters; During the training process, the state and action of the agent at each moment are evaluated through the neural network value function. First, the learner's high-level thinking data sample tasks are preliminarily processed, and then the current state of all tasks is input. and actions , using the neural network value function Output the evaluation value at this moment; define a loss function , which is used to measure the difference between the output of the neural network value function and the actual reward. Subsequently, the gradient descent method is used to update the parameters of the neural network value function. , the specific update rule formula is: （13） Among them, ξ is the update step size of the neural network value function parameters; By constructing an optimized objective function to maximize the weighted sum of the expected cumulative internal and external rewards, the objective function formula is: （14） in, Indicates that the agent is in state Next action Internal rewards obtained when Represented by the auxiliary function The external rewards generated, represents the weight coefficient that regulates the relative importance of internal and external rewards, and T represents the time step of the training round; The above training process is repeated continuously until the task faced by the agent reaches the predetermined indicators.

7. The multi-agent task planning method for developing high-order thinking under a hybrid game according to claim 1, characterized in that: Step S23 specifically includes: First, set up a multi-task coupling relationship matrix, and use the outputs of a multi-task strategy network of a certain agent to form a multi-task coupling relationship matrix. Suppose the strategy network of task R has output, task The policy network has outputs, and p, q = 1, 2, ..., n, n is the total number of tasks; then construct a Multi-task coupling relationship matrix ,in , , if the oth output of task R is equal to task There is a coupling relationship between the lth output of , the corresponding element of the output is 1, and the rest are 0; Then, each agent forms a multi-agent coupling relationship matrix. According to the actual application situation, it is judged whether there is a collaborative relationship between the agents, which is shown in the multi-agent coupling relationship matrix. There are N agents in the agent cluster, so an N×N multi-agent coupling relationship matrix C is constructed, where 1≤i≤N, 1≤j≤N. If there is a collaborative relationship between agent i and agent j, the corresponding element is output as 1, and the rest are 0. According to the multi-task coupling relationship matrix and the multi-agent coupling relationship matrix C, through matrix operations, from Extract the information closely related to the current task of the current agent from C and input it into the evaluation network of the policy network. The formula is: （15） Among them, I represents the input vector of the evaluation network, represents the element-by-element multiplication operation of the multi-task coupling relationship matrix and the multi-agent coupling relationship matrix, and W is the weight matrix; After receiving the input vector I, the evaluation network processes it and outputs the evaluation result of the current task. According to the evaluation result, combined with the current state and task goal of the intelligent agent, a heuristic search task coupling planning algorithm is adopted based on the greedy strategy to generate a task plan that meets actual needs.

8. The multi-agent task planning method for developing high-order thinking under a hybrid game according to claim 7, characterized in that: The specific heuristic search task coupling planning algorithm is as follows: Initialization, set the initial state of the agent to , the task target is G, the current time is t=0, and the task planning is an empty set ; Evaluation function, define the evaluation function f(S,P,G), the formula is: （16） in, Indicates the distance from the current state S to the task target G, represents the cost of task planning P, represents the execution time of task plan P, , , Both represent weight coefficients, which are used to adjust the relative importance of each indicator in the evaluation function; Heuristic search: From the current state Start by generating a set of all next actions ; For each action , after executing the action, the new state is obtained , and update the mission plan ; Calculate the evaluation function , and select the action that makes the evaluation function value optimal as the optimal action for the current step; Repeat the above steps until the task goal G is reached or no new actions can be generated; Output task planning, the final task planning P is from the initial state To the optimal solution of task goal G.

9. The multi-agent task planning method for developing high-order thinking under a hybrid game according to claim 1, characterized in that: Step S3 specifically includes: S31. The setting of the theoretical framework of multi-agent migration strategy is as follows: First, set a series of source tasks and initialize them as mutually orthogonal vector sets ,in Representative The vector representation of each source task is processed one by one using the orthogonalization algorithm to obtain the orthogonalized task representation set. ; Next, search for a task representation similar to the new task in the orthogonalized task representation set, denoted as ; Then, initialize a task representation vector z and fix the source task representation y to build a parameterized forward model that predicts the transfer function and the loss of the reward function Y(v,c), where v and Represent the current state and the next state respectively, and c represents the executed action; the task representation vector z is updated using the gradient descent optimization method so that the new task can be retained in the task space; then, the task representation is further learned by modeling the transition function and the reward function; a population-invariant network structure is designed, and for non-interactive actions, the traditional deep network is directly applied to calculate its Q value. The specific formula is: （17） Indicates the Q value between the current state v and the executed action c in a non-interactive environment; Represents the functional relationship between the current state v and the executed action c in a non-interactive environment; For interactive actions, a shared network is used , the observation part related to the corresponding entity And the connection z represented by the task is taken as input, and the Q value estimation of the corresponding interaction action is output, the formula is: （18） Indicates the Q value between the current state v and the executed action c in an interactive environment; Finally, the calculated Q value is combined with the task representation and input into the hybrid network to generate new tasks and perform task coupling planning; S32. The embodied development of learners’ higher-order thinking based on multi-agent transfer strategies, specifically: Define the source task planning set as = , the new task planning set is ; A migration evaluation model based on task feature similarity is adopted, and the formula is: （19） in, Represents the first A mission plan, represents the hth task plan in the new task plan set, Indicates The extraction function of the task feature maps the task planning to the feature space, represents the similarity measure between features, Indicates The weight of a feature reflects the importance of the feature in migration evaluation, and d represents the total number of features.

10. The multi-agent task planning method for developing high-order thinking under a hybrid game according to claim 9, characterized in that: The orthogonalization algorithm in step S31 is specifically: a. Initialize the orthogonal vector set to an empty set; b. For each original vector , calculate its inner product with each vector in the orthogonalized vector set to obtain the projection component; c. From the original vector Subtract all projection components from the vector to obtain the orthogonalized vector '; d. Add to the set of orthogonal vectors; e. Repeat steps b to d until all original vectors are orthogonalized.

Images