Multi-agent reinforcement learning method and system based on large language model enhancement

Abstract

The invention discloses a multi-agent reinforcement learning method and system based on large language model enhancement, and belongs to the technical field of agent optimization. The method comprises the steps that environment information is obtained, the environment information is input into an enhanced large model, a decision tree is obtained, the large model comprises a planning module, a programming module, a description module and an analysis module, and the large model is enhanced through multi-agent debate; and a guiding strategy is generated based on the decision tree, knowledge of the guiding strategy is migrated into the agent model by adopting a strategy migration algorithm, an enhanced agent model is obtained, and the strategy migration algorithm comprises a same-strategy migration algorithm and a different-strategy migration algorithm. The method improves the learning efficiency and accuracy of the intelligent agent.

Description

Technical Field

[0001] This application belongs to the field of intelligent agent optimization technology, and in particular relates to a multi-agent reinforcement learning method and system based on large language model enhancement. Background Technology

[0002] Multi-agent reinforcement learning, as a core research branch in artificial intelligence, has crucial application value in complex decision-making scenarios such as intelligent game competition, multi-robot cooperative control, and distributed resource scheduling. However, with increasing task complexity and the growing number of agents, traditional methods face increasingly severe challenges in terms of exploration efficiency, cooperative stability, and computational overhead.

[0003] Taking dynamic adversarial scenarios such as multi-agent soccer simulation as an example, agents need to achieve efficient collaboration and rapid decision-making in a real-time environment. Traditional multi-agent reinforcement learning methods often suffer from slow training convergence and limited performance due to sparse rewards, inefficient policy exploration, and difficulty in coordinating agent behaviors. Existing large language model enhancement methods attempt to alleviate these problems by introducing their powerful semantic understanding and policy generation capabilities. Based on the intervention method of the large language model, they can be mainly divided into two categories: direct interaction and indirect guidance. The direct interaction method uses the real-time reasoning ability of the large language model to replace or assist the agent in making decisions at each step. Although this method can provide high-quality decision examples, in tasks with a large number of agents and long decision steps, frequent calls to the large language model will lead to an exponential increase in computational overhead, making it difficult to deploy in practical systems.

[0004] Existing methods suffer from inherent defects in highly dynamic scenarios requiring real-time collaboration, such as football simulation. These defects include limited policy quality, susceptibility to local optima, and difficulty in dynamically adapting to multi-agent collaborative patterns. Consequently, these methods lead to technical problems such as low agent exploration efficiency, unstable collaborative behavior, and difficulty in achieving overall training convergence. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a multi-agent reinforcement learning method and system based on large language model enhancement, which improves the learning efficiency and accuracy of the agents.

[0006] Firstly, this application provides a multi-agent reinforcement learning method based on large language model enhancement, the method comprising: Environmental information is acquired and input into an enhanced large model to obtain a decision tree. The large model includes a planning module, a programming module, a description module, and an analysis module. The large model is enhanced through multi-agent debate. Based on the decision tree, a guiding policy is generated, and the knowledge of the guiding policy is transferred to the agent model using a policy transfer algorithm to obtain an enhanced agent model. The policy transfer algorithm includes same-policy transfer algorithm and different-policy transfer algorithm.

[0007] According to one embodiment of this application, the step of inputting environmental information into the enhanced large model to obtain a decision tree includes: Environmental information is input into the planning module, and pseudocode is generated by calling the prior knowledge of the large model; The pseudocode is input into the programming module for code conversion, resulting in executable Python code; Input Python code and environmental information into the description module to interact and collect trajectory information, and obtain description information in natural language format; The executable Python code and its description in natural language format are input into the analysis module. The executable Python code is evaluated based on the description information, and a decision tree is generated.

[0008] According to one embodiment of this application, the step of inputting environmental information into the planning module and generating pseudocode by calling prior knowledge of the large model includes: Environmental information, including the initial location and site attributes, is input into the planning module. The planning module is preset to be an expert in the target field. Based on the planning module, sample prompts are generated. The environmental information and the sample prompts are combined and processed. The prior knowledge of the large language model is invoked to identify the key tactical logic and action sequence applicable to the current environment. Based on the identified logic and action sequence, pseudocode is generated.

[0009] According to one embodiment of this application, the step of inputting pseudocode into a programming module for code conversion to obtain executable Python code includes: The programming module is preset to the role of a programming expert, and environmental interaction specifications are set, including observation format, available action set and environmental interaction interface; The pseudocode, the environment interaction specifications, and the preset code format requirements are input into the programming module, which converts the pseudocode into executable Python code that conforms to the environment interaction interface call format.

[0010] According to one embodiment of this application, the step of inputting Python code and environment information into the description module for interaction and collection of trajectory information to obtain description information in natural language format includes: The Python code is fused with environmental information to obtain trajectory data in NPZ format. The trajectory data in NPZ format includes multiple NumPy arrays, each of which is used to store observation, action, reward and corresponding time step information. The trajectory data in NPZ format is input into the description module, and the data of each NumPy array is split to obtain structured JSON data; Key information is extracted from the structured JSON data and converted into natural language descriptive text to obtain descriptive information in natural language format.

[0011] According to one embodiment of this application, the step of inputting executable Python code and natural language format description information into the analysis module, evaluating the executable Python code based on the description information, and generating a decision tree includes: The description information in natural language format, the executable Python code, and the results after code execution are input into the analysis module, which is preset to be a data analyst with professional capabilities. The analysis module analyzes the description information in the natural language format, evaluates the effect of the executable Python code, and generates an evaluation result, which includes an analysis of advantages and disadvantages and suggestions for improvement. The evaluation results are sent to the planning module to optimize the executable Python code and iteratively generate a performance-optimized decision tree.

[0012] According to one embodiment of this application, the same-strategy migration algorithm includes: The action probability distribution and KL divergence are calculated based on the guidance strategy, and the action probability distribution and KL divergence are used as the strategy distance metric. The KL divergence is added as a regularization term to the policy network loss function to update the guidance policy; By dynamically balancing the weights of the autonomous and autonomous strategies through the autonomous strategy adjustment factor and the guidance strategy decay factor, the influence of the guidance is gradually reduced as the training process progresses, thereby performing policy transfer on the agent model. The heterogeneous policy migration algorithm includes: replacing the agent's behavior policy with a guiding policy and interacting with the environment with a preset probability to obtain interaction data; The interaction data is stored in the experience replay buffer, and the data is sampled from the buffer to train the agent model; By controlling the proportion of guidance strategy usage through a guidance strategy decay factor, the intervention of guidance strategy is gradually reduced as the training process progresses, thereby enabling policy transfer to the agent model.

[0013] Secondly, this application provides a multi-agent reinforcement learning system based on a large language model, the system comprising: An acquisition module is used to acquire environmental information and input the environmental information into the enhanced large model to obtain a decision tree. The large model includes a planning module, a programming module, a description module, and an analysis module. The large model is enhanced through multi-agent debate. The processing module is used to generate a guiding policy based on a decision tree, and to transfer the knowledge of the guiding policy to the agent model using a policy transfer algorithm to obtain an enhanced agent model. The policy transfer algorithm includes a same-policy transfer algorithm and a different-policy transfer algorithm.

[0014] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-agent reinforcement learning method based on large language model enhancement as described in the first aspect above.

[0015] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-agent reinforcement learning method based on large language model enhancement as described in the first aspect above.

[0016] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the multi-agent reinforcement learning method based on large language model enhancement as described in the first aspect.

[0017] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the multi-agent reinforcement learning method based on large language model enhancement as described in the first aspect above.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0019] The present invention provides a multi-agent reinforcement learning method based on large language model enhancement, which has the following advantages over existing technologies: (1) This invention, by adopting a unified decision tree generation framework and policy transfer mechanism, can efficiently and stably inject the prior knowledge of a large language model into the multi-agent reinforcement learning process within a single framework. Through the collaboration of four modules—planning, programming, description, and analysis—and a multi-agent debate mechanism, the reliability of the generated policy of the large language model is improved. Furthermore, by using two transfer algorithms—one with the same policy and one with a different policy—the generated decision tree is used as a guiding policy to adaptively balance prior knowledge learning and autonomous exploration optimization. This can effectively solve the problems of low exploration efficiency and unstable collaboration among multi-agents, while significantly reducing the huge computational overhead caused by direct interaction of the large language model, and reducing the situation where agents get trapped in the local optimum of the guiding policy and the large language model interacts directly with the environment.

[0020] (2) In the same strategy algorithm, the present invention calculates the KL divergence between the action probability distribution of the guiding strategy and the action probability distribution of the agent to obtain the action distance between the two actions, which guides the update direction of the agent's target network; in the different strategy algorithm, the guiding strategy replaces the agent to interact with the environment to obtain trajectory data, and puts the trajectory data into the buffer pool so that the agent can be trained according to the historical experience of the guiding strategy, so that multiple agents can learn the methods of cooperation and exploration of rewards, reducing the problem of multiple agents having difficulty in cooperation and exploration of sparse reward environments.

[0021] (3) This invention drives the large language model to generate decision strategies through task description, and further derives decision tree code. It collects state-action trajectory data generated by the strategy generated by the large language model in the process of interacting with the environment, and converts the trajectory data into natural language to provide the large language model for further analysis and improvement, thereby improving the accuracy of the large language model. It also improves the performance of the large language model by using the multi-agent debate method and reduces the problem of cognitive degradation in the process of self-reflection. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts of the multi-agent reinforcement learning method based on large language model enhancement provided in the embodiments of this application; Figure 2 This is the second flowchart of the multi-agent reinforcement learning method based on large language model enhancement provided in the embodiments of this application; Figure 3 This is a schematic diagram of the process of generating a decision tree from a large language model provided in an embodiment of this application; Figure 4 This is a schematic diagram of the planning module processing provided in the embodiments of this application; Figure 5 This is a schematic diagram of the programming module processing provided in the embodiments of this application; Figure 6 This is a flowchart illustrating the processing of the description module provided in the embodiments of this application; Figure 7 This is a schematic diagram of the analysis module processing provided in the embodiments of this application; Figure 8 This is a schematic diagram of the framework for multi-agent debate provided in an embodiment of this application; Figure 9 This is a schematic diagram of the framework for policy transfer in reinforcement learning with the same policy provided in an embodiment of this application; Figure 10 This is a graph showing the change in rewards during 15 iterations of the self-reflection and multi-agent debate method provided in this application embodiment; Figure 11 This is a comparison chart of the changes in the number of goals during the training process of the same strategy algorithm provided in the embodiments of this application; Figure 12 This is a comparison chart of the number of goals scored by the same strategy migration algorithm provided in the embodiments of this application and the built-in AI. Figure 13 This is a schematic diagram of the framework for heterogeneous policy transfer in reinforcement learning provided in an embodiment of this application; Figure 14 This is a comparison chart of the changes in the number of goals during the training process of the heterogeneous strategy algorithm provided in this application embodiment; Figure 15 This is a comparison chart of the different strategy migration algorithm provided in this application embodiment and the number of goals scored by the built-in AI; Figure 16 This is a schematic diagram of the structure of a multi-agent reinforcement learning system based on a large language model enhancement provided in an embodiment of this application; Figure 17 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] Large language model-enhanced reinforcement learning is a method that uses the multimodal information processing, semantic generation, and logical reasoning capabilities of pre-trained large language models to provide prior knowledge to assist in the training of traditional reinforcement learning. Compared to traditional reinforcement learning methods, large language model-enhanced reinforcement learning can leverage the generalization knowledge of large language models to improve the capabilities of reinforcement learning algorithms. Based on the tasks handled by the large language model in the algorithm, it can be mainly classified into the following categories: (1) Information Processing: In task scenarios involving language or visual features, agents face cognitive bottlenecks in cross-modal semantic understanding. To alleviate the obstacles for agents in processing multimodal data, large language models can undertake the task of parsing language or visual features. The specific solution is that by extracting key feature information in the task, large language models can transform environmental information represented by natural language or visual information into the required task-specific language, thereby accelerating the training speed of the agent and reducing the learning difficulty.

[0025] (2) Reward design: In many complex tasks, there is a problem of sparse rewards or even no rewards at all. The large language model can use its inherent knowledge to build two different models to solve this problem: ① Indirect reward model, which transforms the environmental state into reward value based on the information of the environment during training; ② Direct reward model, which directly generates reward function based on code, and the agent directly obtains reward value based on the reward function designed by the large language model.

[0026] (3) Policy Optimization: To address the challenges of poor initial policy performance and long interaction time with the environment in reinforcement learning, large language models can provide policy optimization. There are two main solutions: ① Action Generation: Large language models reconstruct reinforcement learning as a sequential decision problem, driving the generation of conditional action policies through reward feedback. With large-scale pre-training data, large language models demonstrate excellent semantic understanding capabilities, which can accelerate policy convergence in offline reinforcement learning. ② Action Guidance: As a knowledge-based guide, large language models can generate a concise set of example actions. By simplifying the original action space into an optimized subset containing prior knowledge, exploration efficiency can be significantly improved. When this knowledge is integrated into the agent through policy regularization, better sample utilization can be achieved.

[0027] Figure 1 This is one of the flowcharts illustrating the multi-agent reinforcement learning method based on large language model enhancement provided in this application embodiment, such as... Figure 1 As shown, the multi-agent reinforcement learning method based on large language model enhancement includes steps 110 and 120.

[0028] Step 110: Obtain environmental information and input it into the enhanced large model to obtain a decision tree. The large model includes a planning module, a programming module, a description module, and an analysis module. The large model is enhanced through multi-agent debate. Figure 2 This is the second flowchart illustrating the multi-agent reinforcement learning method based on large language model enhancement provided in this application embodiment, as follows: Figure 2 As shown, multi-agent reinforcement learning methods based on large language models include methods for generating decision trees from large language models and reinforcement learning methods for policy transfer using decision trees.

[0029] In some embodiments, inputting environmental information into the enhanced large model to obtain a decision tree includes: Environmental information is input into the planning module, and pseudocode is generated by calling the prior knowledge of the large model; Figure 3 This is a schematic diagram of the process of generating a decision tree from a large language model according to an embodiment of this application, such as... Figure 3 As shown, the large language model comprises four modules: planning, programming, description, and analysis. It employs a self-reflective approach overall, while using multi-agent debate in the analysis module to address the cognitive regression issues arising from self-reflection.

[0030] Using football tactics as the background of a large language model, in a single iteration, the planning module generates pseudocode for football tactics based on the existing knowledge of the large language model. The programming module then converts the pseudocode into executable Python code. The code generated by the programming module interacts with the environment and collects trajectory data, which is then provided to the description module. The description module converts the symbolic trajectory data into natural language and provides it to the analysis module for analysis. The analysis module evaluates the quality of the generated code based on the trajectory data in natural language form.

[0031] In multi-agent reinforcement learning, it is difficult for different agents to learn how to cooperate with each other. To address this problem, when generating decision trees, corresponding policies are generated based on the positions and responsibilities of different players. The prior knowledge of a large language model provides the agents with a feasible plan for mutual cooperation.

[0032] In some embodiments, the step of inputting environmental information into the planning module and generating pseudocode by invoking prior knowledge of the large model includes: Environmental information, including the initial location and site attributes, is input into the planning module. The planning module is preset to be an expert in the target field. Based on the planning module, sample prompts are generated. The environmental information and the sample prompts are combined and processed. The prior knowledge of the large language model is invoked to identify the key tactical logic and action sequence applicable to the current environment. Based on the identified logic and action sequence, pseudocode is generated.

[0033] Figure 4 This is a flowchart illustrating the planning module processing provided in an embodiment of this application, as shown below. Figure 4 As shown, the planning module is responsible for generating basic decision-making methods. To facilitate processing by the programming module, the decision-making methods output by the planning module are not in natural language form, but in pseudocode form. Separating the decision-making construction from the programming tasks is necessary in this process. Directly outputting the decision code using a large language model can easily lead to performance degradation; outputting natural language or pseudocode can mitigate this problem.

[0034] Before generating decision-making methods, the planning module needs to be initially configured. In this stage, the planning module is configured as a futsal expert, and basic field information is set. The input to the planning module is the basic field information, including the initial positions of the players and field information; the output is pseudocode for specific tactics, including tactics used by different players and special handling for specific conditions. To improve the performance of the planning module, a few-shot hint technique is used in this environment, providing the planning module with some excellent tactical methods. The planning module can further optimize its strategy based on these hints. In subsequent iterations, the planning module will also update its strategy based on historical data and data from the analysis module.

[0035] In this embodiment, the environmental information can be converted into pseudocode through the design and planning module, which facilitates subsequent processing.

[0036] The pseudocode is input into the programming module for code conversion, resulting in executable Python code; In some embodiments, the step of inputting pseudocode into a programming module for code conversion to obtain executable Python code includes: The programming module is preset to the role of a programming expert, and environmental interaction specifications are set, including observation format, available action set and environmental interaction interface; The pseudocode, the environment interaction specifications, and the preset code format requirements are input into the programming module, which converts the pseudocode into executable Python code that conforms to the environment interaction interface call format.

[0037] Figure 5 This is a flowchart illustrating the processing of the programming module provided in an embodiment of this application, such as... Figure 5As shown, the main task of the programming module is to convert the pseudocode generated by the planning module into executable code. The programming module is initially configured as a programming expert capable of handling the conversion from pseudocode to Python code. During initial setup, the programming module needs to be provided with the Google Football observation format, usable actions, and environmental interaction methods. The input to this stage is the tactical pseudocode from the planning module, the required Google Football API, the required code format, and the output is executable code that meets the requirements.

[0038] Input Python code and environmental information into the description module to interact and collect trajectory information, and obtain description information in natural language format; In some embodiments, the step of inputting Python code and environment information into the description module for interaction and collection of trajectory information to obtain description information in natural language format includes: The Python code is fused with environmental information to obtain trajectory data in NPZ format. The trajectory data in NPZ format includes multiple NumPy arrays, each of which is used to store observation, action, reward and corresponding time step information. The trajectory data in NPZ format is input into the description module, and the data of each NumPy array is split to obtain structured JSON data; Key information is extracted from the structured JSON data and converted into natural language descriptive text to obtain descriptive information in natural language format.

[0039] Figure 6 This is a flowchart illustrating the processing of the description module provided in the embodiments of this application, such as... Figure 6 As shown, the description module does not directly depend on the large language model. In this framework, the description module is an automated tool for converting trajectory data into natural language, resembling a data pipeline in behavior. Its input is trajectory data in NPZ format, and its final output is a natural language description.

[0040] The initial input NPZ data contains multiple NumPy arrays, including observations of the environment, interactive actions, rewards, and corresponding time steps. When processing the data, the data in each NumPy array is first split into JSON data. This step converts the trajectory data into a more structured format, facilitating the subsequent conversion to natural language. After obtaining the JSON data, the description module further extracts key information from the JSON data and converts it into natural language, addressing the issue of JSON data being vague and unclear for large language models.

[0041] The executable Python code and its description in natural language format are input into the analysis module. The executable Python code is evaluated based on the description information, and a decision tree is generated.

[0042] In some embodiments, the step of inputting executable Python code and natural language description information into the analysis module, evaluating the executable Python code based on the description information, and generating a decision tree includes: The description information in natural language format, the executable Python code, and the results after code execution are input into the analysis module, which is preset to be a data analyst with professional capabilities. The analysis module analyzes the description information in the natural language format, evaluates the effect of the executable Python code, and generates an evaluation result, which includes an analysis of advantages and disadvantages and suggestions for improvement. The evaluation results are sent to the planning module to optimize the executable Python code and iteratively generate a performance-optimized decision tree.

[0043] Figure 7 This is a schematic diagram of the analysis module processing provided in the embodiments of this application, such as... Figure 7 As shown, the analysis module is responsible for using the trajectory data transformed by the description module to analyze and evaluate the effectiveness of the tactics designed by the planning module, and to propose corresponding suggestions. The analysis module is designed as a historical data analyst for football matches, analyzing and evaluating the advantages and disadvantages of the tactics used by reading textually described historical football match data, and proposing improvement methods. The inputs to this stage are the trajectory data described in natural language by the description module, the tactics designed by the planning module, and the match results. The outputs are the analysis of the trajectory data, the evaluation of the tactics, and the improvement methods. After the analysis module completes its analysis, the output data will be provided to the planning module to begin the next iteration.

[0044] It's important to note that the basic framework of large language generation decision-making knowledge is based on self-reflection. While self-reflection can effectively improve the performance of large language models, it also introduces the problem of cognitive degradation. This degradation is most pronounced in the analysis module, where it tends to over-rely on previous judgments, even if those judgments might be incorrect. If the analysis module makes an incorrect judgment, it will further affect the planning module in the next iteration, creating a chain reaction that reduces the performance of the large language model's decision-making. To address this issue, a multi-agent debate method is introduced into the analysis module.

[0045] Figure 8 This is a schematic diagram of the framework for multi-agent debate provided in an embodiment of this application, such as... Figure 8 As shown, this involves three roles: affirmative debaters, negative debaters, and a judge. After the topic of the debate is given, the affirmative debaters first state their viewpoints based on the topic, followed by the negative debaters refuting or supplementing the affirmative debaters' statements. After multiple iterations of affirmative and negative debates, the judge summarizes and determines whether to adopt the arguments of the affirmative or negative debaters, thus determining the final approach. Within this framework, the judge can judge whether the current approach is sufficiently reasonable based on the progress of the debate. If it is reasonable, the debate can be terminated; otherwise, iterative debates are required.

[0046] In this embodiment, the large language model is driven by the task description to generate decision strategies, and the decision tree code is further derived. The state-action trajectory data generated by the strategy generated by the large language model during its interaction with the environment is collected, and the trajectory data is converted into natural language and provided to the large language model for further analysis and improvement, thereby improving the accuracy of the large language model. The performance of the large language model is improved by using a multi-agent debate method, and the problem of cognitive degradation in the self-reflection process is reduced.

[0047] Step 120: Generate a guiding policy based on the decision tree, and use a policy transfer algorithm to transfer the knowledge of the guiding policy to the agent model to obtain an enhanced agent model. The policy transfer algorithm includes same-policy transfer algorithm and different-policy transfer algorithm.

[0048] In some embodiments, the same-strategy migration algorithm includes: The action probability distribution and KL divergence are calculated based on the guidance strategy, and the action probability distribution and KL divergence are used as the strategy distance metric. The KL divergence is added as a regularization term to the policy network loss function to update the guidance policy; By dynamically balancing the weights of the autonomous and autonomous strategies through the autonomous strategy adjustment factor and the guidance strategy decay factor, the influence of the guidance is gradually reduced as the training process progresses, thereby performing policy transfer on the agent model. The heterogeneous policy migration algorithm includes: replacing the agent's behavior policy with a guiding policy and interacting with the environment with a preset probability to obtain interaction data; The interaction data is stored in the experience replay buffer, and the data is sampled from the buffer to train the agent model; By controlling the proportion of guidance strategy usage through a guidance strategy decay factor, the intervention of guidance strategy is gradually reduced as the training process progresses, thereby enabling policy transfer to the agent model.

[0049] To enable an agent to learn the decision-making methods of a large language model, cope with environments with sparse rewards, and surpass the initial performance of the large language model's decision tree through further training, a policy transfer method is used. This allows the agent in reinforcement learning to learn the decision-making methods of the large language model. During policy transfer, the agent's policy network is affected. In heterogeneous policy reinforcement learning algorithms, data collection is treated as a separate task; the behavioral policy and the target policy are not the same. A guiding policy is used to direct the agent's behavioral policy and its interaction with the environment. Therefore, the actual impact of policy transfer is on the agent's behavioral policy.

[0050] In reinforcement learning algorithms with the same policy, since the agent's behavioral policy and target policy must be the same, if the policy transfer affects the agent's behavioral policy, the optimization objective will fail. In methods using importance sampling, the policy transfer can also affect the behavioral policy, leading to significant differences between the old and new policies when calculating importance weights, resulting in gradient explosion during network updates. To mitigate these issues, a guiding policy is used to direct the agent's policy network updates, ensuring that the actual policy transfer affects the agent's target policy.

[0051] Based on the actual differences between similar and dissimilar policies in reinforcement learning, the proposed algorithm includes two types of multi-agent reinforcement learning algorithms: similar and dissimilar policies. These will be elaborated on in detail below.

[0052] Figure 9 This is a schematic diagram of the framework for policy transfer in reinforcement learning with the same policy provided in an embodiment of this application, such as... Figure 9 As shown, the basic framework of the same-policy transfer algorithm is as follows: First, using behavior cloning technology, the agent is trained using a decision tree generated by a large language model, whose decision-making ability is close to that provided by the large language model. This cloned agent can serve as a guiding policy, providing an expert-guided action probability distribution, thereby calculating the action distance between the agent and the guiding policy, guiding the agent's updates, and enabling it to learn the inherent knowledge of the large language model more quickly. During data collection, the actual trained agent's behavior policy interacts with the environment and collects trajectory data, evaluating different state-action pairs. During policy updates, the guiding policy guides the actual trained agent's target policy updates, allowing the agent to learn the prior knowledge of the large language model while autonomously exploring the environment. As the performance of the actual trained agent gradually improves, and it can stably obtain rewards from reward-sparse environments, the guiding policy will reduce its guidance level until the agent fully learns the thinking ability of the large language model, and on this basis, can further improve its own performance, ultimately surpassing the decision-making performance of the large language model. The same-policy multi-agent reinforcement learning algorithm based on large language model policy transfer is shown in Algorithm 1: The core idea is to use a guiding policy to direct the update direction of the target policy network. The algorithm first initializes the policy parameters and a buffer to store interaction data. It then interacts with the environment using the agent's own behavioral policy and stores the collected trajectory data in the buffer. Subsequently, the guiding policy is used to calculate the action probability for each state-action transition in the buffer. By calculating the action distance between the agent's policy and the guiding policy, the update of the target policy parameters is influenced. The core of Algorithm 1 lies in minimizing the loss function shown below, allowing the agent's target policy to learn the guiding policy. The formula for calculating the loss function is as follows: in, The agent's policy in the state The probability of each action taken is the probability of using a random strategy. The guiding strategy is in the state The probability of the following action, The KL divergence is used to measure the action distance between the guidance policy and the agent's policy. This represents the baseline loss function of the HAPPO algorithm used. Its calculation method employs importance sampling and generalized advantage estimation, and uses the PPO-pruning algorithm for loss optimization.

[0053] To achieve knowledge transfer through the guidance strategy, the algorithm calculates the action probability of the guidance strategy. Action probability of behavioral strategies The KL divergence guides the direction of network updates, making the agent's policy more closely approximate the guiding policy; using parameters , and The extent to which the guiding strategy affects the agent. First, using... Calculate the KL divergence and benchmark loss function Balance to an order of magnitude, then adjust the factor through autonomous strategy. Controlling the degree of autonomous learning of the agent during actual training; using a decay factor through a guidance strategy. The extent to which the control guidance strategy affects the agent.

[0054] In the early stages of training, adjust the autonomous strategy regulation factor. When the value reaches a low level, the guiding strategy decay factor will be reduced. The value is adjusted to a high level. This setting aims to allow the trained agent to quickly learn the guided policy in the early stages of training and to stably obtain rewards in reward-sparse environments as soon as possible. When the agent can obtain relatively stable rewards in reward-sparse environments, the value is linearly increased. Simultaneously, linear decay This allows the agent being trained to focus more on the benefits derived from autonomous policies. Regarding the autonomous policy adjustment factor... The formula for the timetable used is as follows: Similarly, for the guide strategy decay factor The formula for the timetable used is as follows: in, The number of training iterations. The attenuation rate, It is an autonomous strategy regulating factor The initial value, This is the guide strategy decay factor. The initial value, and They are respectively less than and The constant value of , in the The iteration to the... It remains constant throughout the next iteration. This corresponds to the critical time point at which the intervention strategy is terminated. Specifically, the initial phase is set... The normalization constraint ensures that during the iteration process... It is always equal to 1. This normalization design effectively reduces the overall numerical decay or increase during training iterations, significantly improving the numerical stability of the training process.

[0055] The designed linear iterative adjustment factor achieves stable policy transfer for the agent. In the early stages of training, high-weighted guidance enables the agent to quickly learn the thought processes of a large language model. Gradually, the learning shifts towards a reinforcement-driven approach, establishing a dynamic balance between learning the guidance strategy and gaining decision-making autonomy. When the value reaches 0, the system terminates the influence of the guidance strategy on the agent's strategy, and the agent finally develops independent decision-making capabilities.

[0056] Table 1 shows the performance of the two baseline algorithms and the same-policy transfer algorithm after training. The data comes from a round-robin competition using the trained agents. The three algorithms used in the table are Heterogeneous-Agent Proximal Policy Optimization (HAPPO), Behavior Clone + Heterogeneous-Agent Proximal Policy Optimization (BC+HAPPO), and Policy Transfer Algorithm. These represent the cases of not using the large language model decision knowledge, directly using the large language model decision knowledge, and transferring the large language model decision knowledge through policy transfer, respectively.

[0057] A detailed analysis of the training effects of the three algorithms reveals that using the single HAPPO algorithm is limited by the sparse environmental reward signal, failing to explore an effective goal-scoring strategy within a limited number of training iterations. The BC+HAPPO algorithm, through behavioral cloning, allows the agent to learn prior knowledge of the large language model via a pre-defined policy network, increasing the average number of goals from 0.00 to 0.73 compared to the single HAPPO algorithm. However, it still struggles against Google Soccer's built-in AI, with a win rate of only 0.3%. This demonstrates that strategies using traditional imitation learning methods are insufficient for agents to fully learn prior knowledge of the large language model. The proposed policy transfer algorithm guides the agent to continuously learn prior knowledge of the large language model through policy guidance, achieving significant breakthroughs across multiple datasets. Compared to the BC+HAPPO algorithm, it increases the average number of goals by 0.91 per game and improves the win rate by 41.7%.

[0058] Table 1 Experimental results of the same-policy migration algorithm Figure 10 This is a graph showing the reward changes of the self-reflection and multi-agent debate method provided in this application over 15 iterations, as shown in the example. Figure 10As shown, the self-reflection method effectively improves the overall reward compared to the single-step direct decision generation method during iterations. With each iteration, the maximum reward value of the self-reflection method reaches 0.9, a significant improvement from the minimum of -1.3. The average reward increases from -1.30 to -0.42, a 67.7% increase. However, performance oscillations occur due to the insensitivity of large language models to excessively long contextual information and the uncertainty of the environment. Furthermore, the self-reflection method exhibits cognitive regression; in iterations 12-15, it fails to learn from better-performing historical strategies and instead persists with inferior methods. In contrast, while the multi-agent debate method also experiences occasional oscillations, its overall performance is slightly better than the self-reflection method, with a maximum reward value reaching 1.6. The average reward increases from -1.30 to -0.03, a 97.7% improvement. Moreover, the multi-agent debate method learns from excellent historical strategies, preventing cognitive regression.

[0059] Figure 11 This is a comparison chart of goal count changes during training of the same policy algorithm provided in this application. Specific analysis reveals that using the single HAPPO algorithm, lacking the assistance of prior knowledge from a large language model, fails to obtain stable reward feedback in the exploration environment for an extended period, resulting in learning stagnation. The BC+HAPPO algorithm, due to the agent employing a pre-defined policy network, significantly outperforms the single HAPPO algorithm at the beginning of training; however, it experiences significant stagnation in subsequent training, making further improvement based on the large language model's policies difficult. Using the proposed policy transfer algorithm, by calculating the difference in action probabilities between the agent and the guiding policy, it continuously provides the agent with prior knowledge from the large language model during training, achieving better transfer effects and subsequent improvements.

[0060] During the training process of the policy transfer algorithm, the training effect fluctuates significantly due to the continuous adjustment of the policy adjustment factor, but the overall trend is upward. In the early stage of training, the agent quickly learns the prior knowledge of the large language model under the guidance of the guiding policy, and the number of goals scored increases rapidly. Subsequently, the guiding policy decay factor reduces the magnitude of guidance, and the agent's policy network updates rely more on autonomous value judgments, resulting in fluctuations in the number of goals scored. In the later stage of training, the agent network gradually matures and takes over the policy updates, at which point the agent's training effect again shows a stable upward trend. Compared with the BC+HAPPO method, the policy transfer method effectively overcomes the problem that the agent is unable to further improve upon the policy obtained from the large language model.

[0061] Figure 12This is a comparison chart of the goal count between the same strategy transfer algorithm provided in this application and the built-in AI in Google Soccer. In the early stages of training, the agent struggles to outperform the built-in AI in Google Soccer; in the middle stages of training, the agent's goal count gradually approaches the performance of the built-in AI in Google Soccer; in the final stages of training, the agent's goal count eventually reaches almost the same level as the built-in AI in Google Soccer, and even slightly surpasses it.

[0062] Figure 13 This is a schematic diagram of the framework for heterogeneous policy transfer in reinforcement learning provided in an embodiment of this application, such as... Figure 13 As shown, the basic framework of the hetero-policy transfer algorithm is also based on training the guiding policy using behavior cloning techniques. The key difference from the same-policy method is that the hetero-policy mechanism allows the guiding policy to directly replace the agent's behavior policy. The actions made by the guiding policy can be placed in a buffer pool, thereby guiding the optimization of the agent's policy.

[0063] The guiding strategy involves, during the data collection phase, replacing the actual trained agent's behavioral strategy to interact with the environment, collecting trajectory data and storing it in a buffer pool. The agent's subsequent target policy will learn from data sampled from this buffer pool. In the initial training phase, because the guiding strategy significantly outperforms the untrained behavioral strategy, it temporarily takes over the behavioral strategy entirely. During this phase, the target policy quickly learns the thinking patterns of the large language model through the sampled data. As training progresses, the behavioral strategy begins to autonomously interact with the environment and collect trajectory data. This phase employs a hybrid interaction mechanism: the guiding strategy still interacts with the environment and collects data, but some interactions are performed by the agent's behavioral strategy, and the proportion of interactions performed by the agent gradually increases as training progresses. This design effectively reduces the agent's tendency to get trapped in local optima of the pre-trained strategy; without autonomous exploration, the agent struggles to break through the strategies provided by the large language model. As training progresses, the proportion of autonomous sampling by the actually trained agent increases linearly, achieving a smooth transition from learning the large language model's strategy to complete autonomy, ultimately surpassing the decision-making performance of the large language model. Algorithm 2 shows a heterogeneous multi-agent reinforcement learning algorithm based on policy transfer from a large language model. The core idea is to use a guiding policy to demonstrate actions for the target policy. The algorithm first initializes policy parameters and a buffer to store interaction data. During each policy iteration, it first calculates the guiding policy decay factor for the current iteration number, and then uses a random policy to determine whether the object interacting with the environment is the guiding policy or the agent's behavioral policy. Unlike transfer algorithms based on the same policy, sampling environmental data using a policy different from the target policy does not lead to abnormal policy updates.

[0064] Decay factor through guiding strategy The degree to which the guiding strategy affects the agent is controlled. In the early stages of training, the guiding strategy decay factor... This value is set to a relatively large value. The purpose of this setting is to guide the agent through action demonstrations using a guided strategy when the agent cannot consistently gain rewards from interacting with the environment. Simultaneously, the guided strategy's action demonstrations serve as tactical instruction for agents playing different roles. Specifically, the guided strategy will perform different actions based on the different player's position, thus better achieving multi-agent collaboration. Without guidance from a guided strategy, multi-agent reinforcement learning tends to greedily pursue environmental rewards and struggles to learn action coordination.

[0065] As training progresses, a linear decay is set. This allows the agent being trained to focus more on the benefits of autonomous strategies. By reducing the influence of the guiding policy, the agent being trained becomes more independent in the decision-making process and emphasizes its own learning strategy. Regarding the guiding policy decay factor... The formula for the timetable used is as follows: Similar to transfer algorithms with the same policy, this design enables stable policy transfer for the agent, establishing a dynamic balance between learning the guided policy and gaining decision-making autonomy. When the value reaches 0, the system terminates the influence of the guidance strategy on the agent's strategy, allowing the agent to eventually develop independent decision-making capabilities.

[0066] Table 2 shows the performance of the two baseline algorithms and the heterogeneous policy transfer algorithm after training. The data comes from a round-robin competition using the trained agents. The three algorithms used in the table are the Soft Actor-Critic (SAC) algorithm, the Behavior Clone + Soft Actor-Critic (BC+SAC) algorithm, and the policy transfer algorithm. These represent the cases of not using the large language model decision knowledge, directly using the large language model decision knowledge, and transferring the large language model decision knowledge through policy transfer, respectively.

[0067] A detailed analysis of the training effects of the three algorithms reveals that using the single SAC algorithm can achieve a very small number of goals, but still requires extensive training to learn effective attacking strategies. The BC+SAC algorithm, however, allows the agent to learn prior knowledge from a large language model through a pre-defined policy network, achieving a win rate of 16.0%, but its average goal difference still falls short of Google Soccer's built-in AI. The proposed policy transfer algorithm provides action demonstrations to the agent through guided policies, allowing the agent to learn strategies through experience replay. It shows significant breakthroughs across multiple datasets and surpasses Google Soccer's built-in AI in terms of average goal difference, achieving a win rate of 52.4%.

[0068] Table 2 Experimental Results of Heterogeneous Policy Transfer Algorithm Figure 14 This is a comparison chart of the number of goals scored during the training process of the heterogeneous policy algorithms provided in this application. A detailed analysis of the goal score trends of the three algorithms reveals that using the single SAC algorithm results in poor training performance due to the lack of prior knowledge from the large language model. Using the BC+SAC algorithm, the agent employs a pre-defined policy network, which initially performs well. However, due to the agent's lack of a mature value network, the number of goals decreases in subsequent training. The proposed policy transfer algorithm demonstrates actions to the agent using a guiding policy, allowing the agent to learn the policy of the large language model through experience replay, achieving better transfer performance and subsequent improvement.

[0069] In the early stages of training the policy transfer algorithm, although the agent's goal count steadily increased, it remained low. This is because the data sampled by the agent in the buffer pool was relatively random, making it difficult to provide the agent with a complete game strategy in the early stages of training. However, as training progressed and sufficient data was sampled, the agent was able to learn a more complete strategy, leading to a rapid improvement in training performance and quickly surpassing the performance of BC+SAC.

[0070] Figure 15 This is a comparison chart of the goal count between the heterogeneous strategy transfer algorithm provided in this application and the built-in AI in Google Soccer. In the early stages of training, the agent's performance gradually improves but it is still difficult to beat the built-in AI in Google Soccer; in the later stages of training, the agent's goal count increases rapidly and eventually surpasses the performance of the built-in AI in Google Soccer.

[0071] In this embodiment, in the same-policy algorithm, the action distance between the two actions is obtained by calculating the KL divergence between the action probability distribution of the guiding policy and the action probability distribution made by the agent, which guides the update direction of the agent's target network. In the different-policy algorithm, the guiding policy is used to replace the agent in interacting with the environment to obtain trajectory data, and the trajectory data is put into a buffer pool so that the agent can be trained according to the historical experience of the guiding policy. This enables multiple agents to learn the methods of cooperation and exploration of rewards, reducing the problems of difficulty in cooperation among multiple agents and difficulty in exploring sparse reward environments.

[0072] The multi-agent reinforcement learning method based on large language models provided in this application, by employing a unified decision tree generation framework and policy transfer mechanism, can efficiently and stably inject the prior knowledge of the large language model into the multi-agent reinforcement learning process within a single framework. Through the collaboration of four modules—planning, programming, description, and analysis—and a multi-agent debate mechanism, the reliability of the generated policy from the large language model is improved. Furthermore, by using both same-policy and different-policy transfer algorithms, the generated decision tree serves as the guiding policy, adaptively balancing prior knowledge learning and autonomous exploration optimization. This effectively addresses the problems of low exploration efficiency and unstable collaboration among multi-agents, while significantly reducing the huge computational overhead caused by direct interaction of the large language model, reducing the number of agents trapped in local optima of the guiding policy, and reducing the number of instances where the large language model directly interacts with the environment.

[0073] The multi-agent reinforcement learning method based on large language model enhancement provided in this application can be implemented by a multi-agent reinforcement learning system based on large language model enhancement. This application uses the execution of the multi-agent reinforcement learning method based on large language model enhancement by a multi-agent reinforcement learning system based on large language model enhancement as an example to illustrate the multi-agent reinforcement learning system based on large language model enhancement provided in this application.

[0074] This application also provides a multi-agent reinforcement learning system based on a large language model, such as... Figure 16 As shown, the multi-agent reinforcement learning system based on a large language model includes an acquisition module 1610 and a processing module 1620.

[0075] The acquisition module 1610 is used to acquire environmental information and input the environmental information into the enhanced large model to obtain a decision tree. The large model includes a planning module, a programming module, a description module, and an analysis module. The large model is enhanced through multi-agent debate. The processing module 1620 is used to generate a guiding policy based on a decision tree, and to transfer the knowledge of the guiding policy to the agent model using a policy transfer algorithm to obtain an enhanced agent model. The policy transfer algorithm includes a same-policy transfer algorithm and a different-policy transfer algorithm.

[0076] The multi-agent reinforcement learning system based on a large language model, as provided in the embodiments of this application, can efficiently and stably inject the prior knowledge of the large language model into the multi-agent reinforcement learning process within a single framework by adopting a unified decision tree generation framework and policy transfer mechanism. Through the collaboration of four modules—planning, programming, description, and analysis—and a multi-agent debate mechanism, the reliability of the generated policy from the large language model is improved. Furthermore, by employing two transfer algorithms—one with the same policy and one with a different policy—the generated decision tree is used as a guiding policy, adaptively balancing prior knowledge learning and autonomous exploration optimization. This effectively solves the problems of low exploration efficiency and unstable collaboration among multi-agents, while significantly reducing the huge computational overhead caused by direct interaction of the large language model, reducing the number of agents trapped in local optima of the guiding policy, and reducing the number of instances where the large language model directly interacts with the environment.

[0077] The multi-agent reinforcement learning system based on large language model enhancement provided in this application embodiment can achieve… Figures 1 to 15 The various processes implemented in the multi-agent reinforcement learning method based on large language models will not be described in detail here to avoid repetition.

[0078] In some embodiments, such as Figure 17 As shown, this application embodiment also provides an electronic device 1700, including a processor 1701, a memory 1702, and a computer program stored on the memory 1702 and executable on the processor 1701. When the program is executed by the processor 1701, it implements the various processes of the above-described embodiment of the multi-agent reinforcement learning method based on large language model enhancement, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0079] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

Claims

1. A multi-agent reinforcement learning method based on large language model enhancement, characterized in that, The method includes: Environmental information is acquired and input into an enhanced large model to obtain a decision tree. The large model includes a planning module, a programming module, a description module, and an analysis module. The large model is enhanced through multi-agent debate. Based on the decision tree, a guiding policy is generated, and the knowledge of the guiding policy is transferred to the agent model using a policy transfer algorithm to obtain an enhanced agent model. The policy transfer algorithm includes same-policy transfer algorithm and different-policy transfer algorithm.

2. The multi-agent reinforcement learning method based on large language model enhancement according to claim 1, characterized in that, The process of inputting environmental information into the enhanced large model to obtain a decision tree includes: Environmental information is input into the planning module, and pseudocode is generated by calling the prior knowledge of the large model; The pseudocode is input into the programming module for code conversion, resulting in executable Python code; Input Python code and environmental information into the description module to interact and collect trajectory information, and obtain description information in natural language format; The executable Python code and its description in natural language format are input into the analysis module. The executable Python code is evaluated based on the description information, and a decision tree is generated.

3. The multi-agent reinforcement learning method based on large language model enhancement according to claim 2, characterized in that, The step of inputting environmental information into the planning module and generating pseudocode by calling prior knowledge from the large model includes: Environmental information, including the initial location and site attributes, is input into the planning module. The planning module is preset to be an expert in the target field. Based on the planning module, sample prompts are generated. The environmental information and the sample prompts are combined and processed. The prior knowledge of the large language model is invoked to identify the key tactical logic and action sequence applicable to the current environment. Based on the identified logic and action sequence, pseudocode is generated.

4. The multi-agent reinforcement learning method based on large language model enhancement according to claim 2, characterized in that, The process of inputting pseudocode into a programming module for code conversion to obtain executable Python code includes: The programming module is preset to the role of a programming expert, and environmental interaction specifications are set, including observation format, available action set and environmental interaction interface; The pseudocode, the environment interaction specifications, and the preset code format requirements are input into the programming module, which converts the pseudocode into executable Python code that conforms to the environment interaction interface call format.

5. The multi-agent reinforcement learning method based on large language model enhancement according to claim 2, characterized in that, The process of inputting Python code and environment information into the description module for interaction and collection of trajectory information to obtain description information in natural language format includes: The Python code is fused with environmental information to obtain trajectory data in NPZ format. The trajectory data in NPZ format includes multiple NumPy arrays, each of which is used to store observation, action, reward and corresponding time step information. The trajectory data in NPZ format is input into the description module, and the data of each NumPy array is split to obtain structured JSON data; Key information is extracted from the structured JSON data and converted into natural language descriptive text to obtain descriptive information in natural language format.

6. The multi-agent reinforcement learning method based on large language model enhancement according to claim 2, characterized in that, The step involves inputting executable Python code and natural language description information into the analysis module, evaluating the executable Python code based on the description information, and generating a decision tree, including: The description information in natural language format, the executable Python code, and the results after code execution are input into the analysis module, which is preset to be a data analyst with professional capabilities. The analysis module analyzes the description information in the natural language format, evaluates the effect of the executable Python code, and generates an evaluation result, which includes an analysis of advantages and disadvantages and suggestions for improvement. The evaluation results are sent to the planning module to optimize the executable Python code and iteratively generate a performance-optimized decision tree.

7. The multi-agent reinforcement learning method based on large language model enhancement according to claim 6, characterized in that, The same-strategy migration algorithm includes: The action probability distribution and KL divergence are calculated based on the guidance strategy, and the action probability distribution and KL divergence are used as the strategy distance metric. The KL divergence is added as a regularization term to the policy network loss function to update the guidance policy; By dynamically balancing the weights of the autonomous and autonomous strategies through the autonomous strategy adjustment factor and the guidance strategy decay factor, the influence of the guidance is gradually reduced as the training process progresses, thereby performing policy transfer on the agent model. The heterogeneous policy migration algorithm includes: replacing the agent's behavior policy with a guiding policy and interacting with the environment with a preset probability to obtain interaction data; The interaction data is stored in the experience replay buffer, and the data is sampled from the buffer to train the agent model; By controlling the proportion of guidance strategy usage through a guidance strategy decay factor, the intervention of guidance strategy is gradually reduced as the training process progresses, thereby enabling policy transfer to the agent model.

8. A multi-agent reinforcement learning system based on large language model enhancement, implemented using the multi-agent reinforcement learning method based on large language model enhancement as described in any one of claims 1 to 7, characterized in that, The system includes: An acquisition module is used to acquire environmental information and input the environmental information into the enhanced large model to obtain a decision tree. The large model includes a planning module, a programming module, a description module, and an analysis module. The large model is enhanced through multi-agent debate. The processing module is used to generate a guiding policy based on a decision tree, and to transfer the knowledge of the guiding policy to the agent model using a policy transfer algorithm to obtain an enhanced agent model. The policy transfer algorithm includes a same-policy transfer algorithm and a different-policy transfer algorithm.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-agent reinforcement learning method based on large language model enhancement as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-agent reinforcement learning method based on large language model enhancement as described in any one of claims 1 to 7.

Images