Multi-agent control system and control method based on large language model

Abstract

The invention provides a multi-agent control system based on a large language model. The system comprises an analogue simulation subsystem and a coordination control subsystem. The analogue simulation subsystem comprises a data acquisition module, an initial reward function generation module, an error correction module, a dense reward function generation module and a strategy network updating module, wherein the data acquisition module is used for acquiring a multi-agent reinforcement learning training code; the initial reward function generation module is used for generating an initial reward function code; the error correction module is used for generating executable reward function codes; the dense reward function generation module is used for generating dense reward function codes; and the strategy network updating module is used for obtaining the strategy network with the maximized reward. The coordination control subsystem comprises a data collection module and a strategy distribution module; wherein the data collection module is used for receiving observation data; and the strategy distribution module is used for guiding multiple agents to execute a control task by adopting a strategy network generation action for maximizing rewards based on the observation data.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to multi-agent control technology in the field of artificial intelligence technology, and more specifically to a multi-agent control system and control method based on a large language model. Background Art

[0002] During the training process of a multi-agent collaborative decision-making system, the system's optimization goal is to maximize the collective behavioral decision rewards accumulated by the multi-agents during their interaction with the environment. Based on the resulting policy network corresponding to the maximized behavioral decision rewards, the corresponding actions are generated to guide the multi-agents to complete the control task. To this end, it is crucial to establish an effective feedback strategy to guide the formation of the desired behavioral patterns of the multi-agents. This directly affects the collaborative effectiveness and training efficiency of the multi-agent collaborative decision-making system. A reasonable feedback strategy can ensure that the multi-agent collaborative decision-making system quickly achieves the basic conditions for the expected multi-agent collaboration mode.

[0003] As the complexity of real-world environments and the number of agents increase, the interactions between them become more complex, significantly increasing the difficulty of training multi-agent collaborative decision-making systems. This problem is particularly acute in scenarios with sparse feedback signals: the joint action space of the agents expands exponentially, and sparse feedback signals make it difficult to provide effective behavioral guidance during the training process. This makes it difficult for the system to form a stable collaborative model within a reasonable period of time, ultimately affecting task execution efficiency.

[0004] In response to the training dilemma caused by sparse feedback, existing technologies mainly adopt two optimization solutions: the first is to reconstruct the feedback strategy through manual experience, and professionals artificially enhance the sparse feedback signals based on domain knowledge, and construct a dense feedback strategy with behavioral guidance through repeated experiments. Although this method can obtain effective guidance signals, it has the problems of long manual adjustment cycle and high professional threshold, especially in complex scenarios, which requires a lot of trial and error costs. The second is feedback optimization based on expert behavior data, which reversely deduces the feedback strategy by collecting a large amount of expert behavior data. Although this method can construct a dense feedback strategy that explains expert behavior, it faces the bottlenecks of high expert data collection cost and limited sample coverage, and it is difficult to ensure the adaptability of the feedback strategy in dynamically changing scenarios.

[0005] In summary, existing optimization schemes for sparse feedback all have shortcomings. Optimization schemes based on manual experience to reconstruct feedback strategies require professionals to invest a significant amount of time in strategy adjustments, resulting in long adjustment cycles and high professional barriers to entry. Furthermore, complex scenarios require repeated trial and error, leading to increased implementation costs. While feedback optimization schemes based on expert behavior samples can construct strategies through reverse deduction, they are limited by practical constraints such as the high cost of expert data collection and limited sample coverage. Furthermore, they struggle to cope with the decreased strategy adaptability caused by dynamic scenario changes.

[0006] It should be noted that this background information is provided solely to introduce relevant information of the present invention to facilitate understanding of the technical solution of the present invention. It does not necessarily constitute prior art. In the absence of evidence demonstrating that the relevant information was disclosed prior to the filing date of the present invention, the relevant information should not be considered prior art. Summary of the Invention

[0007] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-agent control system based on a large language model and a multi-agent control method.

[0008] The objectives of the present invention are achieved through the following technical solutions.

[0009] According to a first aspect of the present invention, a multi-agent control system based on a large language model is provided, the system comprising: a simulation subsystem, which includes a data acquisition module, an initial reward function generation module, an error correction module, a dense reward function generation module and a policy network update module; wherein: the data acquisition module is used to obtain a multi-agent reinforcement learning training code under a sparse reward scenario corresponding to a control task; wherein the sparse reward scenario indicates that rewards are given in specific circumstances to evaluate the actions taken by the multi-agent when performing the control task; a pre-trained policy network is provided in the multi-agent reinforcement learning training code; the initial reward function generation module is used to generate a function code prompt word based on the multi-agent reinforcement learning training code, and generate an initial reward function code based on the function code prompt word using a preset large language model; The error correction module is used to add the initial reward function code to the multi-agent reinforcement learning training code and perform multiple error correction processes to obtain an executable reward function code; the dense reward function generation module is used to add the executable reward function code to the multi-agent reinforcement learning training code and perform multiple reinforcement learning training to obtain a dense reward function code; the policy network update module is used to add the dense reward function code to the multi-agent reinforcement learning training code and perform multiple reinforcement learning training to obtain a policy network that maximizes the reward; a coordinated control subsystem, which includes a data collection module and a policy distribution module; wherein: the data collection module is used to receive observation data collected by multiple agents from the environment; the policy distribution module is used to generate actions based on the observation data using a policy network that maximizes the reward to guide the multiple agents to perform control tasks.

[0010] In some embodiments of the present invention, the initial reward function generation module is configured to generate the initial reward function in the following manner: based on the multi-agent reinforcement learning training code in the sparse reward scenario corresponding to the control task, determine the task goal description and the reward function generation instructions and constraints; wherein the task goal description indicates the scenario in which the multi-agent performs the control task and the expected effect of performing the control task; the reward function generation instructions and constraints indicate the rules that need to be followed to generate the initial reward function; extract the environment description and background knowledge from the multi-agent reinforcement learning training code; wherein the environment description indicates the environment in which the multi-agent is located and the interaction between the multi-agent and the environment, and the background knowledge indicates the programming language used in the multi-agent reinforcement learning training code; use the task goal description, reward function generation instructions and constraints, environment description and background knowledge as function code prompts, and generate the initial reward function code based on the function code prompt using a preset large language model.

[0011] In some embodiments of the present invention, the error correction module is configured to perform an error correction process in the following manner: the reward function code obtained after the previous error correction process is added to the multi-agent reinforcement learning training code for execution to obtain the current syntax error and runtime error; based on the current syntax error and runtime error, the preset large language model is used to correct the reward function code obtained after the previous error correction process to obtain the current reward function code; wherein, the error correction module uses the reward function code obtained after multiple error correction processes that does not contain syntax errors and runtime errors as the executable reward function code.

[0012] In some embodiments of the present invention, the dense reward function generation module is configured to perform a reinforcement learning training in the following manner: adding the executable reward function code obtained from the previous reinforcement learning training to the multi-agent reinforcement learning training code to perform the current reinforcement learning training to obtain the current reinforcement learning training result; generating a current feedback prompt word based on the current reinforcement learning training result, and updating the executable reward function code obtained from the previous reinforcement learning training using a preset large language model based on the current feedback prompt word to obtain the current executable reward function code; wherein the current feedback prompt word is used to indicate the improvement direction of the executable reward function code in the current reinforcement learning training; wherein the executable reward function code that can realize the control task obtained after multiple reinforcement learning trainings is used as the dense reward function code.

[0013] In some embodiments of the present invention, the preset large language model is GPT-4o, Claude, kimi or deepseek.

[0014] In some embodiments of the present invention, the multi-agent is composed of multiple agents, wherein each agent includes a rigid body component, a joint component that directly controls the movement of the rigid body component, and an actuator that drives the joint component to control the movement of the rigid body component.

[0015] In some embodiments of the present invention, local sensors are configured on the joint components of each intelligent agent to collect observation data from the environment; wherein the observation data includes the state of the intelligent agent itself and the state of the environment.

[0016] According to a second aspect of the present invention, a multi-agent control method is provided for controlling multi-agents to perform control tasks, the method comprising: using the control system according to the first aspect of the present invention to control multi-agents to perform control tasks

[0017] Compared with the existing technology, the advantages of the present invention are: an initial reward function generation module, an error correction module and a dense reward function generation module are set to jointly generate a dense feedback strategy suitable for specific control tasks. By combining different modules, a dense feedback strategy is automatically generated without the need for professionals to invest a lot of time in strategy adjustment, nor is there a need to collect expert behavior data for strategy optimization. It can quickly generate dense feedback strategies suitable for different control tasks, thereby providing effective behavior guidance for multiple agents to perform specific control tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of the composition of a control system according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the winning rate of multi-agent reinforcement learning training in a 1c3s5z scenario using different reward functions according to an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the winning rate of multi-agent reinforcement learning training in an MMM scenario using different reward functions according to an embodiment of the present invention;

[0022] Figure 4 Schematic diagram of the winning rate of multi-agent reinforcement learning training in the MMM2 scenario using different reward functions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] As mentioned in the background technology section, existing optimization solutions for sparse feedback all have shortcomings. Among them, the optimization solution based on manual experience to reconstruct the feedback strategy requires professionals to invest a lot of time to adjust the strategy, which has the disadvantages of long adjustment cycle and high professional threshold. In addition, there is the problem of repeated trial and error in complex scenarios, which leads to a surge in implementation costs. Although the feedback optimization solution based on expert behavior samples can construct strategies through reverse deduction, it is limited by practical constraints such as the high cost of expert data collection and limited sample coverage. At the same time, it is difficult to deal with the problem of reduced strategy adaptability caused by dynamic scene changes.

[0025] In order to solve the above problems, the inventors analyzed the problems existing in the prior art and found that the initial feedback strategy can be generated with the help of the existing large language model, and the initial feedback strategy is added to the reinforcement learning training code for multiple reinforcement learning trainings so that the large language model can adjust and optimize the initial feedback strategy according to the training results to obtain a dense feedback strategy, and then the dense feedback strategy is added to the reinforcement learning training code to obtain a policy network that maximizes the reward, and the obtained policy network that maximizes the reward is used to generate corresponding actions to guide multiple agents to complete the control task. Based on this, the inventors proposed a control system, and set an initial reward function generation module, an error correction module and a dense reward function generation module in the system to jointly generate a dense feedback strategy suitable for specific control tasks. By combining different modules, the dense feedback strategy is automatically generated without the need for professionals to invest a lot of time in strategy adjustment, nor does it need to collect expert behavior data for strategy optimization. It can quickly generate dense feedback strategies suitable for different control tasks, thereby providing effective behavior guidance for multiple agents to perform specific control tasks. Among them, the initial reward function generation module is used to generate an initial reward function for initial feedback strategy matching based on a preset large language model; the error correction module is used to add the initial reward function code to the multi-agent reinforcement learning training code to perform multiple error correction processes to generate error-free executable reward function code; the dense reward function generation module is used to add the executable reward function code to the multi-agent reinforcement learning training code to control the multi-agent to perform multiple reinforcement learning trainings to obtain dense reward function code for dense feedback strategy matching.

[0026] In summary, if Figure 1As shown, the present invention provides a multi-agent control system based on a large language model, the system comprising: a simulation subsystem, which includes a data acquisition module, an initial reward function generation module, an error correction module, a dense reward function generation module and a policy network update module; wherein: the data acquisition module is used to obtain a multi-agent reinforcement learning training code under a sparse reward scenario corresponding to a control task; wherein the sparse reward scenario indicates that rewards are given in specific circumstances to evaluate the actions taken by the multi-agent when performing the control task; a pre-trained policy network is provided in the multi-agent reinforcement learning training code; the initial reward function generation module is used to generate a function code prompt word based on the multi-agent reinforcement learning training code, and generate an initial reward function code based on the function code prompt word using a preset large language model; the error correction module The module is used to add the initial reward function code to the multi-agent reinforcement learning training code to perform multiple error correction processes to obtain an executable reward function code; the dense reward function generation module is used to add the executable reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning training to obtain a dense reward function code; the policy network update module is used to add the dense reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning training to obtain a policy network that maximizes the reward; a coordinated control subsystem, which includes a data collection module and a policy distribution module; wherein: the data collection module is used to receive observation data collected by multiple agents from the environment; the policy distribution module is used to generate actions based on the observation data using a policy network that maximizes the reward to guide the multiple agents to perform control tasks.

[0027] In order to better understand the present invention, each subsystem in the control system is described in detail below in conjunction with specific embodiments.

[0028] 1. Simulation Subsystem

[0029] The simulation subsystem includes a data acquisition module, an initial reward function generation module, an error correction module, a dense reward function generation module and a policy network update module. Each module is described in detail below.

[0030] 1.1 Data Acquisition Module

[0031] The data acquisition module is used to obtain multi-agent reinforcement learning training code under a sparse reward scenario corresponding to the control task; wherein the sparse reward scenario indicates that rewards are given in specific situations to evaluate the actions taken by the multi-agent when performing the control task; the multi-agent reinforcement learning training code is provided with a pre-trained policy network.

[0032] It should be noted that a sparse reward function is also provided in the multi-agent reinforcement learning training code under the sparse reward scenario corresponding to the control task. The sparse reward function indicates that rewards are given under specific circumstances to evaluate the actions taken by the multi-agent when performing the control task. For example, when the control task is a robot assembly task, a positive reward is given only after the robot completes the assembly task, or a negative reward or zero is given when the robot fails to assemble; for example, when the control task is a robot walking test (the walking test means controlling the robot to walk steadily for a distance), a positive reward is given only after the robot completes the test task, or a negative reward or zero is given when the robot fails the walking test. It should also be noted that the pre-trained policy network is an existing neural network model used to generate corresponding actions to control the multi-agent to perform the control task, and the selection of the policy network is determined according to the actual control task to be performed, and the present invention does not impose any special restrictions.

[0033] 1.2. Initial Reward Function Generation Module

[0034] The initial reward function generation module is used to generate function code prompt words based on the multi-agent reinforcement learning training code, and generate the initial reward function code based on the function code prompt words using a preset large language model.

[0035] According to one embodiment of the present invention, the initial reward function generation module is configured to generate the initial reward function in the following manner: based on the multi-agent reinforcement learning training code in the sparse reward scenario corresponding to the control task, determine the task goal description and the reward function generation instructions and constraints; wherein the task goal description indicates the scenario in which the multi-agent performs the control task and the expected effect of performing the control task; the reward function generation instructions and constraints indicate the rules that need to be followed to generate the initial reward function; extract the environment description and background knowledge from the multi-agent reinforcement learning training code; wherein the environment description indicates the environment in which the multi-agent is located and the interaction relationship between the multi-agent and the environment, and the background knowledge indicates the programming language used in the multi-agent reinforcement learning training code; use the task goal description, reward function generation instructions and constraints, environment description and background knowledge as function code prompts, and generate the initial reward function code based on the function code prompt using a preset large language model.

[0036] Among them, the task objective description includes the multi-agent task scenario described in natural language, the goals expected to be achieved through multi-agent reinforcement learning training, the timing of intensive rewards, and the function name, parameters, return value, and other information of the expected reward function.

[0037] The reward function generation instructions and constraints indicate that the preset large language model must generate the initial reward function code as required, and during the code generation process, only the variables, attributes, and Python packages already in the multi-agent reinforcement learning training code can be used.

[0038] The environment description includes each agent's configuration information, the actions each agent can take, and each agent's observation attributes. Configuration information may include the number and type of agents, hardware parameters, communication capabilities, capability constraints, and strategy patterns. The actions each agent can take represent specific action types, such as discrete, continuous, and compound. Each agent's observation attributes represent the agent's observations of the environment, such as spatial perception, coordinate perception, and noise perception. It's important to note that the environment description is expressed in Python, which offers a higher level of abstraction. The relevant code is often directly provided in multi-agent reinforcement learning training code, reducing manual summarization and induction.

[0039] Background knowledge includes information about Python packages used in the multi-agent reinforcement learning training code. It should be noted that the reason for providing information about Python packages used in the multi-agent reinforcement learning training code (primarily those used in the sparse reward function) is to allow the large language model to learn useful functions as background knowledge, thereby improving the quality of the generated dense reward function code.

[0040] It should be noted that the function code prompt words are set based on actual needs and are not particularly limited in the present invention.

[0041] 1.3 Error Correction Module

[0042] The error correction module is used to add the initial reward function code to the multi-agent reinforcement learning training code and perform multiple error correction processes to obtain an executable reward function code.

[0043] According to one embodiment of the present invention, the error correction module is configured to perform an error correction process in the following manner: the reward function code obtained after the previous error correction process is added to the multi-agent reinforcement learning training code for execution to obtain the current syntax error and runtime error; based on the current syntax error and runtime error, the preset large language model is used to correct the reward function code obtained after the previous error correction process to obtain the current reward function code; wherein, the error correction module uses the reward function code obtained after multiple error correction processes that does not contain syntax errors and runtime errors as the executable reward function code.

[0044] It should be noted that the reason for setting up an error correction module is that the initial reward function code generated by the initial reward function generation module may contain formal errors and syntactic errors. Therefore, the initial reward function code needs to be error corrected to obtain an executable reward function code without syntactic errors and runtime errors.

[0045] 1.4. Dense Reward Function Generation Module

[0046] The dense reward function generation module is used to add the executable reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning trainings to obtain the dense reward function code.

[0047] According to one embodiment of the present invention, the dense reward function generation module is configured to perform a reinforcement learning training in the following manner: adding the executable reward function code obtained from the previous reinforcement learning training to the multi-agent reinforcement learning training code to perform the current reinforcement learning training to obtain the current reinforcement learning training result; generating a current feedback prompt word based on the current reinforcement learning training result, and updating the executable reward function code obtained from the previous reinforcement learning training using a preset large language model based on the current feedback prompt word to obtain the current executable reward function code; wherein the current feedback prompt word is used to indicate the improvement direction of the executable reward function code in the current reinforcement learning training; wherein the executable reward function code that can realize the control task obtained after multiple reinforcement learning trainings is used as the dense reward function code.

[0048] It should be noted that the dense reward function generation module does not update the policy network parameters when performing reinforcement learning training. It should also be noted that the reinforcement learning training method is a technology well known to those skilled in the art and will not be described in detail in this invention.

[0049] It should also be noted that if the policy network obtained after multiple reinforcement learning trainings based on the dense reward function code can generate actions to guide multiple agents in performing the control task, and the multiple agents complete the control task based on the actions generated by the policy network, then the dense reward function code can achieve the control task. For example, taking the robot assembly task as an example, if the policy network obtained after multiple reinforcement learning trainings based on the dense reward function code can generate actions to guide the robot in performing the assembly task, and the robot completes the assembly task based on the actions generated by the policy network, then the dense reward function code can achieve the control task.

[0050] According to one embodiment of the present invention, the preset large language model is GPT-4o, Claude, kimi or deepseek.

[0051] It should be noted that existing optimization schemes for sparse feedback require a lot of manpower, are inefficient, and require a lot of expert knowledge. Considering the shortcomings of existing technologies, the present invention introduces a large language model. The large language model is trained using data from multiple fields. It can understand function code generation tasks and provide relevant knowledge. The large language model performs well in function code generation and can generate high-quality function code and code error correction based on preset function code prompts and feedback prompts, thereby greatly reducing the design difficulty and manual input of sparse feedback strategies. Moreover, the use of a large language model can achieve zero-sample generation, without the need for any examples or expert data. Existing optimization schemes for sparse feedback require the collection of large amounts of expert behavior data to reversely deduce feedback strategies, but the cost of obtaining expert data is high and there are also application limitations.

[0052] 1.5. Policy Network Update Module

[0053] The policy network update module is used to add the dense reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning trainings to obtain a policy network that maximizes the reward.

[0054] 2. Coordination Control Subsystem

[0055] The coordination control subsystem includes a data collection module and a strategy distribution module. The following describes these two modules separately.

[0056] 2.1 Data Collection Module

[0057] The data collection module is used to receive observation data collected by multiple agents from the environment.

[0058] 2.2 Strategy Distribution Module

[0059] The strategy allocation module is used to generate actions based on the observed data using a strategy network that maximizes rewards to guide multiple agents to perform control tasks.

[0060] According to one embodiment of the present invention, the multi-agent is composed of multiple agents, wherein each agent includes a rigid body component, a joint component that directly controls the movement of the rigid body component, and an actuator that drives the joint component to control the movement of the rigid body component.

[0061] According to one embodiment of the present invention, each joint component of the intelligent agent is equipped with a local sensor to collect observation data from the environment; wherein the observation data includes the state of the intelligent agent itself and the state of the environment.

[0062] In order to verify the difference between the present invention and the prior art, a comparative experiment was conducted using the StarCraft Multi-Agent Challenge (SMAC) environment.

[0063] In a comparative experiment, two groups of agents, friendly and enemy, were set up to compete against each other. The enemy agents were always controlled by a handcrafted heuristic policy built into the game, while the friendly agents were controlled by a policy network generated by the control system described in the previous examples, a policy network generated by a manually designed dense reward function, and a policy network generated by a sparse reward function.

[0064] The control system described in the present invention generates a policy network as follows. Before describing the policy network generation process, let's first explain the background knowledge behind generating this policy network. In the StarCraft Multi-Agent Challenge (SMAC) environment, a straightforward mission objective is typically used as a reward: winning the battle. However, this reward is too sparse, available only after a round. Furthermore, this sparse reward cannot successfully guide multi-agent reinforcement learning training, and thus cannot generate a policy network capable of guiding multi-agent combat. Therefore, it is necessary to use the control system described in the aforementioned embodiment to generate a policy network capable of guiding multi-agent combat. The specific generation process is as follows.

[0065] First, the data acquisition module in the control system obtains the multi-agent reinforcement learning training code in the sparse reward scenario corresponding to the control task, where the multi-agent reinforcement learning training code includes an abstract representation of the environment and a background knowledge part; the abstract representation of the environment includes code representations such as the class definition and initialization parameters shown in Table 1, the observation and state parameters shown in Table 2, the sparse reward function configuration shown in Table 3, the map and unit matching shown in Table 4, and the action space definition shown in Table 5.

[0066] Table 1

[0067]

[0068] Table 2

[0069]

[0070] Table 3

[0071]

[0072] Table 4

[0073]

[0074] Table 5

[0075]

[0076] Then, the initial reward function generation module in the control system generates function code prompt words based on the multi-agent reinforcement learning training code, and generates the initial reward function code based on the function code prompt words using a preset large language model.

[0077] Then, the error correction module in the control system adds the initial reward function code to the multi-agent reinforcement learning training code and performs multiple error correction processes to obtain an executable reward function code.

[0078] Then, the dense reward function generation module in the control system adds the executable reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning trainings to obtain the dense reward function code.

[0079] Finally, the policy network update module adds the dense reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning trainings to obtain a policy network that maximizes the reward.

[0080] After obtaining the dense reward function generated by the present invention in the StarCraft Multi-Agent Challenge (SMAC) environment based on the above steps, we used this policy network to conduct experiments in the three scenarios of 1c3s5z, MMM, and MMM2. In these three scenarios, we used the mappo multi-agent reinforcement learning algorithm and conducted experiments under 5 random seeds. The solid line in the experimental results represents the average win rate of the 5 experiments. The same parameter configuration was used in the experiment, and the experiment was repeated using sparse rewards, the manually designed dense reward function provided in SMAC, and the dense reward function generated by the present invention. Figure 2-Figure 4 And the experimental results are shown in Table 6.

[0081] Among them, 1c3s5z means that our side has 1 Colossus, 3 Stalkers, and 5 Zealots, and the enemy has the same configuration.

[0082] MMM said that our side was equipped with 1 Medivac, 2 Marauders and 7 Marines, and the enemy had the same configuration.

[0083] MMM2 indicates that our side is equipped with 1 Medivac, 2 Marauders, and 7 Marines; while the enemy side is equipped with 1 Medivac, 3 Marauders, and 8 Marines. In this scenario, the enemy has more Marauders and Marines than our side, which gives the enemy greater firepower and survivability in a head-on confrontation.

[0084] Figure 2It indicates that in the 1c3s5z scenario, sparse rewards, the manually designed dense reward function provided in SMAC, and the dense reward function generated by the present invention are used to guide multi-agent reinforcement learning training respectively. The number of training steps is 10,000,000 (10M) steps, and a battle test is performed every 80,000 steps. The current policy network obtained by the current training is used in this battle test. A total of 32 battle tests are performed, and the winning rate is calculated.

[0085] Figure 3 It indicates that in the MMM scenario, sparse rewards, the manually designed dense reward function provided in SMAC, and the dense reward function generated by the present invention are used to guide multi-agent reinforcement learning training. The number of training steps is 10,000,000 (10M). A battle test is performed every 80,000 steps. The current policy network obtained by the current training is used in this battle test. A total of 32 battle tests are performed, and the winning rate is calculated.

[0086] Figure 4 It indicates that in the MMM2 scenario, sparse rewards, the manually designed dense reward function provided in SMAC, and the dense reward function generated by the present invention are used to guide multi-agent reinforcement learning training. The number of training steps is 10,000,000 (10M), and a battle test is performed every 80,000 steps. The current policy network obtained by the current training is used in this battle test. A total of 32 battle tests are performed, and the winning rate is calculated.

[0087] Table 6 shows the winning rate of our team after controlling our agent to perform 32 rounds of battle in the 1c3s5z, MMM, and MMM2 scenarios using the strategy network obtained after reinforcement learning training using sparse rewards, the manually designed dense reward function provided in SMAC, and the dense reward function generated by the present invention.

[0088] Combine Figure 2 As can be seen from Table 6, in the simple scenario 1c3s5z, the use of sparse rewards did not enable the multi-agent reinforcement learning algorithm to converge. Although the dense reward function generated by the control system of the present invention and the manually designed dense reward function had comparable convergence speeds, the dense reward function generated by the control system of the present invention was significantly superior to the manually designed dense reward function in terms of the final winning rate.

[0089] Combine Figure 3As shown in Table 6, in the similarly simple MMM scenario, sparse rewards failed to enable convergence of the multi-agent reinforcement learning algorithm. However, the dense reward function generated using the control system described in this invention significantly outperformed the manually designed dense reward function in terms of both convergence speed and ultimate win rate. Compared to the 1c3s5z scenario, where both the friendly and enemy teams consist of three types of units and the coordination between team members is more complex, the dense reward function generated using the control system described in this invention performed better in this complex scenario.

[0090] Combine Figure 4 As shown in Table 6, in the MMM2 scenario, sparse rewards did not enable the multi-agent reinforcement learning algorithm to converge. Although the dense reward function generated by the control system of the present invention and the manually designed dense reward function had comparable convergence speeds, the dense reward function generated by the control system of the present invention significantly outperformed the manually designed dense reward function in terms of the final winning rate. Moreover, MMM2 is a complex scenario in SMAC, and the difficulty of learning collaboration among our units is higher.

[0091] Comprehensive experimental results show that the policy network obtained through reinforcement learning training based on the dense reward function generated by the control system described in the present invention has a higher winning rate than the manually designed dense reward function and sparse reward function. This shows that the control system proposed in the present invention can effectively generate a dense reward function for guiding multi-agent reinforcement learning training and generate a policy network that can realize control tasks based on the dense reward function.

[0092] Table 6

[0093]

[0094] Based on the control system proposed in the aforementioned embodiment, the present invention also proposes a multi-agent control method for controlling multiple agents to perform control tasks. The method includes: using the control system described in the aforementioned embodiment to control multiple agents to perform control tasks.

[0095] Compared with the existing technology, the present invention can overcome the shortcomings of existing optimization solutions for sparse feedback. While reducing manual input, it no longer requires professional expert data. It can combine different modules to automatically generate dense feedback strategies to guide multi-agent reinforcement learning to generate corresponding policy networks.

[0096] The beneficial effects of the present invention are that an initial reward function generation module, an error correction module and a dense reward function generation module are set up to jointly generate a dense feedback strategy suitable for specific control tasks. By combining different modules, a dense feedback strategy is automatically generated without the need for professionals to invest a lot of time in strategy adjustment, nor is there a need to collect expert behavior data for strategy optimization. It can quickly generate dense feedback strategies suitable for different control tasks, thereby providing effective behavior guidance for multiple agents to perform specific control tasks.

[0097] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0098] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0099] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove having instructions stored thereon, and any suitable combination thereof.

[0100] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-agent control system based on a large language model, characterized in that: The system comprises: The simulation subsystem includes a data acquisition module, an initial reward function generation module, an error correction module, a dense reward function generation module, and a policy network update module; wherein: The data acquisition module is used to acquire multi-agent reinforcement learning training code for a sparse reward scenario corresponding to a control task; wherein the sparse reward scenario indicates that rewards are given in specific circumstances to evaluate the actions taken by the multi-agent when performing the control task; the multi-agent reinforcement learning training code is provided with a pre-trained policy network; The initial reward function generation module is used to generate function code prompt words based on the multi-agent reinforcement learning training code, and generate the initial reward function code based on the function code prompt words using a preset large language model; The error correction module is used to add the initial reward function code to the multi-agent reinforcement learning training code and perform multiple error correction processes to obtain an executable reward function code; The dense reward function generation module is used to add the executable reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning trainings to obtain the dense reward function code; The policy network update module is used to add the dense reward function code to the multi-agent reinforcement learning training code to perform multiple reinforcement learning trainings to obtain a policy network that maximizes the reward; The coordination control subsystem includes a data collection module and a strategy distribution module; wherein: The data collection module is used to receive observation data collected by multiple agents from the environment; The strategy allocation module is used to generate actions based on the observed data using a strategy network that maximizes rewards to guide multiple agents to perform control tasks.

2. The system according to claim 1, wherein: The initial reward function generation module is configured to generate the initial reward function in the following manner: Based on the multi-agent reinforcement learning training code for the sparse reward scenario corresponding to the control task, determine the task objective description and reward function generation instructions and constraints. The task objective description indicates the scenario in which the multi-agent performs the control task and the expected effect of performing the control task; the reward function generation instructions and constraints indicate the rules that need to be followed to generate the initial reward function. Extracting environment descriptions and background knowledge from the multi-agent reinforcement learning training code; the environment description indicates the environment in which the multi-agent is located and the interaction between the multi-agent and the environment, and the background knowledge indicates the programming language used in the multi-agent reinforcement learning training code; The task objective description, reward function generation instructions and constraints, environment description and background knowledge are used as function code prompts, and the initial reward function code is generated based on the function code prompts using a preset large language model.

3. The system according to claim 2, characterized in that The error correction module is configured to perform an error correction process in the following manner: Add the reward function code obtained after the previous error correction process to the multi-agent reinforcement learning training code to execute it to obtain the current syntax error and runtime error; Based on the current syntax error and runtime error, the reward function code obtained after the previous error correction process is corrected using the preset large language model to obtain the current reward function code; The error correction module uses the reward function code obtained after multiple error correction processes and having no syntax errors and runtime errors as the executable reward function code.

4. The system according to claim 3, characterized in that The dense reward function generation module is configured to perform a reinforcement learning training as follows: Add the executable reward function code obtained from the previous reinforcement learning training to the multi-agent reinforcement learning training code to execute the current reinforcement learning training and obtain the current reinforcement learning training result; Generate a current feedback prompt word based on the current reinforcement learning training result, and update the executable reward function code obtained from the previous reinforcement learning training using a preset large language model based on the current feedback prompt word to obtain the current executable reward function code; wherein the current feedback prompt word is used to indicate the improvement direction of the executable reward function code in the current reinforcement learning training; Among them, the executable reward function code that can realize the control task obtained after multiple reinforcement learning trainings is used as the dense reward function code.

5. The system according to claim 4, characterized in that The preset large language model is GPT-4o, Claude, kimi or deepseek.

6. The system according to claim 5, characterized in that The multi-agent is composed of multiple agents, wherein each agent includes a rigid body component, a joint component that directly controls the movement of the rigid body component, and an actuator that drives the joint component to control the movement of the rigid body component.

7. The system according to claim 6, characterized in that Each joint component of the agent is equipped with local sensors to collect observation data from the environment; the observation data includes the agent's own state and the environment state.

8. A multi-agent control method for controlling multi-agents to perform control tasks, characterized in that: The method comprises: A control system as described in any one of claims 1 to 7 is used to control multiple agents to perform control tasks.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to claim 8.

10. An electronic device, characterized in that: include: one or more processors, and memory; Wherein, each processor is configured with a system as described in any one of claims 1-7, which is used to control multiple agents to perform control tasks.

Images