Multi-agent reward function automatic generation and optimization method and system

Abstract

The invention provides a multi-agent reward function automatic generation and optimization method and system, and relates to the technical field of artificial intelligence and reinforcement learning. According to the method, state description information of an environment is extracted through an environment context construction module, and then a plurality of reward function candidates are evaluated at the same time through a parallelized reward function evaluation module by utilizing a reward function code generated by a large language model; afterwards, statistical information in the training process is converted into structured natural language feedback through a reflection report generation module, so that the large language model can understand the training dynamics and improve the reward function in a targeted manner, and then context-based directional rewriting of the reward function is realized; and finally, through an iterative optimization process, generating a reward function which not only conforms to a task target but also has good training characteristics. According to the method, the code generation capability of the large language model is combined with the reinforcement learning training process, so that automatic generation and iterative optimization of the reward function are realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and reinforcement learning, and in particular to a method and system for automatic generation and optimization of multi-agent reward functions based on a large language model. Background Technology

[0002] In reinforcement learning (RL) tasks, an appropriate reward function is crucial for learning the desired policy. Real-world task evaluation metrics are often sparse or non-smooth (e.g., returning 1 only on success and 0 otherwise), and using them directly as learning objectives can lead to training difficulties. Therefore, in engineering, "reward shaping" or manual trial-and-error design is often used to accelerate learning.

[0003] However, recent studies have shown that reward functions obtained through trial and error and manual parameter tuning are prone to overfitting to specific algorithms / hyperparameters, and may constitute invalid or erroneous task specifications (i.e., the reward does not reflect the designer's true intention), thus leading to unpredictable or unreproducible learning results and unfair comparative evaluation problems.

[0004] Traditional methods for automatically generating reward functions (such as template-based / parameter-search) are limited by predefined reward templates and expressiveness, making it difficult to cover complex behaviors or high-dimensional manipulations. Existing technologies, such as EUREKA (an automated reward design framework based on a large language model), can automatically generate and iteratively improve white-box reward programs by using the environment source code as context and performing evolutionary search and reward reflection. However, the specific application and systematic implementation of this framework in multi-agent scenarios still need improvement, particularly in handling multi-agent cooperation and competition, and integrating human feedback. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for automatic generation and optimization of multi-agent reward functions, aiming to solve the technical problem of: how to provide a general method that can automatically generate high-quality, interpretable reward function code, and can adaptively improve during training dynamics, while being compatible with the characteristics of multi-agent systems, so as to overcome the problems of time-consuming and error-prone manual design, limited expression of traditional automated methods, and insufficient adaptability of existing solutions in multi-agent scenarios.

[0006] On one hand, this invention provides a method for automatic generation and optimization of multi-agent reward functions, comprising the following steps: S1, environment context construction: including acquiring and processing program code or state description information of the target multi-agent reinforcement learning environment to form context information containing observable and interactive variables; S2, reward function generation: including inputting the context information and task target description information into a large language model, which generates reward function code; S3, reward function evaluation: including using the reward function code to perform reinforcement learning training in the target multi-agent reinforcement learning environment, and evaluating the performance of the trained reward function code based on a preset evaluation index to obtain evaluation results; S4, reflection report generation: including generating a structured natural language reflection report based on the evaluation results and statistical information of each component of the reward function during training; S5, context update and iteration: including feeding the reflection report back to the large language model, which, together with the context information, constitutes a new input to guide the large language model to generate improved reward function code in subsequent iterations; repeating steps S2 to S5 until the termination condition is met.

[0007] Preferably, in step S2, the large language model is a large language model capable of generating executable program code.

[0008] Preferably, in step S3, the reward function evaluation further includes using a graphics processor for parallel computation to evaluate multiple reward function codes simultaneously.

[0009] Preferably, in step S4, the statistical information includes the numerical sequences and statistics of each component of the reward function at different time points during the training process.

[0010] Preferably, in step S5, the termination condition is reaching a preset number of iterations or the evaluation result being better than a preset threshold.

[0011] Preferably, the method further includes the following step: S6, Human Feedback Integration: This includes receiving and processing natural language feedback information from human input, and integrating the feedback information into a reflection report or as independent input to a large language model.

[0012] Preferably, in step S2, the reward function code includes a local reward component for a single agent and a global reward component for a team of agents.

[0013] Preferably, in step S3, the reward function evaluation includes using a pool of opponents containing self-adversarial strategies to evaluate the trained reward function code to test its robustness.

[0014] On the other hand, the present invention provides a system for the above-mentioned method for automatic generation and optimization of multi-agent reward functions, comprising: an environment context construction module, including an interface for communicating with a large language model service, the environment context construction module being configured to execute the environment context construction step; a reward function generation module, including a large language model, the large language model being configured to execute the reward function generation step; a reward function evaluation module, including a graphics processor, the graphics processor being configured to execute the reward function evaluation step; a reflection report generation module, including a data statistician and a natural language template library, the data statistician statistically analyzing each component of the reward function during training, the natural language template library being configured to generate a structured natural language reflection report; and an iteration control module, including an interface for communicating with a large language model service, the iteration control module being configured to execute the context update and iteration steps.

[0015] Preferably, it also includes a human feedback interface module, including an interface for communicating with the large language model service, the human feedback interface module being configured to perform the human feedback integration step.

[0016] Implementing this invention offers the following advantages: By combining the code generation capabilities of a large language model with the reinforcement learning training process, this invention achieves automatic generation and iterative optimization of reward functions. First, the environment context construction module extracts state description information of the environment, providing the large language model with an accurate programming context and ensuring that the generated reward function code correctly references environment variables. Second, utilizing the large language model to generate reward function code enables zero-shot generation of richly expressive reward function code, significantly reducing the workload and subjective errors of manual design. Subsequently, the reflection report generation module transforms statistical information from the training process into structured natural language feedback, enabling the large language model to understand training dynamics and specifically improve the reward function. This achieves context-based targeted rewriting of the reward function, which is more efficient than random search or one-time generation and can accurately correct training defects. Finally, through iterative optimization, the system can gradually improve the quality of the reward function, generating a reward function that both meets the task objectives and possesses good training characteristics.

[0017] Furthermore, the reward function of this invention also supports human feedback integration, allowing human preferences and safety constraints to be incorporated into the optimization process in the form of natural language, achieving human preference alignment without gradient calculation, thus enhancing the practicality and security of the system.

[0018] Furthermore, special processing for multi-agent scenarios includes hierarchical reward structures, opponent pool evaluation, and collaborative indicator monitoring to ensure that the generated reward function can effectively drive individual learning and team collaboration, adapting to complex multi-agent interaction environments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a method for automatically generating and optimizing multi-agent reward functions based on a large language model, as shown in one embodiment. Figure 2 As one embodiment, in the GRF task, under the scenarios of empty goal close and 3 vs 1 with keeper, the trend graph of the average environmental return rate of the present invention and different SOTAs changes with time step; Figure 3 As one embodiment, in the GRF task, under the corner and pass and shoot with keeper scenarios, the trend of the average environmental return rate of the present invention and different SOTA algorithms changes with time step; Figure 4 In one embodiment, in the GRF task, under the run-to-score with keeper scenario, the trend graph of the average environmental reward rate of the present invention and different SOTA algorithms as a function of time step is shown. Figure 5 As one embodiment, in the SMAC task, under the 3m and 2s3z scenarios, the trend graph of the success rate of the present invention and different SOTA algorithms as a function of time step; Figure 6 As one embodiment, in the SMAC task, under the scenarios of 3s5z and 5mvs6m, the trend graph of the success rate of the present invention and different SOTA algorithms as a function of time step; Figure 7 In one embodiment, in the SMAC task, under the 6hvs8z scenario, the trend of the success rate of the present invention and different SOTA algorithms as a function of time step is shown. Figure 8 for Figures 2-7 The definition of curve identifiers corresponding to different algorithms; Figure 9 This is a schematic diagram comparing the success rates of three different reward function design methods—Ours, Human, and Ours with Human—in different scenarios of the SMAC task, as shown in one embodiment. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0021] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] Example 1 This embodiment provides a method for automatically generating and optimizing multi-agent reward functions for soccer robots.

[0023] In this embodiment, as Figure 1 The method for automatically generating and optimizing multi-agent reward functions, as shown, includes the following steps: S1. Context construction: This includes acquiring and processing program code or state description information of the target multi-agent reinforcement learning environment to form context information containing observable and interactive variables. S2. Reward function generation: This includes inputting the context information and task objective description information into the Large Language Model (LLM), and having the Large Language Model generate reward function code. S3. Reward function evaluation: This includes using the reward function code to perform reinforcement learning training in the target multi-agent reinforcement learning environment, and evaluating the performance of the trained reward function code based on a preset evaluation metric to obtain the evaluation result. S4. Reflection Report Generation: This includes generating a structured natural language reflection report based on the evaluation results and statistical information of each component of the reward function during the training process. S5. Context Update and Iteration: This includes feeding back the reflection report to the large language model, which, together with the context information, constitutes a new input to guide the large language model to generate improved reward function code in subsequent iterations; repeating steps S2 to S5 until the termination condition is met.

[0024] In this embodiment, in step S3, the reward function evaluation further includes using a graphics processor for parallel computation to simultaneously evaluate multiple reward function codes.

[0025] In this embodiment, in step S4, the statistical information includes the numerical sequences and statistics of each component of the reward function at different time points during the training process.

[0026] In this embodiment, in step S5, the termination condition is reaching a preset number of iterations or the evaluation result being better than a preset threshold.

[0027] In this embodiment, the construction of the environment context specifically includes: This process automatically extracts code snippets or schemas (data schemas, structured definitions describing observations, actions, and other data, such as JSON schemas) that only expose state / action / observation fields from a target multi-agent soccer environment (e.g., based on Unity / Isaac / self-developed simulations). This text is then used as context input for the LLM, ensuring the context remains within the LLM window size and does not leak internal simulation parameters (e.g., physics parameters). This ensures the LLM can recognize existing variable names (e.g., ball_pos, agent_positions, stamina) and directly reference them when generating reward code.

[0028] In this embodiment, by directly using the environment source code as context input and leveraging the code generation capabilities of LLM, the system can generate executable and expressive reward functions with zero-shot capability on most tasks, significantly reducing human resource input.

[0029] If the source code cannot be provided, the user must provide the observation / action dictionary and API (Application Programming Interface) specifications. This includes, for example, an automatic extraction script, a field mapping table, field data type specifications (tensor / float / bool), and common kinematic function interface templates.

[0030] In this embodiment, the large language model in step S2 is a coding LLM (transformed into a large language model), which can generate executable program code.

[0031] In this embodiment, the reward function is generated using Coding LLM to produce executable Python reward function code. The output format is required to be (total_reward, components_dict), and the sub-items of each award are exposed as a dictionary (for later review). The initial prompt includes a task description, environment fields, format and code style constraints (see EUREKA's Prompt1 / Prompt3), generating a candidate award program with K samples.

[0032] In this embodiment, by outputting the reward function as readable Python code and exposing the reward sub-item, it is easy to review, debug, and embed security constraints, thereby effectively avoiding the problems of IRL (Inverse Reinforcement Learning) and the inexplicability of black-box rewards.

[0033] Here, `total_reward` represents the weighted sum (scalar) of all reward items; `components_dict` represents exposing each individual reward sub-item (component) as a dictionary pair. `Prompt1` and `Prompt3` are two core prompt templates defined in the EUREKA paper; `Prompt1` is used for initial generation, and `Prompt3` is used for reflective iterative improvement.

[0034] In this embodiment, the reward function code needs to be compatible with TorchScript (a sub-language provided by PyTorch, used to convert dynamic Python models into static, serializable, and efficient representations) to ensure that it can be directly JIT (Just-In-Time compilation) and executed efficiently in a high-performance simulation environment, avoiding performance bottlenecks caused by the characteristics of dynamic Python.

[0035] Reward function generation quality assurance mechanism: (1) Structured Prompt Constraints (Input-side Reduction): Explicitly define a "code contract" in the Prompt to reduce generation randomness through mandatory templates. Dependency declarations: explicitly specify the libraries that can be called (such as NumPy, Torch) to avoid LLM calling non-existent private packages.

[0036] (2) Static automated compliance testing (pre-compilation): Before running the code, use static analysis tools to perform an initial screening to avoid basic syntax errors: (2.1) AST Syntax Parsing: The code tree is parsed using Python's AST module. If parsing fails (SyntaxError), it is directly deemed invalid;

[0037] (2.2) Feature extraction and verification: Check whether the generated code contains the necessary return structure (i.e., whether it contains return total_reward, components_dict) through AST. (2.3) Security Sandbox Audit: Check for dangerous operations (such as import os, shutil or network requests) to ensure code security.

[0038] (3) Dynamic verification and anomaly injection (pre-training): Before formally entering the training cycle of multi-agent reinforcement learning (MARL), a trial run is conducted: (3.1) Mock data testing: Construct a set of random state and action vectors and input them into the generated function; (3.2) Exception handling mechanism: Wrap function calls in try-except blocks, and capture traceback information if a runtime error (such as TypeError, ZeroDivisionError) occurs; (3.3) Fitness penalty: For code that fails to run, the system not only records the error, but also feeds it back to the reflection module, and assigns a very low penalty score to ensure that the sample is eliminated in the evolutionary algorithm or iteration.

[0039] In this embodiment, the reward function evaluation specifically includes: Parallelized RL training and evaluation are performed for each candidate award, using GPU-accelerated simulations (such as Isaac Gym or Isaac Sim) to obtain policy training curves and intermediate checkpoints in a reasonable time.

[0040] Evaluation metrics include task metrics F (such as win rate, goal difference), training stability (success rate as a function of epochs), and statistics for each reward component (mean / variance / time series). Parallel execution ensures that sufficient samples are available for selection within the evolutionary loop.

[0041] In this embodiment, RL training uses the PPO (Proximal Policy Optimization) algorithm with unified hyperparameters and training time window, and records sub-statistics of checkpoints for use by the reflection module.

[0042] In other embodiments, the algorithm used for RL training can also be a MARL (Multi-Agent Reinforcement Learning) algorithm, such as QMIX (Q-learning with Mixtures of Experts), VDN (Value-Decomposition Network), or MapPO (Multi-Agent Proximal Policy Optimization).

[0043] In this embodiment, the generation of the reflection report specifically includes: Make a "readable" summary of the training process of the candidate reward function - put the statistics of each sub-item of each award at each checkpoint and the task metric curve into a structured natural language, and send this text along with the code of the "best candidate" into the LLM request for targeted rewriting.

[0044] In this embodiment, training process statistics and task metric curves are converted into natural language feedback and returned to the LLM, achieving context-specific rewriting. This is more efficient and better able to correct training defects than simple template search or one-time generation, thereby improving the quality of the final reward function.

[0045] Targeted rewriting is used to guide the generation of reflective content and problem diagnosis. The diagnostic rules for targeted rewriting are as follows: If the success rate remains 0% for an extended period, rewrite the entire process. If a sub-item has a constant value, it may not be able to be optimized. It is recommended to scale or rewrite it. If the range of a certain sub-item is too large, it needs to be renormalized, etc.

[0046] In this embodiment, the reflection report is generated by the reflection report generation module, which may use EUREKA's Reward Reflection mechanism.

[0047] The output format of Reward Reflection is shown below.

[0048] In this embodiment, the statistics for each award sub-item at each checkpoint include the maximum value, average value, minimum value, variance, and a time-series summary.

[0049] In this embodiment, the task metric curve includes success rate, reward, etc.

[0050] In this embodiment, context update and iteration specifically include: The search is performed using an iterative evolutionary strategy (the algorithm is derived from EUREKA): In each round, K rewards are sampled from the LLM, the optimal reward function is evaluated, and the optimal reward function, along with the reflection summary of the optimal reward function, is filled back into the LLM for "mutation" to generate candidates for the next round; this is repeated for N rounds, and multiple random restarts are performed to avoid local optima.

[0051] In this embodiment, by performing multiple random restarts in the evolutionary search, evaluating candidates using an adversary pool and various RL design contexts, and using performance under different algorithms / hyperparameters as reflective input, the problem of overfitting to a single algorithm / hyperparameter, which is common in human trial and error, is mitigated.

[0052] In this embodiment, the context update and iteration algorithm process specifically includes: initialization: Start with an initial reward function prompt, or let the LLM generate initial candidates directly from scratch.

[0053] Each iteration: Starting from the code and context of the current best reward function, sample K new candidate reward functions (sample K rewards) to be generated by LLM.

[0054] Each of these K candidates is evaluated (performance is typically measured by running a reinforcement learning agent on the target task, such as success rate, reward, etc.).

[0055] Select the best performer in this round as the current optimal reward function.

[0056] Feedback and Mutation: The code for the optimal reward function, along with the reflection summary (i.e., the readable statistics and diagnostic text of the training process generated by the aforementioned module), is fed back to the LLM. The LLM is then instructed to rewrite or modify it to generate K new candidates for the next round. In this embodiment, the rewriting emphasizes incremental improvement, that is, making minor modifications based on existing excellent candidates, rather than generating them completely randomly.

[0057] Multiple cycles: Repeat the above process N times to gradually evolve a better reward function.

[0058] Random Restarts: To avoid getting trapped in local optima, multiple independent complete search processes are performed, each starting with a different initial hint or random seed, and finally the globally optimal reward function is selected from all restarts.

[0059] Algorithm features and advantages: It combines sample diversity (by sampling K candidates at once) with the ability to incrementally improve within the context (by guiding the LLM to make targeted rewrites through reflective feedback). This evolutionary strategy has been proven effective in various robotic manipulation tasks in the EUREKA paper, generating reward functions that are close to human-designed ones.

[0060] In this embodiment, the recommended hyperparameter example (default value) is as follows: K = 16: 16 candidates are sampled in each round.

[0061] N = 5: Each search path undergoes 5 rounds of iteration.

[0062] restarts = 5: Perform 5 independent random restarts.

[0063] The hyperparameters (K=16, N=5, restarts=5) mentioned above are recommended default values ​​based on typical multi-agent soccer environments (such as 5v5). Their values ​​are determined as follows: K balances generation diversity with computational cost; experience shows that fewer than 8 candidates are insufficient to cover effective exploration, while more than 32 results in excessive evaluation overhead; N=5 is a trade-off between computational budget and observed convergence speed; restarts=5 is used to escape local optima with a higher probability. In practical applications, users can dynamically adjust these parameters according to environmental complexity (such as the number of agents and the dimension of observations), computational resources, and the required solution quality. For example, for more complex environments, the values ​​of K and restarts can be appropriately increased.

[0064] In this embodiment, the pseudocode (excerpt) of the context update and iteration algorithm is as follows:

[0065] The overall structure and flow of this pseudocode are as follows: Input (Required): task l: Task description (usually a task objective described in natural language).

[0066] env code M: Environment source code (provided directly to the LLM as context to facilitate the generation of executable reward code).

[0067] LLM: Large Language Model (used for sampling to generate candidate reward functions).

[0068] fitness F: Fitness function (a fitness / evaluation metric used to evaluate the quality of the reward function, typically obtained by running an RL agent in the environment, such as success rate, cumulative reward, etc.).

[0069] init_prompt P0: Initial prompt (used to guide the LLM in generating candidates for the first time).

[0070] Hyperparameters: N_iterations: The number of iterations for each search path.

[0071] K_samples: The number of candidates sampled from the LLM in each round (corresponding to diversity).

[0072] num_restarts: The number of random restarts (used to avoid local optima).

[0073] Main loop: Outer loop: Performs num_restarts random restarts, each time starting independently from the initial prompt P0 to explore different search paths and finally find the global optimum.

[0074] Inner initialization: In each restart, set the current best reward (best_reward) to None and the best score (best_score) to -∞.

[0075] The middle loop performs N_iterations rounds of iterative evolution.

[0076] Using LLM, K_samples candidate reward functions are generated based on the current prompt, task l, and environment code M.

[0077] Call the `evaluate_candidates` function to evaluate these candidates: return scores (based on fitness F) and reward reflections (usually statistical summaries and diagnostic texts of the RL training process, such as the contribution of each reward item, failure modes, etc.).

[0078] Find the best index i_best (the one with the highest score) for this round.

[0079] If the best score in this round is higher than the current overall best, then update best_reward and best_score (tracking the global best within this restart).

[0080] Update tip: augment(prompt, reflections[i_best]) — Adds reflections of the best candidate in this round to the prompt, achieving incremental improvement (in-context improvement) and guiding the next round of LLM rewriting.

[0081] Output: return global_best_reward: The globally best reward function selected from all restarts.

[0082] It's important to note that in EUREKA, LLM needs to read the complete environment source code as context to generate reward functions. However, the complete source code is often very large, leading to an excessive number of input tokens (increasing cost and latency), and may also contain sensitive kernel code (posing a risk of information leakage). In this case, a context extraction script can automatically preprocess the source code, extracting only the necessary parts for LLM (observation space, action space, state variables, and other key fields), pruning irrelevant code, and converting it into a structured text format for efficient reading and understanding by LLM.

[0083] In this embodiment, the iterative process of EUREKA requires repeatedly evaluating the quality of a large number of candidate reward functions (by training the policy and calculating fitness scores in a simulation environment). This is slow if simulated using a single thread or CPU; therefore, parallel evaluation can be employed. Specifically, using a GPU-accelerated large-scale parallel simulation environment (such as Isaac Gym) can run tens of thousands of environment instances simultaneously on a single or multiple GPUs, rapidly training and evaluating multiple candidate reward functions in parallel, significantly shortening the evolutionary cycle time. The parallelized reward function evaluation module simultaneously evaluates multiple candidate reward functions, accelerating the optimization process.

[0084] In this embodiment, the reward function generated by LLM may have defects, leading to dangerous behaviors in the trained policy, such as reward hacking: the agent obtains high scores through cheating methods, such as infinite loop actions to farm rewards. Before formal deployment, a security constraint validator needs to be added: through static code analysis (checking for infinite terms and anomalous incentives in the reward formula) and sandbox simulation (running tests in an isolated environment), potentially dangerous reward items are automatically detected and filtered out. To prevent reward hacking and dangerous behaviors, the reward function integrates the following security mechanisms:

[0085] (1) Static code analysis: Before the reward function code is executed, it automatically checks for dangerous function calls (such as direct calls to the underlying physics engine API of the environment), infinite loops, and maximum / minimum values ​​(such as e). 10 or -e 10 And the risk of dividing by zero.

[0086] (2) Dynamic sandbox testing: In an isolated test environment, a simple pre-trained strategy is used to run the reward function for several steps and monitor its reward value. If the reward value spikes or drops abnormally in a very short time, the reward function is deemed to have a deceptive risk and is discarded.

[0087] (3) Semantic constraint check: Define safety constraints according to task requirements (such as prohibiting or encouraging high collision force) and convert them into code inspection rules (such as monitoring collision force indicators, and issuing an alarm if the reward item is positively correlated with the collision force).

[0088] In this embodiment, the reward function typically consists of multiple sub-items, making the training process complex and opaque. Visualization tools, such as logs and interpretability dashboards, can be used to plot the contribution curve (time series graph) of each reward sub-item during training in real time, and generate automated reflection reports for human experts to review. Humans can directly edit or intervene in the reward function, improving the system's interpretability and the efficiency of human supervision.

[0089] Example 2 Based on Embodiment 1, this embodiment provides a method for automatically generating and optimizing multi-agent reward functions that includes human feedback integration.

[0090] In this embodiment, the method for automatic generation and optimization of multi-agent reward functions further includes the following steps: S6. Human Feedback Integration: This includes receiving and processing natural language feedback information from human input, and integrating the feedback information into the reflection report or as independent input to the large language model.

[0091] In this embodiment, when the objective metric F (fitness score) for the real task is unavailable (e.g., the “aesthetic teamwork” in a football match is difficult to quantify) or the computational cost is too high, human-provided textual reflection or video contrast preference can be used to replace or supplement the original automated feedback.

[0092] In this embodiment, the textual reflection is manually input by a human, such as "The team passes too many balls but lacks shots, appearing conservative."

[0093] In this embodiment, the video comparison preference is subjectively selected by humans, such as "strategy A is more aggressive than strategy B, I prefer strategy A".

[0094] LLM directly generates improved reward function code within a new cue context that includes information about human reflection or preferences, thereby achieving gradient-free in-context reinforcement learning from human feedback (in-context RLHF).

[0095] In this embodiment, safety constraints can be explicitly added to human reflection or system prompts, such as prohibiting excessive collision actions (to avoid the risk of injury caused by high-intensity fouls) and setting a penalty cap for stamina depletion (to prevent strategies from excessively consuming stamina and becoming unsustainable).

[0096] Security constraints are incorporated into the reflective text, guiding LLMs to proactively add corresponding penalty terms when generating new reward functions, thereby improving the deployability and security of the final policy.

[0097] In this embodiment, the integration of human feedback shifts the entire reward evolution framework from purely automated evaluation to a human-machine collaborative model. This enables the handling of subjective and difficult-to-quantify goals (such as the aesthetics and entertainment value of strategies), while naturally incorporating safety constraints through human feedback. This is particularly important in tasks requiring both "spectacular" and "safe" aspects, such as football. The resulting strategies are not only high-performing but also better aligned with human expectations and real-world deployment requirements.

[0098] Example 3 This embodiment provides a multi-agent reward function automatic generation and optimization system based on any of the above embodiments, for implementing the above-mentioned multi-agent reward function automatic generation and optimization method.

[0099] In this embodiment, the multi-agent reward function automatic generation and optimization system includes: The system includes an environment context construction module, comprising an interface for communicating with a large language model service, configured to execute the environment context construction step; a reward function generation module, comprising a large language model, configured to execute the reward function generation step; a reward function evaluation module, comprising a graphics processor, configured to execute the reward function evaluation step; a reflection report generation module, comprising a data statistician and a natural language template library, wherein the data statistician statistically analyzes the components of the reward function during training, and the natural language template library is configured to generate a structured natural language reflection report; and an iteration control module, comprising an interface for communicating with a large language model service, configured to execute the context update and iteration steps.

[0100] In this embodiment, the multi-agent reward function automatic generation and optimization system further includes a human feedback interface module, which includes an interface for communicating with a large language model service. The human feedback interface module is configured to perform the human feedback integration step.

[0101] Example 4 Based on the above embodiments, this embodiment provides a method for adapting the automatic design and iteration method of multi-agent reward functions to a multi-agent football robot (MARL-football).

[0102] In this embodiment, the multi-agent soccer task has three core characteristics: team collaboration, competitive confrontation, and local / global evaluation.

[0103] The three characteristics mentioned above make it difficult to directly apply single-agent reward design methods. Therefore, it is necessary to adapt them to multi-agent systems in order to generate reward functions that are more suitable for the multi-agent reinforcement learning (MARL) environment.

[0104] In this embodiment, in step S2, the reward function code includes a local reward component for a single agent and a global reward component for a team of agents.

[0105] In this embodiment, in step S3, the reward function evaluation includes using a pool of opponents containing self-adversarial strategies to evaluate the trained reward function code in order to test its robustness.

[0106] In this embodiment, multi-agent adaptation specifically includes the following levels of adaptation: (1) Hierarchical / Role-Based Reward Structure In this embodiment, the generated reward function code must support a combination of per-agent sub-rewards and global team-level rewards.

[0107] Example structure: Reward components are organized in dictionary form, such as reward_components['agent_3']['pass_success'] which represents the reward for a successful pass by a specific agent 3 (agent_3); team_reward also exists to represent the reward for team cooperation (such as goals, possession rate).

[0108] Final total reward: total_reward = agent_local_reward + α * team_reward (α is an adjustable weight).

[0109] In this embodiment, by setting a hierarchical / role-based reward structure, individual skill learning (such as individual ball control and shooting) and team collaboration (such as passing coordination and positioning) can be driven simultaneously, avoiding selfish behavior caused by purely individual rewards or individual inertia caused by purely team rewards.

[0110] (2) Opponent Modeling and Adversarial Robustness In this embodiment, when evaluating candidate reward functions, it is not sufficient to use only fixed opponents; instead, an opponent pool must be introduced, employing either league training or self-play strategies for evaluation. This prevents the reward function from overfitting to a specific opponent's strategy, ensuring that the generated rewards maintain high performance and robustness when facing opponents of different levels and styles.

[0111] (3) Convergence and Stability Monitoring In this embodiment, in order to address the common training instability problem in multi-agent reinforcement learning (MARL), additional collaboration metrics (such as cooperation rate, number of conflicts between teammates, blocking events, etc.) are recorded during the evaluation phase, and the collaboration metrics are input into the LLM as text of the reflection report.

[0112] In this embodiment, convergence / stability monitoring can guide LLM to adjust the reward function in a targeted manner, suppress undesirable cooperative behaviors (such as teammates blocking each other or deliberately committing fouls to steal the ball), and promote stable convergence and the emergence of high-quality team strategies.

[0113] (4) Reward Normalization and Scale Adaptation In this embodiment, during the reward function evaluation or reflection report generation stage, the magnitude differences of each sub-reward item are automatically detected (e.g., a pass reward might be in the range of 0-1, while a distance penalty might be between -100 and 0), and this statistical information is fed back to the LLM to prompt it to adjust the weights or scaling factors. This avoids a particular reward item becoming too large and dominating the optimization process, causing other important items to be ignored, and ensures that the trainer can optimize all objectives in a balanced manner.

[0114] In this embodiment, through the adaptation of the above four levels, the reward function generated by LLM can not only quickly drive individual skill learning, but also effectively catalyze the emergence of advanced collaborative strategies at the team level, thereby better adapting to highly competitive and collaborative multi-agent environments such as football.

[0115] Example 5 Based on the above embodiments, this embodiment provides a specific implementation plan for an automatic design and iteration method of multi-agent reward functions in 5v5 multi-agent soccer.

[0116] In this embodiment, the process of automatic design and iteration of multi-agent reward functions in 5v5 multi-agent soccer is as follows: (1) Extract environmental fields (ball position, parameters of each agent (position, speed, orientation, physical strength, etc.), game time, pass / shoot determination, etc.) and construct the LLM context.

[0117] (2) Generate K=16 reward function programs from an LLM (such as GPT-4) using an initial prompt (including format constraints), in the format (reward_total, components_dict).

[0118] (3) Train each candidate strategy for several rounds in parallel using the baseline MARL algorithm (such as MAPPO or QMIX, selected based on whether it is intensive training), and record the objective metrics of the task (such as win rate / goal difference) and the sequence of each component.

[0119] (4) Generate reward reflection text and feed it back to the LLM, requiring the LLM to rewrite based on the reflection text; repeat the evolution iteration N times and restart multiple times, and finally select the best reward function. (5) Test the selected reward function in the opponent pool (including historical strategies and self-adversarial strategies). If the performance is stable, deploy it to a higher fidelity simulation / real machine test. At the same time, run the safety constraint verifier before deployment.

[0120] In this embodiment, when task metrics are difficult to express using objective metrics for a single task or need to reflect human aesthetics (such as elegant passing, avoiding high foul rates): human textual feedback or video preference comparison is introduced during the reflection phase. Human review conclusions (e.g., "preferring solid passing → reducing high-speed advance rewards") are added to the prompt in natural language format to guide the LLM to generate a more aligned reward function to adapt to human subjective preference indicators.

[0121] In this embodiment, during the evaluation phase, multiple training contexts (different RL algorithms, different discount factors γ, and different rewriting strategies) are used simultaneously for each candidate reward function, and the performance of multiple contexts is used as the comprehensive fitness. The reflection report includes "comparing the performance differences of different contexts" to encourage LLM to generate a reward function that is robust across contexts, thereby reducing the risk of overfitting.

[0122] In this embodiment, if computational resources are limited, only lightweight simulations or shorter training epochs can be run initially to quickly screen candidate reward functions, followed by a longer period of high-fidelity evaluation of the selected optimal reward function. Simultaneously, the reflective rewriting mechanism is retained to improve search efficiency when computation is constrained.

[0123] Example 6 Based on the above embodiments, this embodiment provides a verification embodiment of an automatic design and iteration method for multi-agent reward functions.

[0124] In this embodiment, the effectiveness of the algorithm proposed in this invention was verified through simulation experiments in multiple high-dimensional, highly dynamic complex multi-agent environments (Google Research Football and StarCraft Multi-Agent Challenge, SMAC).

[0125] The experimental environment is set up as follows: Google Research Football (GRF) Task: Typical football game scenarios were selected as follows: (a) empty goal close, (b) 3 vs 1 with keeper, (c) corner, (d) pass and shoot with keeper, and (e) run to score with keeper. These scenarios cover different difficulty levels, from simple scoring to complex cooperative defense, and are designed to test the algorithm's strategy learning ability under sparse rewards.

[0126] StarCraft Multi-Agent Challenge (SMAC) Missions: (a) 3m, (b) 2s3z, (c) 3s5z, (d) 5mvs6m, and (e) 6hvs8z were selected as multi-heterogeneous agent battle scenarios for evaluation, focusing on verifying the performance of the algorithm at the level of large-scale cooperation and micro-management.

[0127] Performance analysis: like Figures 2-4 As shown, in all test scenarios of the GRF task, the achieved Average Environment Return and Success Rate are significantly better than existing state-of-the-art (SOTA) algorithms (including Eureka, QMIX, VDN, and DRND). Meanwhile, as... Figures 5-7 As shown, in all test scenarios of the SMAC task, the multi-agent reward function automatic generation and optimization method proposed in this invention (marked as "Ours" in the figure, red curve) achieves a significantly higher success rate than existing state-of-the-art algorithms (including Eureka, QMIX, VDN, DRND, etc.). Figures 2-7 As can be seen, compared with existing state-of-the-art (SOTA) algorithms, the automatic design and iteration method of multi-agent reward function in this invention has higher environmental rewards and success rate:

[0128] In this embodiment, Figure 8 To Figures 2-7 The curve identifiers corresponding to different algorithms are as follows: Figure 8 The red curve (the first curve from top to bottom) in the diagram represents the algorithm proposed in this invention and is labeled "Ours". Figure 8 The yellow curve (the second curve from the top) is labeled Eureka, which is a MARL method. Figure 8 The blue curve (the third curve from the top) is labeled QMIX (Monotonic Value Function Factorization for Deep Multi-Agent Reinforcement Learning). QMIX is a method for decomposing monotonic value functions for deep multi-agent reinforcement learning. Figure 8 The purple curve (the fourth curve from the top) is labeled VDN (Value Decomposition Networks), which is a value decomposition network method. Figure 8 The green curve (the fifth curve from the top) is labeled DRND (Diversity Regularizedsomethin), which is a diversity regularization exploration method.

[0129] Experimental results show that the multi-agent reward function automatic generation and optimization method of this invention has a steeper slope of the ascent curve in the early stage of training. In extremely difficult scenarios such as 6hvs8z, when the comparison algorithm is still in a performance plateau or improving slowly, this invention can quickly reach a higher success rate threshold in fewer time steps, demonstrating excellent convergence speed and proving its high efficiency in reward function optimization and policy iteration. In highly adversarial tasks such as 3s5z and 6hvs8z, this invention exhibits excellent convergence stability with smaller curve fluctuations, demonstrating extremely high robustness in complex scenarios, and the final stable performance peak is significantly higher than other benchmark algorithms.

[0130] In summary, this invention effectively solves the problem of difficult reward modeling in complex tasks through a reward function generation and optimization mechanism guided by a large model. Experimental data quantitatively demonstrate that this algorithm not only possesses stronger policy discovery capabilities in multi-agent cooperative tasks, but also achieves leading levels in the field in terms of convergence speed and final execution performance.

[0131] To further verify the key role of LLM in reward function design, this embodiment sets up three sets of variables for comparative experiments: reward function independently designed by LLM (Ours), reward function designed by humans (Human), and reward function optimized by LLM based on human priors (Ours with Human).

[0132] Experimental results (such as) Figure 9 As shown in the figure: In most scenarios (such as 3s5z, 6hvs8z), the reward function independently generated by LLM significantly outperforms the reward function designed manually, demonstrating the superiority of LLM-independent design and its outstanding ability to uncover potential incentives in complex environments. In extremely difficult tasks (such as 5m vs 6m), by using the reward logic defined by human experts as the prior input LLM for a second iteration, Ours with Human algorithm shows stronger robustness and a higher upper limit of win rate.

[0133] Experimental data quantitatively demonstrate that, compared to traditional manually designed reward functions, the proposed method for automatic generation and optimization of multi-agent reward functions can significantly reduce the cost of manual modeling and effectively overcome the cognitive limitations inherent in manually designed reward functions.

[0134] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for automatic generation and optimization of multi-agent reward functions, characterized in that, Includes the following steps: S1. Context construction: This includes acquiring and processing program code or state description information of the target multi-agent reinforcement learning environment to form context information containing observable and interactive variables. S2. Reward function generation: This includes inputting the context information and task target description information into the large language model, and generating reward function code from the large language model. S3. Reward function evaluation: This includes using the reward function code for reinforcement learning training in the target multi-agent reinforcement learning environment, and evaluating the performance of the trained reward function code based on a preset evaluation metric to obtain the evaluation result. S4. Reflection Report Generation: This includes generating a structured natural language reflection report based on the evaluation results and statistical information of each component of the reward function during the training process. S5. Context Update and Iteration: This includes feeding the reflection report back to the large language model, which, together with the context information, constitutes a new input to guide the large language model to generate improved reward function code in subsequent iterations. Repeat steps S2 to S5 until the termination condition is met.

2. The method for automatic generation and optimization of multi-agent reward functions according to claim 1, characterized in that, In step S2, the large language model is a large language model capable of generating executable program code.

3. The method for automatic generation and optimization of multi-agent reward functions according to claim 1, characterized in that, In step S3, the performance evaluation of the trained reward function code based on preset evaluation metrics includes using a graphics processor for parallel computation to simultaneously evaluate multiple reward function codes.

4. The method for automatic generation and optimization of multi-agent reward functions according to claim 1, characterized in that, In step S4, the statistical information includes the numerical sequences and statistics of each component of the reward function at different time points during the training process.

5. The method for automatic generation and optimization of multi-agent reward functions according to claim 1, characterized in that, In step S5, the termination condition is reaching a preset number of iterations or the evaluation result being better than a preset threshold.

6. The method for automatic generation and optimization of multi-agent reward functions according to claim 1, characterized in that, It also includes the following steps: S6. Human Feedback Integration: This includes receiving and processing natural language feedback information from human input, and integrating the feedback information into the reflection report or as independent input to the large language model.

7. The method for automatic generation and optimization of multi-agent reward functions according to claim 1, characterized in that, In step S2, the reward function code includes a local reward component for a single agent and a global reward component for a team of agents.

8. The method for automatic generation and optimization of multi-agent reward functions according to claim 7, characterized in that, In step S3, the reward function evaluation includes evaluating the trained reward function code using a pool of opponents containing self-adversarial strategies to test its robustness.

9. A system for implementing the automatic generation and optimization method of multi-agent reward function as described in any one of claims 1 to 8, characterized in that, include: An environment context building module, including an interface for communicating with a large language model service, is configured to perform the environment context building steps; The reward function generation module includes a large language model, which is configured to perform the reward function generation step; A reward function evaluation module includes a graphics processor configured to perform the reward function evaluation step; The reflection report generation module includes a data statistician and a natural language template library. The data statistician counts the components of the reward function during training, and the natural language template library is configured to generate structured natural language reflection reports. An iteration control module, including an interface for communicating with a large language model service, is configured to execute the context update and iteration steps.

10. The system according to claim 9, characterized in that, It also includes a human feedback interface module, which includes an interface for communicating with the large language model service. The human feedback interface module is used to receive and process natural language feedback information input by humans, and to integrate the feedback information into the reflection report or as independent input to the large language model.

Images