Large language model collaborative decision-making method and system based on reinforcement learning driving

Abstract

The invention discloses a large language model collaborative decision-making method and system based on reinforcement learning driving, and belongs to the technical field of strategic game decision-making, and the method comprises the steps: building a large language model strategy: converting a game state into a structured text prompt, and generating an executable micro-action sequence; constructing a tool strategy: generating a predefined macro action based on the micro action sequence; and collaborative decision-making: cooperatively scheduling the large language model strategy and the tool strategy by adopting a near-end strategy optimization algorithm. An action self-adaptive adjustment mechanism is introduced, and the execution length of the micro action and the macro action is dynamically adjusted according to the game stage; and a layered reward function is designed, and dense and effective training signals are provided in combination with the overall winning rate reward and the local performance reward. According to the method, the problems of semantic conflicts, resource conflicts and attention offset existing in tool calling of a large language model are effectively solved, and the decision-making efficiency, the strategy coordination and the training stability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language models and strategic game decision-making, and in particular to a large language model collaborative decision-making method and system driven by reinforcement learning. BACKGROUND

[0002] With the increasing application of large language models (LLM) in the field of strategic game decision-making, tool calling (ToolCalling) as a key enhancement technology for improving model decision-making capabilities has gradually become an important research direction. Although large language models have powerful semantic understanding and generation capabilities, their internal reasoning mechanisms often fail to meet the high real-time and high complexity requirements of strategic decision-making.

[0003] Traditional methods rely on prompt engineering to coordinate tools and native strategies, and by introducing structured tool calling, the functional boundaries of the model can be effectively expanded, and decision-making efficiency and task execution capabilities can be improved. Large language models not only need to fully understand the functional characteristics of various external tools, but also need to clearly distinguish the differences between them and the native strategy to achieve their collaborative use. The contradiction of traditional methods is that after introducing tool calling, the effective calling rate of the tool is still insufficient, and the decision-making behavior of the large model is mostly concentrated in the original action space, resulting in the overall strategy performance not being improved. This shows that traditional methods have structural defects in coordinating native strategies and external tool calling, and lack effective fusion mechanisms. The specific problems are as follows: (1) Semantic conflict: the model may have difficulty distinguishing between executing native actions or calling tools with similar semantic descriptions at a specific decision-making moment, leading to confusion in decision-making logic.

[0004] (2) Resource conflict: although external tools can improve task effectiveness, their calling is often accompanied by high time and computational resource overhead. If not called at the appropriate time, it will result in resource waste and affect overall decision-making efficiency.

[0005] (3) Attention shift: due to the repetition of text descriptions of tool actions and native actions in prompt words, the large model's attention is easily over-concentrated on co-occurring words, leading to the generation of actions that do not conform to the current context, interfering with the accuracy of the decision-making process.

[0006] In summary, existing methods lack effective fusion mechanisms, resulting in low tool calling rates and limited strategy performance improvements. Therefore, there is an urgent need for an intelligent decision-making method that can dynamically coordinate large language models and tool calling. SUMMARY

[0007] The purpose of this invention is to overcome the problems existing in the prior art and provide a collaborative decision-making method and system based on reinforcement learning-driven large language models. This method can effectively integrate the fine-grained decision-making capabilities of large language models with the efficient macro actions of predefined tool modules in complex strategic game environments. Dynamic coordination is achieved through reinforcement learning algorithms, thereby improving decision-making efficiency and win rate.

[0008] The objective of this invention is achieved through the following technical solution: Firstly, a collaborative decision-making method for large language models driven by reinforcement learning is provided, including: The strategy for building a large language model is to convert game states into structured text prompts and generate executable micro-action sequences. Tool construction strategy: Generate predefined macro actions based on the micro-action sequences; Collaborative decision-making: The large language model strategy and tool strategy are coordinated and scheduled using a near-end strategy optimization algorithm.

[0009] In some embodiments, converting the game state into a structured text prompt includes: The game state is converted into structured hints using a digest chain.

[0010] In some embodiments, each of the macro actions consists of a series of micro action sequences, and the tool strategy is designed in a hierarchical manner.

[0011] In some embodiments, the collaborative decision-making specifically includes: An adaptive action adjustment mechanism is introduced to dynamically adjust the execution length of micro-actions and macro-actions according to the game stage; We design a hierarchical reward function that combines overall win rate rewards with local performance rewards to optimize the training process.

[0012] In some embodiments, the action adaptive adjustment mechanism includes: Micro-motion adjustment: Limit the maximum length of the generated sequence according to the game stage; Macro-action adjustment: Dynamically shorten the length of macro-actions based on time thresholds, and gradually transfer decision-making power to the large language model.

[0013] In some embodiments, the formula for calculating the overall win rate reward is: Where t represents the game duration, and As the baseline value, and These are the attenuation coefficient and the growth coefficient, respectively.

[0014] In some embodiments, the local performance bonus includes: Population congestion rate: reward the agent for efficient population supply management.

[0015] Average population utilization rate: measure the balance of resource allocation.

[0016] Attack frequency: encourage offensive behavior by rewarding the damage value caused to the enemy.

[0017] In some embodiments, the proximal policy optimization algorithm employs a clipped target function and a trust region mechanism.

[0018] In a second aspect, a large language model collaborative decision-making method driven by reinforcement learning is provided, comprising: A large language model strategy construction module for converting game states into structured text prompts and generating executable micro-action sequences; A tool strategy construction module for generating predefined macro-actions based on the micro-action sequences; A collaborative decision-making module for collaboratively scheduling the large language model strategy and tool strategy using a proximal policy optimization algorithm.

[0019] It should be further noted that the technical features corresponding to the above-mentioned option embodiments can be combined or replaced with each other to form new technical solutions without conflict.

[0020] Compared with the prior art, the present application has the following advantages: 1. Improve decision-making efficiency and strategy coordination: The present application uses a reinforcement learning driven large language model and tool invocation collaborative strategy framework to collaboratively schedule the large language model and tool invocation, effectively integrating micro-actions and macro-actions, overcoming semantic and resource conflicts, and improving decision-making efficiency and strategy coordination.

[0021] 2. Reduce training complexity: The present application is based on PPO algorithm design, with high sample utilization rate and training stability, reducing training complexity and sample demand, and adapting to complex environments. 3. Dynamically optimize the decision-making process: The present application introduces an action adaptive adjustment mechanism to flexibly control the length of the action sequence according to the game stage, dynamically optimizing the length of the action sequence, and balancing high-level strategic planning and low-level fine-grained control.

[0022] 4. Relieve the reward sparsity problem: The present application designs a hierarchical reward function to provide immediate and intensive feedback signals, alleviating the reward sparsity and delay problem, and stabilizing the training process.

[0023] 5. Enhance long-term planning ability: The present application supplements the large language model's ability through predefined tool strategies, making up for its shortcomings in long-term planning, and achieving collaborative optimization. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A flowchart of a large language model collaborative decision-making method based on reinforcement learning driving according to the present application is shown in FIG. 1. Figure 2 A schematic diagram of the action adaptive adjustment mechanism according to the present application is shown in FIG. 2.

[0025] Figure 3 A flowchart of a reinforcement learning driven large model and tool invocation collaborative framework according to the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0026] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various configurations. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0027] It should be noted that the defects in the above prior art solutions are the result of the inventors' practice and careful study, therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present application to solve the above problems should be the contribution of the inventors to the present application, and should not be understood as technical content known to those skilled in the art.

[0028] To solve the technical problems pointed out in the background art, the embodiments provided by the present application are as follows: In an exemplary embodiment, the present application provides a large language model collaborative decision-making method based on reinforcement learning driving by analyzing the challenges of large language models in tool invocation integration in complex environments such as TSC2, through a reinforcement learning framework to coordinate large language model strategies and external tool strategies, as shown in FIG. 1, which includes: Figure 1 Building a large language model strategy: converting game state to structured text prompt and generating executable micro-action sequence; Building a tool strategy: generating predefined macro actions based on the micro action sequence; Collaborative decision-making: using proximal policy optimization algorithm to coordinate and schedule the large language model strategy and tool strategy.

[0029] Specifically, in building a large language model strategy, the specific implementation process is as follows: The large language model strategy is built based on the TSC2 environment, using all 72 micro actions ( - ​). The game state information is converted into structured text prompts through the Chain of Summarization of TSC2, and after inputting into a large language model (such as Qwen1.5-7B), executable micro-action sequences are generated. This step is responsible for fine-grained context-aware decision-making and can generate flexible micro-operation sequences according to the current game state. Its core decision-making process can be formally represented as a mapping from state space to action sequence: (1) wherein, represents the structured game state obtained through the Chain of Summarization at time step t, represents the LLM output action, L is the length of the micro-action sequence generated by the model, ∈ {C0, C1,..., C71} is a specific micro-action. This design enables the agent to go beyond the limitations of single-step decision-making and achieve multi-step, forward-looking strategy planning.

[0030] Specifically, in the construction of tool strategies, the specific implementation process is as follows: The designed tool strategy is a macro-action, which encapsulates a series of micro-actions into a reliable and meaningful "skill" or "strategy fragment", providing the agent with pre-packaged high-level "options". This greatly reduces the effective decision-making space, as the agent does not need to explore all combinations of basic actions from scratch. Tool strategies are supported by a dedicated database ε, which contains five independent macro-actions (A1-A5). Each macro-action is defined as a fixed sequence of 72 TSC2 micro-actions, representing an optimized sub-strategy for a specific task. A macro-action can be defined as: (2) Here, is the number of micro-actions contained in the macro-action Based on these macro-actions, the invention designs two types of high-level tool invocation strategies: Strategy I (combining A1-A3) and Strategy II (integrating A4-A5). This hierarchical abstraction not only reduces decision-making complexity, but also embeds prior knowledge, guiding the agent to learn advanced tactical combinations more quickly.

[0031] Exemplarily, the PPO (Proximal Policy Optimization) algorithm is used as the core training mechanism in collaborative decision-making, mainly based on its excellent performance in handling high-dimensional state spaces and complex action spaces. PPO can achieve efficient strategy optimization while ensuring training stability through its unique clipping objective function and trust region mechanism, effectively avoiding the training collapse problems that are prone to occur in traditional policy gradient methods. The core part of its objective function is as follows: (3) where, denotes the probability ratio of new and old policies, is the advantage function estimate, and ε is a hyperparameter that limits the magnitude of each policy update. In addition, the sample efficiency advantage of PPO makes it particularly suitable for reinforcement learning tasks with high sample collection costs, as it supports multiple small batch updates using the same batch of samples.

[0032] Further, an action adaptive adjustment mechanism is introduced in the collaborative decision-making, which dynamically adjusts the execution length of micro-actions and macro-actions according to the game stage, specifically including: Micro-action adjustment: In the adaptive adjustment mechanism, the length of the generated sequence is dynamically controlled according to the current game stage to adjust the micro-action (MIAS). This ensures that in the early stage of the game, the large language model with parameters Θ does not generate too long or inefficient sequences; while in the later stage when fine micro-operation is crucial, the model is allowed to produce longer and more accurate control sequences.

[0033] Macro-action adjustment: Complementary to the micro-action adjustment mechanism, the macro-action (MAA) is adaptively shortened over time. Although macro-actions are highly efficient when executing fixed strategies (such as economic construction sequences) in the early stages of the game, static macro-actions can become a burden in the dynamic game later. By gradually shortening the macro-action length, this mechanism gradually transfers decision-making power from static tool modules to more flexible large language models, enabling the model to balance high-level planning and fine-grained adaptability.

[0034] Further, a hierarchical reward function is designed in the collaborative decision-making, which is a mechanism that decomposes complex overall tasks into different levels or phased goals and designs specific reward signals for each goal. The hierarchical reward mechanism can significantly improve the efficiency of reinforcement learning by providing richer and more timely training signals. The invention decomposes rewards into overall win rate rewards and local performance rewards to optimize the training process. Specifically including: Overall win rate reward: This is the main reward based on the win rate, which contains a time-dependent component. Winning a positive reward, but its value decreases with the length of the game, to encourage the agent to win faster; failure produces a penalty term that increases with the length of the game, to punish continuous losing games. This mechanism prevents the agent from accumulating small rewards by delaying the game. The overall win rate reward is calculated as follows: (4) where t represents the length of the game, and are the baseline values, and are the decay coefficient and the growth coefficient, respectively.

[0035] Local performance rewards: To provide more immediate and intensive feedback signals, the invention selects three key performance indicators as dimension-specific rewards: 1. Population blocking rate (RPB): reward agents for efficient population supply management.

[0036] 2. Average population utilization (APU): measure the balance of resource allocation.

[0037] 3. Attack frequency (AF): encourage offensive behavior by rewarding damage caused to the enemy.

[0038] The influence of these rewards changes dynamically over time. According to empirical research, the importance of RPB and APU significantly decreases after reaching a certain game time T, and their weights will be significantly reduced or removed in the total reward calculation. All dimension-specific rewards are normalized, value clipped, and logarithmically scaled to stabilize the learning process and prevent a single reward from dominating the training signal. The hierarchical reward total is obtained by adding the overall reward and the dimension-specific reward: (5) where represents the overall reward, and is the processed local reward.

[0039] Figure 2 Based on the above method, in an example, as shown in the specific steps of the action adaptive adjustment mechanism are as follows: Let x represent the original sequence length of generating micro-actions. The stage-dependent adjustment function is specified, and the maximum allowed sequence length is: (6) The final adjusted micro-action sequence (7) is then padded with empty micro-actions C7 (denoted as E) to maintain a fixed queue length k = 10. This ensures the consistency of the PPO agent action queue input size.

[0040] Let represent the original macro-action sequence. Its time-adjusted sequence is defined as: (8) Here ​​denotes a concatenation operation. Each macro action A i is independently truncated by a time-dependent function f i (t) with the following form: (9) where and denote the maximum and minimum length allowed, respectively, and is its decay threshold. Thus, the final adjusted macro action sequence embodies a controlled shortening of action length over time, thereby enabling a gradual shift in strategic influence as the game progresses, with the micro action sequence adapting accordingly.

[0041] In another example, as shown in Figure 3 , the overall flow of the reinforcement learning-driven large model and tool invocation collaborative framework is given as follows: S1. In the initialization phase, the algorithm performs a series of preparations: define the game stage, divide the game process into early, middle, and late stages to adopt different strategies in different stages. Set an action queue with a length of 10 to smooth and manage the execution of actions. Initialize the reward function ( ), set multiple local reward functions to evaluate the goodness of actions from different dimensions. Set the macro action time threshold ( ), set a maximum execution time for macro actions to prevent them from occupying too many resources.

[0042] S2. The observation space of the PPO network is derived from the text information of the game environment. The game state data is obtained through the python-sc2 interface , and converted into a structured abstract through a prompt engineering, as shown in equation 1. Then, it is encoded into a high-dimensional embedding vector by the pre-trained model MPNet as the input state of the PPO network. The action space is defined as a discrete set of three meta-actions: large language model strategy (generating micro action sequence), tool strategy I (calling a predefined macro action sequence), and tool strategy II (calling another set of predefined macro action sequence).

[0043] S3. The algorithm selects from the encoded state vector , the large language model strategy (generating a series of fine "micro actions" relying on the large language model) and the tool strategy (calling a powerful "macro action" encapsulated from the external tool database E.) as the output of the PPO network. The action of the large language model strategy is , and the action of the tool strategy is .

[0044] S4. If LLM strategy is selected: the large language model generates a micro-action sequence MIAS. The algorithm adaptively truncates the micro-action sequence according to the current game stage (early, middle, late) to ensure the appropriate duration of actions, as shown in Equation 6 or Table 1. If tool strategy is selected: the algorithm retrieves the corresponding macro-action MAA from the tool database E and applies the time decay mechanism to adaptively adjust the macro-action, as shown in Equation 9.

[0045] Table 1. Illustration of adaptive adjustment of micro-action sequence S5. Regardless of whether a micro-action sequence or a macro-action is generated, the system will use a specific null action Fill the action queue to a fixed length k = 10 to ensure that the data dimension input to the subsequent reinforcement learning network is uniform.

[0046] S6. The algorithm calculates a total reward for this round by synthesizing feedback from multiple dimensions , as shown in Equation 5. This reward is "layered", meaning it considers both immediate and long-term benefits.

[0047] S7. The algorithm uses the meta-action or , state vector and calculated reward R' to update its value network and policy network parameters through the PPO (Proximal Policy Optimization) algorithm.

[0048] In another exemplary embodiment, a large language model collaborative decision-making method driven by reinforcement learning is provided, comprising: a large language model strategy construction module for converting game states into structured text prompts and generating executable micro-action sequences; a tool strategy construction module for generating predefined macro-actions based on the micro-action sequences; a collaborative decision-making module for adopting a proximal policy optimization algorithm to collaboratively schedule the large language model strategy and tool strategy.

[0049] The above specific embodiments are detailed descriptions of the present application, which cannot be considered as limiting the specific embodiments of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, a number of simple deductions and substitutions can be made, which should be considered as belonging to the protection scope of the present application.

Claims

1. A method for collaborative decision-making of a large language model driven by reinforcement learning, characterized in that, The method comprises the following steps: Building a large language model strategy: converting game states into structured text prompts and generating executable micro-action sequences; Building a tool strategy: generating predefined macro-actions based on the micro-action sequences; Collaborative decision-making: using a proximal policy optimization algorithm to coordinate the large language model strategy and the tool strategy.

2. The method of claim 1, wherein the method is characterized by, The game state is converted into a structured text prompt, which comprises: Convert the game state into a structured prompt through an abstract chain.

3. The method of claim 1, wherein, Each macro-action is composed of a series of micro-action sequences, and the tool strategy is designed hierarchically.

4. The method of claim 1, wherein, The collaborative decision-making specifically comprises: Introducing an action adaptive adjustment mechanism to dynamically adjust the execution length of micro-actions and macro-actions according to the game stage; Designing a hierarchical reward function to combine overall win rate rewards and local performance rewards to optimize the training process.

5. The method of claim 4, wherein, The action adaptive adjustment mechanism comprises: Micro-action adjustment: limiting the maximum length of the sequence according to the game stage; Macro-action adjustment: dynamically shortening the length of macro-actions according to the time threshold, gradually transferring decision-making power to the large language model.

6. The method of claim 4, wherein the method is characterized by, The calculation formula of the overall win rate reward is: ； wherein t represents the length of the game, With is the reference value, and are the attenuation coefficient and the growth coefficient, respectively.

7. The method of claim 4, wherein the method is characterized by, The local performance reward comprises: Population congestion rate: reward the agent for efficient population supply management; Average population utilization rate: measure the balance of resource allocation; Attack frequency: encourage offensive behavior by rewarding damage value to the enemy.

8. The method of claim 1, wherein the method is characterized by, The proximal policy optimization algorithm uses a clipping objective function and a trust region mechanism.

9. A method for collaborative decision-making of a large language model driven by reinforcement learning, characterized in that, The method comprises the following steps: A large language model strategy construction module is used to convert game states into structured text prompts and generate executable micro-action sequences; A tool strategy construction module is used to generate predefined macro-actions based on the micro-action sequences; A collaborative decision-making module is used to coordinate the large language model strategy and the tool strategy using a proximal policy optimization algorithm.

Images