Large model reasoning game ability optimization method based on reinforcement learning
Through multi-agent autonomous gaming and REINFORCE algorithm optimization, the dependence on labeled data in large model training is resolved, performance improvement and generalization of logical reasoning capabilities in gaming scenarios are achieved, and data acquisition costs are reduced.
Patent Information
- Application Number
- CN202510970186.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-15
- Publication Date
- 2025-09-19
AI Technical Summary
Existing technologies require a large amount of high-quality labeled data to improve the capabilities of large models, which is particularly costly in gaming scenarios and difficult to train effectively.
The environment is built using the rules and logic of multi-person reasoning games, and reinforcement learning is performed through game data constructed through multi-agent autonomous games. The REINFORCE algorithm is used to optimize large model strategies, and KL divergence constraints and proximal optimization are introduced to form a closed loop of generation-training-optimization.
It enables autonomous improvement of large models without additional data, improves the performance of reasoning games, generalizes logical reasoning capabilities to other reasoning tasks, and reduces data acquisition costs.
Smart Images
Figure CN120671852A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of game reasoning technology, and in particular to a method for optimizing large-model reasoning game capabilities based on reinforcement learning. Background Art
[0002] With the development of technology, supervised fine-tuning methods face a significant bottleneck in improving the capabilities of large models: a large amount of high-quality labeled data is required, but the cost of obtaining and labeling this data is high, especially in the gaming scenario, where this problem is even more prominent.
[0003] Therefore, it is necessary to provide a reinforcement learning-based optimization method for the large model's reasoning game capabilities, allowing the large model to perform reinforcement learning during the multi-agent autonomous game, so as to achieve autonomous improvement of the large model without additional data, improve the performance of the large model in text reasoning games, and generalize the logical reasoning capabilities to other reasoning tasks. Summary of the Invention
[0004] The purpose of this invention is to provide a method for optimizing the reasoning game capabilities of large models based on reinforcement learning, allowing the large model to perform reinforcement learning during the autonomous game of multiple agents, so as to achieve autonomous improvement of the large model without additional data, improve the performance of the large model in text reasoning games, and generalize the logical reasoning capabilities to other reasoning tasks.
[0005] In order to solve the problems existing in the prior art, the present invention provides a method for optimizing the reasoning game capabilities of a large model based on reinforcement learning, comprising the following steps:
[0006] Use the rules and logic of multiplayer reasoning games to train large-scale reasoning game capabilities, build an environment that includes role allocation, game stages, and victory and defeat determination conditions, and provide environmental rewards;
[0007] Each agent in the large model will be modeled as a Player object, which contains the role type, survival status, predefined prompt words for each stage, and game history;
[0008] Construct game data through interactions between multiple agents;
[0009] Use game data to train and optimize the reasoning capabilities of large models.
[0010] Optionally, in the optimization method of the large model reasoning game capability based on reinforcement learning, the environmental reward includes game rewards and large model output format rewards. The game reward is determined by whether the camp in which the large model is located wins the final game; the large model output format reward is used to limit the large model output format to comply with the thinking-first-output-later execution action.
[0011] Optionally, in the optimization method of large-model reasoning game capabilities based on reinforcement learning, at each stage of the game, the agent will generate a response for the current stage through predefined stage prompt words, and the observations, original responses, executed actions and reward information of each step will be saved to form complete game data.
[0012] Optionally, in the optimization method for large-model reasoning game capabilities based on reinforcement learning, the REINFORCE algorithm is used to optimize the large-model strategy.
[0013] Optionally, in the optimization method for large-scale model reasoning game capabilities based on reinforcement learning, the REINFORCE algorithm is optimized. The REINFORCE algorithm optimization method is as follows:
[0014] Introducing KL divergence constraints to limit the difference between the new strategy and the initial strategy, preventing the strategy from deviating too far from the original model and causing performance degradation;
[0015] Introducing a proximal optimization approach by using the probability ratio between the new and old strategies and clipping the probability ratio to limit the update amplitude at each step;
[0016] The two steps of KL divergence constraint and proximal optimization are repeated, and the large model is continuously updated by training the game data and reward information generated in each round to improve the large model's ability to reason about games and complete the optimization.
[0017] Compared with the prior art, the present invention has the following advantages:
[0018] (1) The present invention can utilize a predefined game environment and construct game data through multi-agent autonomous gaming to conduct reinforcement learning training. This method significantly reduces the reliance on manually labeled data and can also continuously optimize the capabilities of large models through interactions between agents, thereby achieving stable performance improvements.
[0019] (2) Through the reinforcement learning process of iterative autonomous games, the present invention improves the performance of the large model in reasoning games and achieves an increase in the winning rate compared to the untrained large model. The large model can better understand complex game rules and formulate reasonable strategies.
[0020] (3) The reinforcement learning training of this invention not only improves the performance of large models in reasoning games, but also successfully generalizes their logical reasoning capabilities to other general reasoning tasks. It has been verified that performance improvements can still be achieved on general reasoning datasets such as StrategyQA.
[0021] (4) This invention generates training data through autonomous game play, completely avoiding reliance on manually labeled data and significantly reducing data acquisition costs. Continuous interaction and optimization among multiple agents form a closed loop of “generation-training-optimization,” enabling large models to achieve autonomous improvement without the need for additional data. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A flowchart of an optimization method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following is a more detailed description of the specific embodiments of the present invention with reference to schematic diagrams. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the drawings are greatly simplified and not to exact scale, and are only used for the purpose of conveniently and clearly illustrating the embodiments of the present invention.
[0024] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0025] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0026] With the development of technology, supervised fine-tuning methods face a significant bottleneck in improving the capabilities of large models: a large amount of high-quality labeled data is required, but the cost of obtaining and labeling this data is high, especially in the gaming scenario, where this problem is even more prominent.
[0027] In order to solve the problems existing in the prior art, the present invention provides a method for optimizing the reasoning game capabilities of a large model based on reinforcement learning, such as Figure 1 As shown, the optimization method includes the following steps:
[0028] S1: Use the rules and logic of multiplayer reasoning games (such as Werewolf) to train large-scale reasoning game skills. This involves setting up an environment, including role assignments (such as werewolves, villagers, witches, and prophets), game stages (such as nighttime action, daytime discussion, and daytime voting), and victory and defeat determination criteria, along with environmental rewards.
[0029] Environmental rewards: include game rewards and large model output format rewards. Game rewards are determined by whether the large model's camp wins the final game. Large model output format rewards are used to limit the large model's output format to the "think first, then output" execution action, which meets the following requirements: <think> Thought Process< / think> <action> Perform an action< / action> format.
[0030] S2: Each agent in the large model will be modeled as a Player object, which contains information such as role type, survival status, predefined prompt words for each stage, and game history;
[0031] S3: Generate autonomous game data: Construct game data through interactions between multiple agents. At each stage of the game, the agent will generate a response for the current stage using predefined stage prompts. The observations, original responses, executed actions, and reward information for each step will be saved to form complete game data.
[0032] In one embodiment, the game phase will sequentially go through the night werewolf discussion, werewolf action, witch action, prophet action and daytime discussion and voting phase.
[0033] S4: Training reinforcement learning: Use game data to train and optimize the reasoning capabilities of large models.
[0034] Preferably, the present invention uses the REINFORCE algorithm to optimize the large model strategy. The core idea is to improve the model strategy by maximizing the cumulative return. In addition, in order to enhance the stability of learning training, the present invention optimizes the REINFORCE algorithm. The REINFORCE algorithm optimization method is as follows:
[0035] Introducing KL divergence constraints to limit the difference between the new strategy and the initial strategy, preventing the strategy from deviating too far from the original model and causing performance degradation;
[0036] The proximal optimization method is introduced to optimize the REINFORCE algorithm by using the probability ratio between the new and old strategies and clipping the probability ratio to limit the update amplitude of each step.
[0037] The two steps of KL divergence constraint and proximal optimization are repeated, and the large model is continuously updated by training the game data and reward information generated in each round to improve the reasoning game ability of the large model and complete the optimization of the large model.
[0038] In summary, the present invention has the following advantages compared with the prior art:
[0039] (1) The present invention can utilize a predefined game environment and construct game data through multi-agent autonomous gaming to conduct reinforcement learning training. This method significantly reduces the reliance on manually labeled data and can also continuously optimize the capabilities of large models through interactions between agents, thereby achieving stable performance improvements.
[0040] (2) Through the reinforcement learning process of iterative autonomous games, the present invention improves the performance of the large model in reasoning games and achieves an increase in the winning rate compared to the untrained large model. The large model can better understand complex game rules and formulate reasonable strategies.
[0041] (3) The reinforcement learning training of this invention not only improves the performance of large models in reasoning games, but also successfully generalizes their logical reasoning capabilities to other general reasoning tasks. It has been verified that performance improvements can still be achieved on general reasoning datasets such as StrategyQA.
[0042] (4) This invention generates training data through autonomous game play, completely avoiding reliance on manually labeled data and significantly reducing data acquisition costs. Continuous interaction and optimization among multiple agents form a closed loop of “generation-training-optimization,” enabling large models to achieve autonomous improvement without the need for additional data.
[0043] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.
Claims
1. A method for optimizing the ability of large-scale model reasoning games based on reinforcement learning, characterized in that: The following steps are involved: Use the rules and logic of multiplayer reasoning games to train large-scale reasoning game capabilities, build an environment that includes role allocation, game stages, and victory and defeat determination conditions, and provide environmental rewards; Each agent in the large model will be modeled as a Player object, which contains the role type, survival status, predefined prompt words for each stage, and game history; Construct game data through interactions between multiple agents; Use game data to train and optimize the reasoning capabilities of large models.
2. The method for optimizing large-scale model reasoning game capabilities based on reinforcement learning according to claim 1, characterized in that: Environmental rewards: These include game rewards and large model output format rewards. Game rewards are determined by whether the large model's faction wins the final game. Large model output format rewards are used to limit the large model's output format to a "think before output" execution action.
3. The method for optimizing large-scale model reasoning game capabilities based on reinforcement learning as claimed in claim 1, characterized in that: At each stage of the game, the agent will generate a response for the current stage using predefined stage prompts. The observations, original responses, executed actions, and reward information for each step will be saved to form complete game data.
4. The method for optimizing large-scale model reasoning game capabilities based on reinforcement learning according to claim 1, characterized in that: Use the REINFORCE algorithm to optimize large model strategies.
5. The method for optimizing large-scale model reasoning game capabilities based on reinforcement learning according to claim 1, characterized in that: Optimize the REINFORCE algorithm. The optimization method of the REINFORCE algorithm is as follows: Introducing KL divergence constraints to limit the difference between the new strategy and the initial strategy, preventing the strategy from deviating too far from the original model and causing performance degradation; Introducing a proximal optimization approach by using the probability ratio between the new and old strategies and clipping the probability ratio to limit the update amplitude at each step; The two steps of KL divergence constraint and proximal optimization are repeated, and the large model is continuously updated by training the game data and reward information generated in each round to improve the large model's ability to reason about games and complete the optimization.