Game strategy optimization method based on reinforcement learning

Through the game strategy optimization method based on reinforcement learning, the problem of low self-adjustment efficiency in existing technology in complex and dynamic environments is solved, the flexibility and adaptability of strategies are achieved, applicable to multiple game types, and the science of strategies is improved Sex and precision.

CN118940819APending Publication Date: 2024-11-12HAINAN ZHITA INVESTMENT CO LTD
View PDF 0 Cites 12 Cited by

Patent Information

Application Number
CN202411163432.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing game strategy optimization methods have poor self-adjustment efficiency in changing environments, making it difficult to adapt to complex and dynamic game environments, especially in single agents, multi-agents and collaborative games.

Method used

The game strategy optimization method based on reinforcement learning is adopted, and the strategies are adjusted in real time to adapt to environmental changes by defining the game model, selecting appropriate reinforcement learning algorithms, initializing strategies, collecting data, adjusting hyperparameters, training processes, strategy evaluation and iterative training.

Benefits of technology

Self-adjustment in a changing environment to improve the effectiveness and adaptability of strategies. It is suitable for single agents, multi-agents and collaborative games, enhancing the scientificity and accuracy of strategies and avoiding overfitting and instability , promotes innovation in game theory.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940819A_ABST
    Figure CN118940819A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of game strategies, and particularly relates to a game strategy optimization method based on reinforcement learning, and the method comprises the following steps: S1, defining a game model: determining the basic structure of the game, including game players, action space and reward structure, defining the strategy space of each game player, all possible action strategies can be adopted. According to the method, self-adjustment can be carried out in a constantly changing environment, a complex and dynamic game environment can be adapted, a strategy is learned and optimized from interaction data through training and simulation, the flexibility of the method is allowed to be applied to single-agent, multi-agent and cooperative games, environment changes are coped by adjusting hyper-parameters and the strategy in real time, and the method is suitable for large-scale popularization and application. The scientificity and accuracy of the strategy are improved through data collection and analysis of the system, and innovation of the game theory is promoted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of game strategies, and in particular relates to a game strategy optimization method based on reinforcement learning. Background Art

[0002] Game strategy refers to the plans and methods developed in game theory to achieve the best results in competitive or cooperative situations. Common game strategies include Nash equilibrium, zero-sum game strategy, mixed strategy, prisoner's dilemma, dominant strategy and reverse reasoning. Nash equilibrium means that in the game, all participants know the strategies of other participants and no one can get a better result by unilaterally changing their own strategy; zero-sum game strategy is a strategy choice made when one participant's score is equal to the loss of another participant; mixed strategy is a random choice between different strategies to deal with uncertainty; prisoner's dilemma reveals the potential benefits of cooperation and the suboptimal results that may result from individual rationality; dominant strategy is a strategy that can achieve better results than other strategies in all cases; reverse reasoning determines one's best strategy by reasoning about the strategies that the opponent may adopt. Game strategies are widely used in fields such as economics, political science, military strategy and market competition to analyze and predict the behavior of individuals or groups in complex interactions.

[0003] The existing game strategy optimization methods have poor self-adjustment efficiency, are not convenient for self-adjustment in a changing environment, are not easy to apply to single-agent, multi-agent and collaborative games, and are difficult to adapt to complex and dynamic game environments. To this end, we propose a game strategy optimization method based on reinforcement learning to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a game strategy optimization method based on reinforcement learning, which can self-adjust in a constantly changing environment, adapt to complex and dynamic game environments, learn and optimize strategies from interactive data through training and simulation, and its flexibility allows application to single-agent, multi-agent and collaborative games, and respond to environmental changes by adjusting hyperparameters and strategies in real time. Systematic data collection and analysis improves the scientificity and accuracy of strategies and also promotes innovation in game theory.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A game strategy optimization method based on reinforcement learning, the game strategy optimization method comprising the following steps:

[0007] S1. Define the game model: determine the basic structure of the game, including players, action space, and reward structure, and define the strategy space of each player, that is, all possible action strategies that can be taken;

[0008] S2. Select reinforcement learning algorithms: Select algorithms based on single agent and multi-agent, including Q-learning, DeepQ-Networks, policy gradient methods, Actor-Critic methods, independent Q-learning, and collaboration-based policy optimization methods;

[0009] S3. Initialization: Initialize the strategy for each player, using a value-based approach to initialize the value function;

[0010] S4. Collect data: Record game data during training, including player actions, reward signals, and strategy updates. Analyze the collected data to identify problems or opportunities for improvement in the strategy.

[0011] S5. Adjust and optimize hyperparameters: During the training process, adjust the hyperparameters of the reinforcement learning algorithm, and optimize the strategy training process based on the results of the hyperparameter adjustment to improve training efficiency and strategy quality;

[0012] S6. Training process: simulate the game process, players play according to the current strategy, obtain rewards and feedback, and update the strategy based on the feedback;

[0013] S7. Strategy evaluation: Evaluate the performance of the trained strategy in the game through battle testing and profit calculation, and improve and adjust the strategy based on the evaluation results;

[0014] S8. Iterative training: perform multiple rounds of training and strategy updates;

[0015] S9. Practical application: Apply the optimized strategy to the actual gaming environment, monitor the strategy performance in the actual application, and make further adjustments as needed.

[0016] In a preferred embodiment, the definition of the game model also includes determining the participants in the game, where the players can be individuals, teams or automated systems, determining the rewards or penalties that the players will receive after performing specific actions in the game, clarifying the rules for the game, including the order of actions in each round, the legality of actions, and state transition rules, determining all possible states in the game, describing the interaction methods between the players, including competition, cooperation, and mixed strategies, determining the players' understanding of game information, including complete information or incomplete information, clarifying the goals pursued by each player in the game, determining the goals of the optimization strategy, verifying the accuracy of the model through actual or simulated games, adjusting the definition of the game model based on the verification results, and modifying the parts that do not conform to the actual situation.

[0017] In a preferred embodiment, the selection of reinforcement learning algorithm also includes determining the complexity of state space and action space, understanding the immediacy and delay of rewards, determining whether offline training or online training is required, determining whether there is an environmental model, and if there is a model, a model-based method can be used; if there is no model, a model-independent method is used to evaluate the computing resource requirements of the selected algorithm, including training time and storage requirements, evaluate the convergence and stability of the algorithm, test the selected algorithm in an actual or simulated environment, verify its performance in the game, and evaluate the performance of the algorithm, including convergence speed, strategy quality, and computational efficiency.

[0018] In a preferred embodiment, the selection of reinforcement learning algorithm also includes determining the complexity of state space and action space, understanding the immediacy and delay of rewards, determining whether offline training or online training is required, determining whether there is an environmental model, and if there is a model, a model-based method can be used; if there is no model, a model-independent method is used to evaluate the computing resource requirements of the selected algorithm, including training time and storage requirements, evaluate the convergence and stability of the algorithm, test the selected algorithm in an actual or simulated environment, verify its performance in the game, and evaluate the performance of the algorithm, including convergence speed, strategy quality, and computational efficiency.

[0019] In a preferred embodiment, the data collection also includes recording the actions taken by each player at each time step. For a multi-agent environment, the actions of each agent are recorded, the immediate rewards obtained for each action and any delayed rewards are recorded, the environmental state at each time step, including the current state and state transitions in the game, the current strategy parameters or strategy probability distribution are recorded, and the changes in the strategy during the training process are understood. A data structure is selected to store the recorded data, including lists, dictionaries, and databases. A data recording mechanism is implemented during the training process, including the use of callback functions, loggers, or real-time data storage systems. A data storage method is selected according to the amount of data and usage requirements, including local files, databases, or cloud storage. The data is backed up and protected, duplicate, erroneous, or incomplete data is removed, the data is converted into a format suitable for analysis, and necessary standardization or normalization is performed. The stability of the strategy during the training process is analyzed, and the effectiveness of the strategy is evaluated, including average rewards, winning rates, and returns.

[0020] In a preferred embodiment, the adjustment and optimization of hyperparameters also includes determining the step size in the strategy update process, determining the impact of future rewards on current actions, determining the size of the experience replay data set, determining the amount of data used for each strategy update, setting the range of parameter values, and traversing all combinations to find the optimal hyperparameters, randomly selecting parameter values, and finding a hyperparameter combination with better performance through multiple experiments, optimizing hyperparameters based on the Bayesian criterion, using a model to predict performance and conduct minimum necessary experiments, simulating the evolutionary process, and finding the optimal solution through reproduction, mutation, selection and elimination. According to the designed hyperparameter search strategy, experiments are set up, multiple training and evaluation are performed, and the hyperparameter combinations and performance indicators of each experiment are recorded. By comparing the results of different hyperparameter combinations, it is analyzed which hyperparameters have a greater impact on performance, and the hyperparameters are adjusted based on the analysis results, including reducing the learning rate, increasing the experience replay capacity, testing the performance of the strategy in different environments, and evaluating the generalization ability of the strategy.

[0021] In a preferred embodiment, the training process also includes starting training using a preset initial strategy or a random strategy, each player selects an action according to the current strategy, and in each time step, the player's choice is recorded. After each action, the state of the environment changes, and the state information is updated. The reward is calculated based on the player's action and current state, and the reward signal is recorded. The state, action and reward of each time step are recorded, and the parameters or probability distribution of the current strategy are recorded. The loss function is calculated based on the reward signal and the strategy goal. The loss function includes the TD error in Q-learning, the product of the logarithmic probability and the reward in the policy gradient, and the prediction error of the current strategy is calculated, that is, the gap between the value estimate of the current strategy and the actual reward. The policy parameters are updated using gradient descent or other optimization algorithms. According to the results of the policy update, the policy is adjusted, and samples are randomly drawn from the experience replay pool to reduce the correlation between samples. The samples in the experience replay are used to update the policy or value function.

[0022] In a preferred embodiment, the strategy evaluation also includes evaluating the total reward, average reward, and reward variance of the strategy, testing the winning rate or success rate of the strategy in a game environment, calculating the cumulative benefits of the strategy in different scenarios or time periods, competing the trained strategy with other strategies, including baseline strategies, random strategies, and opponent strategies, testing the performance of the strategy under different environmental conditions, comparing it with the existing best strategy or other comparison strategies, analyzing the relative performance of the strategy, identifying reasons why the strategy performs poorly in certain situations, analyzing whether the reward mechanism needs to be adjusted, checking whether the strategy has instability or overfitting, and adjusting the strategy parameters or improving the strategy model based on the evaluation results.

[0023] In a preferred embodiment, the iterative training also includes selecting an initial strategy, including a random strategy, a simple strategy or a pre-trained strategy, initializing a policy network or a value function, preparing for training, running multiple rounds of games in a training environment, collecting state, action, and reward data, calculating the loss function of the strategy based on the collected data, updating the strategy parameters using gradient descent or other optimization algorithms, applying a policy update rule or a value function update rule, and adjusting the strategy to improve future performance.

[0024] In a preferred embodiment, the actual application also includes ensuring that the actual gaming environment is prepared and has similarities with the training environment, integrating the optimized strategy into the actual application system, including necessary software and hardware interfaces, importing the strategy parameters, models or networks obtained during the training process into the actual application system, and conducting preliminary strategy testing in the actual environment.

[0025] The technical effects achieved by the present invention are:

[0026] Reinforcement learning algorithms can adjust themselves in a constantly changing environment, so that strategies can cope with various complex and dynamic game environments. Through training and simulating the game process, the algorithm can learn from a large amount of interactive data and continuously optimize the strategy to improve the effectiveness of the strategy. By iteratively training and evaluating the strategy, it can find the strategy that performs best in a specific game environment. This optimization not only takes into account the current performance, but also adapts to potential changes and uncertainties.

[0027] This method can be applied to various game models, including single-agent, multi-agent and collaborative games. The flexibility of reinforcement learning allows it to adapt to different game complexities and strategy spaces. According to the specific needs and complexity of the game, a suitable reinforcement learning algorithm can be selected to improve the efficiency and effect of strategy optimization. During the training process, the algorithm can adjust hyperparameters and strategies in real time, and continuously optimize strategies based on feedback, so that it can respond to environmental changes in a timely manner in practical applications.

[0028] By systematically collecting and analyzing game data, we can identify deficiencies in strategies and make targeted improvements. Data-driven methods improve the scientificity and accuracy of strategy optimization. By comprehensively evaluating and testing strategies, we can ensure the effectiveness and robustness of strategies and avoid overfitting and strategy instability. By applying reinforcement learning to game strategy optimization, we can not only improve strategies in practical applications, but also promote the development and innovation of game theory. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of a game strategy optimization method based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0031] See also Figure 1 As shown, the present invention provides a game strategy optimization method based on reinforcement learning, and the game strategy optimization method comprises the following steps:

[0032] S1. Define the game model: determine the basic structure of the game, including players, action space, and reward structure, and define the strategy space of each player, that is, all possible action strategies that can be taken;

[0033] S2. Select reinforcement learning algorithms: Select algorithms based on single agent and multi-agent, including Q-learning, DeepQ-Networks, policy gradient methods, Actor-Critic methods, independent Q-learning, and collaboration-based policy optimization methods;

[0034] S3. Initialization: Initialize the strategy for each player, using a value-based approach to initialize the value function;

[0035] S4. Collect data: Record game data during training, including player actions, reward signals, and strategy updates. Analyze the collected data to identify problems or opportunities for improvement in the strategy.

[0036] S5. Adjust and optimize hyperparameters: During the training process, adjust the hyperparameters of the reinforcement learning algorithm, and optimize the strategy training process based on the results of the hyperparameter adjustment to improve training efficiency and strategy quality;

[0037] S6. Training process: simulate the game process, players play according to the current strategy, obtain rewards and feedback, and update the strategy based on the feedback;

[0038] S7. Strategy evaluation: Evaluate the performance of the trained strategy in the game through battle testing and profit calculation, and improve and adjust the strategy based on the evaluation results;

[0039] S8. Iterative training: perform multiple rounds of training and strategy updates;

[0040] S9. Practical application: Apply the optimized strategy to the actual gaming environment, monitor the strategy performance in the actual application, and make further adjustments as needed.

[0041] Defining a game model also includes determining the participants in the game, which can be individuals, teams, or automated systems; determining the rewards or penalties that players will receive after performing specific actions in the game; clarifying the rules for conducting the game, including the order of actions in each round, the legality of actions, and the state transition rules; determining all possible states in the game; describing the interaction between players, including competition, cooperation, and mixed strategies; determining the extent of players' understanding of game information, including complete information or incomplete information; clarifying the goals pursued by each player in the game; determining the goals of the optimization strategy; verifying the accuracy of the model through actual or simulated games; adjusting the definition of the game model based on the verification results; and modifying the parts that do not conform to the actual situation.

[0042] Selecting a reinforcement learning algorithm also includes determining the complexity of the state space and action space, understanding the immediacy and delay of rewards, determining whether offline training or online training is required, determining whether there is an environmental model, and if there is a model, using model-based methods; if there is no model, using model-independent methods, evaluating the computing resource requirements of the selected algorithm, including training time and storage requirements, evaluating the convergence and stability of the algorithm, testing the selected algorithm in a real or simulated environment, verifying its performance in the game, and evaluating the performance of the algorithm, including convergence speed, strategy quality, and computational efficiency.

[0043] Selecting a reinforcement learning algorithm also includes determining the complexity of the state space and action space, understanding the immediacy and delay of rewards, determining whether offline training or online training is required, determining whether there is an environmental model, and if there is a model, using model-based methods; if there is no model, using model-independent methods, evaluating the computing resource requirements of the selected algorithm, including training time and storage requirements, evaluating the convergence and stability of the algorithm, testing the selected algorithm in a real or simulated environment, verifying its performance in the game, and evaluating the performance of the algorithm, including convergence speed, strategy quality, and computational efficiency.

[0044] Collecting data also includes recording the actions taken by each player at each time step. For multi-agent environments, record the actions of each agent, record the immediate reward obtained for each action and any delayed rewards, record the state of the environment at each time step, including the current state and state transitions in the game, record the current strategy parameters or strategy probability distribution, understand how the strategy changes during the training process, select data structures to store the recorded data, including lists, dictionaries, and databases, implement data recording mechanisms during training, including using callback functions, loggers, or real-time data storage systems, select data storage methods based on data volume and usage requirements, including local files, databases, or cloud storage, back up and protect data, remove duplicate, erroneous, or incomplete data, convert data into a format suitable for analysis, perform necessary standardization or normalization, analyze whether the strategy is stable during training, and evaluate the effectiveness of the strategy, including average reward, win rate, and revenue.

[0045] Adjusting and optimizing hyperparameters also includes determining the step size during the policy update process, determining the impact of future rewards on current actions, determining the size of the experience replay data set, determining the amount of data used for each policy update, setting the range of parameter values, and traversing all combinations to find the optimal hyperparameters, randomly selecting parameter values, and finding the best performing hyperparameter combination through multiple experiments. Hyperparameter optimization is performed based on the Bayesian criterion, using models to predict performance and conduct minimal necessary experiments, simulating the evolutionary process, and finding the optimal solution through reproduction, mutation, selection, and elimination. According to the designed hyperparameter search strategy, experiments are set up, and multiple training and evaluation are performed. The hyperparameter combinations and performance indicators of each experiment are recorded. By comparing the results of different hyperparameter combinations, it is analyzed which hyperparameters have a greater impact on performance. Based on the results of the analysis, the hyperparameters are adjusted, including reducing the learning rate, increasing the experience replay capacity, testing the performance of the strategy in different environments, and evaluating the generalization ability of the strategy.

[0046] The training process also includes starting training using a preset initial strategy or a random strategy. Each player selects an action based on the current strategy. In each time step, the player's choice is recorded. After each action, the state of the environment changes, and the state information is updated. The reward is calculated based on the player's action and current state, and the reward signal is recorded. The state, action and reward of each time step are recorded, and the parameters or probability distribution of the current strategy are recorded. The loss function is calculated based on the reward signal and the strategy goal. The loss function includes the TD error in Q-learning, the product of the log probability and reward in the policy gradient, and the prediction error of the current strategy is calculated, that is, the gap between the value estimate of the current strategy and the actual reward. The policy parameters are updated using gradient descent or other optimization algorithms. According to the results of the policy update, the policy is adjusted. Samples are randomly drawn from the experience replay pool to reduce the correlation between samples, and the samples in the experience replay are used to update the policy or value function.

[0047] Strategy evaluation also includes evaluating the total reward, average reward, and reward variance of the strategy, testing the winning rate or success rate of the strategy in a game environment, calculating the cumulative benefits of the strategy in different scenarios or time periods, and competing the trained strategy with other strategies, including baseline strategies, random strategies, and opponent strategies. Testing the performance of the strategy under different environmental conditions, comparing it with the best existing strategy or other comparison strategies, analyzing the relative performance of the strategy, identifying reasons why the strategy performs poorly in certain situations, analyzing whether the reward mechanism needs to be adjusted, checking whether the strategy has instability or overfitting, and adjusting strategy parameters or improving the strategy model based on the evaluation results.

[0048] Iterative training also includes selecting an initial strategy, including a random strategy, a simple strategy, or a pre-trained strategy, initializing the policy network or value function, preparing for training, running multiple rounds of games in the training environment, collecting state, action, and reward data, calculating the loss function of the strategy based on the collected data, updating the policy parameters using gradient descent or other optimization algorithms, applying the policy update rule or the value function update rule, and adjusting the strategy to improve future performance.

[0049] Practical applications also include ensuring that the actual gaming environment is prepared and has similarities to the training environment, integrating the optimized strategy into the actual application system, including the necessary software and hardware interfaces, importing the strategy parameters, models or networks obtained during the training process into the actual application system, and conducting preliminary strategy testing in the actual environment.

[0050] In the present invention, the reinforcement learning algorithm can adjust itself in a constantly changing environment, so that the strategy can cope with various complex and dynamic game environments. Through training and simulating the game process, the algorithm can learn from a large amount of interactive data and continuously optimize the strategy, thereby improving the effectiveness of the strategy. By iteratively training and evaluating the strategy, the strategy that performs best in a specific game environment can be found. This optimization not only takes into account the current performance, but also adapts to potential changes and uncertainties.

[0051] This method can be applied to various game models, including single-agent, multi-agent and collaborative games. The flexibility of reinforcement learning allows it to adapt to different game complexities and strategy spaces. According to the specific needs and complexity of the game, a suitable reinforcement learning algorithm can be selected to improve the efficiency and effect of strategy optimization. During the training process, the algorithm can adjust hyperparameters and strategies in real time, and continuously optimize strategies based on feedback, so that it can respond to environmental changes in a timely manner in practical applications.

[0052] By systematically collecting and analyzing game data, we can identify deficiencies in strategies and make targeted improvements. Data-driven methods improve the scientificity and accuracy of strategy optimization. By comprehensively evaluating and testing strategies, we can ensure the effectiveness and robustness of strategies and avoid overfitting and strategy instability. By applying reinforcement learning to game strategy optimization, we can not only improve strategies in practical applications, but also promote the development and innovation of game theory.

[0053] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A game strategy optimization method based on reinforcement learning, characterized in that: The game strategy optimization method comprises the following steps: S1. Define the game model: determine the basic structure of the game, including players, action space, and reward structure, and define the strategy space of each player, that is, all possible action strategies that can be taken; S2. Select reinforcement learning algorithms: Select algorithms based on single agent and multi-agent, including Q-learning, Deep Q-Networks, policy gradient methods, Actor-Critic methods, independent Q-learning, and collaboration-based policy optimization methods; S3. Initialization: Initialize the strategy for each player, using a value-based approach to initialize the value function; S4. Collect data: Record game data during training, including player actions, reward signals, and strategy updates. Analyze the collected data to identify problems or opportunities for improvement in the strategy. S5. Adjust and optimize hyperparameters: During the training process, adjust the hyperparameters of the reinforcement learning algorithm, and optimize the strategy training process based on the results of the hyperparameter adjustment to improve training efficiency and strategy quality; S6. Training process: simulate the game process, players play according to the current strategy, obtain rewards and feedback, and update the strategy based on the feedback; S7. Strategy evaluation: Evaluate the performance of the trained strategy in the game through battle testing and profit calculation, and improve and adjust the strategy based on the evaluation results; S8. Iterative training: perform multiple rounds of training and strategy updates; S9. Practical application: Apply the optimized strategy to the actual gaming environment, monitor the strategy performance in the actual application, and make further adjustments as needed.

2. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The definition of the game model also includes determining the participants in the game, where the players can be individuals, teams or automated systems, determining the rewards or penalties that the players will receive after performing specific actions in the game, clarifying the rules for the game, including the order of actions in each round, the legality of actions, and state transition rules, determining all possible states in the game, describing the interaction methods between players, including competition, cooperation, and mixed strategies, determining the players' understanding of game information, including complete information or incomplete information, clarifying the goals pursued by each player in the game, determining the goals of the optimization strategy, verifying the accuracy of the model through actual or simulated games, adjusting the definition of the game model based on the verification results, and modifying the parts that do not conform to the actual situation.

3. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The selection of reinforcement learning algorithm also includes determining the complexity of state space and action space, understanding the immediacy and delay of rewards, determining whether offline training or online training is required, determining whether there is an environmental model, and if there is a model, a model-based method can be used; if there is no model, a model-independent method is used, evaluating the computing resource requirements of the selected algorithm, including training time and storage requirements, evaluating the convergence and stability of the algorithm, testing the selected algorithm in an actual or simulated environment, verifying its performance in the game, and evaluating the performance of the algorithm, including convergence speed, strategy quality, and computational efficiency.

4. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The initialization also includes assigning equal probability to each action, initializing the strategy using heuristic methods, clarifying all possible strategies for each player, using parameterized strategies, initializing the weights and bias parameters of the network to random values ​​or specific distributions, for random strategies, setting the probability distribution of each action, initializing the value functions of all states or state-action pairs to zero, setting an initial value for each state, setting initial network weights for value functions using neural networks or other function approximation methods, verifying the rationality of the initialization strategy, checking whether the initialized value function is reasonable, adjusting the method of initializing the strategy and value function based on feedback from initial training, starting preliminary training using the initialized strategy and value function, observing the performance of the algorithm, and evaluating the impact of the initialization settings on the training process and the final strategy.

5. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The data collection also includes recording the actions taken by each player at each time step. For a multi-agent environment, the actions of each agent are recorded, the immediate rewards obtained for each action and any delayed rewards are recorded, the environmental state at each time step, including the current state and state transitions in the game, the current strategy parameters or strategy probability distribution, understanding the changes in the strategy during the training process, selecting data structures to store the recorded data, including lists, dictionaries, and databases, implementing a data recording mechanism during the training process, including using callback functions, loggers, or real-time data storage systems, selecting data storage methods based on data volume and usage requirements, including local files, databases, or cloud storage, backing up and protecting data, removing duplicate, erroneous, or incomplete data, converting data into a format suitable for analysis, performing necessary standardization or normalization processing, analyzing whether the strategy is stable during the training process, and evaluating the effectiveness of the strategy, including average rewards, winning rates, and returns.

6. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The adjustment and optimization of hyperparameters also includes determining the step size in the strategy update process, determining the impact of future rewards on current actions, determining the size of the experience replay data set, determining the amount of data used for each strategy update, setting the range of parameter values, and traversing all combinations to find the optimal hyperparameters, randomly selecting parameter values, and finding a hyperparameter combination with better performance through multiple experiments, optimizing hyperparameters based on the Bayesian criterion, using a model to predict performance and conduct minimal necessary experiments, simulating the evolutionary process, and finding the optimal solution through reproduction, mutation, selection, and elimination. According to the designed hyperparameter search strategy, experiments are set up, multiple training and evaluations are conducted, and the hyperparameter combinations and performance indicators of each experiment are recorded. By comparing the results of different hyperparameter combinations, it is analyzed which hyperparameters have a greater impact on performance, and the hyperparameters are adjusted based on the analysis results, including reducing the learning rate, increasing the experience replay capacity, testing the performance of the strategy in different environments, and evaluating the generalization ability of the strategy.

7. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The training process also includes starting training using a preset initial strategy or a random strategy, each player selects an action according to the current strategy, recording the player's choice at each time step, and after each action, the state of the environment changes, updating the state information, calculating rewards based on the player's actions and current state, and recording reward signals, recording the state, actions and rewards at each time step, recording the parameters or probability distribution of the current strategy, calculating a loss function based on the reward signal and the strategy goal, the loss function including the TD error in Q-learning, the product of the logarithmic probability and reward in the policy gradient, calculating the prediction error of the current strategy, that is, the gap between the value estimate of the current strategy and the actual reward, updating the strategy parameters using gradient descent or other optimization algorithms, adjusting the strategy based on the results of the strategy update, randomly extracting samples from the experience replay pool to reduce the correlation between samples, and using samples in the experience replay to update the strategy or value function.

8. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The strategy evaluation also includes evaluating the total reward, average reward, and reward variance of the strategy, testing the winning rate or success rate of the strategy in a game environment, calculating the cumulative benefits of the strategy in different scenarios or time periods, competing the trained strategy with other strategies, including baseline strategies, random strategies, and opponent strategies, testing the performance of the strategy under different environmental conditions, comparing it with the existing best strategy or other comparison strategies, analyzing the relative performance of the strategy, identifying reasons why the strategy performs poorly in certain situations, analyzing whether the reward mechanism needs to be adjusted, checking whether the strategy has instability or overfitting, and adjusting strategy parameters or improving the strategy model based on the evaluation results.

9. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The iterative training also includes selecting an initial strategy, including a random strategy, a simple strategy or a pre-trained strategy, initializing a strategy network or a value function, preparing for training, running multiple rounds of games in a training environment, collecting state, action, and reward data, calculating the loss function of the strategy based on the collected data, updating the strategy parameters using gradient descent or other optimization algorithms, applying a strategy update rule or a value function update rule, and adjusting the strategy to improve future performance.

10. The game strategy optimization method based on reinforcement learning according to claim 1, characterized in that: The actual application also includes ensuring that the actual game environment is prepared and has similarities with the training environment, integrating the optimized strategy into the actual application system, including necessary software and hardware interfaces, importing the strategy parameters, models or networks obtained during the training process into the actual application system, and conducting preliminary strategy testing in the actual environment.

Citation Information

Cited By

  • Received data analysis and judgment system for data interface

    CN119293705A

  • Multistage deduction interaction data processing method and system based on intelligent game

    CN119514697A

  • System for developing operation planning optimization algorithm under intelligent game

    CN119918572A

  • An Operational Optimization Algorithm Development System under Intelligent Games

    CN119918572B

  • System security management method

    CN120105460A